buildParquetCopySQL() gains an append argument that emits the DuckDB
APPEND copy option. This lets a PARTITION_BY write add new files to an
existing (hive-partitioned) directory instead of failing "directory is not
empty", which is what a coord-array append needs (DuckDBArray).writeDuckDBTableParquet() / buildTableSelectSQL())
now types the __index__ column the same way the in-memory writer does,
instead of always emitting a BIGINT row_number(). It CASTs the index to a
type chosen by range (narrowed on a fresh write, index_max honored, or pinned
to part 0's on-disk type on append), so a resource written or appended across
both write paths keeps one consistent __index__ type. Previously an
in-memory part 0 (narrowed) plus a lazy append (BIGINT) produced a
schema-inconsistent, unreadable resource, and the same table had a different
index type depending on which path wrote it.DuckDBSelfHits() now fails loudly when nnode (or a supplied node id)
exceeds the 32-bit integer range, instead of letting as.integer() silently
coerce it to NA (which corrupted graph reconstruction). Graphs with more
than ~2.1e9 nodes are not yet supported; the error says so explicitly.validateAppendOffset() now accepts a whole-number append offset above the
32-bit integer range instead of coercing it via as.integer() (which produced
NA), and buildTableSelectSQL() emits the offset as a full integer literal
rather than a 32-bit %d. Together these let a resource with more than ~2.1e9
rows stream without the append offset overflowing to NA. Offsets within the
32-bit range are still returned as integer, so index-column narrowing is
unchanged for the common case.transform() test is now skipped on R (< 4.6.0). The package requires
R (>= 4.6.0); on older R the paired S4Vectors ships a transform() whose
internal evaluation-frame stack-walk fails when transform() is called as a
lazily-forced argument promise. This avoids a spurious test ERROR on the
R-oldrel build of an unsupported R version.table() methods for DuckDBTable and DuckDBColumn now declare x as
a formal argument (function(x, ...)), conforming to the table generic in
Bioconductor-devel BiocGenerics, which dispatches on x
(setGeneric("table", signature = "x")) rather than on .... Under the
previous function(...) definition the package failed to install on
Bioc-devel with a conformMethod error ("formal arguments ... omitted in the
method definition cannot be in the signature"). Behavior is unchanged, and the
new signature also conforms against the ...-dispatch generic in the current
Bioconductor release.configureCloud() wires up remote object-storage reads on the shared
connection: it installs + loads the DuckDB httpfs extension up front (so a
firewalled environment fails early with actionable guidance rather than
mid-read) and applies s3_* credential / region settings (s3_region,
s3_access_key_id, s3_secret_access_key, s3_session_token, s3_endpoint,
s3_url_style, s3_use_ssl) resolved from R options (DuckDBDataFrame.s3_*)
or BIOCDUCKDB_S3_* environment variables, applied after httpfs loads
(DuckDB does not know the s3_* settings until then). See ?DuckDBConnection.
A dataset backed by a remote object-storage directory (s3://, gs://,
http(s)://, …) now resolves correctly. Because the VFS cannot be listed with
list.files(), the connection wrapper detects a remote URI, calls
configureCloud(), and lets DuckDB's httpfs glob expand
read_parquet('<uri>/**') — reads only; writing to object storage stays
unsupported (see BiocDuckDB::writeParquet).
configureOutOfCore() sets the DuckDB out-of-core engine knobs
(memory_limit, threads, temp_directory, preserve_insertion_order) on
the shared connection from R options or BIOCDUCKDB_* environment variables
(see ?DuckDBConnection). It runs automatically the first time
acquireDuckDBConn() creates the connection, so a large aggregation / sort /
join can spill instead of being OOM-killed, and a very large export can drop
insertion-order buffering.BIOCDUCKDB_EXTENSION_REPOSITORY (previously
MODL_DUCKDB_EXTENSION_REPOSITORY); the DuckDB-native
DUCKDB_EXTENSION_REPOSITORY fallback is unchanged.writeParquet(..., cluster_by = ) clusters rows on write so DuckDB row-group zonemaps
prune range queries. cluster_by is a character vector (lexicographic ordering),
zorder(cols) (a Morton / Z-order space-filling code over any number of numeric columns,
lowered SQL-side to a generated bit-interleave ORDER BY — no extension needed), or
hilbert(cols) (the native DuckDB spatial ST_Hilbert, exactly two numeric columns,
better locality, requires the spatial extension). Space-filling keys compute per-column
extents with one cheap MIN/MAX scan and bake them into a pushed-down ORDER BY, so the
DuckDB write path never materializes the table. A single physical order clusters one key:
supplying cluster_by overrides the default __index__ ordering. New exported
zorder() / hilbert() constructors and clusterSort() (the in-memory counterpart used
by the materializing data.frame / DataFrame write path). This generalizes the
coord-indexed points layout (DuckDBSpatial) to any multi-dimensional range-queried table
(spatial points, embeddings / reducedDims, genomic-interaction bins).loadExtension(), which installs (if needed) and loads a DuckDB
extension on a connection. It was already used internally to load spatial /
httpfs; exporting it lets companion packages ensure their required extension
is available on the shared connection — e.g. DuckDBSpatial now installs and
loads spatial on load via loadExtension(acquireDuckDBConn(), "spatial").DuckDBTable() and DuckDBDataFrame() gain a collevels argument that
restores factor (and ordered-factor) columns on materialization. Levels are
carried on the object and applied lazily when a column is collected, so
readParquet() can recover factors recorded in a product's schema. Columns
that are cast or otherwise transformed away from a character type are left
untouched.HUGEINT/UHUGEINT) and wide
DECIMAL (precision > 15) collapse to double, and unsigned 64-bit
(UBIGINT) collapses to signed integer64.NA compiled to SQL
NOT IN (..., NULL), which evaluates to UNKNOWN for all rows and returned
an empty result. NA-valued keys are now handled explicitly and never emitted
inside an IN list, reproducing base-R %in% semantics.integer64 keys. Because is.integer()
is FALSE for integer64, the fast-path was previously skipped for exactly
the BIGINT / row-number keycols where it matters most, falling back to an
IN list.SEMI JOIN rather than an
INNER JOIN against the temporary key table. SEMI JOIN is the correct
membership primitive: it neither duplicates rows when the key repeats nor
appends the join column.IN list
to a temporary-table SEMI/ANTI JOIN (from 10000 to 256). Large inline IN
lists dominate DuckDB's SQL compile time, so the crossover belongs in the low
hundreds.INSTALL spatial/httpfs from the public CDN
on every connect (which made an optional, spatial-only concern everyone's
problem). Instead acquireDuckDBConn() now:
BIOCDUCKDB_EXTENSION_REPOSITORY (or the DuckDB-native
DUCKDB_EXTENSION_REPOSITORY);LOADs only extensions already present in the extension directory
(a local, offline-safe operation), so a pre-provisioned or air-gapped cache
just works.loadExtension() is retained as a helper for explicit
or vendored loading.DuckDBTable
abstraction, SQL translation, connection model, and extension points, for
developers).DUCKDB_EXTENSION_DIRECTORY), so extension use works on shared or
read-only R libraries; optional extensions degrade gracefully.BiocCheck: replaced sapply() with
setNames(lapply(...)) where names must be preserved, used seq_len() in
place of 1:n, and tidied condition-signal messages.