.writeDataFrameParquet() now fails loudly when a part's __index__ would
exceed the 32-bit range but index_max was not supplied, instead of silently
writing a floating-point (float64) key spine. Producers streaming past 2^31
rows must declare index_max (or Inf) so the column is typed int64
consistently across parts.
writeDatapackage() validates its inputs at the write seam (every resource
must be a descriptor with a single-string name and path; names must be
unique), so a malformed manifest fails fast at write time rather than
surfacing as an obscure error at read time.
writeParquet() (flat / data-frame path) can now write a resource with more
than ~2.1e9 rows without the __index__ column overflowing 32-bit integers. A
new optional index_max argument declares the resource's total index range;
when it exceeds the 32-bit limit the __index__ column is typed as a 64-bit
integer up front (part 0 included) so every streamed append part shares one
type, instead of narrowing part 0 to int32 and overflowing later parts on
cast (mirrors the coord-array max_dim typing). Pass index_max = Inf when
the total row count is unknown before streaming (e.g. graph edges). Small
resources are unaffected --- the index still narrows to the smallest integer
type. Requires the companion DuckDBDataFrame validateAppendOffset() fix.writeDatapackage() function, which assembles and writes a
Frictionless datapackage.json envelope from a list of resource descriptors.
The experiment-level writeParquet() methods (SummarizedExperiment,
MultiAssayExperiment) now single-source their manifest assembly through it,
and producers that build resources incrementally --- streaming a dataset too
large to hold in memory, or promoting from another store --- can emit a
conformant manifest without reconstructing an in-memory Bioconductor object.
NULL descriptors (returned by append/streaming parts) are dropped, so
accumulated resource lists can be passed straight through. This is the write
half of the ingest contract; the read half is readParquet() together with
the DuckDBMatrix()/DuckDBArray()/DuckDBTable() constructors, which attach
existing Parquet in place.writeDatapackage() for assembling a manifest and the DuckDB-backed
constructors for attaching existing coord-array Parquet in place.\value sections (roxygen @return) to the DuckDBDataFrame-spatial,
DuckDBDualSubset-class, DuckDBMatrix-scran, DuckDBMatrix-scuttle, and
MultiAssaySpatialExperiment-spatial man pages, documenting the values
returned by the spatial query, dual-subset, scran, scuttle, and MASE spatial
I/O methods. Resolves the R CMD BiocCheck "missing \value" WARNING.linkSpatialMap(), spatialViews(),
spatialCoordinateSystems(), validateSpatialMap()) now have runnable
examples backed by a minimal spatial MultiAssaySpatialExperiment fixture
bundled under inst/extdata/spatial_mase/ (generated by
inst/scripts/make-spatial-mase-fixture.R). This raises the share of
exported-object man pages with runnable examples above the 80% BiocCheck
threshold, resolving the R CMD BiocCheck ERROR.writeParquet() now right-sizes the internal integer index columns of
data-frame resources (the __index__/__sample__/__feature__ key spine and
graph-edge from/to) to their smallest Arrow integer type. This narrowing
was previously unreachable: it was gated on !flat_part, but
setupFlatParquetWrite() always resolves flat_part = TRUE, so data-frame
index columns were written as int32. The coord-array assay path already
narrowed, so only the tabular resources were affected.
scibis
(_optimize_integer_columns / _narrow_index): user data columns and
Hive partition-group columns keep their declared dtype, so a write→read
round-trip is dtype-preserving.reconcileParquetSchema(), whose result was previously
discarded) so every part of a streamed resource shares one integer width.arrowType in each resource's schema now reflects the narrowed
width, matching the physical Parquet.readParquet() can now read a dataset from remote object storage (s3://,
gs://, http(s)://, …). The datapackage.json sidecar is fetched over DuckDB
httpfs (jsonlite::read_json cannot fetch object-storage schemes) after
DuckDBDataFrame::configureCloud() installs/loads httpfs and applies any
s3_* credentials; the resource reads then prune row groups over the VFS the
same as on local disk. Local paths are unchanged.writeParquet() refuses a remote object-storage path with an actionable error
("write to a local directory and upload it"). Object stores have no atomic
directory swap and the append/refuse-to-overwrite guards rely on local file
existence, so BiocDuckDB writes stay local + whole-object; upload the written
directory afterwards (e.g. aws s3 cp --recursive).options(DuckDBDataFrame.threads = 1L) in setup.R) so parallel
floating-point reductions accumulate in a fixed order and tight-tolerance
expectations stay reproducible run-to-run. This is a test-only change; real
sessions use all cores.MultiAssaySpatialExperiment whose
spatial layers are DuckDB-backed (R/MultiAssaySpatialExperiment-query.R). MASE
stays DuckDB-free; this is BiocDuckDB's DuckDB engine operating on a MASE, using
DuckDBSpatial (Suggests) for spatial SQL.
spatialViews() registers each spatial layer and the spatialMap junction as
on-the-fly DuckDB temp views (lazy layers as views over their rendered SQL; no
materialization).linkSpatialMap() links assay observations to their spatial layer rows through
the full spatialMap key (assay, colname, element_type, region,
instance_id), returning a lazy DuckDBDataFrame.validateSpatialMap() checks referential integrity via DuckDB anti-joins
(orphan instance_id/colname, unknown layer, duplicate instance_id);
reports by default, errors with strict = TRUE.spatialElementJoin() runs an ST_* cross-element spatial join (via
DuckDBSpatial), first aligning both elements into a common coordinate system
through the coordinate-transform graph when coordinate_system is given.spatialCoordinateSystems() and per-element transforms stored in
metadata(mase)$transforms ("<element_type>/<region>" -> {cs -> transform},
RFC-5-shaped) round-trip through the standard MASE metadata annotations.writeParquet(..., cluster_by = ) threads a clustering key (DuckDBDataFrame::zorder() /
hilbert(), or a character vector) through every writeParquet method to the primitive
writers, so rows are physically ordered on write for DuckDB row-group zonemap pruning. The
lazy DuckDBTable/DuckDBDataFrame path lowers it SQL-side (no materialization); the
materializing data.frame/DataFrame path reorders in memory via
DuckDBDataFrame::clusterSort(). Requires DuckDBDataFrame (>= 0.9.27).sf / DataFrame with a geometry
column, e.g. a ShapesLayerList layer) to Parquet no longer errors with "flat
append ('append', 'part') is not supported for sf objects". Since flat writes
became the default, every such write was a flat part 0, which the sf branch of
.writeDataFrameParquet mistook for an append/part continuation. The guard now
rejects only a genuine append or a subsequent part (part > 0), matching the
lazy geometry write path, so a normal geometry write goes through
DuckDBSpatial::writeGeoParquet.readParquet() now restores factor columns (including ordered factors) in
the flat-table read paths, using the categories/categoriesOrdered recorded
in the product schema on write. Previously these columns came back as
character. (Factor restoration inside GRanges/SelfHits mcols is not yet
covered.)writeParquet()/readParquet() workflow,
the Frictionless/coordinate storage layout, operations pushed to SQL, genomic
coordinates, single-cell data, and the filter-realize-analyze pattern.scuttle/scran methods on a
DuckDBMatrix compared with in-memory and HDF5Array, rendered from
precomputed full-scale results (10x 1.3M brain cells) so the vignette builds
quickly.inst/scripts/ with the offline benchmark generator
(run_scran_scuttle_benchmarks.R) and the vignette table helpers
(make_timings_table.R), following the HDF5Array performance-vignette
precompute pattern.