Changes in version 0.99.5 - .writeDataFrameParquet() now fails loudly when a part's __index__ would exceed the 32-bit range but index_max was not supplied, instead of silently writing a floating-point (float64) key spine. Producers streaming past 2^31 rows must declare index_max (or Inf) so the column is typed int64 consistently across parts. - writeDatapackage() validates its inputs at the write seam (every resource must be a descriptor with a single-string name and path; names must be unique), so a malformed manifest fails fast at write time rather than surfacing as an obscure error at read time. Changes in version 0.99.4 Bug fixes - writeParquet() (flat / data-frame path) can now write a resource with more than ~2.1e9 rows without the __index__ column overflowing 32-bit integers. A new optional index_max argument declares the resource's total index range; when it exceeds the 32-bit limit the __index__ column is typed as a 64-bit integer up front (part 0 included) so every streamed append part shares one type, instead of narrowing part 0 to int32 and overflowing later parts on cast (mirrors the coord-array max_dim typing). Pass index_max = Inf when the total row count is unknown before streaming (e.g. graph edges). Small resources are unaffected --- the index still narrows to the smallest integer type. Requires the companion DuckDBDataFrame validateAppendOffset() fix. Changes in version 0.99.3 New features - Added the exported writeDatapackage() function, which assembles and writes a Frictionless datapackage.json envelope from a list of resource descriptors. The experiment-level writeParquet() methods (SummarizedExperiment, MultiAssayExperiment) now single-source their manifest assembly through it, and producers that build resources incrementally --- streaming a dataset too large to hold in memory, or promoting from another store --- can emit a conformant manifest without reconstructing an in-memory Bioconductor object. NULL descriptors (returned by append/streaming parts) are dropped, so accumulated resource lists can be passed straight through. This is the write half of the ingest contract; the read half is readParquet() together with the DuckDBMatrix()/DuckDBArray()/DuckDBTable() constructors, which attach existing Parquet in place. Documentation - Documented the storage layout as a targetable Frictionless contract in the package vignette (the new "Targeting the storage contract" section), covering writeDatapackage() for assembling a manifest and the DuckDB-backed constructors for attaching existing coord-array Parquet in place. Changes in version 0.99.2 Documentation - Added \value sections (roxygen @return) to the DuckDBDataFrame-spatial, DuckDBDualSubset-class, DuckDBMatrix-scran, DuckDBMatrix-scuttle, and MultiAssaySpatialExperiment-spatial man pages, documenting the values returned by the spatial query, dual-subset, scran, scuttle, and MASE spatial I/O methods. Resolves the R CMD BiocCheck "missing \value" WARNING. Changes in version 0.99.1 Bug fixes - The spatial-map query functions (linkSpatialMap(), spatialViews(), spatialCoordinateSystems(), validateSpatialMap()) now have runnable examples backed by a minimal spatial MultiAssaySpatialExperiment fixture bundled under inst/extdata/spatial_mase/ (generated by inst/scripts/make-spatial-mase-fixture.R). This raises the share of exported-object man pages with runnable examples above the 80% BiocCheck threshold, resolving the R CMD BiocCheck ERROR. Changes in version 0.9.29 Bug fixes - writeParquet() now right-sizes the internal integer index columns of data-frame resources (the __index__/__sample__/__feature__ key spine and graph-edge from/to) to their smallest Arrow integer type. This narrowing was previously unreachable: it was gated on !flat_part, but setupFlatParquetWrite() always resolves flat_part = TRUE, so data-frame index columns were written as int32. The coord-array assay path already narrowed, so only the tabular resources were affected. - Scoped to internal index columns to match scibis (_optimize_integer_columns / _narrow_index): user data columns and Hive partition-group columns keep their declared dtype, so a write→read round-trip is dtype-preserving. - On append, each part's index columns are pinned to the existing part-0 schema (via reconcileParquetSchema(), whose result was previously discarded) so every part of a streamed resource shares one integer width. - The recorded arrowType in each resource's schema now reflects the narrowed width, matching the physical Parquet. Changes in version 0.9.28 New features - readParquet() can now read a dataset from remote object storage (s3://, gs://, http(s)://, …). The datapackage.json sidecar is fetched over DuckDB httpfs (jsonlite::read_json cannot fetch object-storage schemes) after DuckDBDataFrame::configureCloud() installs/loads httpfs and applies any s3_* credentials; the resource reads then prune row groups over the VFS the same as on local disk. Local paths are unchanged. Changes - writeParquet() refuses a remote object-storage path with an actionable error ("write to a local directory and upload it"). Object stores have no atomic directory swap and the append/refuse-to-overwrite guards rely on local file existence, so BiocDuckDB writes stay local + whole-object; upload the written directory afterwards (e.g. aws s3 cp --recursive). Changes in version 0.9.27 Changes - The test harness pins DuckDB to a single thread (options(DuckDBDataFrame.threads = 1L) in setup.R) so parallel floating-point reductions accumulate in a fixed order and tight-tolerance expectations stay reproducible run-to-run. This is a test-only change; real sessions use all cores. Changes in version 0.9.26 New features - Cross-element DuckDB query layer over a MultiAssaySpatialExperiment whose spatial layers are DuckDB-backed (R/MultiAssaySpatialExperiment-query.R). MASE stays DuckDB-free; this is BiocDuckDB's DuckDB engine operating on a MASE, using DuckDBSpatial (Suggests) for spatial SQL. - spatialViews() registers each spatial layer and the spatialMap junction as on-the-fly DuckDB temp views (lazy layers as views over their rendered SQL; no materialization). - linkSpatialMap() links assay observations to their spatial layer rows through the full spatialMap key (assay, colname, element_type, region, instance_id), returning a lazy DuckDBDataFrame. - validateSpatialMap() checks referential integrity via DuckDB anti-joins (orphan instance_id/colname, unknown layer, duplicate instance_id); reports by default, errors with strict = TRUE. - spatialElementJoin() runs an ST_* cross-element spatial join (via DuckDBSpatial), first aligning both elements into a common coordinate system through the coordinate-transform graph when coordinate_system is given. - spatialCoordinateSystems() and per-element transforms stored in metadata(mase)$transforms ("/" -> {cs -> transform}, RFC-5-shaped) round-trip through the standard MASE metadata annotations. - writeParquet(..., cluster_by = ) threads a clustering key (DuckDBDataFrame::zorder() / hilbert(), or a character vector) through every writeParquet method to the primitive writers, so rows are physically ordered on write for DuckDB row-group zonemap pruning. The lazy DuckDBTable/DuckDBDataFrame path lowers it SQL-side (no materialization); the materializing data.frame/DataFrame path reorders in memory via DuckDBDataFrame::clusterSort(). Requires DuckDBDataFrame (>= 0.9.27). Bug fixes - Writing an in-memory geometry table (an sf / DataFrame with a geometry column, e.g. a ShapesLayerList layer) to Parquet no longer errors with "flat append ('append', 'part') is not supported for sf objects". Since flat writes became the default, every such write was a flat part 0, which the sf branch of .writeDataFrameParquet mistook for an append/part continuation. The guard now rejects only a genuine append or a subsequent part (part > 0), matching the lazy geometry write path, so a normal geometry write goes through DuckDBSpatial::writeGeoParquet. Changes in version 0.9.23 Bug fixes - readParquet() now restores factor columns (including ordered factors) in the flat-table read paths, using the categories/categoriesOrdered recorded in the product schema on write. Previously these columns came back as character. (Factor restoration inside GRanges/SelfHits mcols is not yet covered.) Documentation - Restructured the vignettes into a user-first set, replacing Constructing Experiment Objects with BiocDuckDB and scuttle and scran integration for DuckDBMatrix: - Introduction to BiocDuckDB --- the writeParquet()/readParquet() workflow, the Frictionless/coordinate storage layout, operations pushed to SQL, genomic coordinates, single-cell data, and the filter-realize-analyze pattern. - Benchmarking BiocDuckDB --- the SQL-optimized scuttle/scran methods on a DuckDBMatrix compared with in-memory and HDF5Array, rendered from precomputed full-scale results (10x 1.3M brain cells) so the vignette builds quickly. - Added inst/scripts/ with the offline benchmark generator (run_scran_scuttle_benchmarks.R) and the vignette table helpers (make_timings_table.R), following the HDF5Array performance-vignette precompute pattern. - Rewrote the README.