This vignette is for developers: it documents how the DuckDB backend satisfies the DelayedArray seed contract and how array operations become SQL, so you can maintain the package, extend it, or implement a similar backend. For day-to-day use see Introduction to DuckDBArray; for performance see Benchmarking DuckDBArray.
The design follows the pattern DelayedArray prescribes for backends
(see its Implementing a DelayedArray backend vignette): a
lightweight seed class that knows how to read blocks of
data, wrapped by the standard DelayedArray /
DelayedMatrix classes that supply the lazy-evaluation
machinery.
DuckDBArray layers three classes on the DuckDBDataFrame foundation:
| Class | Extends | Role |
|---|---|---|
DuckDBArraySeed |
Array |
the backend seed: wraps a DuckDBTable, reads
blocks |
DuckDBArray |
DelayedArray |
user-facing N-dimensional array |
DuckDBMatrix |
DelayedMatrix |
2-D specialization |
The seed holds no data; it holds a DuckDBTable (from
DuckDBDataFrame)
describing a Parquet file in coordinate (COO) form —
one or more key columns giving each entry’s position and a
single data column giving its value. DelayedArray calls the
seed’s extract_array() /
extract_sparse_array() methods to materialize only the
blocks an operation needs; everything else (subsetting, arithmetic,
log1p(), …) is recorded lazily by DelayedArray and resolved
against the seed on demand.
A DelayedArray seed must answer three questions: its shape, and how
to read a dense or a sparse block. DuckDBArraySeed
implements all three.
dim() and dimnames()The dimensions come from the key columns: each key column is one dimension, its levels are that dimension’s extent and names.
array_df <- expand.grid(i = 1:4, j = 1:3, k = 1:2)
array_df$value <- rpois(nrow(array_df), lambda = 2)
array_path <- tempfile(fileext = ".parquet")
arrow::write_parquet(array_df, array_path)
seed <- DuckDBArraySeed(array_path, datacol = "value",
keycols = list(i = 1:4, j = 1:3, k = 1:2))
dim(seed)
#> [1] 4 3 2extract_array()extract_array(seed, index) returns an ordinary array for
the requested index (a list of integer subscripts, one per
dimension). Internally it subsets the underlying
DuckDBTable to those positions, materializes the result
with as.data.frame(), scatters the values into an array of
the block’s shape, and fills absent coordinates with the seed’s
fill value.
extract_sparse_array()Because the data is stored as COO, sparse extraction is natural: the
seed returns a COO_SparseArray (from SparseArray)
built directly from the rows DuckDB returns, with no densification.
sparse_df <- data.frame(i = c(1, 1, 2, 4), j = c(1, 3, 2, 3),
k = c(1, 1, 2, 1), value = c(5, 3, 8, 7))
sparse_path <- tempfile(fileext = ".parquet")
arrow::write_parquet(sparse_df, sparse_path)
sparse_seed <- DuckDBArraySeed(sparse_path, datacol = "value",
keycols = list(i = 1:4, j = 1:3, k = 1:2))
extract_sparse_array(sparse_seed, list(1:4, 1:3, 1))
#> <4 x 3 x 1 SparseArray> of type "double" [nzcount=3 (25%)]:
#> ,,1
#> j
#> i 1 2 3
#> 1 5 0 3
#> 2 0 0 0
#> 3 0 0 0
#> 4 0 0 7is_sparse() on the seed reports whether sparse
extraction is available, which is how DelayedArray decides to call
extract_sparse_array() in preference to
extract_array().
DuckDBArray() / DuckDBMatrix() wrap a seed
(or construct one from a path) in the standard DelayedArray classes:
arr <- DuckDBArray(seed)
class(arr)
#> [1] "DuckDBArray"
#> attr(,"package")
#> [1] "DuckDBArray"
seedClass <- class(seed(arr))
seedClass
#> [1] "DuckDBArraySeed"
#> attr(,"package")
#> [1] "DuckDBArray"From here the full DelayedArray
API applies unchanged — lazy arithmetic, subsetting,
blockApply(), realize() — because
DuckDBArray is a DelayedArray.
The performance story rests on one idea: rather than let
DelayedArray’s generic block machinery loop over blocks, DuckDBArray
overrides the reductions so they become single SQL aggregations on the
DuckDBTable. A rowSums() on a
DuckDBMatrix is
and rowVars() the same with
VAR_SAMP(value). The matrixStats
/ MatrixGenerics
methods (rowSums/colSums,
rowMeans, rowVars, rowSds,
rowMins/rowMaxs,
rowNnzs/colNnzs, rowDeviances)
are all implemented this way, at the DuckDBTable level, so
DuckDB does the scan and the arithmetic and only the (small) per-margin
result crosses back into R.
Operations without an aggregation form (element-wise math,
subsetting) fall back to DelayedArray’s lazy machinery over
extract_array() — correct for any backend, and cheap
because the seed reads only the needed blocks.
For large arrays, createDimTables() builds per-dimension
lookup tables that map indices to grid partitions, letting DuckDB prune
to just the Parquet partitions a query touches (and enabling
independent, parallel processing of partitions).
library(S4Arrays)
grid <- RegularArrayGrid(dim(state.x77), c(10, 4))
dim_tables <- createDimTables(state.x77, grid = grid)
dim_tables
#> $index1
#> index1_group
#> 1 1
#> 2 1
#> 3 1
#> 4 1
#> 5 1
#> 6 1
#> 7 1
#> 8 1
#> 9 1
#> 10 1
#> 11 2
#> 12 2
#> 13 2
#> 14 2
#> 15 2
#> 16 2
#> 17 2
#> 18 2
#> 19 2
#> 20 2
#> 21 3
#> 22 3
#> 23 3
#> 24 3
#> 25 3
#> 26 3
#> 27 3
#> 28 3
#> 29 3
#> 30 3
#> 31 4
#> 32 4
#> 33 4
#> 34 4
#> 35 4
#> 36 4
#> 37 4
#> 38 4
#> 39 4
#> 40 4
#> 41 5
#> 42 5
#> 43 5
#> 44 5
#> 45 5
#> 46 5
#> 47 5
#> 48 5
#> 49 5
#> 50 5
#>
#> $index2
#> index2_group
#> 1 1
#> 2 1
#> 3 1
#> 4 1
#> 5 2
#> 6 2
#> 7 2
#> 8 2Dimension tables are attached to a seed via the dimtbls=
argument (this is what writeParquet() in BiocDuckDB
does when it writes an experiment object) and read back with
dimtbls().
A DuckDBArray interoperates with the rest of the
DelayedArray ecosystem: it can be realized to another backend or to an
ordinary array, and combined in delayed expressions with other
seeds.
Because the on-disk representation is plain Parquet, the same files are readable outside R (Python, Julia, DuckDB itself, cloud query engines) — a property the BiocDuckDB package relies on for its language-neutral on-disk experiment format.
sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so; LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] DuckDBArray_0.99.2 DelayedArray_0.39.3 SparseArray_1.13.2
#> [4] S4Arrays_1.13.0 abind_1.4-8 MatrixGenerics_1.25.0
#> [7] matrixStats_1.5.0 Matrix_1.7-5 DuckDBDataFrame_0.99.6
#> [10] IRanges_2.47.2 S4Vectors_0.51.5 BiocGenerics_0.59.10
#> [13] generics_0.1.4 bit64_4.8.2 BiocStyle_2.41.0
#>
#> loaded via a namespace (and not attached):
#> [1] sass_0.4.10 lattice_0.22-9 digest_0.6.39
#> [4] magrittr_2.0.5 evaluate_1.0.5 grid_4.6.1
#> [7] blob_1.3.0 fastmap_1.2.0 jsonlite_2.0.0
#> [10] DBI_1.3.0 BiocManager_1.30.27 purrr_1.2.2
#> [13] jquerylib_0.1.4 duckdb_1.5.4.3 cli_3.6.6
#> [16] rlang_1.3.0 dbplyr_2.6.0 XVector_0.53.0
#> [19] withr_3.0.3 cachem_1.1.0 yaml_2.3.12
#> [22] otel_0.2.0 tools_4.6.1 dplyr_1.2.1
#> [25] assertthat_0.2.1 buildtools_1.0.0 vctrs_0.7.3
#> [28] R6_2.6.1 lifecycle_1.0.5 bit_4.6.0
#> [31] arrow_25.0.0 pkgconfig_2.0.3 bslib_0.11.0
#> [34] pillar_1.11.1 glue_1.8.1 xfun_0.60
#> [37] tibble_3.3.1 tidyselect_1.2.1 sys_3.4.3
#> [40] knitr_1.51 htmltools_0.5.9 rmarkdown_2.31
#> [43] maketools_1.3.2 compiler_4.6.1