Introduction

The Introduction to DuckDBArray vignette shows how a DuckDBMatrix behaves like an ordinary matrix while keeping its data on disk. This vignette asks the follow-up question: what does that cost, and what does it buy? We compare the DuckDB backend against three alternatives on real single-cell data:

1. an in-memory sparse matrix (Matrix’s dgCMatrix) — the fastest option when data fits in RAM, and our baseline;
2. HDF5Array — the long-standing Bioconductor backend for out-of-memory single-cell analysis;
3. TileDBArray — a tile-based on-disk backend.

The headline results below were produced offline on the full 10x Genomics 1.3 million brain-cell dataset (see Benchmark setup) and are rendered here from a bundled results file, so this vignette builds quickly. The A small, live comparison section runs a miniature version at build time so you can see the mechanics end to end.

The short version: for the columnar aggregations that dominate single-cell QC and feature selection, DuckDB matches an in-memory sparse matrix while staying on disk, and clearly beats the other disk-backed backends — without block-size tuning.

What DuckDBArray optimizes

The reason is that DuckDBArray turns array summaries into SQL aggregations that DuckDB executes directly over the Parquet file. Existing Bioconductor code picks these up automatically: calling rowSums() on a DuckDBMatrix is a SUM ... GROUP BY.

Standard MatrixGenerics row/column summaries:

Plus two helpers for count-based feature selection:

A small, live comparison

To show the mechanics without a large download, we build a small sparse matrix and run one summary on both an in-memory dgCMatrix and a DuckDBMatrix, confirming the results agree.

library(DuckDBArray)
library(Matrix)
library(MatrixGenerics)

set.seed(1L)
m <- Matrix(rpois(2000 * 400, lambda = 0.3), nrow = 2000, ncol = 400,
            sparse = TRUE)
rownames(m) <- paste0("Gene", seq_len(nrow(m)))
colnames(m) <- paste0("Cell", seq_len(ncol(m)))

path <- tempfile()
writeCoordArray(m, path)
mat <- DuckDBMatrix(path, datacol = "value",
                    keycols = list(index1 = setNames(seq_len(nrow(m)), rownames(m)),
                                   index2 = setNames(seq_len(ncol(m)), colnames(m))),
                    dimtbls = createDimTables(m))

## same answer, one in memory and one queried from disk
all.equal(rowVars(m), rowVars(mat))
#> [1] "Attributes: < names for current but not for target >"             
#> [2] "Attributes: < Length mismatch: comparison on first 0 components >"
#> [3] "target is numeric, current is array"
all.equal(unname(rowDeviances(m, family = "binomial")),
          unname(rowDeviances(mat, family = "binomial")))
#> [1] TRUE

At this size the in-memory matrix is faster — there is nothing to gain from going to disk. The advantage appears at scale, which is what the offline benchmark measures.

Benchmark setup

The full benchmark uses the 10x Genomics 1.3 million brain-cell dataset (27,998 x 1,306,127 UMI counts), the same dataset used by the HDF5Array performance vignette and available through ExperimentHub (accession EH1039). The headline table uses a 200,000-cell subset — the same size the HDF5Array performance vignette headlines, so the numbers are directly comparable; inst/scripts/ can run other cell counts.

Results

Rendered from the bundled offline results (inst/scripts/benchmark_results.rds); regenerate with inst/scripts/run_vignette_benchmarks.R (see that script’s header).

Best effort (full core budget)

Single-threaded (one core)

Configuration (200000 cells, 16-core budget, 1024 MB blocks). In-memory: single-threaded (Matrix). HDF5Array: SnowParam(8) over blocks; HDF5 file locking disabled; 1024 MB blocks. TileDBArray: SnowParam(8) x 2 threads/worker; file locks off; 1024 MB blocks. DuckDB: internal threads (SET threads = 16; serial = 1).

Main takeaways

• DuckDB’s real advantage is zero configuration. It reaches its performance by autotuning; matching it with the disk-backed backends takes SnowParam workers, per-worker contexts, thread budgeting, file-lock flags, and block-size tuning.
• Basic statistics match in-memory, on disk. For colSums and rowVars, DuckDB is on par with an in-memory dgCMatrix while never loading the matrix.
• Feature selection can beat in-memory. rowDeviances computes its aggregation entirely in SQL, avoiding the intermediate objects the dgCMatrix path builds.
• Parallelism helps the disk backends unevenly. With 8 SnowParam workers, TileDBArray speeds up well (rowVars ~5x, rowDeviances ~4x); HDF5Array’s gains are smaller and operation-dependent (~7x on rowNnzs, ~2x on rowDeviances, but essentially none on rowVars), reflecting contention on its single file. DuckDB shows no such unevenness — it scales ~11–13x across every operation, automatically. Compare the two regime tables.

Backend comparison

Running your own benchmarks

inst/scripts/ reproduces these numbers on your hardware and extends them to larger cell counts:

• run_vignette_benchmarks.R — the operations timed above; writes benchmark_results.rds.
• normalize_and_PCA.R, run_comparison.sh — the fuller normalization/PCA sweep, comparable to the HDF5Array performance vignette.

See inst/scripts/README.md for details. For higher-level scuttle/scran benchmarks (QC, normalization, variance modelling, marker detection), see the BiocDuckDB package.

Session information

sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats4    stats     graphics  grDevices utils     datasets  methods  
#> [8] base     
#> 
#> other attached packages:
#>  [1] DuckDBArray_0.99.2     DelayedArray_0.39.3    SparseArray_1.13.2    
#>  [4] S4Arrays_1.13.0        abind_1.4-8            MatrixGenerics_1.25.0 
#>  [7] matrixStats_1.5.0      Matrix_1.7-5           DuckDBDataFrame_0.99.6
#> [10] IRanges_2.47.2         S4Vectors_0.51.5       BiocGenerics_0.59.10  
#> [13] generics_0.1.4         bit64_4.8.2            BiocStyle_2.41.0      
#> 
#> loaded via a namespace (and not attached):
#>  [1] sass_0.4.10              lattice_0.22-9           digest_0.6.39           
#>  [4] magrittr_2.0.5           sparseMatrixStats_1.25.0 evaluate_1.0.5          
#>  [7] grid_4.6.1               blob_1.3.0               fastmap_1.2.0           
#> [10] jsonlite_2.0.0           DBI_1.3.0                BiocManager_1.30.27     
#> [13] purrr_1.2.2              jquerylib_0.1.4          duckdb_1.5.4.3          
#> [16] cli_3.6.6                rlang_1.3.0              dbplyr_2.6.0            
#> [19] XVector_0.53.0           withr_3.0.3              cachem_1.1.0            
#> [22] yaml_2.3.12              otel_0.2.0               tools_4.6.1             
#> [25] dplyr_1.2.1              assertthat_0.2.1         buildtools_1.0.0        
#> [28] vctrs_0.7.3              R6_2.6.1                 lifecycle_1.0.5         
#> [31] bit_4.6.0                arrow_25.0.0             pkgconfig_2.0.3         
#> [34] bslib_0.11.0             pillar_1.11.1            Rcpp_1.1.2              
#> [37] glue_1.8.1               xfun_0.60                tibble_3.3.1            
#> [40] tidyselect_1.2.1         sys_3.4.3                knitr_1.51              
#> [43] htmltools_0.5.9          rmarkdown_2.31           maketools_1.3.2         
#> [46] compiler_4.6.1