Introduction

The Introduction to BiocDuckDB vignette shows how an experiment can keep its assays on disk while presenting the standard Bioconductor API. This vignette asks the follow-up question: can the standard single-cell analysis methods run directly on that on-disk representation, and how fast?

BiocDuckDB implements the common scuttle and scran generics for DuckDBMatrix as SQL-optimized queries, so QC, normalization, variance modelling, and marker detection run on the Parquet-backed matrix without realizing it into memory. We compare those against the same generics on an in-memory dgCMatrix and on HDF5Array.

The headline results below were produced offline on the 10x Genomics 1.3 million brain-cell dataset (see Benchmark setup) and are rendered here from a bundled results file, so this vignette builds quickly. The A small, live comparison section runs a miniature version at build time.

What BiocDuckDB optimizes

Each method is implemented as per-gene / per-group SQL aggregation on the DuckDBMatrix, so the scan and the arithmetic happen in DuckDB and only the (small) result crosses back into R.

scuttle — QC, normalization, pseudo-bulk:

Function	Description
`perCellQCMetrics` / `perFeatureQCMetrics`	library size, detected genes / mean, detection rate
`librarySizeFactors` / `normalizeCounts`	size factors and normalization
`summarizeAssayByGroup`	pseudo-bulk aggregation (`GROUP BY`)

scran — variance modelling and markers:

Function	Description
`modelGeneVar` / `modelGeneVarByPoisson`	decompose technical vs biological variance
`correlatePairs`	pairwise gene correlations
`pairwiseTTests` / `findMarkers`	pairwise DE and candidate markers

A small, live comparison

To show the mechanics without a large download, we build a small sparse matrix and run one QC metric on both an in-memory dgCMatrix and a DuckDBMatrix, confirming the results agree.

library(BiocDuckDB)
library(DuckDBArray)
library(Matrix)
library(scuttle)

set.seed(1L)
m <- as(Matrix(rpois(2000 * 400, lambda = 0.3), nrow = 2000, ncol = 400,
               sparse = TRUE), "dgCMatrix")
rownames(m) <- paste0("Gene", seq_len(nrow(m)))
colnames(m) <- paste0("Cell", seq_len(ncol(m)))

path <- tempfile()
writeParquet(t(m), path)
mt <- t(m)
mat <- DuckDBMatrix(path, datacol = "value",
    keycols = list(index2 = setNames(seq_len(ncol(mt)), colnames(mt)),
                   index1 = setNames(seq_len(nrow(mt)), rownames(mt))),
    dimtbls = createDimTables(mt))

## same answer, one in memory and one queried from disk
qc_mem <- perCellQCMetrics(m)
qc_ddb <- perCellQCMetrics(mat)
all.equal(qc_mem$sum, qc_ddb$sum)
#> [1] TRUE

At this size the in-memory matrix is faster — there is nothing to gain from going to disk. The advantage appears at scale, which is what the offline benchmark measures.

Benchmark setup

The full benchmark uses the 10x Genomics 1.3 million brain-cell dataset (available through ExperimentHub, accession EH1039), subset to 12,500 cells — the size used by the original comparison. Each operation runs on three backends: an in-memory Matrix dgCMatrix, an HDF5Array, and a DuckDBMatrix. The in-memory and HDF5 backends run single-threaded; DuckDBMatrix autotunes DuckDB’s internal threads up to the core budget. The variance and marker operations run on log-normalized counts produced by each backend.

Results

Rendered from the bundled offline results (inst/scripts/benchmark_results.rds); regenerate with inst/scripts/run_scran_scuttle_benchmarks.R (see that script’s header).

Elapsed seconds per operation. Speedups > 1 favor DuckDB.
Operation	In-memory (s)	HDF5Array (s)	DuckDB (s)	vs HDF5Array (x)	vs in-memory (x)
perCellQCMetrics	0.56	3.09	0.10	30.6	5.5
perFeatureQCMetrics	0.77	4.44	0.09	48.3	8.4
summarizeAssayByGroup	0.11	1.25	0.58	2.2	0.2
normalizeCounts	0.78	1.95	0.15	13.0	5.2
modelGeneVar	0.28	10.42	1.02	10.3	0.3
correlatePairs	54.98	77.66	0.61	127.1	90.0
pairwiseTTests	6.98	15.95	7.11	2.2	1.0
findMarkers	21.67	31.17	22.16	1.4	1.0

Configuration: 27,998 genes x 12,500 cells, 16-core budget. In-memory: dgCMatrix. HDF5Array: 10x/HDF5 backend. DuckDB: DuckDBMatrix over Parquet (autotuned threads). All backends run the same scran/scuttle generics.

Main takeaways

The generics run unchanged on disk. perCellQCMetrics(), modelGeneVar(), findMarkers() and the rest dispatch to SQL-optimized methods for DuckDBMatrix, so existing scran/scuttle code works on a Parquet-backed matrix without changes or realization.
Against the out-of-core baseline, DuckDB wins every operation. Compared with HDF5Array — the fair comparison, since both keep the matrix on disk — DuckDB is faster on all eight operations, from ~1.4x (findMarkers) to well over 100x (correlatePairs).
QC and normalization beat in-memory too. perCellQCMetrics, perFeatureQCMetrics, and normalizeCounts are pure SUM/AVG aggregations and run several times faster than an in-memory dgCMatrix while never loading the matrix.
correlatePairs is the standout. Its sparse-aware SQL avoids the dense intermediates the other backends build, making it roughly two orders of magnitude faster than both HDF5Array and in-memory.
Some steps still favor in-memory at this scale. summarizeAssayByGroup and modelGeneVar are faster in memory (though DuckDB still beats HDF5Array); the DuckDB advantage on these grows as data outgrows RAM. Marker detection (pairwiseTTests, findMarkers) is on par with in-memory, since the per-gene statistics after the SQL GROUP BY are the same in every backend.

Consult the rendered table above for the measured numbers on your build’s bundled results.

When this matters

The value is compounding: because these methods run on the DuckDB-backed object directly, an analysis can go from raw counts through QC, normalization, feature selection, and marker detection before ever realizing the matrix into memory — realizing only the small, filtered result it actually needs. That is what makes the filter, realize, analyze pattern from the introduction practical on datasets far larger than RAM.

Running your own benchmarks

inst/scripts/run_scran_scuttle_benchmarks.R reproduces these numbers on your hardware (BENCH_NCELLS, BENCH_CORES; set BENCH_SYNTHETIC=1 to smoke-test without the EH1039 download). It writes benchmark_results.rds, which this vignette renders via inst/scripts/make_timings_table.R. For the lower-level matrix operations (colSums, rowVars, rowDeviances), see the DuckDBArray benchmarking vignette.

Session information

sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats4    stats     graphics  grDevices utils     datasets  methods  
#> [8] base     
#> 
#> other attached packages:
#>  [1] scuttle_1.23.1              SingleCellExperiment_1.35.2
#>  [3] SummarizedExperiment_1.43.0 Biobase_2.73.1             
#>  [5] BiocDuckDB_0.99.5           DuckDBGRanges_0.99.1       
#>  [7] GenomicRanges_1.65.1        Seqinfo_1.3.0              
#>  [9] DuckDBArray_0.99.2          DelayedArray_0.39.3        
#> [11] SparseArray_1.13.2          S4Arrays_1.13.0            
#> [13] abind_1.4-8                 MatrixGenerics_1.25.0      
#> [15] matrixStats_1.5.0           Matrix_1.7-5               
#> [17] DuckDBDataFrame_0.99.6      IRanges_2.47.2             
#> [19] S4Vectors_0.51.5            BiocGenerics_0.59.10       
#> [21] generics_0.1.4              bit64_4.8.2                
#> [23] BiocStyle_2.41.0           
#> 
#> loaded via a namespace (and not attached):
#>  [1] tidyselect_1.2.1                   blob_1.3.0                        
#>  [3] dplyr_1.2.1                        arrow_25.0.0                      
#>  [5] fastmap_1.2.0                      bluster_1.23.0                    
#>  [7] duckdb_1.5.4.3                     digest_0.6.39                     
#>  [9] rsvd_1.0.5                         lifecycle_1.0.5                   
#> [11] cluster_2.1.8.2                    sf_1.1-1                          
#> [13] statmod_1.5.2                      magrittr_2.0.5                    
#> [15] compiler_4.6.1                     rlang_1.3.0                       
#> [17] sass_0.4.10                        tools_4.6.1                       
#> [19] igraph_2.3.3                       yaml_2.3.12                       
#> [21] knitr_1.51                         dqrng_0.4.1                       
#> [23] bit_4.6.0                          classInt_0.4-11                   
#> [25] BiocParallel_1.47.0                KernSmooth_2.23-26                
#> [27] withr_3.0.3                        purrr_1.2.2                       
#> [29] sys_3.4.3                          grid_4.6.1                        
#> [31] beachmat_2.29.0                    e1071_1.7-17                      
#> [33] edgeR_4.11.4                       MultiAssayExperiment_1.39.0       
#> [35] cli_3.6.6                          rmarkdown_2.31                    
#> [37] otel_0.2.0                         metapod_1.21.0                    
#> [39] rjson_0.2.23                       DBI_1.3.0                         
#> [41] cachem_1.1.0                       proxy_0.4-29                      
#> [43] assertthat_0.2.1                   parallel_4.6.1                    
#> [45] BiocManager_1.30.27                XVector_0.53.0                    
#> [47] vctrs_0.7.3                        jsonlite_2.0.0                    
#> [49] BiocSingular_1.29.0                BiocNeighbors_2.7.2               
#> [51] irlba_2.3.7                        maketools_1.3.2                   
#> [53] magick_2.9.1                       locfit_1.5-9.12                   
#> [55] limma_3.69.2                       jquerylib_0.1.4                   
#> [57] units_1.0-1                        glue_1.8.1                        
#> [59] codetools_0.2-20                   ScaledMatrix_1.21.0               
#> [61] tibble_3.3.1                       pillar_1.11.1                     
#> [63] htmltools_0.5.9                    MultiAssaySpatialExperiment_0.99.2
#> [65] R6_2.6.1                           dbplyr_2.6.0                      
#> [67] evaluate_1.0.5                     lattice_0.22-9                    
#> [69] SpatialExperiment_1.23.0           scran_1.41.1                      
#> [71] bslib_0.11.0                       class_7.3-23                      
#> [73] Rcpp_1.1.2                         xfun_0.60                         
#> [75] buildtools_1.0.0                   pkgconfig_2.0.3

Benchmarking BiocDuckDB