Package 'DuckDBArray' reference manual

Title:	DuckDB-Backed DelayedArray Implementation
Description:	Provides DuckDB-backed implementations of DelayedArray and DelayedMatrix classes for efficient, out-of-memory operations on large array and matrix datasets. The package includes DuckDBArray, DuckDBArraySeed, and DuckDBMatrix classes with optimized matrixStats methods. Supports sparse arrays, grid partitioning, and parallel processing for scalable genomic data analysis.
Authors:	Patrick Aboyoun [aut, cre], Genentech, Inc. [cph]
Maintainer:	Patrick Aboyoun <[email protected]>
License:	MIT + file LICENSE
Version:	0.99.2
Built:	2026-07-23 11:10:39 UTC
Source:	https://github.com/BiocStaging/DuckDBArray

Create Dimension Lookup Tables

Description

Creates lookup tables for array dimensions that map dimension names to their corresponding indices and grid group identifiers. These tables are useful for understanding the structure of partitioned arrays and for reconstructing the original array from coordinate format data.

Usage

createDimTables(
  x,
  indexcols = names(dimnames(x)) %||% sprintf("index%d", seq_along(dim(x))),
  grid = defaultAutoGrid(COO_SparseArray(dim(x))),
  grid_suffix = "_group",
  BPPARAM = getAutoBPPARAM()
)
createDimTables(
  x,
  indexcols = names(dimnames(x)) %||% sprintf("index%d", seq_along(dim(x))),
  grid = defaultAutoGrid(COO_SparseArray(dim(x))),
  grid_suffix = "_group",
  BPPARAM = getAutoBPPARAM()
)

Arguments

x

An array-like object with dimensions and dimension names. Must be coercible to a SparseArray object. Common examples include matrices, arrays, and table objects.

indexcols

A character vector of column names for the dimensions of the array.

grid

An optional ArrayGrid to use for partitioning the array. If NULL, uses defaultAutoGrid(COO_SparseArray(dim(x))).

grid_suffix

A character string to append to the partitioning directories if the grid contains more than one cell. Defaults to "_group".

BPPARAM

A BiocParallelParam object to use for parallel processing when grid contains multiple cells. Defaults to getAutoBPPARAM().

Details

This function is particularly useful when working with partitioned arrays created by writeCoordArray. The returned lookup tables provide a mapping between dimension names and their positions in the original array, as well as identify which grid partition contains each dimension element.

When the grid contains only one cell, the lookup tables contain only the dimension names and their indices. When multiple grid cells are present, additional columns indicate which grid group each dimension element belongs to.

Value

A named list of data frames, one for each dimension in x. Each data frame contains:

The dimension names as row names
A column with the corresponding indices
A column with the grid group identifier (if grid has multiple cells)

The list is named according to indexcols.

Author(s)

Patrick Aboyoun

Examples

# Write the state.x77 matrix to multiple Parquet files using grid partitioning
tf <- tempfile()
state_grid <- RegularArrayGrid(dim(state.x77), c(10, 4))
writeCoordArray(state.x77, file.path(tf, "state"), grid = state_grid)
dimtbls <- createDimTables(state.x77, grid = state_grid)
list.files(tf, full.names = TRUE, recursive = TRUE)

# Write the state.x77 matrix to multiple Parquet files using grid partitioning
tf <- tempfile()
state_grid <- RegularArrayGrid(dim(state.x77), c(10, 4))
writeCoordArray(state.x77, file.path(tf, "state"), grid = state_grid)
dimtbls <- createDimTables(state.x77, grid = state_grid)
list.files(tf, full.names = TRUE, recursive = TRUE)

DuckDBArray objects

Description

The DuckDBArray class is a DelayedArray subclass for representing and operating on a DuckDB table.

All the operations available for DelayedArray objects work on DuckDBArray objects.

Value

The DuckDBArray() constructor returns a DuckDBArray object. The accessors return the component of x named by the accessor (e.g. dim(x) returns an integer vector of the array dimensions).

Constructor

DuckDBArray(conn, datacol, keycols, dimtbls = NULL, type = NULL):

Creates a DuckDBArray object.

conn: Either a character vector containing the paths to parquet, csv, or gzipped csv data files; a string that defines a duckdb read_* data source; a DuckDBDataFrame object; or a tbl_duckdb_connection object.
datacol: Either a string specifying the column from conn or a named expression that will be evaluated in the context of conn that defines the values in the array.
keycols: Either a character vector of column names from conn that will specify the dimension names, or a named list of character vectors where the names of the list specify the dimension names and the character vectors set the distinct values for the dimension names.
dimtbls: Either NULL, a named DataFrameList object, or an environment containing a named DataFrameList "dimtbls" element that specifies the dimension tables associated with the keycols. The name of the list elements match the names of the keycols list. Additionally, the DataFrame objects have row names that match the distinct values of the corresponding keycols list element and columns that define partitions in the data table for efficient querying.
type: String specifying the type of the data values; one of "logical", "integer", "integer64", "double", or "character". If NULL, it is determined by inspecting the data.

Accessors

In the code snippets below, x is a DuckDBArray object:

dim(x):: An integer vector of the array dimensions.
dimnames(x), dimnames(x) <- value:: Get or set the array dimension names.
dimtbls(x, drop = TRUE), dimtbls(x) <- value:: Get or set the list of dimension tables used to define partitions for efficient queries. If drop = TRUE, then it returns a named DataFrameList object, else it returns an environment containing a dimtbls named DataFrameList element.
type(x), type(x) <- value:: Get or set the data type of the array elements; one of "logical", "integer", "integer64", "double", or "character".

Subsetting

In the code snippets below, x is a DuckDBArray object:

x[i, j, ..., drop = TRUE]:: Returns a new DuckDBArray object. Empty dimensions are dropped if drop = TRUE.

Transposition

In the code snippets below, x is a DuckDBArray object:

aperm(a, perm):: Returns a new DuckDBArray object with the dimensions permuted according to the perm vector.
t(x):: For two-dimensional arrays, returns a new DuckDBArray object with the dimensions transposed.

Author(s)

Patrick Aboyoun

Examples

# Create a data.frame from the Titanic data
df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- Titanic[as.matrix(df)]

# Write data to a parquet file
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)

pqarray <- DuckDBArray(tf, datacol = "fate", keycols = c("Class", "Sex", "Age", "Survived"))

# Create a data.frame from the Titanic data
df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- Titanic[as.matrix(df)]

# Write data to a parquet file
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)

pqarray <- DuckDBArray(tf, datacol = "fate", keycols = c("Class", "Sex", "Age", "Survived"))

DuckDBArray row / column summarization methods

Description

Row / column summarization methods for DuckDBArray objects.

Value

Method return types are documented in the sections above.

Row / Column Summarization Methods

In the code snippets below, x is a DuckDBArray object:

rowCounts(x, value = TRUE):

Calculates the row counts of x that are equal to value.

value: The value to count.

colCounts(x, value = TRUE):

Calculates the column counts of x that are equal to value.

value: The value to count.

rowMaxs(x):

Calculates the row maxima of x.

colMaxs(x):

Calculates the column maxima of x.

rowMeans(x, dims = 1):

Calculates the row means of x.

dims: An integer specifying which dimensions to average over, namely dims + 1, ....

colMeans(x, dims = 1):

Calculates the column means of x.

dims: An integer specifying which dimensions to average over, namely 1:dims.

rowMins(x):

Calculates the row minima of x.

colMins(x):

Calculates the column minima of x.

rowSums(x, dims = 1):

Calculates the row sums of x.

dims: An integer specifying which dimensions to sum over, namely dims + 1, ....

colSums(x, dims = 1):

Calculates the column sums of x.

dims: An integer specifying which dimensions to sum over, namely 1:dims.

rowSds(x):

Calculates the row standard deviations of x.

colSds(x):

Calculates the column standard deviations of x.

rowVars(x):

Calculates the row variances of x.

colVars(x):

Calculates the column variances of x.

Author(s)

Patrick Aboyoun

Examples

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
pqarray <- DuckDBArray(tf, datacol = "fate",
                       keycols = c("Class", "Sex", "Age", "Survived"))
colMeans(pqarray)
rowSums(pqarray, dims = 2L)

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
pqarray <- DuckDBArray(tf, datacol = "fate",
                       keycols = c("Class", "Sex", "Age", "Survived"))
colMeans(pqarray)
rowSums(pqarray, dims = 2L)

Common operations on DuckDBArray objects

Description

Common operations on DuckDBArray objects.

Value

Method return types are documented in the sections above.

Group Generics

DuckDBArray objects have support for S4 group generic functionality:

Arith: "+", "-", "*", "^", "%%", "%/%", "/"
Compare: "==", ">", "<", "!=", "<=", ">="
Logic: "&", "|", "!"
Ops: "Arith", "Compare", "Logic"
Math: "abs", "sign", "sqrt", "ceiling", "floor", "trunc", "log", "log10", "log2", "acos", "acosh", "asin", "asinh", "atan", "atanh", "exp", "expm1", "cos", "cosh", "sin", "sinh", "tan", "tanh", "gamma", "lgamma"
Summary: "max", "min", "range", "prod", "sum", "any", "all"

See S4groupGeneric for more details.

Numerical Data Methods

In the code snippets below, x is a DuckDBArray object:

is.finite(x):

Returns a DuckDBArray containing logicals that indicate which values are finite.

is.infinite(x):

Returns a DuckDBArray containing logicals that indicate which values are infinite.

is.nan(x):

Returns a DuckDBArray containing logicals that indicate which values are Not a Number.

sweep(x, MARGIN, STATS, FUN = "/"):

Sweeps out array summaries from x. Applies FUN to each element of x using the corresponding value from STATS based on MARGIN.

MARGIN: integer specifying the dimension (1 for rows, 2 for columns)
STATS: numeric vector with length equal to the extent of dimension MARGIN
FUN: function to be used to carry out the sweep

Sparsity Methods

In the code snippets below, x is a DuckDBArray object:

is_nonzero(x):: Returns a DuckDBArray containing logicals that indicate if the values in each of the columns of x are non-zero.
nzcount(x):: Returns the total number of non-zero values.
nzvals(x):: Returns the non-zero values.
is_sparse(x):: Returns TRUE since data are stored in a sparse array representation.

Author(s)

Patrick Aboyoun

Examples

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
pqarray <- DuckDBArray(tf, datacol = "fate",
                       keycols = c("Class", "Sex", "Age", "Survived"))
is_sparse(pqarray)
rowSums(pqarray)
pqarray + 1L

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
pqarray <- DuckDBArray(tf, datacol = "fate",
                       keycols = c("Class", "Sex", "Age", "Survived"))
is_sparse(pqarray)
rowSums(pqarray)
pqarray + 1L

DuckDBArraySeed objects

Description

DuckDBArraySeed is a low-level helper class for representing a pointer to a DuckDB table.

Note that a DuckDBArraySeed object is not intended to be used directly. Most end users will typically create and manipulate a higher-level DuckDBArray object instead. See ?DuckDBArray for more information.

Value

The DuckDBArraySeed() constructor returns a DuckDBArraySeed object; the accessors return the component of x named by the accessor.

Constructor

DuckDBArraySeed(conn, datacol, keycols, dimtbls = NULL, type = NULL):

Creates a DuckDBArraySeed object.

conn: Either a character vector containing the paths to parquet, csv, or gzipped csv data files; a string that defines a duckdb read_* data source; a DuckDBDataFrame object; or a tbl_duckdb_connection object.
datacol: Either a string specifying the column from conn or a named expression that will be evaluated in the context of conn that defines the values in the array.
keycols: Either a character vector of column names from conn that will specify the dimension names, or a named list of character vectors where the names of the list specify the dimension names and the character vectors set the distinct values for the dimension names.
dimtbls: Either NULL, a named DataFrameList object, or an environment containing a named DataFrameList "dimtbls" element that specifies the dimension tables associated with the keycols. The name of the list elements match the names of the keycols list. Additionally, the DataFrame objects have row names that match the distinct values of the corresponding keycols list element and columns that define partitions in the data table for efficient querying.
type: String specifying the type of the data values; one of "logical", "integer", "integer64", "double", or "character". If NULL, it is determined by inspecting the data.

Accessors

In the code snippets below, x is a DuckDBArraySeed object:

dim(x):: An integer vector of the array dimensions.
dimnames(x), dimnames(x) <- value:: Get or set the array dimension names.
dimtbls(x, drop = TRUE), dimtbls(x) <- value:: Get or set the list of dimension tables used to define partitions for efficient queries. If drop = TRUE, then it returns a named DataFrameList object, else it returns an environment containing a dimtbls named DataFrameList element.
type(x), type(x) <- value:: Get or set the data type of the array elements; one of "logical", "integer", "integer64", "double", or "character".

Subsetting

In the code snippets below, x is a DuckDBArraySeed object:

x[i, j, ..., drop = TRUE]:: Returns a new DuckDBArraySeed object. Empty dimensions are dropped if drop = TRUE.

Transposition

In the code snippets below, x is a DuckDBArraySeed object:

aperm(a, perm):: Returns a new DuckDBArraySeed object with the dimensions permuted according to the perm vector.
t(x):: For two-dimensional arrays, returns a new DuckDBArraySeed object with the dimensions transposed.

Author(s)

Patrick Aboyoun

Examples

# Create a data.frame from the Titanic data
df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])

# Write data to a parquet file
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)

pqaseed <- DuckDBArraySeed(tf, datacol = "fate", keycols = c("Class", "Sex", "Age", "Survived"))

# Create a data.frame from the Titanic data
df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])

# Write data to a parquet file
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)

pqaseed <- DuckDBArraySeed(tf, datacol = "fate", keycols = c("Class", "Sex", "Age", "Survived"))

DuckDBArraySeed row / column summarization methods

Description

Row / column summarization methods for DuckDBArraySeed objects.

Value

Method return types are documented in the sections above.

Row / Column Summarization Methods

In the code snippets below, x is a DuckDBArraySeed object:

rowCounts(x, value = TRUE):

Calculates the row counts of x that are equal to value.

value: The value to count.

colCounts(x, value = TRUE):

Calculates the column counts of x that are equal to value.

value: The value to count.

rowMaxs(x):

Calculates the row maxima of x.

colMaxs(x):

Calculates the column maxima of x.

rowMeans(x, dims = 1):

Calculates the row means of x.

dims: An integer specifying which dimensions to average over, namely dims + 1, ....

colMeans(x, dims = 1):

Calculates the column means of x.

dims: An integer specifying which dimensions to average over, namely 1:dims.

rowMins(x):

Calculates the row minima of x.

colMins(x):

Calculates the column minima of x.

rowSums(x, dims = 1):

Calculates the row sums of x.

dims: An integer specifying which dimensions to sum over, namely dims + 1, ....

colSums(x, dims = 1):

Calculates the column sums of x.

dims: An integer specifying which dimensions to sum over, namely 1:dims.

rowSds(x):

Calculates the row standard deviations of x.

colSds(x):

Calculates the column standard deviations of x.

rowVars(x):

Calculates the row variances of x.

colVars(x):

Calculates the column variances of x.

Author(s)

Patrick Aboyoun

Examples

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
seed <- DuckDBArraySeed(tf, datacol = "fate",
                        keycols = c("Class", "Sex", "Age", "Survived"))
colMeans(seed)
rowSums(seed, dims = 2L)

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
seed <- DuckDBArraySeed(tf, datacol = "fate",
                        keycols = c("Class", "Sex", "Age", "Survived"))
colMeans(seed)
rowSums(seed, dims = 2L)

Common operations on DuckDBArraySeed objects

Description

Common operations on DuckDBArraySeed objects.

Value

Method return types are documented in the sections above.

Group Generics

DuckDBArraySeed objects have support for S4 group generic functionality:

Arith: "+", "-", "*", "^", "%%", "%/%", "/"
Compare: "==", ">", "<", "!=", "<=", ">="
Logic: "&", "|", "!"
Ops: "Arith", "Compare", "Logic"
Math: "abs", "sign", "sqrt", "ceiling", "floor", "trunc", "log", "log10", "log2", "acos", "acosh", "asin", "asinh", "atan", "atanh", "exp", "expm1", "cos", "cosh", "sin", "sinh", "tan", "tanh", "gamma", "lgamma"
Summary: "max", "min", "range", "prod", "sum", "any", "all"

See S4groupGeneric for more details.

Numerical Data Methods

In the code snippets below, x is a DuckDBArraySeed object:

is.finite(x):

Returns a DuckDBArraySeed containing logicals that indicate which values are finite.

is.infinite(x):

Returns a DuckDBArraySeed containing logicals that indicate which values are infinite.

is.nan(x):

Returns a DuckDBArraySeed containing logicals that indicate which values are Not a Number.

sweep(x, MARGIN, STATS, FUN = "/"):

Sweeps out array summaries from x. Applies FUN to each element of x using the corresponding value from STATS based on MARGIN.

MARGIN: integer specifying the dimension (1 for rows, 2 for columns)
STATS: numeric vector with length equal to the extent of dimension MARGIN
FUN: function to be used to carry out the sweep

Sparsity Methods

In the code snippets below, x is a DuckDBArraySeed object:

is_nonzero(x):: Returns a DuckDBArraySeed containing logicals that indicate if the values in each of the columns of x are non-zero.
nzcount(x):: Returns the total number of non-zero values.
is_sparse(x):: Returns TRUE since data are stored in a sparse array representation.

Author(s)

Patrick Aboyoun

Examples

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
seed <- DuckDBArraySeed(tf, datacol = "fate",
                        keycols = c("Class", "Sex", "Age", "Survived"))
is_sparse(seed)
nzcount(seed)
seed + 1L

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
seed <- DuckDBArraySeed(tf, datacol = "fate",
                        keycols = c("Class", "Sex", "Age", "Survived"))
is_sparse(seed)
nzcount(seed)
seed + 1L

DuckDBMatrix objects

Description

The DuckDBMatrix class is a DelayedMatrix subclass for representing and operating on a DuckDB table.

All the operations available for DelayedMatrix objects work on DuckDBMatrix objects.

Value

The DuckDBMatrix() constructor returns a DuckDBMatrix object; the accessors return the component of x named by the accessor.

Constructor

DuckDBMatrix(conn, datacol, row, col, keycols = c(row, col), dimtbls = NULL, type = NULL):

Creates a DuckDBMatrix object.

conn: Either a character vector containing the paths to parquet, csv, or gzipped csv data files; a string that defines a duckdb read_* data source; a DuckDBDataFrame object; or a tbl_duckdb_connection object.
datacol: Either a string specifying the column from conn or a named expression that will be evaluated in the context of conn that defines the values in the matrix.
row: Either a string that specifies the column in conn that specifies the row names of the matrix, or a named list containing a single character vector that defines the column in conn for the row names and its values.
col: Either a string that specifies the column in conn that specifies the column names of the matrix, or a named list containing a single character vector that defines the column in conn for the column names and its values.
keycols: An optional character vector that define the names of the columns in conn for the rows and columns of the matrix, or a named list of character vectors where the names of the list define rows and columns and the character vectors define distinct values for the rows and columns.
dimtbls: Either NULL, a named DataFrameList object, or an environment containing a named DataFrameList "dimtbls" element that specifies the dimension tables associated with the keycols. The name of the list elements match the names of the keycols list. Additionally, the DataFrame objects have row names that match the distinct values of the corresponding keycols list element and columns that define partitions in the data table for efficient querying.
type: String specifying the type of the data values; one of "logical", "integer", "integer64", "double", or "character". If NULL, it is determined by inspecting the data.

Accessors

In the code snippets below, x is a DuckDBMatrix object:

dim(x):: An integer vector of the array dimensions.
dimnames(x), dimnames(x) <- value:: Get or set the array dimension names.
dimtbls(x, drop = TRUE), dimtbls(x) <- value:: Get or set the list of dimension tables used to define partitions for efficient queries. If drop = TRUE, then it returns a named DataFrameList object, else it returns an environment containing a dimtbls named DataFrameList element.
type(x), type(x) <- value:: Get or set the data type of the array elements; one of "logical", "integer", "integer64", "double", or "character".

Subsetting

In the code snippets below, x is a DuckDBMatrix object:

x[i, j, ..., drop = TRUE]:: Returns a new DuckDBMatrix object. Empty dimensions are dropped if drop = TRUE.

Transposition

In the code snippets below, x is a DuckDBMatrix object:

aperm(a, perm):: Returns a new DuckDBMatrix object with the dimensions permuted according to the perm vector.
t(x):: Returns a new DuckDBMatrix object with the rows and columns transposed.

Author(s)

Patrick Aboyoun

Examples

# Create a data.frame from a matrix
df <- data.frame(
  rowname = rep(rownames(state.x77), times = ncol(state.x77)),
  colname = rep(colnames(state.x77), each = nrow(state.x77)),
  value = as.vector(state.x77)
)

# Write data to a parquet file
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)

pqmat <- DuckDBMatrix(tf, row = "rowname", col = "colname", datacol = "value")

# Create a data.frame from a matrix
df <- data.frame(
  rowname = rep(rownames(state.x77), times = ncol(state.x77)),
  colname = rep(colnames(state.x77), each = nrow(state.x77)),
  value = as.vector(state.x77)
)

# Write data to a parquet file
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)

pqmat <- DuckDBMatrix(tf, row = "rowname", col = "colname", datacol = "value")

DuckDBMatrix utility methods

Description

Matrix multiplication methods for DuckDBMatrix objects.

Value

Method return types are documented in the sections above.

Matrix Multiplication Methods

In the code snippets below, x is a DuckDBMatrix object and y is an ordinary matrix or vector:

x %*% y:: Matrix multiplication via SQL aggregation.
crossprod(x, y):: Cross-product, equivalent to t(x) %*% y.
tcrossprod(x, y):: Transposed cross-product, equivalent to x %*% t(y).
crossprod(x):: Self cross-product via SQL self-join.
tcrossprod(x):: Self transposed cross-product via SQL self-join.

Author(s)

Patrick Aboyoun

Examples

state_df <- data.frame(
  index1 = rep(rownames(state.x77), times = ncol(state.x77)),
  index2 = rep(colnames(state.x77), each = nrow(state.x77)),
  value = as.vector(state.x77)
)
state_df <- subset(state_df, value != 0)
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(state_df, tf)
pqmat <- DuckDBMatrix(tf, datacol = "value",
                      keycols = list(index1 = rownames(state.x77),
                                     index2 = colnames(state.x77)))
y <- rep(1, ncol(pqmat))
pqmat %*% y
dim(crossprod(pqmat))

state_df <- data.frame(
  index1 = rep(rownames(state.x77), times = ncol(state.x77)),
  index2 = rep(colnames(state.x77), each = nrow(state.x77)),
  value = as.vector(state.x77)
)
state_df <- subset(state_df, value != 0)
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(state_df, tf)
pqmat <- DuckDBMatrix(tf, datacol = "value",
                      keycols = list(index1 = rownames(state.x77),
                                     index2 = colnames(state.x77)))
y <- rep(1, ncol(pqmat))
pqmat %*% y
dim(crossprod(pqmat))

DuckDBTable row / column summarization methods

Description

Row / column summarization methods for DuckDBTable objects.

Value

Method return types are documented in the sections above.

Row / Column Summarization Methods

In the code snippets below, x is a DuckDBTable object:

rowCounts(x, value = TRUE):

Calculates the row counts of x that are equal to value.

value: The value to count.

colCounts(x, value = TRUE):

Calculates the column counts of x that are equal to value.

value: The value to count.

rowMaxs(x):

Calculates the row maxima of x.

colMaxs(x):

Calculates the column maxima of x.

rowMeans(x, dims = 1):

Calculates the row means of x.

dims: An integer specifying which dimensions to average over, namely dims + 1, ....

colMeans(x, dims = 1):

Calculates the column means of x.

dims: An integer specifying which dimensions to average over, namely 1:dims.

rowMins(x):

Calculates the row minima of x.

colMins(x):

Calculates the column minima of x.

rowSums(x, dims = 1):

Calculates the row sums of x.

dims: An integer specifying which dimensions to sum over, namely dims + 1, ....

colSums(x, dims = 1):

Calculates the column sums of x.

dims: An integer specifying which dimensions to sum over, namely 1:dims.

rowSds(x):

Calculates the row standard deviations of x.

colSds(x):

Calculates the column standard deviations of x.

rowVars(x):

Calculates the row variances of x.

colVars(x):

Calculates the column variances of x.

Sweep Method

In the code snippets below, x is a DuckDBTable object:

sweep(x, MARGIN, STATS, FUN = "/"):

Sweeps out array summaries from x. Applies FUN to each element of x using the corresponding value from STATS based on MARGIN.

MARGIN: integer specifying the dimension (1 for rows, 2 for columns)
STATS: numeric vector with length equal to the extent of dimension MARGIN
FUN: function to be used to carry out the sweep

Author(s)

Patrick Aboyoun

Examples

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
tbl <- DuckDBTable(tf, datacols = "fate",
                   keycols = c("Class", "Sex", "Age", "Survived"))
rowSums(tbl)
colMeans(tbl)

df <- do.call(expand.grid, c(dimnames(Titanic), stringsAsFactors = FALSE))
df$fate <- as.integer(Titanic[as.matrix(df)])
df <- df[df$fate != 0L, ]
tf <- tempfile(fileext = ".parquet")
on.exit(unlink(tf))
arrow::write_parquet(df, tf)
tbl <- DuckDBTable(tf, datacols = "fate",
                   keycols = c("Class", "Sex", "Age", "Survived"))
rowSums(tbl)
colMeans(tbl)

Row-wise Deviance Statistics

Description

Compute per-row (per-gene) deviance statistics for feature selection in single-cell RNA-seq data. Genes with high deviance are likely to be informative for downstream analysis.

Usage

rowDeviances(x, family = c("binomial", "poisson"), ...)

## S4 method for signature 'matrix'
rowDeviances(x, family = c("binomial", "poisson"), ...)

## S4 method for signature 'dgCMatrix'
rowDeviances(x, family = c("binomial", "poisson"), ...)

## S4 method for signature 'DelayedMatrix'
rowDeviances(
  x,
  family = c("binomial", "poisson"),
  ...,
  grid = NULL,
  as.sparse = NA,
  BPPARAM = getAutoBPPARAM()
)

## S4 method for signature 'DuckDBMatrix'
rowDeviances(x, family = c("binomial", "poisson"), ...)
rowDeviances(x, family = c("binomial", "poisson"), ...)

## S4 method for signature 'matrix'
rowDeviances(x, family = c("binomial", "poisson"), ...)

## S4 method for signature 'dgCMatrix'
rowDeviances(x, family = c("binomial", "poisson"), ...)

## S4 method for signature 'DelayedMatrix'
rowDeviances(
  x,
  family = c("binomial", "poisson"),
  ...,
  grid = NULL,
  as.sparse = NA,
  BPPARAM = getAutoBPPARAM()
)

## S4 method for signature 'DuckDBMatrix'
rowDeviances(x, family = c("binomial", "poisson"), ...)

Arguments

x

A matrix-like object with genes in rows and cells in columns. Typically a count matrix (not log-transformed).

family

Character string specifying the null model distribution. Either "binomial" (default) or "poisson".

...

Additional arguments (currently unused).

grid

For DelayedMatrix objects, an optional ArrayGrid object specifying the block structure. Defaults to colAutoGrid(x) for column-wise processing since deviance is additive across cells.

as.sparse

For DelayedMatrix objects, logical indicating whether blocks should be returned as sparse matrices. Default is NA, which lets blockApply decide based on data sparsity. Set to TRUE to force sparse blocks or FALSE to force dense blocks.

BPPARAM

For DelayedMatrix objects, a BiocParallelParam object specifying the parallel backend. Defaults to getAutoBPPARAM().

Details

The deviance statistic measures how poorly each gene fits a simple null model where expression is constant across cells (after accounting for library size). Genes with high deviance show more variation than expected under the null model and are good candidates for highly variable gene selection.

For family = "binomial", the null model treats each UMI as a Bernoulli trial with gene-specific success probability. This is the closest approximation to multinomial sampling.

For family = "poisson", the null model assumes Poisson-distributed counts with rate proportional to library size. This is faster to compute and often gives similar results to binomial.

Value

A numeric vector of deviance statistics, one per row of x. Names are preserved from rownames(x) if present.

Author(s)

Patrick Aboyoun

References

Townes FW, Hicks SC, Aryee MJ, and Irizarry RA (2019). Feature Selection and Dimension Reduction for Single Cell RNA-Seq based on a Multinomial Model. Genome Biology https://doi.org/10.1186/s13059-019-1861-6

Examples

# Generate example count data
set.seed(123)
mat <- matrix(rpois(1000, lambda = 5), nrow = 100, ncol = 10)
rownames(mat) <- paste0("Gene", 1:100)

# Compute deviances
dev_binom <- rowDeviances(mat, family = "binomial")
dev_pois <- rowDeviances(mat, family = "poisson")

# Select top variable genes
top_genes <- head(order(dev_binom, decreasing = TRUE), 20)

# Generate example count data
set.seed(123)
mat <- matrix(rpois(1000, lambda = 5), nrow = 100, ncol = 10)
rownames(mat) <- paste0("Gene", 1:100)

# Compute deviances
dev_binom <- rowDeviances(mat, family = "binomial")
dev_pois <- rowDeviances(mat, family = "poisson")

# Select top variable genes
top_genes <- head(order(dev_binom, decreasing = TRUE), 20)

Row and Column Non-Zero Counts

Description

Count the number of non-zero elements per row or column of a matrix-like object. This is useful for filtering genes by detection rate (number of cells where the gene is expressed) or filtering cells by complexity (number of genes detected).

Usage

rowNnzs(x, value = vector(type(x), 1L), ...)

colNnzs(x, value = vector(type(x), 1L), ...)

## S4 method for signature 'ANY'
rowNnzs(x, value = vector(type(x), 1L), ...)

## S4 method for signature 'DuckDBArray'
rowNnzs(
  x,
  value = vector(type(x), 1L),
  na.rm = FALSE,
  dims = 1,
  ...,
  useNames = TRUE
)

## S4 method for signature 'ANY'
colNnzs(x, value = vector(type(x), 1L), ...)

## S4 method for signature 'DuckDBArray'
colNnzs(
  x,
  value = vector(type(x), 1L),
  na.rm = FALSE,
  dims = 1,
  ...,
  useNames = TRUE
)
rowNnzs(x, value = vector(type(x), 1L), ...)

colNnzs(x, value = vector(type(x), 1L), ...)

## S4 method for signature 'ANY'
rowNnzs(x, value = vector(type(x), 1L), ...)

## S4 method for signature 'DuckDBArray'
rowNnzs(
  x,
  value = vector(type(x), 1L),
  na.rm = FALSE,
  dims = 1,
  ...,
  useNames = TRUE
)

## S4 method for signature 'ANY'
colNnzs(x, value = vector(type(x), 1L), ...)

## S4 method for signature 'DuckDBArray'
colNnzs(
  x,
  value = vector(type(x), 1L),
  na.rm = FALSE,
  dims = 1,
  ...,
  useNames = TRUE
)

Arguments

x

A matrix-like object.

value

The value to treat as "zero" (i.e., elements NOT equal to this value are counted). Defaults to the zero value for the storage mode of x.

...

Additional arguments passed to rowCounts or colCounts.

na.rm

If TRUE, missing values are excluded first.

dims

A single integer specifying the dimension(s) over which to operate. For rowNnzs, dims = 1 counts non-zeros in each row. For colNnzs, dims = 1 counts non-zeros in each column. Only used for DuckDBArray objects.

useNames

If TRUE, the names of the input are preserved in the output.

Details

These functions are convenience wrappers around rowCounts and colCounts that count elements not equal to zero. The zero value is determined by the storage mode of x:

For logical matrices: FALSE
For integer matrices: 0L
For numeric matrices: 0
For character matrices: ""

Value

An integer vector with length equal to nrow(x) for rowNnzs or ncol(x) for colNnzs.

Author(s)

Patrick Aboyoun

Examples

# Dense matrix
mat <- matrix(c(0L, 1L, 2L, 0L, 0L, 3L), nrow = 2)
rowNnzs(mat)
colNnzs(mat)

# Dense matrix
mat <- matrix(c(0L, 1L, 2L, 0L, 0L, 3L), nrow = 2)
rowNnzs(mat)
colNnzs(mat)

Write an Array-Like Object in Coordinate Format

Description

An arrow::write_dataset wrapper function to write array-like objects in coordinate format. If dimension names are present, they are substituted for the corresponding indices.

Usage

writeCoordArray(
  x,
  path,
  indexcols = names(dimnames(x)) %||% sprintf("index%d", seq_along(dim(x))),
  datacol = "value",
  grid = defaultAutoGrid(COO_SparseArray(dim(x))),
  grid_suffix = "_group",
  arrowtype = NULL,
  max_dim = NULL,
  append = FALSE,
  along = NULL,
  offset = 0L,
  group_offset = 0L,
  ...
)

## S4 method for signature 'ANY'
writeCoordArray(
  x,
  path,
  indexcols = names(dimnames(x)) %||% sprintf("index%d", seq_along(dim(x))),
  datacol = "value",
  grid = defaultAutoGrid(COO_SparseArray(dim(x))),
  grid_suffix = "_group",
  arrowtype = NULL,
  max_dim = NULL,
  append = FALSE,
  along = NULL,
  offset = 0L,
  group_offset = 0L,
  BPPARAM = getAutoBPPARAM(),
  ...
)

## S4 method for signature 'DuckDBArray'
writeCoordArray(
  x,
  path,
  indexcols = names(dimnames(x)) %||% sprintf("index%d", seq_along(dim(x))),
  datacol = "value",
  grid = defaultAutoGrid(COO_SparseArray(dim(x))),
  grid_suffix = "_group",
  arrowtype = NULL,
  max_dim = NULL,
  append = FALSE,
  along = NULL,
  offset = 0L,
  group_offset = 0L,
  ...
)
writeCoordArray(
  x,
  path,
  indexcols = names(dimnames(x)) %||% sprintf("index%d", seq_along(dim(x))),
  datacol = "value",
  grid = defaultAutoGrid(COO_SparseArray(dim(x))),
  grid_suffix = "_group",
  arrowtype = NULL,
  max_dim = NULL,
  append = FALSE,
  along = NULL,
  offset = 0L,
  group_offset = 0L,
  ...
)

## S4 method for signature 'ANY'
writeCoordArray(
  x,
  path,
  indexcols = names(dimnames(x)) %||% sprintf("index%d", seq_along(dim(x))),
  datacol = "value",
  grid = defaultAutoGrid(COO_SparseArray(dim(x))),
  grid_suffix = "_group",
  arrowtype = NULL,
  max_dim = NULL,
  append = FALSE,
  along = NULL,
  offset = 0L,
  group_offset = 0L,
  BPPARAM = getAutoBPPARAM(),
  ...
)

## S4 method for signature 'DuckDBArray'
writeCoordArray(
  x,
  path,
  indexcols = names(dimnames(x)) %||% sprintf("index%d", seq_along(dim(x))),
  datacol = "value",
  grid = defaultAutoGrid(COO_SparseArray(dim(x))),
  grid_suffix = "_group",
  arrowtype = NULL,
  max_dim = NULL,
  append = FALSE,
  along = NULL,
  offset = 0L,
  group_offset = 0L,
  ...
)

Arguments

x

An array-like object.

path

The path to write the array-like object to.

indexcols

A character vector of column names for the dimensions of the array.

datacol

A character string specifying the column name containing the array values in the resulting table. Defaults to "value".

grid

An optional ArrayGrid to use for partitioning the array. If NULL, uses defaultAutoGrid(COO_SparseArray(dim(x))).

grid_suffix

A character string to append to the partitioning directories if the grid contains more than one cell. Defaults to "_group".

arrowtype

An optional DataType for the value column (e.g. arrow::int32(), arrow::uint16(), arrow::float64()). When NULL and append = FALSE, the type is inferred from the data: the narrowest type capable of representing the range of non-zero values if they are all integer-valued, and otherwise double. When NULL and append = TRUE, the type is read from an existing parquet file under path so that the appended slab always matches the existing value-column schema. When supplied explicitly in append mode, it is verified to match the existing schema; a mismatch aborts the call before any file is written.

max_dim

Optional integer vector of length length(dim(x)) giving an upper bound on the coordinates in each dimension. Controls the arrow type chosen for each index column: the narrowest unsigned integer type that can represent 0..max_dim[j] is picked for column indexcols[j]. When NULL and append = FALSE, dim(x) is used. When NULL and append = TRUE, the index-column types are read from an existing parquet file under path so that the appended slab always matches the existing index-column schema. When supplied explicitly in append mode, the resulting types are verified to match the existing schema; a mismatch aborts the call before any file is written. The caller does not need to know the exact final extent, only an upper bound on it; setting max_dim too high only costs a slightly wider integer type on disk.

append

Logical; when TRUE, write additional hive-style partitions into an existing directory previously created by writeCoordArray. Append mode is only supported for hive-partitioned output (i.e. length(grid) > 1L); each cell of the supplied grid produces a new partition directory along dimension along. Before writing, writeCoordArray performs several corruption-prevention checks against the existing dataset:

path must already exist and be laid out as a hive partitioning (top-level entries must all be paste0(indexcols[1], grid_suffix, "=", <integer>) subdirectories).
At least one parquet file must exist under path, and its schema must contain every indexcols[j] and datacol.
The value-column and every index-column arrow type must match the existing schema. If arrowtype/max_dim were supplied, a mismatch errors; if not, the existing types are adopted.
No partition directory that this call would write to may already exist (otherwise group_offset is too small and the call would silently overwrite previously-written data).

Because arrow's dataset reader requires a uniform schema across files, these checks make silent schema drift and silent partition collisions impossible. Defaults to FALSE.

along

Integer index of the dimension along which to append new partitions. Required when append = TRUE.

offset

Non-negative integer added to the coordinates in dimension along for every non-zero written in this call. Typically the total extent already written along along by previous calls. Defaults to 0L.

group_offset

Non-negative integer added to the partition group index along dimension along (i.e. the value embedded in the paste0(indexcols[along], grid_suffix) subdirectory) so that new partition directories do not collide with existing ones. Typically the total number of partition groups already written along along by previous calls. Defaults to 0L.

...

Additional arguments to pass to arrow::write_dataset. Note that existing_data_behavior defaults to "error" (rather than arrow's default of "overwrite_or_ignore") so that a re-run into a populated path fails loudly instead of silently clobbering or merging existing data. Pass existing_data_behavior = "delete_matching" or similar to override.

BPPARAM

A BiocParallelParam object to use for parallel processing when grid contains multiple cells (ANY method only). Defaults to getAutoBPPARAM(). Note: The DuckDBArray method does not support this parameter because DuckDB connections are not thread-safe for concurrent execution. The DuckDBArray method uses sequential execution and relies on DuckDB's internal parallelism for performance.

Value

Called for its side effect of writing x to path as coordinate-format Parquet; returns NULL invisibly.

Author(s)

Patrick Aboyoun

Examples

# Write the state.x77 matrix to multiple Parquet files using grid partitioning
tf <- tempfile()
state_grid <- RegularArrayGrid(dim(state.x77), c(10, 4))
writeCoordArray(state.x77, file.path(tf, "state"), grid = state_grid)
list.files(tf, full.names = TRUE, recursive = TRUE)

# Append mode: write two column-slabs as distinct hive partitions without
# first materializing cbind(slab1, slab2). Omitting 'arrowtype' and
# 'max_dim' is safe -- both are read from the existing schema so the
# appended slab is guaranteed to match.
tf2 <- tempfile()
slab1 <- state.x77[, 1:4]
slab2 <- state.x77[, 5:8]
grid1 <- RegularArrayGrid(dim(slab1), c(10L, 2L))  # 2 col partitions
grid2 <- RegularArrayGrid(dim(slab2), c(10L, 2L))  # 2 more col partitions
writeCoordArray(slab1, file.path(tf2, "state"), grid = grid1)
writeCoordArray(slab2, file.path(tf2, "state"), grid = grid2,
                append = TRUE, along = 2L,
                offset = ncol(slab1), group_offset = dim(grid1)[2L])

# Pinning 'arrowtype' / 'max_dim' is still useful when the first slab's
# inferred schema is narrower than you want the final dataset to have
# (e.g. a small first slab that picks uint8 but you know later slabs
# will need uint16). Pin both on every call for a fully-predictable
# dataset schema.
tf3 <- tempfile()
int_slab   <- state.x77[, c("Population", "Income")]
float_slab <- state.x77[, c("Illiteracy", "Life Exp")]
grid_i <- RegularArrayGrid(dim(int_slab),   c(10L, 2L))
grid_f <- RegularArrayGrid(dim(float_slab), c(10L, 2L))
max_dim <- c(nrow(state.x77), ncol(state.x77))
writeCoordArray(int_slab, file.path(tf3, "state"), grid = grid_i,
                arrowtype = arrow::float64(), max_dim = max_dim)
writeCoordArray(float_slab, file.path(tf3, "state"), grid = grid_f,
                arrowtype = arrow::float64(), max_dim = max_dim,
                append = TRUE, along = 2L,
                offset = ncol(int_slab),
                group_offset = dim(grid_i)[2L])

# Write the state.x77 matrix to multiple Parquet files using grid partitioning
tf <- tempfile()
state_grid <- RegularArrayGrid(dim(state.x77), c(10, 4))
writeCoordArray(state.x77, file.path(tf, "state"), grid = state_grid)
list.files(tf, full.names = TRUE, recursive = TRUE)

# Append mode: write two column-slabs as distinct hive partitions without
# first materializing cbind(slab1, slab2). Omitting 'arrowtype' and
# 'max_dim' is safe -- both are read from the existing schema so the
# appended slab is guaranteed to match.
tf2 <- tempfile()
slab1 <- state.x77[, 1:4]
slab2 <- state.x77[, 5:8]
grid1 <- RegularArrayGrid(dim(slab1), c(10L, 2L))  # 2 col partitions
grid2 <- RegularArrayGrid(dim(slab2), c(10L, 2L))  # 2 more col partitions
writeCoordArray(slab1, file.path(tf2, "state"), grid = grid1)
writeCoordArray(slab2, file.path(tf2, "state"), grid = grid2,
                append = TRUE, along = 2L,
                offset = ncol(slab1), group_offset = dim(grid1)[2L])

# Pinning 'arrowtype' / 'max_dim' is still useful when the first slab's
# inferred schema is narrower than you want the final dataset to have
# (e.g. a small first slab that picks uint8 but you know later slabs
# will need uint16). Pin both on every call for a fully-predictable
# dataset schema.
tf3 <- tempfile()
int_slab   <- state.x77[, c("Population", "Income")]
float_slab <- state.x77[, c("Illiteracy", "Life Exp")]
grid_i <- RegularArrayGrid(dim(int_slab),   c(10L, 2L))
grid_f <- RegularArrayGrid(dim(float_slab), c(10L, 2L))
max_dim <- c(nrow(state.x77), ncol(state.x77))
writeCoordArray(int_slab, file.path(tf3, "state"), grid = grid_i,
                arrowtype = arrow::float64(), max_dim = max_dim)
writeCoordArray(float_slab, file.path(tf3, "state"), grid = grid_f,
                arrowtype = arrow::float64(), max_dim = max_dim,
                append = TRUE, along = 2L,
                offset = ncol(int_slab),
                group_offset = dim(grid_i)[2L])

Package 'DuckDBArray'

Help Index

Create Dimension Lookup Tables

Description

Usage

Arguments

Details

Value

Author(s)

See Also

Examples

DuckDBArray objects

Description

Value

Constructor

Accessors

Subsetting

Transposition

Author(s)

See Also

Examples

DuckDBArray row / column summarization methods

Description

Value

Row / Column Summarization Methods

Author(s)

See Also

Examples

Common operations on DuckDBArray objects

Description

Value

Group Generics

Numerical Data Methods

Sparsity Methods

Author(s)

See Also

Examples

DuckDBArraySeed objects

Description

Value

Constructor

Accessors

Subsetting

Transposition

Author(s)

See Also

Examples

DuckDBArraySeed row / column summarization methods

Description

Value

Row / Column Summarization Methods

Author(s)

See Also

Examples

Common operations on DuckDBArraySeed objects

Description

Value

Group Generics

Numerical Data Methods

Sparsity Methods

Author(s)

See Also

Examples

DuckDBMatrix objects

Description

Value

Constructor

Accessors

Subsetting

Transposition

Author(s)

See Also

Examples

DuckDBMatrix utility methods

Description

Value

Matrix Multiplication Methods

Author(s)

See Also

Examples