Package: MsBackendParquet
Authors: Ossama Edbali [aut, cre] (ORCID: https://orcid.org/0000-0003-0132-8668), Ralf Johannes
Maria Weber [aut] (ORCID: https://orcid.org/0000-0002-8796-4771), BioFAIR UK [fnd]
(BioFAIR Pathfinder Project)
Compiled: Tue Oct 13
19:33:42 2026
The MsBackendParquet
package provides a backend for Spectra
objects that stores mass spectrometry (MS) data in an on-disk dataset
built on Apache Parquet.
Datasets are written with the Apache Arrow R bindings and
queried with DuckDB, keeping
the spectra metadata together with the
mz/intensity peak values in a columnar format
that supports compression, dictionary encoding and Hive-style
partitioning.
A dataset is a directory identified by an mzStack.json
manifest, holding one or more runs. A run is one source of
spectra and owns a contiguous block of the dataset’s spectrum ids. A run
is either native (converted from a raw MS data file, with
peaks stored alongside the metadata) or mzpeak, an index
over external HUPO-PSI
mzPeak archives that are read where they are and never modified (see
Querying collections of mzPeak archives below). The manifest
records which kind each run is, so the two never have to be told apart
by inspecting the directory.
To keep access fast, the (small) non-peak spectra variables are held in a bounded in-memory index while the peak data is read from disk on demand. As a result, metadata filters cost about as much as with an in-memory backend, while the bulk peak storage stays out of core.
The backend extends MsBackendCached from Spectra
so it benefits from caching of user-defined spectra variables while
keeping the on-disk dataset read-only.
MsBackendMzR from Spectra
keeps the spectra metadata in memory and reads peaks from the raw mzML
files on demand: nothing has to be converted, but every peak access
parses the files again. MsBackendSql
stores spectra in an SQL database, which suits data shared from a
database server. SpectraStash
saves and restores whole Spectra objects in interoperable
formats such as HDF5. It is for exchanging an object, not for querying
it on disk.
MsBackendParquet converts once, like
MsBackendSql, and then queries a database engine, DuckDB.
But its storage is plain Parquet files that need no server and that
other tools, such as Python, DuckDB or Spark, read directly. It can also
index mzPeak archives where they are, without converting them.
The package is installed from Bioconductor:
if (!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("MsBackendParquet")Reading raw mzML files needs the mzR package,
which
BiocManager::install("MsBackendParquet", dependencies = TRUE)
installs too. Datasets made from mzPeak archives or from in-memory data
do not need it.
The examples below use a DDA data file of a pesticide mix, provided by the MsDataHub package. The file is downloaded once and cached locally.
MsDataHub returns the file under its cache name, and the
id of a run is taken from the name of the file it was imported from.
Copying the file to a named path first therefore gives the dataset
readable run ids:
fl_named <- file.path(tempdir(), "PestMix1_DDA.mzML")
invisible(file.copy(fl, fl_named, overwrite = TRUE))mzMLToParquet() converts raw MS data files into a
Parquet dataset. It validates the input files, writes the dataset and
returns an initialised backend, ready to be wrapped in a
Spectra object:
path <- tempfile()
be <- mzMLToParquet(fl_named, path = path, verbose = FALSE)
#> Importing data ...
#> duckdb keeps downloaded extensions and secrets in a temporary directory:
#> ℹ /tmp/Rtmpjg3R4y/duckdb
#> This is removed when the R session ends.
#> • Extensions are re-downloaded each session.
#> • Secrets are lost.
#> ℹ Run duckdb(shared_home = TRUE) (or create ~/.duckdb) to keep them (suitable for most users).
#> ℹ Run duckdb(shared_home = FALSE) to accept the temporary directory (and silence this message).
#> ℹ See ?duckdb_storage for details and alternatives.
sps <- Spectra(be)
sps
#> MSn data (Spectra) with 7602 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 1 NA 1
#> 3 1 NA 1
#> 4 1 NA 1
#> 5 1 NA 1
#> ... ... ... ...
#> 7598 1 NA 1
#> 7599 1 NA 1
#> 7600 1 NA 1
#> 7601 1 NA 1
#> 7602 1 NA 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c45b20d11Each input file becomes its own run, written to
<path>/spectra/run_id=<id>/, so a filter that
selects whole files prunes at the directory level:
mzMLToParquet() imports MS data via
MsBackendMzR() and writes the spectra metadata together
with the mz/intensity values as Parquet
list<double> columns.
A Spectra object that is already loaded, with any
backend, can instead be moved onto Parquet with
setBackend(). The destination is given as
path:
sps_mem <- Spectra(fl_named, source = MsBackendMzR())
sps_pq <- setBackend(sps_mem, MsBackendParquet(), path = tempfile())
sps_pq
#> MSn data (Spectra) with 7602 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 1 NA 1
#> 3 1 NA 1
#> 4 1 NA 1
#> 5 1 NA 1
#> ... ... ... ...
#> 7598 1 NA 1
#> 7599 1 NA 1
#> 7600 1 NA 1
#> 7601 1 NA 1
#> 7602 1 NA 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c65080ce0
#> Processing:
#> Switch backend from MsBackendMzR to MsBackendParquet [Tue Oct 13 19:33:52 2026]spectraData(), peaksData(),
mz(), intensity() and related methods work the
same way as for any other MsBackend. The spectra variables
of the dataset are:
spectraVariables(sps)
#> [1] "msLevel" "rtime"
#> [3] "acquisitionNum" "scanIndex"
#> [5] "dataStorage" "dataOrigin"
#> [7] "centroided" "smoothed"
#> [9] "polarity" "precScanNum"
#> [11] "precursorMz" "precursorIntensity"
#> [13] "precursorCharge" "collisionEnergy"
#> [15] "isolationWindowLowerMz" "isolationWindowTargetMz"
#> [17] "isolationWindowUpperMz" "spectrum_id_"
#> [19] "run_id" "spectrum_index"
#> [21] "spectrumId" "totIonCurrent"
#> [23] "basePeakMz" "basePeakIntensity"
#> [25] "lowMz" "highMz"
#> [27] "peaksCount" "injectionTime"
#> [29] "filterString" "scanWindowLowerLimit"
#> [31] "scanWindowUpperLimit" "instrumentConfigurationId"
#> [33] "spectrumType" "basePeakMZ_1"
#> [35] "electronBeamEnergy" "ionisationEnergy"
#> [37] "lowMZ_1" "highMZ_1"
#> [39] "mergedScan" "mergedResultScanNum"
#> [41] "mergedResultStartScanNum" "mergedResultEndScanNum"
#> [43] "ionMobilityDriftTime"Values are returned as usual, with the peak data read from disk on demand:
spectraData(sps, c("msLevel", "rtime", "precursorMz"))
#> DataFrame with 7602 rows and 3 columns
#> msLevel rtime precursorMz
#> <integer> <numeric> <numeric>
#> 1 1 0.231 NA
#> 2 1 0.351 NA
#> 3 1 0.471 NA
#> 4 1 0.591 NA
#> 5 1 0.711 NA
#> ... ... ... ...
#> 7598 1 899.491 NA
#> 7599 1 899.613 NA
#> 7600 1 899.747 NA
#> 7601 1 899.872 NA
#> 7602 1 899.993 NA
head(peaksData(sps)[[1L]])
#> mz intensity
#> [1,] 50.01579 0.03076322
#> [2,] 52.01793 0.16344352
#> [3,] 53.00306 0.38972694
#> [4,] 53.01328 1.38231599
#> [5,] 53.84987 1.00965071
#> [6,] 53.93832 0.03967578
head(tic(sps))
#> [1] 168989 167674 193191 146445 145577 156807Filter methods are translated into DuckDB SQL predicates that are pushed down to the Parquet scan. DuckDB prunes whole row groups using their min/max statistics, and skips whole partitions for Hive-partitioned datasets, so a range or set filter reads only the data it needs. When the filter column is held in the in-memory metadata index (see The in-memory metadata cache below), the predicate is evaluated in memory instead.
Selecting spectra by MS level and by retention time, which is given in seconds:
filterMsLevel(sps, 2L)
#> MSn data (Spectra) with 2975 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 137.9639 1
#> 2 2 56.9419 1
#> 3 2 89.9449 1
#> 4 2 207.0294 1
#> 5 2 55.0083 1
#> ... ... ... ...
#> 2971 2 56.0495 1
#> 2972 2 67.0289 1
#> 2973 2 142.9611 1
#> 2974 2 53.0129 1
#> 2975 2 91.5022 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c45b20d11
#> Processing:
#> Filter: select MS level(s) 2 [Tue Oct 13 19:33:52 2026]
filterRt(sps, c(200, 300))
#> MSn data (Spectra) with 717 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 1 NA 1
#> 3 1 NA 1
#> 4 1 NA 1
#> 5 1 NA 1
#> ... ... ... ...
#> 713 1 NA 1
#> 714 1 NA 1
#> 715 2 320.107 1
#> 716 1 NA 1
#> 717 2 320.107 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c45b20d11
#> Processing:
#> Filter: select retention time [200..300] on MS level(s) [Tue Oct 13 19:33:52 2026]Filters return a Spectra object and can be chained. Each
call narrows the predicate rather than reading data, so the combination
is answered by a single scan:
MS2 spectra can be selected by precursor m/z, either over a range or against a set of values with a mass tolerance in ppm:
filterPrecursorMzRange(sps, c(278, 279))
#> MSn data (Spectra) with 9 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 278.094 1
#> 2 2 278.091 1
#> 3 2 278.095 1
#> 4 2 278.092 1
#> 5 2 278.091 1
#> 6 2 278.092 1
#> 7 2 278.091 1
#> 8 2 278.091 1
#> 9 2 278.092 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c45b20d11
#> Processing:
#> Filter: select spectra with a precursor m/z within [278, 279] [Tue Oct 13 19:33:52 2026]
ms2 <- filterPrecursorMzValues(sps, 278.093, ppm = 30)
ms2
#> MSn data (Spectra) with 9 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 278.094 1
#> 2 2 278.091 1
#> 3 2 278.095 1
#> 4 2 278.092 1
#> 5 2 278.091 1
#> 6 2 278.092 1
#> 7 2 278.091 1
#> 8 2 278.091 1
#> 9 2 278.092 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c45b20d11
#> Processing:
#> Filter: select spectra with precursor m/z matching 278.093 [Tue Oct 13 19:33:52 2026]
range(rtime(ms2))
#> [1] 268.587 387.849filterPrecursorMzValues() accepts several values at
once, which become one OR-ed predicate and therefore one scan rather
than one scan per value:
filterPrecursorMzValues(sps, c(278.093, 124.086), tolerance = 0.01)
#> MSn data (Spectra) with 63 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 124.086 1
#> 2 2 278.094 1
#> 3 2 278.091 1
#> 4 2 278.095 1
#> 5 2 124.086 1
#> ... ... ... ...
#> 59 2 124.086 1
#> 60 2 124.086 1
#> 61 2 124.086 1
#> 62 2 124.085 1
#> 63 2 124.087 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c45b20d11
#> Processing:
#> Filter: select spectra with precursor m/z matching 278.093, 124.086 [Tue Oct 13 19:33:52 2026]None of the filters above read peak data. They are answered from the spectra variables, and the resulting predicate is carried into any later peak access.
Peak data is read when it is asked for, and only for the spectra that survived the filters. The nine MS2 spectra selected above share one precursor, so their fragments can be compared directly:
lengths(ms2)
#> [1] 25 14 12 4 8 4 1 18 6
pk <- peaksData(ms2)[[which.max(lengths(ms2))]]
head(pk[order(-pk[, "intensity"]), ])
#> mz intensity
#> [1,] 164.0541 1.1601784
#> [2,] 278.0952 1.0027263
#> [3,] 161.1078 0.4363745
#> [4,] 120.0820 0.2798612
#> [5,] 130.0637 0.1656372
#> [6,] 149.1082 0.1439338containsMz() from Spectra
reports which of those spectra contain a peak at a given m/z:
Because the peaks are stored columnwise, DuckDB reads only the
mz and intensity columns, and only the rows
belonging to the selected spectra.
createMsBackendParquetDataset() writes a dataset without
opening it and exposes the layout options: the reader backend, chunk
size, compression codec, Parquet row-group size, and Hive
partitioning.
path_part <- tempfile()
createMsBackendParquetDataset(path = path_part, x = fl,
partitioning = "msLevel",
compression = "zstd")
#> Importing data ...
be_part <- backendInitialize(MsBackendParquet(), path = path_part)
length(be_part)
#> [1] 7602partitioning adds a further Hive level below
the run. The partition directories carry mzPeak column names (see
How the dataset interfaces with mzPeak’s model below):
list.files(list.files(file.path(path_part, "spectra"), full.names = TRUE))
#> [1] "ms_level=1" "ms_level=2"The same function writes an existing DataFrame of
spectra data when it is passed as data instead of
x.
mzPeak is the HUPO-PSI successor to mzML. An mzPeak archive holds one MS run as a folder (or an uncompressed ZIP) of Parquet files plus a JSON manifest naming them. That structure is well suited to distributing and archiving individual runs, but inefficient for queries spanning thousands of runs. Answering such a query would require opening and parsing each archive and its manifest.
MsBackendParquet bridges the two.
createMzPeakDataset() registers a set of archives as one
dataset. The package ships two small archives for this purpose:
archives <- system.file("extdata", c("QC01.mzpeak", "QC02.mzpeak"),
package = "MsBackendParquet")
qc_path <- tempfile()
createMzPeakDataset(archives[1L], path = qc_path, verbose = FALSE)Further archives join the same dataset later. Only the index grows:
addMzPeakArchives(qc_path, archives[2L], verbose = FALSE)
be_qc <- backendInitialize(MsBackendParquet(), path = qc_path)
sps_qc <- Spectra(be_qc)
sps_qc
#> MSn data (Spectra) with 10 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> 5 1 NA 1
#> 6 2 405 1
#> 7 1 NA 1
#> 8 2 401 1
#> 9 1 NA 1
#> 10 2 403 1
#> ... 36 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c5ee261df
#> mzPeak archives: 2 run(s), signal read as: auto
table(basename(dataOrigin(sps_qc)))
#>
#> QC01.mzpeak QC02.mzpeak
#> 6 4With two runs in the dataset, filterDataOrigin() selects
one of them:
filterDataOrigin(sps_qc, grep("QC02", dataOrigin(sps_qc), value = TRUE)[1L])
#> MSn data (Spectra) with 4 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> ... 36 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c5ee261df
#> mzPeak archives: 2 run(s), signal read as: auto
#> Processing:
#> Filter: select data origin(s) /tmp/Rtmpjg3R4y/file179c5ee261df/runs/QC02.mzpeak [Tue Oct 13 19:33:54 2026]The archives are neither modified nor copied. Ingest reads each
archive’s manifest, its spectrum metadata table and its scan, precursor
and selected-ion side tables, and writes a small derived index next to
them, around 3% of the size of the archives. It never reads a byte of
signal data, so the cost scales with the number of spectra rather than
the size of the data. Peaks are then read from the archives themselves,
on demand. Pass link = "copy" to place a copy of each
archive inside the dataset directory instead of referencing it in
place.
Because the archives stay exactly as they were, they remain readable by every other mzPeak tool, and exporting a run back out of the dataset is a directory copy rather than a conversion.
Several aspects of the data are translated or transformed on input, so it is worth being precise about what changes.
time in minutes; Spectra expects
rtime in seconds. The conversion, and every other
column rename and re-encoding (scan_polarity ±1,
spectrum_representation as a CURIE, the isolation window as
a target plus offsets, and so on), happens in a single SQL view. The
same view serves a natively converted dataset: its spectra/
files are written with mzPeak’s column names and encodings too, so both
kinds are directly comparable and either can be exported back to an
archive mechanically. spectrum_index is always the
dataset’s unique 0-based key; a spectrum’s position within its source
file (scanIndex, which restarts per file) is kept in
scan_index. The package adds only a handful of reserved
columns (spectrum_id_, acquisition_num_, the
n_* counts).n_scans, n_selected_ions and
n_precursors. The archive still holds the full detail, but
a spectrum with several scans is summarised in the index.representation, which defaults to "auto":be_centroid <- backendInitialize(MsBackendParquet(), path = qc_path,
representation = "centroid")
be_centroid
#> MsBackendParquet with 10 spectra
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> 5 1 NA 1
#> 6 2 405 1
#> 7 1 NA 1
#> 8 2 401 1
#> 9 1 NA 1
#> 10 2 403 1
#> ... 36 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c5ee261df
#> mzPeak archives: 2 run(s), signal read as: centroidNA.Archives that use mzPeak’s chunked signal layout are rejected with a clear message; only the point layout is supported so far.
Filters on spectra variables work as they do for any other dataset. Finding spectra that contain a peak at a given m/z is different: it has to look at the signal, and mzPeak stores signal ordered by spectrum, which means a filter on m/z cannot skip anything, because every block of rows spans the whole mass range.
filterContainsMz() runs that search inside the storage
engine rather than pulling peaks into R:
hits <- filterContainsMz(be_qc, c(201, 10101), ppm = 20)
length(hits)
#> [1] 2
basename(dataOrigin(hits))
#> [1] "QC01.mzpeak" "QC02.mzpeak"filterContainsMz() works on the backend, and the result
is wrapped in a Spectra object like any other backend:
Spectra(hits)
#> MSn data (Spectra) with 2 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 401 1
#> 2 1 NA 1
#> ... 36 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c5ee261df
#> mzPeak archives: 2 run(s), signal read as: autoFor datasets where this is a common query, build an m/z-ordered copy of the signal. Sorting is what makes the block statistics useful, so the same search then reads a handful of blocks instead of everything:
On a 5-million-point test dataset this made
filterContainsMz() around eight times faster. A projection
is a cache: it is a derived copy of the signal and so costs roughly as
much disk again, it can be dropped with dropProjection(),
and results are identical with or without it. Only the performance
changes. Build it per run, for the runs you query often.
An experiment records factors about the sample a run came
from, such as subject, timepoint, treatment group or injection order,
that are not in the raw data. runData() stores them in the
dataset, one row per run, keyed by run_id.
runData(be_qc) <- data.frame(run_id = c("QC01", "QC02"),
subject = c("S1", "S2"),
timepoint = c(0L, 6L))
runVariables(be_qc)
#> [1] "subject" "timepoint"The columns then behave like any other spectra variable:
spectraData(be_qc, c("rtime", "subject", "timepoint"))
#> DataFrame with 10 rows and 3 columns
#> rtime subject timepoint
#> <numeric> <character> <integer>
#> 1 1 S1 0
#> 2 2 S1 0
#> 3 3 S1 0
#> 4 4 S1 0
#> 5 5 S1 0
#> 6 6 S1 0
#> 7 1 S2 6
#> 8 2 S2 6
#> 9 3 S2 6
#> 10 4 S2 6and filterSampleData() selects on them:
Spectra(filterSampleData(be_qc, timepoint == 6))
#> MSn data (Spectra) with 4 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> ... 38 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c5ee261df
#> mzPeak archives: 2 run(s), signal read as: autoThe sample metadata is held once per run, not copied onto every
spectrum. Each run owns a contiguous block of
spectrum_id_, so timepoint == 6 is exactly a
range of ids, and filterSampleData() resolves it against a
table of a few thousand rows rather than reading a column of a few
million. The resulting predicate is a range on the column the files are
ordered by, which is the case Parquet’s row-group statistics prune best,
and it is carried into any later peaksData().
Keeping it out of the per-spectrum data also makes it correctable. Sample metadata is the one thing that reliably changes after ingest. Rewriting it touches a few kilobytes and leaves the signal, and any projections built over it, untouched.
Runs with no row get NA, so metadata covering only some
runs is fine. To go further, and link samples to spectra in an
experiment-wide object, pass the same table to MsExperiment
and link the two on run_id:
library(MsExperiment)
samples <- S4Vectors::DataFrame(run_id = c("QC01", "QC02"),
subject = c("S1", "S2"),
timepoint = c(0L, 6L))
mse <- MsExperiment(sampleData = samples, spectra = Spectra(be_qc))
mse <- linkSampleData(mse, with = "sampleData.run_id = spectra.run_id")
mse
#> Object of class MsExperiment
#> Spectra: MS1 (5) MS2 (5)
#> Experiment data: 2 sample(s)
#> Sample data links:
#> - spectra: 2 sample(s) to 10 element(s).
## The spectra of the second sample.
spectra(mse[2])
#> MSn data (Spectra) with 4 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> ... 38 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/Rtmpjg3R4y/file179c5ee261df
#> mzPeak archives: 2 run(s), signal read as: autoThe non-peak spectra variables are typically small compared to the
peak data, so MsBackendParquet keeps them in a bounded,
read-through in-memory index. The cache is held at the package level,
keyed by dataset path and shared by every backend over that dataset, so
it is independent of the individual, serialisable backend objects. It is
warmed on backendInitialize() and dropped automatically
whenever the dataset on disk is (re-)written. It is entirely separate
from the user-modified variables tracked by
MsBackendCached, which always take precedence.
The cache is controlled through three options, shown here with their defaults:
## Enable ("auto", the default) or disable ("off") the metadata cache.
options(MsBackendParquet.cacheMetadata = "auto")
## Columns held by default (the ones the optimised filters need).
options(MsBackendParquet.cacheColumns =
c("msLevel", "rtime", "precursorMz", "dataOrigin"))
## Budget in cells (rows x columns); 5e7 is roughly 400 MB of doubles.
## Requests over budget quietly fall back to reading from disk.
options(MsBackendParquet.cacheMaxCells = 5e7)Two further options tune the underlying DuckDB session.
MsBackendParquet.threads sets how many threads DuckDB uses,
and Arrow too. When it is unset, both use at most two threads, and one
when the registered BiocParallel
backend is serial, so the package does not take every core of a shared
machine. DuckDB reads the option when the package first opens its
connection, and Arrow when the package is loaded, so set it before
either. MsBackendParquet.memoryLimit maps to DuckDB’s
memory_limit, and is left at DuckDB’s default when
unset.
For very large mzML files, the default conversion path goes through a
Spectra object built with MsBackendMzR(),
which materialises a whole chunk of peaks in memory before writing it to
Parquet. The engine = "mzr" option bypasses
Spectra entirely and streams batches of spectra directly
from mzR::openMSfile() into Apache Arrow’s incremental
ParquetFileWriter:
be_stream <- mzMLToParquet(fl, path = tempfile(),
engine = "mzr", batch_size = 1000L,
verbose = FALSE)
length(be_stream)
#> [1] 7602Memory use stays bounded by
batch_size * mean(peaks per spectrum), regardless of total
file size. The streaming engine processes files serially and requires
the mzR package.
Storing the peaks as columnar list<double> values
lets DuckDB skip whole row groups when a query touches only part of the
data, while projection and predicate push-down keep both the columns and
rows read to a minimum. Combined with the in-memory metadata index, this
makes metadata filters roughly as fast as an in-memory backend while the
peak storage stays on disk.
Across a suite of seven queries over 380,100 spectra (50 mzML files),
the Parquet backend is the fastest of the on-disk Spectra
backends on 5 of 7 queries (medians in ms, lower is better):
| query | MsBackendParquet | MsBackendMzR | MsBackendHdf5Peaks | MsBackendOfflineSql |
|---|---|---|---|---|
filterRt(), wide range |
12.24 | 15.44 | 24.05 | 380.19 |
filterRt(), narrow range |
11.23 | 5.33 | 14.76 | 348.17 |
filterPrecursorMzValues(), ppm |
14.09 | 21.66 | 28.66 | 336.33 |
filterRt() and
filterPrecursorMzValues() |
12.56 | 17.25 | 27.75 | 389.22 |
peaksData() of MS1 spectra |
314.82 | 1510 | 1440 | 919.04 |
tic() of MS1 spectra |
506.79 | 1390 | 1490 | 890.82 |
spectraData() of MS1 spectra |
26.09 | 17.27 | 22.30 | 718.91 |
The benchmark scripts live in inst/benchmarks/;
see inst/benchmarks/performance.Rmd for the full numbers,
the configuration sweeps over the tunable options, and the design
considerations behind them.
Because MsBackendParquet objects only contain a file
system path, they are fully serialisable and can be sent to parallel
workers without re-opening connections:
sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04.1 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so; LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] MsExperiment_1.15.0 ProtGenerics_1.45.0 MsDataHub_1.13.3
#> [4] MsBackendParquet_0.99.8 Spectra_1.23.5 BiocParallel_1.47.0
#> [7] S4Vectors_0.51.10 BiocGenerics_0.59.12 generics_0.1.4
#> [10] BiocStyle_2.41.0
#>
#> loaded via a namespace (and not attached):
#> [1] DBI_1.3.0 httr2_1.3.0
#> [3] rlang_1.3.0 magrittr_2.0.5
#> [5] clue_0.3-68 otel_0.2.0
#> [7] matrixStats_1.5.0 compiler_4.6.1
#> [9] RSQLite_3.53.3 png_0.1-9
#> [11] vctrs_0.7.3 reshape2_1.4.5
#> [13] stringr_1.6.0 pkgconfig_2.0.3
#> [15] MetaboCoreUtils_1.21.1 crayon_1.5.3
#> [17] fastmap_1.2.0 dbplyr_2.6.0
#> [19] XVector_0.53.0 rmarkdown_2.32
#> [21] purrr_1.2.2 bit_4.6.0
#> [23] xfun_0.61 MultiAssayExperiment_1.39.1
#> [25] cachem_1.1.0 jsonlite_2.0.0
#> [27] blob_1.3.0 DelayedArray_0.39.8
#> [29] parallel_4.6.1 cluster_2.1.8.3
#> [31] R6_2.6.1 bslib_0.12.0
#> [33] stringi_1.8.9 GenomicRanges_1.65.4
#> [35] jquerylib_0.1.4 Rcpp_1.1.2
#> [37] Seqinfo_1.3.2 assertthat_0.2.1
#> [39] SummarizedExperiment_1.43.0 knitr_1.52
#> [41] IRanges_2.47.5 BiocBaseUtils_1.15.1
#> [43] Matrix_1.7-6 igraph_2.3.4
#> [45] tidyselect_1.2.1 abind_1.4-8
#> [47] yaml_2.3.12 codetools_0.2-20
#> [49] curl_8.0.0 plyr_1.8.9
#> [51] lattice_0.23-1 tibble_3.3.1
#> [53] Biobase_2.73.2 withr_3.0.3
#> [55] KEGGREST_1.53.6 evaluate_1.0.5
#> [57] BiocFileCache_3.3.0 ExperimentHub_3.3.2
#> [59] Biostrings_2.81.9 pillar_1.11.1
#> [61] BiocManager_1.30.27 filelock_1.0.3
#> [63] MatrixGenerics_1.25.0 ncdf4_1.24
#> [65] BiocVersion_3.24.0 glue_1.8.1
#> [67] lazyeval_0.2.3 maketools_1.3.2
#> [69] tools_4.6.1 AnnotationHub_4.3.2
#> [71] sys_3.4.3 data.table_1.18.6.1
#> [73] QFeatures_1.23.2 mzR_2.47.1
#> [75] buildtools_1.0.0 fs_2.1.0
#> [77] grid_4.6.1 tidyr_1.3.2
#> [79] MsCoreUtils_1.25.4 AnnotationDbi_1.75.2
#> [81] duckdb_1.5.6 cli_3.6.6
#> [83] rappdirs_0.3.4 S4Arrays_1.13.2
#> [85] arrow_25.0.1 dplyr_1.2.1
#> [87] AnnotationFilter_1.37.0 sass_0.4.10
#> [89] digest_0.6.39 SparseArray_1.13.4
#> [91] memoise_2.0.1 htmltools_0.5.9
#> [93] lifecycle_1.0.5 httr_1.4.9
#> [95] bit64_4.8.6 MASS_7.3-66