--- title: "Validation: calibration and accuracy" author: "Dharmesh D. Bhuva" package: spiDE output: BiocStyle::html_document: toc_float: true vignette: > %\VignetteIndexEntry{Validation: calibration and accuracy} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- # Can we trust the calls? A test is only useful if its error rate is what it claims. When spiDE reports a $q$-value below 0.05, the expected share of false calls among the calls should be at most 5%. In this article we show how we checked that on real spatial data, where we do not know the truth but can build data in which there is nothing to find, and on simulated data with a known truth, where we can measure accuracy and power. We then report what the checks found for both engines, for the choice of sequencing depth adjustment, for gene sets and for the between-patient test, and we end with what has not been measured. Together with `vignette("spiDE-design")`, this is one of the two places in the package that quote benchmark numbers. How many patients and cells a study needs, and the defaults for small studies, are in that article. The statistics are in `vignette("spiDE-model")`, and the workflow in `vignette("spiDE")`. The studies behind the numbers are described in (Bhuva et al., in preparation). # How calibration is checked The idea is to run spiDE on data where, by construction, there is nothing to find, and to see whether it stays quiet. We make two kinds of such *null* data from each real cohort. The first is **the block null**, in which the neighbourhood is scrambled. In every tissue section we shift the niche densities by a random offset, wrapping around the edges, so that each cell receives the neighbourhood of a cell elsewhere in the same section. The niche maps stay as smooth as real ones, but a cell's expression no longer has anything to do with its neighbourhood. Both the pooled test and the condition-specific test should find nothing. Each shifted map is a full refit. The second is **the permutation null**, in which the condition is scrambled. We shuffle the condition labels between patients, within each slide so that a slide difference cannot pose as a condition difference. The niche effects are real, but they no longer differ between the shuffled groups, so the condition-specific test should find nothing. On each null we read two things: | reading | what it means | pass | |---|---|---| | **spread** | For every gene, the root-mean-square of its null $z$-statistics; the median over genes, in each fifth of genes by expression. A calibrated test gives 1. | 0.90 to 1.10 in every fifth | | **tail** | The fraction of null data sets in which a Benjamini-Hochberg screen at 0.05 made at least one call. | at most 0.10 (block), 0.07 (permutation) | The spread checks the whole distribution of the test statistics, not only the extreme ones. The tail checks the extreme ones, which are what become calls. # The cohorts We used five cohorts, three of them public and two from the YTMA lung cancer cohorts. | cohort | platform, tissue | condition (patients) | bandwidth | |---|---|---|---| | GSE250346 | Xenium, lung (pulmonary fibrosis) | control 9, disease 26 | 60 µm | | GSE282639 | CosMx 1k, lung (IPF) | control 13, IPF 15 | 30 µm | | GSE289194 | CosMx 1k, lymph node | extranodal 20, nodal 24 | 30 µm | | YTMA | CosMx WTA, NSCLC | non-responder 30, responder 25 | 30 µm | | YTMA stage | CosMx WTA, lung adenocarcinoma | early 123, late 40 | 30 µm | Each cohort was analysed in its five best-covered cell types, as both index and niche types, with every other cell adjusted for as one further niche. The three public cohorts carry the conclusions below, and the two YTMA cohorts are shown for completeness. # The slopes engine (the default) The first table is spiDE 0.99.37 at its defaults, on ten block-null maps and 1,000 condition permutations per cohort. A condition tail is given for every triplet ("all") and for the default family ("default"). Small designs are treated in `vignette("spiDE-design")`. | cohort | pooled, block: spread (tail) | condition, block: spread (tail, all / default) | condition, permutation: spread | permutation tail, all / default | |---|---|---|---|---| | GSE250346 | 0.94-0.97 (0) | 0.95-0.96 (0 / 0) | 0.98-0.99 | 0 / 0.002 | | GSE282639 | 0.93-0.98 (0) | 0.97-1.00 (0 / 0.10) | 0.99-1.00 | 0.015 / 0.019 | | GSE289194 | 0.93-0.95 (0) | 0.97-0.99 (0 / 0) | 0.99 | 0.015 / 0.015 | | YTMA | 0.94-0.96 (0) | 0.98 (0 / 0) | 0.99 | 0.010 / 0.028 | | YTMA stage | 0.96-0.98 (0) | 0.98 (0 / 0.20) | 0.99 | 0.012 / 0.013 | Every criterion is met on the three public cohorts and YTMA. GSE282639's default-family block tail is at its limit, at one map of ten. The YTMA stage cohort misses one criterion: its default-family block tail is 0.20, which is two maps of ten, each with a single call at $q$ 0.043 and 0.044 among about 1,200 triplets. On twenty further block maps of the same cohort, which no other step had used, there was no call. The pooled spread sits slightly below 1, so its $p$-values are, if anything, a little conservative. # The sandwich engine We ran the same nulls for the sandwich engine, with 50 permutations per cohort, because each permutation of the sandwich engine is a refit. | cohort | pooled, block: spread (tail) | condition, block: spread (tail) | condition, permutation: spread | tail, all / filtered | |---|---|---|---|---| | GSE250346 | 0.97-1.01 (0) | 0.93-0.98 (0) | 1.00-1.02 | 0 / 0 | | GSE282639 | 0.99-1.01 (0) | 0.96-0.99 (0) | 1.01-1.04 | 0.02 / 0 | | GSE289194 | 1.03-1.06 (0) | 1.03-1.06 (0) | 1.06-1.10 | 0.02 / 0.06 | | YTMA | 0.99-1.00 (0) | 0.99-1.00 (0) | 1.00-1.02 | 0.04 / 0.14 | | YTMA stage | 1.00-1.01 (0.20) | 0.99-1.00 (0) | 1.00-1.02 | 0 / 0 | On the stage cohort (163 patients) a sandwich permutation is a refit of about an hour, so its permutation null has 40 permutations. The sandwich engine runs closer to a spread of 1 but less evenly. It reaches 1.10 on GSE289194, its filtered procedure called in 7 of 50 permutations on YTMA, and it called in 2 of 10 block maps on the stage cohort. It is a useful second opinion, but the slopes engine is the one to rely on. # Accuracy and power, with a known truth Real data cannot tell us how large an effect truly is, or which calls are true. For that, we simulated counts from models fitted to four of these cohorts, using their real cell positions, cell types, patients and niche maps, with gene-specific library-size effects that vary across the tissue. We then planted condition by niche effects, either shared by all patients or varying between them. Six ways in which sequencing depth can relate to the niche were simulated, with 20 data sets per scenario, on GSE289194 and YTMA. These are the two cohorts whose simulations passed a check of realism against their real data (library sizes, detection, dispersion and the genes' response to depth). The simulations of GSE282639 and GSE250346 failed that check and were run on two of the six scenarios, as a description only. These simulations were run with the defaults of spiDE 0.99.32: the condition test within the pooled filter (`procedure = "filtered"`) and the pooled test's capped degrees of freedom, the rule used before spiDE 0.99.37. The defaults of 0.99.37 were measured on the power benchmark and the real-data checks in `vignette("spiDE-design")`. For the condition-specific test (filtered procedure), averaged over data sets: | template | engine | relative bias of the estimate (worst depth scenario) | coverage of 95% intervals | true-positive rate | false discovery proportion | |---|---|---|---|---|---| | GSE289194 | slopes | -0.03 (0.06) | 0.90 | 0.57 | 0.041 | | GSE289194 | sandwich | -0.02 (0.05) | 0.88 | 0.23 | 0.047 | | YTMA | slopes | -0.03 (0.05) | 0.94 | 0.28 | 0.032 | | YTMA | sandwich | -0.01 (0.02) | 0.92 | 0.21 | 0.047 | The estimates are close to the truth: on average they are within about 3% of the planted effect, and within about 6% in the worst depth scenario. The slopes engine finds more true effects, about 2.5 times as many as the sandwich engine on GSE289194, and the gain is largest when effects vary between patients. Both engines keep the false discovery proportion below 5%. On the two simulations that failed the realism check, the slopes engine's filtered procedure ran higher, with 0.05 on GSE282639 and 0.12 on GSE250346. Those simulations do not resemble their real data closely enough to say whether this carries over to real data, and the real-data nulls above show no excess. The pooled test found 26% of the planted pooled effects on GSE289194 and 3% on YTMA with the slopes engine (14% and 5% with the sandwich engine). On YTMA the effects were small against the count noise of whole-transcriptome data. # Sequencing depth A cell's library size mixes technical capture with its biological RNA content, and the RNA content can itself depend on the neighbourhood. We therefore judged depth handling by how accurately each option recovered the planted effects, not by how much library-size signal it removed. We compared four options in the simulations above: - one slope on the log library size (`depth = "loglib"`); - a smooth curve in it (`"nonlinear"`); - a library-size effect that varies across the tissue (`"spatial_spline"`); - SpaNorm's library-size component as a fixed offset. In the slopes engine the four lay within 0.015 of each other in bias, with the same power, and every one was calibrated in every scenario, so the default is the simplest, `"loglib"`. (The sandwich engine with `"loglib"` ran wide on GSE289194, up to 1.13.) The spatial option costs a median of 1.2 to 1.8 times as much to fit, per template, in the slopes engine. One consequence holds for every option. When the neighbourhood raises a cell's total RNA, adjusting for library size also removes that part of the niche effect, so a pooled slope reports expression **relative to the cell's total RNA**. In the simulations this removed most of a niche-driven change in total RNA from the pooled slope. The condition-specific test is affected only by the part of that change that differs between conditions. # Gene sets We checked `spiGSEA()` in the same way with spiDE 0.99.37, on about 6,400 GO biological-process sets per cohort (10 to 500 genes each), with 10 block maps and 200 condition permutations: | cohort | pooled, block: spread (tail) | condition, block: spread (tail) | condition, permutation: spread (tail) | |---|---|---|---| | GSE250346 | 0.90-0.97 (0) | 0.93-0.95 (0) | 0.98 (0) | | GSE282639 | 0.89-0.99 (0) | 0.96-0.97 (0) | 0.99-1.00 (0.005) | | GSE289194 | 0.94-0.98 (0) | 0.95-0.97 (0) | 0.99 (0.010) | | YTMA | 0.97-0.99 (0) | 0.95-0.96 (0) | 0.99 (0.015) | | YTMA stage | 1.01-1.03 (0) | 0.95-0.96 (0) | 0.99 (0.010) | The spread is read per fifth of sets by size, and a tail counts a call at `q.global` < 0.05. No null made more than an occasional call. On GSE250346 and GSE282639 the pooled set test misses the lower bound of the spread criterion. Its largest fifth of sets reaches 0.89 to 0.90, so the test is slightly conservative there. With the capped degrees of freedom used before spiDE 0.99.37 it reached 0.88 to 0.89 on the same two cohorts. The pooled set test uses the proportional rule. # The between-patient test We checked `testNicheAbundance()` with 200 permutations within slide per cohort, each of two kinds. In **condition permutations** the condition labels are shuffled across patients, which is the null of the term `"condition:niche"`. In **niche permutations** each patient's niche abundances are shuffled across patients, which is the null of the term `"niche"`. The tail is the share of permutations with any call at `q.global` < 0.05, against a limit of 0.07: | cohort | condition:niche, no cap | condition:niche, default cap | niche, no cap | niche, default cap | |---|---|---|---|---| | GSE250346 | 0.15 | 0.050 | 0.15 | 0.010 | | GSE282639 | 0.03 | 0.025 | 0.07 | 0.055 | | GSE289194 | 0.07 | 0.040 | 0.14 | 0.075 | | YTMA | 0.07 | 0.025 | 0.145 | 0.060 | | YTMA stage | 0.28 | 0.010 | 0.33 | 0.020 | Without the leverage cap, a single patient at the edge of a niche type's abundances made false calls in up to a third of permutations. The default, `max.leverage = 3`, is the weakest cap that keeps the condition test within its limit on all three public cohorts, and a cap of 2 over-corrects: its null spread falls to 0.56-0.68. At the default the test is slightly conservative on GSE250346 (spread 0.86-0.95) and YTMA stage (0.86-0.88), and calibrated on the other three (0.93-1.02). A finer design breaks it. On the YTMA cohort split into 13 or 24 cell types, with many types that have only 14-36 cells per patient, the condition test still made false calls in 11-35% of permutations at the default cap. Its calls are not reliable when each patient contributes few cells of the index type. # What has not been measured Some questions remain open. - **Bandwidth.** Each cohort was checked at one bandwidth. Comparing results across bandwidths is a sensitivity analysis, and each bandwidth is its own set of tests. - **Many index types at once on small panels.** The checks used five index types per cohort. With many more, the pooled test's multiple-testing burden grows and power falls accordingly, though calibration is not expected to change. - **Very small cohorts on real data.** The smallest real group checked whole had 9 patients. Smaller designs were checked by subsampling the real cohorts' patients and simulated down to 6 patients (3 per condition), as described in `vignette("spiDE-design")`. At 6 patients (3 per condition) the condition-specific test is mostly unavailable. For each gene it needs a Kish effective number of patients, $(\sum w)^2 / \sum w^2$ over that gene's usable patients' weights, of at least 3 in each condition. The effective number never exceeds the patient count and equals it only when the weights are equal, so 3 patients in a group pass only if their weights are equal; in practice a group needs more than 3 patients. Genes that fail return `NA`. # Session information ```{r session, eval = TRUE} sessionInfo() ```