Comprehensive Catalog of the 62 Evaluation Metrics

Overview of the Evaluation Architecture

scSimEval evaluates single-cell simulation methods using an objective, ground-truth-free framework. It includes 62 distinct evaluation measures organized into 8 core biological and computational categories.

Every metric operates directly on the real biological reference data and the synthetic data generated by single-cell and multiomics simulators (scRNA-seq, scATAC-seq, and paired/unpaired co-assays). All 62 metrics in this catalog match identically with the metric identifiers and labels evaluated within the package functions, exported tables, and the interactive scSimEval Studio Shiny application.

Summary Table of the 8 Categories

Category Domain Number of Metrics Primary Focus
I Distributional Properties 14 Statistical distance between real and simulated single-cell count distributions and manifolds
II Correlations & Zero-Inflation 6 Biological overdispersion (BCV), dropout curves, cell correlations, and gene co-expression
III Cellular Structure & Concordance 10 Unsupervised cluster separation (ASW, Dunn, DB, CH) and supervised cell type preservation (ARI, NMI, AMI, V-measure, purity, precision)
IV Batch Effects & Confounder Mixing 7 Technical batch mixing (CellMixS, LISI, Shannon entropy, PCR \(R^2\), Seurat mixing, batch ASW, and ldfDiff)
V Biological Signal & Downstream Fidelity 15 Differentially expressed gene (DEG) preservation, fold-change correlation, null uniformity, classifier accuracy, and Shaky Foundations variance
VI Trajectory & Lineage Dynamics 2 Inferred developmental pseudotime rank correlation and cell lineage branching tree topology
VII Cross-Modal Coupling & Modularity 6 Paired and joint ATAC-RNA regulatory links, FOSCTTM, , label transfer, and \(R_V\) coefficient
VIII Computational Scalability 2 Execution runtime in seconds and peak RAM memory allocation in MiB
Total 62 Complete single-cell and multiomics simulation evaluation suite

Complete Master Reference Table (62 Metrics)

Below is the definitive cross-reference table mapping each metric’s internal identifier, user-facing display label in the Shiny app and figures, category, evaluation direction, and scope:

# Metric ID Shiny App Display Label Category Direction Description
1 KS KS Distance (I) Distributional Properties Lower is better Kolmogorov-Smirnov supremum distance between cumulative distributions
2 Wasserstein Wasserstein Dist. (I) Distributional Properties Lower is better 1D Earth Mover’s Distance measuring work required to transform distributions
3 MAD MAD (I) Distributional Properties Lower is better Median Absolute Deviation discrepancy across cell and gene properties
4 MAE MAE (I) Distributional Properties Lower is better Mean Absolute Error across summary statistics
5 RMSE RMSE (I) Distributional Properties Lower is better Root Mean Squared Error penalizing large deviations in moment statistics
6 OV Overlap Coeff. (I) Distributional Properties Higher is better Kernel density curve area intersection between reference and simulated distributions
7 Bhattacharyya Bhattacharyya Dist. (I) Distributional Properties Lower is better Statistical divergence between discrete count probability distributions
8 ECDF_DiffArea ECDF Diff. Area (I) Distributional Properties Lower is better Integrated area difference between empirical cumulative distribution functions
9 Runs_Statistic Runs Statistic (I) Distributional Properties Lower is better Wald-Wolfowitz runs test absolute \(z\)-score testing random interleaving
10 Fasano_Franceschini_2D_KS Fasano-Franceschini 2D (I) Distributional Properties Lower is better 2D extension of the KS statistic for bivariate feature distributions
11 Peacock_2D_KS Peacock 2D (I) Distributional Properties Lower is better 2D Peacock bivariate distribution test statistic
12 KDE_Bivariate_zstat 2D KDE z-stat (I) Distributional Properties Lower is better 2D Kernel Density Estimation divergence \(z\)-statistic between joint distributions
13 MMD MMD (I) Distributional Properties Lower is better Maximum Mean Discrepancy testing multivariate single-cell manifolds in RKHS
14 Frechet_SingleCell_Distance Frechet SC Dist. (I) Distributional Properties Lower is better Fréchet single-cell distance between multivariate Gaussian manifolds in PC space
15 bcv_discrepancy Biological CV (BCV) (II) Correlations & Zero-Inflation Lower is better Biological Coefficient of Variation discrepancy after Poisson noise subtraction
16 dropout_curve_midpoint_x0 Dropout Midpoint (x0) (II) Correlations & Zero-Inflation Lower is better Logistic dropout curve midpoint (\(x_0\)) estimation error
17 dropout_curve_slope_k Dropout Slope (k) (II) Correlations & Zero-Inflation Lower is better Logistic dropout curve slope (\(k\)) steepness error
18 mean_variance_r2 Mean-Variance R2 (II) Correlations & Zero-Inflation Higher is better Mean-variance trend fit (\(R^2\)) and dropout model goodness-of-fit
19 cell_pearson_cor Cell Pearson Corr. (II) Correlations & Zero-Inflation Higher is better Cell-to-cell Pearson correlation matrix concordance
20 feature_pearson_cor Feature Pearson Corr. (II) Correlations & Zero-Inflation Higher is better Gene-to-gene Pearson co-expression network correlation concordance
21 silhouette_sim Silhouette ASW (III) Cellular Structure & Concordance Higher is better Average Silhouette Width (ASW) cluster separation fidelity
22 dunn_sim Dunn Index (III) Cellular Structure & Concordance Higher is better Dunn index measuring ratio of inter-cluster separation to cluster diameter
23 davies_bouldin_sim Davies-Bouldin (III) Cellular Structure & Concordance Lower is better Davies-Bouldin compactness and separation discrepancy
24 calinski_harabasz_sim Calinski-Harabasz (III) Cellular Structure & Concordance Higher is better Calinski-Harabasz variance ratio criterion between and within clusters
25 ari ARI (III) Cellular Structure & Concordance Higher is better Adjusted Rand Index comparing unsupervised clusters against cell type labels
26 nmi NMI (III) Cellular Structure & Concordance Higher is better Normalized Mutual Information between unsupervised clusters and known cell types
27 ami AMI (III) Cellular Structure & Concordance Higher is better Adjusted Mutual Information adjusted for chance cluster overlap
28 v_measure V-Measure (III) Cellular Structure & Concordance Higher is better Harmonic mean of cluster homogeneity and completeness
29 neighborhood_purity Neighborhood Purity (III) Cellular Structure & Concordance Higher is better Proportion of \(k\)-nearest neighbors sharing identical annotated cell type labels
30 Generative_Precision Generative Precision (III) Cellular Structure & Concordance Higher is better Hungarian optimal matching accuracy between simulated and real cell partitions
31 batch_silhouette Batch Silhouette (IV) Batch Effects & Confounder Mixing Higher is better Average Silhouette Width for batch confounders (lower batch clustering is better)
32 shannon_entropy Shannon Entropy (IV) Batch Effects & Confounder Mixing Higher is better Batch Shannon diversity entropy across local cell neighborhoods
33 pcr_r2 PC Regression R2 (IV) Batch Effects & Confounder Mixing Lower is better Principal Component Regression variance explained by technical batch labels (\(R^2_{\text{batch}}\))
34 cms CMS Score (IV) Batch Effects & Confounder Mixing Higher is better CellMixS cell-specific mixing score testing random batch distributions
35 isi LISI / ISI (IV) Batch Effects & Confounder Mixing Higher is better Local Inverse Simpson Index measuring effective batches per neighborhood
36 seurat_mixing_metric Seurat Mixing (IV) Batch Effects & Confounder Mixing Lower is better Seurat batch mixing metric measuring average rank of same-batch neighbors
37 ldf_diff ldfDiff (IV) Batch Effects & Confounder Mixing Lower is better Local Density Factor discrepancy measuring manifold density distortion
38 SimBench_SMAPE SimBench SMAPE (V) Biological Signal & Downstream Fidelity Lower is better Symmetric Mean Absolute Percentage Error in differentially expressed log2FC
39 SimBench_DE_Fidelity_Score DE Fidelity (1-SMAPE) (V) Biological Signal & Downstream Fidelity Higher is better SimBench composite differential expression fidelity score (\(1 - \text{SMAPE}\))
40 Log2FC_Pearson_Corr Log2FC Pearson Corr (V) Biological Signal & Downstream Fidelity Higher is better Pearson correlation of estimated log2 fold-changes across marker genes
41 Log2FC_Spearman_Corr Log2FC Spearman Corr (V) Biological Signal & Downstream Fidelity Higher is better Spearman rank correlation of estimated log2 fold-changes
42 Top_DEG_Jaccard_Overlap Top DEG Jaccard (V) Biological Signal & Downstream Fidelity Higher is better Jaccard index overlap of top differentially expressed genes between real and sim
43 DEG_Ratio DEG Ratio (Sim/Real) (V) Biological Signal & Downstream Fidelity Matches 1.0 Ratio of identified DEG count in simulated data vs real reference data
44 PValue_Uniformity_Chisq P-Val Unif. (Chisq) (V) Biological Signal & Downstream Fidelity Lower is better \(\chi^2\) divergence test verifying uniform p-value distribution under the null hypothesis
45 Distribution_Score Distribution Score (V) Biological Signal & Downstream Fidelity Higher is better Composite null p-value distribution preservation score without false positives
46 Classifier_Accuracy Classifier Accuracy (V) Biological Signal & Downstream Fidelity Higher is better Random forest / SVM cell type classification accuracy on marker transcripts
47 Classifier_Macro_F1 Classifier Macro-F1 (V) Biological Signal & Downstream Fidelity Higher is better Macro-averaged F1 classification score across rare and common cell populations
48 Classifier_Macro_Recall Classifier Macro-Recall (V) Biological Signal & Downstream Fidelity Higher is better Macro-averaged recall score across biological cell identities
49 Silhouette_Sim Group Silhouette (V) Biological Signal & Downstream Fidelity Higher is better Shaky Foundations cell-type silhouette similarity in DEG marker space
50 Silhouette_Discrepancy Silhouette Discrepancy (V) Biological Signal & Downstream Fidelity Lower is better Shaky Foundations absolute silhouette width discrepancy relative to reference
51 PVE_Group_Sim Group PVE (V) Biological Signal & Downstream Fidelity Higher is better Shaky Foundations Proportion of Variance Explained (PVE) by cell groups
52 PVE_Discrepancy Group PVE Discrepancy (V) Biological Signal & Downstream Fidelity Lower is better Discrepancy in proportion of variance explained by biological condition
53 pseudotime_correlation Pseudotime Corr. (rho) (VI) Trajectory & Lineage Dynamics Higher is better Spearman rank correlation of inferred developmental pseudotime ordering
54 tree_height_rmse Lineage Tree RMSE (VI) Trajectory & Lineage Dynamics Lower is better Root Mean Squared Error of lineage branching hierarchy tree heights
55 cross_modal_accuracy Cross-Modal Accuracy (VII) Cross-Modal Coupling & Modularity Higher is better Cross-modal cell type label transfer accuracy (RNA \(\rightarrow\) ATAC)
56 cross_modal_F1 Cross-Modal F1 (VII) Cross-Modal Coupling & Modularity Higher is better Macro-averaged F1 score of cross-modal cell type label transfer
57 FOSCTTM FOSCTTM (VII) Cross-Modal Coupling & Modularity Lower is better Fraction of Samples Closer Than The True Match in joint multi-omics space
58 match_at_1 (VII) Cross-Modal Coupling & Modularity Higher is better Top-1 cross-modal nearest neighbor exact cell pairing rate
59 rv_coefficient RV Coefficient (VII) Cross-Modal Coupling & Modularity Higher is better Vector correlation coefficient measuring chromatin co-accessibility and expression coupling
60 module_correlation_r Module Corr. (r) (VII) Cross-Modal Coupling & Modularity Higher is better Peak-to-gene linkage and co-regulatory module correlation concordance
61 elapsed_time_seconds Elapsed Time (s) (VIII) Computational Scalability Lower is better Wall-clock execution runtime in seconds required for model training and synthesis
62 peak_memory_mb Peak RAM (MiB) (VIII) Computational Scalability Lower is better Maximum resident set size / peak memory allocated during execution

Category 1: Distribution Properties (14 Metrics)

Evaluates whether single-cell count distributions (library size, mean expression, gene variance, and zero fraction) and higher-dimensional data manifolds in simulated data match real experimental data.

1. Kolmogorov-Smirnov (KS) Distance (KS)

  • Direction: Lower is better (inverted during score normalization).
  • Formula: \(D_{KS} = \sup_x |F_{ref}(x) - F_{sim}(x)|\)
  • Biological Meaning: Measures the maximum vertical distance between the empirical cumulative distribution functions of real reference and simulated properties. A value of 0 indicates identical distributions.

2. Wasserstein Metric (Earth Mover’s Distance 1D) (Wasserstein)

  • Direction: Lower is better.
  • Formula: \(W_1(P, Q) = \int_{-\infty}^{\infty} |F_{ref}(x) - F_{sim}(x)| \, dx\)
  • Biological Meaning: Represents the minimal cost required to transform the simulated distribution into the reference distribution, integrating differences across the full range of counts.

3. Median Absolute Deviation (MAD)

  • Direction: Lower is better (\(|\text{MAD}_{ref} - \text{MAD}_{sim}|\)).
  • Formula: \(\text{MAD} = \text{median}(|x - \text{median}(x)|)\)
  • Biological Meaning: Robust non-parametric estimate of count spread that resists distortion by extreme sequencing outliers.

4. Mean Absolute Error (MAE)

  • Direction: Lower is better.
  • Formula: \(\text{MAE} = \frac{1}{n} \sum_{i=1}^n |x_i - y_i|\)
  • Biological Meaning: Average absolute deviation between cell-level and gene-level summary statistics.

5. Root Mean Squared Error (RMSE)

  • Direction: Lower is better.
  • Formula: \(\text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i - y_i)^2}\)
  • Biological Meaning: Standard quadratic scoring rule penalizing large discrepancies between real and simulated statistical moments.

6. Density Curve Overlap (OV)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Formula: \(\text{Overlap} = \int \min(f_{ref}(x), f_{sim}(x)) \, dx\)
  • Biological Meaning: Shared area under the kernel density estimates of real and simulated data. A value of \(1.0\) represents identical density profiles.

7. Bhattacharyya Distance (Bhattacharyya)

  • Direction: Lower is better.
  • Formula: \(D_B(P, Q) = -\ln(BC(P, Q))\), where \(BC(P, Q) = \sum \sqrt{p(x) q(x)}\)
  • Biological Meaning: Measures statistical overlap and divergence between discrete single-cell count probability distributions.

8. Empirical Cumulative Distribution Area Difference (ECDF_DiffArea)

  • Direction: Lower is better.
  • Formula: \(\Delta_{\text{ECDF}} = \int |F_{ref}(x) - F_{sim}(x)| \, dx\)
  • Biological Meaning: Directly integrates the area enclosed between real and simulated empirical cumulative distribution functions.

9. Wald-Wolfowitz Runs Test Statistic (Runs_Statistic)

  • Direction: Lower is better (\(|Z_{\text{runs}}|\), where values near 0 indicate identical orderings).
  • Biological Meaning: Non-parametric test checking whether interleaved reference and simulated observations mix randomly, confirming they originate from the same continuous distribution.

10. 2D Fasano-Franceschini Test Statistic (Fasano_Franceschini_2D_KS)

  • Direction: Lower is better.
  • Biological Meaning: Computationally efficient bivariate generalization of the Kolmogorov-Smirnov test over quadrants, evaluating paired properties (e.g. library size vs. cell zero fraction).

11. 2D Peacock Bivariate Test Statistic (Peacock_2D_KS)

  • Direction: Lower is better.
  • Biological Meaning: Rigorous two-dimensional Kolmogorov-Smirnov test measuring supremum divergence across all four quadrants in bivariate planes (e.g. mean vs. variance).

12. 2D Bivariate Kernel Density Estimation z-Statistic (KDE_Bivariate_zstat)

  • Direction: Lower is better.
  • Biological Meaning: Non-parametric two-dimensional kernel density divergence statistic assessing the global shape and contours of paired feature distributions.

13. Maximum Mean Discrepancy (MMD)

  • Direction: Lower is better.
  • Formula: \(\text{MMD}^2(P, Q) = \mathbb{E}[k(x, x')] - 2\mathbb{E}[k(x, y)] + \mathbb{E}[k(y, y')]\)
  • Biological Meaning: Kernel-based two-sample test comparing multivariate single-cell distributions in reproducing kernel Hilbert space without projecting down to 1D summaries.

14. Fréchet Single-Cell Distance (Frechet_SingleCell_Distance)

  • Direction: Lower is better.
  • Formula: \(d_F^2 = \|\mu_{ref} - \mu_{sim}\|_2^2 + \text{Tr}(\Sigma_{ref} + \Sigma_{sim} - 2(\Sigma_{ref}\Sigma_{sim})^{1/2})\)
  • Biological Meaning: Multivariate Gaussian manifold divergence in principal component space between empirical and synthesized cell populations.

Category 2: Correlations and Zero Patterns (6 Metrics)

Evaluates whether simulators accurately capture biological overdispersion, zero-inflation dynamics, cell-to-cell dependencies, and gene co-expression networks.

15. Biological Coefficient of Variation Distance (bcv_discrepancy)

  • Direction: Lower is better (\(|\text{BCV}_{ref} - \text{BCV}_{sim}|\)).
  • Formula: \(\text{BCV} = \sqrt{\frac{\sigma^2}{\mu^2} - \frac{1}{\mu}}\)
  • Biological Meaning: Isolates true biological variability across cells after subtracting Poisson technical sampling noise.

16. Logistic Dropout Curve Midpoint Error (dropout_curve_midpoint_x0)

  • Direction: Lower is better (\(|x_{0,\text{ref}} - x_{0,\text{sim}}|\)).
  • Formula: \(P(\text{zero}) = \frac{1}{1 + \exp(-k(x - x_0))}\)
  • Biological Meaning: Evaluates the expression threshold (\(x_0\)) at which a gene has a 50% probability of exhibiting technical dropout (zero count).

17. Logistic Dropout Curve Slope Error (dropout_curve_slope_k)

  • Direction: Lower is better (\(|k_{\text{ref}} - k_{\text{sim}}|\)).
  • Biological Meaning: Evaluates the rate of transition (\(k\)) from unobserved dropouts to consistently detected counts as expression levels increase.

18. Mean-Variance Fit / Dropout Goodness-of-Fit (mean_variance_r2)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Quantifies how accurately the non-linear relationship between mean expression and zero frequency / variance is preserved in simulated counts.

19. Cell-Cell Pearson Correlation Distance (cell_pearson_cor)

  • Direction: Higher is better (or Frobenius discrepancy, inverted).
  • Biological Meaning: Confirms that pairwise cell similarity matrices and cellular manifold neighborhoods are accurately replicated.

20. Feature/Gene-Gene Pearson Correlation Distance (feature_pearson_cor)

  • Direction: Higher is better.
  • Biological Meaning: Assesses whether gene co-expression modules and transcriptional regulatory dependencies maintain their pairwise correlation strength.

Category 3: Cell Structure and Clustering (10 Metrics)

Evaluates whether simulated datasets preserve distinct cell types, cluster separation, and biological partitions without artificial blurring.

21. Average Silhouette Width Discrepancy (silhouette_sim)

  • Direction: Higher is better (similarity) / Lower is better (discrepancy).
  • Formula: \(s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}\)
  • Biological Meaning: Measures how well-separated distinct cell types are relative to within-cluster compactness.

22. Dunn Index Discrepancy (dunn_sim)

  • Direction: Higher is better / Lower discrepancy.
  • Formula: \(D = \frac{\min_{i \neq j} d(C_i, C_j)}{\max_k \text{diam}(C_k)}\)
  • Biological Meaning: Ratio between the smallest inter-cluster distance and largest cluster diameter, evaluating worst-case cluster separation.

23. Davies-Bouldin Index Discrepancy (davies_bouldin_sim)

  • Direction: Lower is better.
  • Formula: \(DB = \frac{1}{k} \sum_{i=1}^k \max_{j \neq i} \left( \frac{\sigma_i + \sigma_j}{d(c_i, c_j)} \right)\)
  • Biological Meaning: Measures similarity between clusters; lower scores reflect compact clusters that are widely separated.

24. Calinski-Harabasz Index Discrepancy (calinski_harabasz_sim)

  • Direction: Higher is better / Lower discrepancy.
  • Formula: \(CH = \frac{\text{SS}_B / (k - 1)}{\text{SS}_W / (N - k)}\)
  • Biological Meaning: Variance ratio criterion measuring overall between-cluster dispersion relative to within-cluster dispersion.

25. Adjusted Rand Index Concordance (ari)

  • Direction: Higher is better (range: \([-1, 1]\), where \(1.0\) is perfect match).
  • Biological Meaning: Evaluates agreement between unsupervised clusters and known biological cell type labels, corrected for chance agreement.

26. Normalized Mutual Information (nmi)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Formula: \(\text{NMI}(U, V) = \frac{2 I(U; V)}{H(U) + H(V)}\)
  • Biological Meaning: Measures shared mutual information between unsupervised cluster assignments and biological cell types.

27. Adjusted Mutual Information (ami)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Information-theoretic concordance between cell partitions adjusted for expected random overlap across unbalanced clusters.

28. \(V\)-Measure (v_measure)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Formula: \(V = \frac{2 \cdot h \cdot c}{h + c}\)
  • Biological Meaning: Harmonic mean of cluster homogeneity (clusters contain single cell types) and completeness (all cells of a given type are assigned together).

29. Neighborhood Purity (neighborhood_purity)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Fraction of \(k\)-nearest neighbors in embedding space that share the same biological cell identity.

30. Generative Precision / Hungarian Optimal Matching (Generative_Precision)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Solves the linear sum assignment problem (Hungarian algorithm) to find optimal one-to-one pairings between simulated clusters and real biological cell types.

Category 4: Batch Effects and Technical Confounders (7 Metrics)

Evaluates whether technical batch variation is realistically synthesized without washing out biological signals.

31. Batch Silhouette ASW (batch_silhouette)

  • Direction: Higher is better for mixed batches (or discrepancy relative to reference batch ASW).
  • Biological Meaning: Average Silhouette Width calculated using batch labels; near-zero values indicate well-mixed batches where cells do not cluster solely by technical batch.

32. Batch Shannon Entropy (shannon_entropy)

  • Direction: Higher is better (range: \([0, \ln(B)]\)).
  • Formula: \(H = -\sum_{b=1}^B p_b \ln(p_b)\)
  • Biological Meaning: Measures batch composition diversity in local single-cell neighborhoods; high entropy indicates thorough mixing across batches.

33. Principal Component Regression Batch Variance (pcr_r2)

  • Direction: Lower is better (or low discrepancy relative to reference).
  • Biological Meaning: Quantifies the proportion of global gene expression variance linearly explained by technical batch covariates (\(R^2_{\text{batch}}\)).

34. CellMixS Mixing Score (cms)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Tests whether the distribution of batch labels in local neighborhoods follows random expectations derived from overall batch proportions.

35. Local Inverse Simpson Index (isi)

  • Direction: Higher is better (range: \([1, B]\)).
  • Formula: \(\text{LISI} = \frac{1}{\sum_{b=1}^B p_b^2}\)
  • Biological Meaning: Effective number of distinct batches represented in each cell’s nearest-neighbor neighborhood.

36. Seurat Mixing Metric (seurat_mixing_metric)

  • Direction: Lower is better.
  • Biological Meaning: Computes the average rank of neighbors belonging to the same batch; lower ranks reflect thorough mixing without batch clustering.

37. Local Density Factor Discrepancy (ldf_diff)

  • Direction: Lower is better.
  • Biological Meaning: Measures whether local density factor differences between real and simulated data remain small, ensuring local manifold geometries are not distorted by batch artifacts.

Category 5: Marker Genes and Biological Signals (15 Metrics)

Assesses whether differentially expressed genes (DEGs), fold-changes, null p-value uniformity, and downstream classification models are faithfully preserved across established benchmarking frameworks (SimBench, Simpipe, and Shaky Foundations).

38. Symmetric Mean Absolute Percentage Error (SimBench_SMAPE)

  • Direction: Lower is better (range: \([0, 200\%]\)).
  • Formula: \(\text{SMAPE} = \frac{100\%}{n} \sum_{i=1}^n \frac{|y_i - \hat{y}_i|}{(|y_i| + |\hat{y}_i|)/2}\)
  • Biological Meaning: Relative percentage error in log2 fold-changes of differentially expressed genes between reference and simulated datasets.

39. SimBench Composite DE Fidelity Score (SimBench_DE_Fidelity_Score)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Formula: \(\text{Score} = 1 - \frac{\text{SMAPE}}{200}\)
  • Biological Meaning: Unified SimBench score capturing overall differential expression effect size retention.

40. Log2 Fold-Change Pearson Correlation (Log2FC_Pearson_Corr)

  • Direction: Higher is better (range: \([-1, 1]\)).
  • Biological Meaning: Measures linear proportionality of estimated DEG effect sizes between real and simulated experiments.

41. Log2 Fold-Change Spearman Correlation (Log2FC_Spearman_Corr)

  • Direction: Higher is better (range: \([-1, 1]\)).
  • Biological Meaning: Measures whether the relative ranking of marker genes by effect size is preserved.

42. Top DEG Jaccard Overlap (Top_DEG_Jaccard_Overlap)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Formula: \(J(A, B) = \frac{|A \cap B|}{|A \cup B|}\)
  • Biological Meaning: Set intersection overlap of the top differentially expressed marker genes between real and simulated data.

43. DEG Detection Ratio (DEG_Ratio)

  • Direction: Matches 1.0 (range: \([0, \infty)\)).
  • Formula: \(\text{Ratio} = \frac{N_{\text{DEG, sim}}}{N_{\text{DEG, ref}}}\)
  • Biological Meaning: Evaluates sensitivity and false discovery balance by checking whether simulated data generates the expected quantity of significant marker genes.

44. Uniform P-Value Distribution Discrepancy (PValue_Uniformity_Chisq)

  • Direction: Lower is better.
  • Biological Meaning: \(\chi^2\) goodness-of-fit test assessing whether non-differentially expressed null genes yield uniform p-values without anti-conservative false positive spikes.

45. Null P-Value Distribution Preservation Score (Distribution_Score)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Composite score verifying that null p-value uniformity is preserved without systematic type I error inflation.

46. Machine Learning Cell Type Classification Accuracy (Classifier_Accuracy)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Accuracy of a machine learning classifier trained on marker genes in identifying cell types.

47. Macro-Averaged F1 Score (Classifier_Macro_F1)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Formula: \(F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}\)
  • Biological Meaning: Harmonic balance between precision and recall across all cell types, preventing common populations from masking rare cell types.

48. Macro-Averaged Recall Score (Classifier_Macro_Recall)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Unweighted average sensitivity across biological cell types in simulated datasets.

49. Shaky Foundations Group Silhouette Similarity (Silhouette_Sim)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Silhouette width concordance in differential marker space according to the Shaky Foundations methodology.

50. Shaky Foundations Silhouette Discrepancy (Silhouette_Discrepancy)

  • Direction: Lower is better.
  • Biological Meaning: Absolute difference in cell-group silhouette separation between real and simulated datasets.

51. Shaky Foundations Group PVE Similarity (PVE_Group_Sim)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Preservation of the Proportion of Variance Explained (PVE) by biological condition/cell type.

52. Shaky Foundations PVE Discrepancy (PVE_Discrepancy)

  • Direction: Lower is better.
  • Biological Meaning: Absolute discrepancy in the proportion of total variance explained by biological group structure.

Category 6: Trajectory & Lineage Dynamics (2 Metrics)

Evaluates whether continuous developmental progressions and branching lineage hierarchies are accurately reconstructed.

53. Pseudotime Rank Correlation (pseudotime_correlation)

  • Direction: Higher is better (range: \([-1, 1]\)).
  • Biological Meaning: Spearman rank correlation of cell progression ordering along developmental trajectories (e.g. stem \(\rightarrow\) progenitor \(\rightarrow\) differentiated), inferred directly from expression counts.

54. Lineage Tree Height Discrepancy (tree_height_rmse)

  • Direction: Lower is better.
  • Biological Meaning: Root Mean Squared Error of lineage tree branch lengths and branching points between real and simulated developmental trees.

Category 7: Cross-Modal Coupling & Modularity (6 Metrics)

Evaluates the joint regulatory linkage between chromatin accessibility (scATAC-seq) and gene expression (scRNA-seq) in multiomics datasets.

55. Cross-Modal Cell Type Transfer Accuracy (cross_modal_accuracy)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Accuracy of predicting cell types in Modality 2 (scATAC-seq) using a classifier trained strictly on Modality 1 (scRNA-seq).

56. Cross-Modal Cell Type Transfer Macro-F1 Score (cross_modal_F1)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Macro-averaged F1 score evaluating cross-modality cell identity transfer across rare and common cell states.

57. Fraction of Samples Closer Than The True Match (FOSCTTM)

  • Direction: Lower is better (range: \([0, 0.5]\), where \(0.0\) indicates perfect alignment and \(0.5\) represents random chance).
  • Formula: \(\text{FOSCTTM} = \frac{1}{2N} \left( \sum_{i=1}^N \frac{|\{j : d(x_i, y_j) < d(x_i, y_i)\}|}{N} + \sum_{j=1}^N \frac{|\{i : d(y_j, x_i) < d(y_j, x_j)\}|}{N} \right)\)
  • Biological Meaning: Measures single-cell alignment accuracy in joint multi-omics latent space for co-assayed cells.

58. Top-1 Cross-Modal Match Rate (match_at_1)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Biological Meaning: Proportion of single cells whose true co-assayed profile is its nearest neighbor (rank 1) in joint embedding space.

59. Regulatory Coupling RV Coefficient (rv_coefficient)

  • Direction: Higher is better (range: \([0, 1]\)).
  • Formula: \(R_V(X, Y) = \frac{\text{Tr}(X X^T Y Y^T)}{\sqrt{\text{Tr}((X X^T)^2) \text{Tr}((Y Y^T)^2)}}\)
  • Biological Meaning: Vector matrix correlation measuring global co-inertia and shared regulatory geometry between chromatin accessibility and transcriptional output.

60. Cross-Modal Module Correlation Fidelity (module_correlation_r)

  • Direction: Higher is better (range: \([-1, 1]\)).
  • Biological Meaning: Evaluates whether peak-to-gene correlations and cis-regulatory co-accessibility modules maintain biological concordance.

Category 8: Computational Scalability (2 Metrics)

Evaluates computational runtime and memory resource consumption across simulators.

61. Wall-Clock Execution Time (elapsed_time_seconds)

  • Direction: Lower is better.
  • Biological Meaning: Total elapsed runtime in seconds required to train the generative model and synthesize simulated single-cell profiles.

62. Peak Memory Allocation (peak_memory_mb)

  • Direction: Lower is better.
  • Biological Meaning: Maximum resident memory (RAM in MiB) consumed during simulation execution.

Automated Score Normalization Pipeline

To compare metrics with completely different units (seconds, megabytes, p-values, correlation coefficients) fairly on the same scale, scSimEval executes an automated two-step normalization:

Raw Metric Value
      │
      ▼
Step 1: Direction Inversion (for lower-is-better metrics)
      Inverted Value = Maximum - Value
      │
      ▼
Step 2: Min-Max Normalization
      Standardized Score = (Value - Minimum) / (Maximum - Minimum)
      │
      ▼
Standardized Score in [0.00, 1.00]
(Higher is always better; 1.00 = Top Performer)

In visual representations (such as plot_benchmark_bubble_matrix()), standardized scores are mapped directly to bubble sizes, and top performers (\(\ge 0.96\)) receive bold highlighted square glyphs.

sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04.1 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] scSimEval_0.99.7 rmarkdown_2.32  
#> 
#> loaded via a namespace (and not attached):
#>   [1] RColorBrewer_1.1-3          sys_3.4.3                  
#>   [3] jsonlite_2.0.0              magrittr_2.0.5             
#>   [5] ggbeeswarm_0.7.3            SuppDists_1.1-9.9          
#>   [7] modeltools_0.2-25           farver_2.1.2               
#>   [9] nloptr_2.2.1                provenance_4.4             
#>  [11] vctrs_0.7.3                 memoise_2.0.1              
#>  [13] minqa_1.2.8                 htmltools_0.5.9            
#>  [15] S4Arrays_1.13.2             curl_8.0.0                 
#>  [17] BiocNeighbors_2.7.3         SparseArray_1.13.4         
#>  [19] overlapping_2.5             sass_0.4.10                
#>  [21] bslib_0.12.0                testthat_3.3.2             
#>  [23] httr2_1.3.0                 cachem_1.1.0               
#>  [25] buildtools_1.0.0            igraph_2.3.4               
#>  [27] lifecycle_1.0.5             pkgconfig_2.0.3            
#>  [29] rsvd_1.0.5                  Matrix_1.7-6               
#>  [31] R6_2.6.1                    fastmap_1.2.0              
#>  [33] rbibutils_2.4.1             MatrixGenerics_1.25.0      
#>  [35] clue_0.3-68                 digest_0.6.39              
#>  [37] S4Vectors_0.51.10           scater_1.41.2              
#>  [39] irlba_2.4.1                 IsoplotR_7.0               
#>  [41] GenomicRanges_1.65.4        RSQLite_3.53.3             
#>  [43] beachmat_2.29.3             philentropy_0.10.0         
#>  [45] filelock_1.0.3              labeling_0.4.3             
#>  [47] abind_1.4-8                 compiler_4.6.1             
#>  [49] proxy_0.4-29                bit64_4.8.6                
#>  [51] withr_3.0.3                 CellMixS_1.29.0            
#>  [53] S7_0.2.2                    BiocParallel_1.47.0        
#>  [55] viridis_0.6.5               DBI_1.3.0                  
#>  [57] MASS_7.3-66                 DelayedArray_0.39.8        
#>  [59] bluster_1.23.1              tools_4.6.1                
#>  [61] vipor_0.4.7                 otel_0.2.0                 
#>  [63] beeswarm_0.4.0              prabclus_2.3-5             
#>  [65] nnet_7.3-21                 glue_1.8.1                 
#>  [67] nlme_3.1-171                grid_4.6.1                 
#>  [69] waddR_1.27.0                cluster_2.1.8.3            
#>  [71] ade4_1.7-24                 generics_0.1.4             
#>  [73] gtable_0.3.6                class_7.3-24               
#>  [75] tidyr_1.3.2                 ScaledMatrix_1.21.0        
#>  [77] BiocSingular_1.29.2         XVector_0.53.0             
#>  [79] flexmix_2.3-21              BiocGenerics_0.59.12       
#>  [81] ggrepel_0.9.8               RANN_2.6.3                 
#>  [83] pillar_1.11.1               limma_3.99.1               
#>  [85] clValid_0.7                 robustbase_0.99-7          
#>  [87] splines_4.6.1               dplyr_1.2.1                
#>  [89] BiocFileCache_3.3.0         lattice_0.23-1             
#>  [91] bit_4.6.0                   tidyselect_1.2.1           
#>  [93] SingleCellExperiment_1.35.2 locfit_1.5-9.12            
#>  [95] scuttle_1.23.2              maketools_1.3.2            
#>  [97] clusterSim_0.51-6           knitr_1.52                 
#>  [99] gridExtra_2.3.1             reformulas_0.4.4           
#> [101] IRanges_2.47.5              Seqinfo_1.3.2              
#> [103] edgeR_4.99.6                SummarizedExperiment_1.43.0
#> [105] stats4_4.6.1                xfun_0.61                  
#> [107] Biobase_2.73.2              statmod_1.5.2              
#> [109] diptest_0.77-2              randtests_1.0.2            
#> [111] brio_1.1.5                  arm_1.15-3                 
#> [113] matrixStats_1.5.0           DEoptimR_1.2-2             
#> [115] yaml_2.3.12                 boot_1.3-32                
#> [117] eva_0.2.7                   evaluate_1.0.5             
#> [119] codetools_0.2-20            MLmetrics_1.1.3            
#> [121] kSamples_1.2-12             kernlab_0.9-33             
#> [123] tibble_3.3.1                cli_3.6.6                  
#> [125] RcppParallel_6.2.1          Rdpack_2.6.6               
#> [127] jquerylib_0.1.4             Rcpp_1.1.2                 
#> [129] dbplyr_2.6.0                coda_0.19-4.1              
#> [131] parallel_4.6.1              ggplot2_4.0.3              
#> [133] blob_1.3.0                  mclust_6.1.3               
#> [135] lme4_2.0-6                  viridisLite_0.4.3          
#> [137] ggridges_0.5.7              scales_1.4.0               
#> [139] e1071_1.7-17                purrr_1.2.2                
#> [141] fpc_2.2-15                  rlang_1.3.0                
#> [143] cowplot_1.2.0