---
title: "The HTAN data model, identifiers and lineage"
author: "Sean Davis"
date: today
format:
  html:
    toc: true
    toc-depth: 3
    fig-width: 7
    fig-height: 4.5
knitr:
  opts_chunk:
    comment: "#>"
    collapse: true
    message: false
    fig.retina: 1
vignette: >
  %\VignetteIndexEntry{The HTAN data model, identifiers and lineage}
  %\VignetteEngine{quarto::html}
  %\VignetteEncoding{UTF-8}
---

```{r}
#| label: setup
#| include: false
library(HTAN)
library(dplyr)
fmt <- function(x) format(x, big.mark = ",", scientific = FALSE)
pct <- function(x, digits = 1) paste0(format(round(100 * x, digits),
    nsmall = digits), "%")
```

Package: [*HTAN*](https://bioconductor.org/packages/HTAN)
`r packageVersion("HTAN")`
([source](https://github.com/seandavi/HTAN)).

HTAN's metadata answer three questions about every data file: what assay
and processing step produced it, which biospecimen it was measured on, and
which participant that biospecimen came from. This vignette shows how the
Phase 1 data model records the answers, how to follow them with *HTAN*, and
where the records are messier than the model suggests. It assumes you have
read the [introduction](HTAN.html).

All output below was computed from the Release 7 snapshot:

```{r}
#| label: con
library(HTAN)
library(dplyr)
con <- htan_con("r7")
```

# The data model in brief

HTAN's Phase 1 data model is a set of metadata templates, one per kind of
record. Each template is called a **component**: `Demographics`,
`Diagnosis` and `Therapy` for clinical data, `Biospecimen` for samples, and
one component per assay and level for files, such as `ScRNA-seqLevel3` or
`ImagingLevel2`. Centers filled in a manifest (a spreadsheet) for each
component, and the DCC validated it against the model before release. The
model was maintained as a CSV in
[ncihtan/data-models](https://github.com/ncihtan/data-models), now
archived, and documented in the HTAN manual
([metadata standards](https://docs.humantumoratlas.org/data_model/metadata_standards/),
[file standards](https://docs.humantumoratlas.org/data_model/file_standards/),
[relationships](https://docs.humantumoratlas.org/data_model/relationships/)).

ISB-CGC loads each component into its own BigQuery table, and the
snapshot keeps those tables as they are. The model itself is in the
snapshot too, as the `schema` table: one row per attribute, with its
description, valid values and the component it belongs to. Here is the
attribute that links a file to the files it came from:

```{r}
#| label: schema
schema <- htan_tbl("schema", con) |> collect()
nrow(schema)
schema |>
    filter(Attribute == "HTAN Parent Data File ID") |>
    select(Attribute, Parent, Description) |>
    as.list()
```

## Clinical data

Phase 1 clinical data are organised in three tiers
([metadata standards](https://docs.humantumoratlas.org/data_model/metadata_standards/)).
Tier 1 follows the NCI Genomic Data Commons clinical model, in seven
components: demographics, diagnosis, exposure, family history, follow-up,
molecular test and therapy. Tier 2 adds disease-agnostic HTAN extensions,
and tier 3 adds disease-specific ones (breast, colorectal, lung, melanoma,
brain and others).

Each clinical record carries the `HTAN_Participant_ID` it describes. To
keep records de-identified, dates are never calendar dates. They are
recorded as a number of days from the participant's index date, which the
center chooses and records.

## The main tables

Row counts are for Release 7:

```{r}
#| label: table-list
#| echo: false
desc <- c(
    clinical_tier1_demographics =
        "Sex, race, ethnicity, vital status; one row per participant",
    clinical_tier1_diagnosis =
        "Primary diagnosis, site, stage, grade; one or more rows per participant",
    clinical_tier1_therapy = "Treatments, with days from index",
    clinical_tier1_followup = "Follow-up visits, progression, comorbidities",
    clinical_tier1_exposure = "Smoking, alcohol and other exposures",
    clinical_tier1_familyhistory = "Family history of cancer",
    clinical_tier1_moleculartest =
        "Clinical molecular tests (e.g. receptor status, mutations)",
    clinical_tier2 = "HTAN disease-agnostic clinical extensions",
    biospecimen = "Every biospecimen and its parent, type, site, preservation",
    id_provenance =
        "One row per file-parent-biospecimen link: the lineage table",
    gc_drs_map = "HTAN file ID to CRDC General Commons DRS URI",
    imaging_channel_metadata =
        "Channel (antibody/marker) lists for imaging files",
    tissueOrOrganOfOrigin_to_wholeOrgan_mapping =
        "Fine-grained site of origin to whole organ",
    phase1_longitudinal_data =
        "Per-participant timeline of biopsies, treatment and vital status",
    schema = "The Phase 1 data model: attributes, descriptions, valid values"
)
htan_tables(con) |>
    filter(name %in% names(desc)) |>
    mutate(description = desc[name]) |>
    select(table = name, rows, description) |>
    arrange(match(table, names(desc))) |>
    knitr::kable(format.args = list(big.mark = ","))
```

The other tables are assay metadata, one per component, named after the
assay and level (`scRNAseq_level4_metadata`, `imaging_level2_metadata`,
`10xvisium_spatialtranscriptomics_scRNAseq_level3_metadata`, ...). The
tier 3 tables are `clinical_tier3_<disease>`, and the Trans-Network
Project SRRS has its own `srrs_biospecimen`, `srrs_clinical_tier2` and
`srrs_imaging_level2_metadata`. Use `htan_tables()` for the full list.

## File components

Every assay table shares a core set of columns (`HTAN_Data_File_ID`,
`Filename`, `File_Format`, `File_Size`, `md5`, `Component`, the Synapse
`entityId`, `HTAN_Center` and `Data_Release`), plus assay-specific fields:
workflow and reference genome for sequencing, channels and microscope for
imaging, capture area and slide for Visium. `htan_files()` stacks the core
columns of all of them. Here are the file components in Release 7, each
with the model's own description from the `schema` table:

```{r}
#| label: components
#| echo: false
key <- function(x) tolower(gsub("[^A-Za-z0-9]", "", x))
comp <- htan_files(con = con) |>
    count(Component, assay, level) |>
    collect()
schema |>
    mutate(key = key(Attribute)) |>
    select(key, Description) |>
    distinct(key, .keep_all = TRUE) |>
    right_join(mutate(comp, key = key(Component)), by = "key") |>
    arrange(assay, level) |>
    select(Component, level, files = n, Description) |>
    knitr::kable(format.args = list(big.mark = ","))
```

# Identifiers

## The grammar

Every participant, biospecimen and file has an HTAN identifier, and all of
them start with the center's prefix (`HTA1` to `HTA16` in Phase 1; see the
[introduction](HTAN.html#phase-1) for the list). The HTAN manual gives the
Phase 1 grammar as
([identifiers](https://docs.humantumoratlas.org/data_model/identifiers/)):

```
<participant_id>        ::= <htan_center_id>_<integer>
<derivative_entity_id>  ::= <participant_id>_<integer>
```

So `HTA11_7862` is participant 7862 of center HTA11 (Vanderbilt), and
anything derived from that participant, whether a biospecimen or a file,
is `HTA11_7862_<integer>`. Phase 1 identifiers do not say whether an ID is a
biospecimen or a file; the table it appears in does. (Phase 2 adds `B` and
`D` markers, as in `HTA209_1_B1` and `HTA209_1_D1`.) A participant number of
`0000` marks data pooled from several participants.

A regular expression splits an ID into its parts:

```{r}
#| label: parse-ids
parse_htan_id <- function(id) {
    m <- regmatches(id, regexec("^(HTA[0-9]+)_([0-9]+)(?:_(.+))?$", id))
    tibble::tibble(
        id = id,
        center = vapply(m, `[`, "", 2L),
        participant = vapply(m, function(x) paste(x[2:3], collapse = "_"),
            ""),
        rest = vapply(m, `[`, "", 4L)
    )
}
parse_htan_id(c("HTA11_7862", "HTA11_7862_2000001011",
    "HTA11_7862_200000101134111", "HTA5_0000_1003255"))
```

The prefix names the center that submitted the data. You can check that
against the `HTAN_Center` column:

```{r}
#| label: prefix-center
htan_files(con = con) |>
    mutate(prefix = regexp_extract(HTAN_Data_File_ID, "^HTA[0-9]+")) |>
    count(prefix, HTAN_Center) |>
    arrange(as.integer(substr(prefix, 4L, 6L))) |>
    collect()
```

Two quirks show up. Files with the PCAPP prefix `HTA2` were submitted
under `HTAN BU`, and a few `HTA5` files under `HTAN SRRS`. And there are
no `HTA16` (CASI) files in Release 7 at all. To group by center, use
`HTAN_Center`; to group by the atlas a participant was enrolled in, use
the prefix.

`regexp_extract()` is a DuckDB function. dbplyr passes functions it does
not know straight through to SQL, so you can use DuckDB's string and regex
functions inside dplyr verbs.

## Do not parse the participant out of a file ID

It is tempting to read a file's participant off its ID. That is wrong often
enough to matter:

```{r}
#| label: prefix-vs-participant
f <- htan_files(con = con) |>
    select(HTAN_Data_File_ID, HTAN_Participant_ID, n_participants) |>
    collect() |>
    mutate(id_participant = sub("^(HTA[0-9]+_[0-9]+).*$", "\\1",
        HTAN_Data_File_ID))
single <- filter(f, n_participants == 1)
mismatch <- filter(single, id_participant != HTAN_Participant_ID)
nrow(mismatch)
mismatch |>
    count(prefix = sub("_.*", "", HTAN_Data_File_ID), sort = TRUE)
head(mismatch, 3)
```

Of the `r fmt(nrow(single))` files that come from one participant,
`r fmt(nrow(mismatch))` (`r pct(nrow(mismatch) / nrow(single))`) carry
another participant's number in their own ID, most of them from
`r names(sort(table(sub("_.*", "", mismatch$HTAN_Data_File_ID)), decreasing = TRUE))[1]`
(CHOP).
Across all files,
`r fmt(sum(grepl("^HTA[0-9]+_0000_", f$HTAN_Data_File_ID)))` have the
pooled participant number `0000`, and
`r fmt(sum(f$n_participants > 1))` are attributed to more than one
participant. A file's participant is whatever `id_provenance` says, and
that is where `htan_files()` gets it.

# The hierarchy: participant, biospecimen, file

## Biospecimens

The `biospecimen` table holds
`r fmt(pull(count(htan_tbl("biospecimen", con))))` rows. Each has an
`HTAN_Parent_ID`. For an originating biospecimen (the biopsy or resection
itself) that is the participant; for a derived biospecimen (a block, a
section, a cell suspension, an extracted analyte) it is the biospecimen it
was cut or extracted from. Follow those links and you have the
biospecimen's ancestry. Counting what each parent ID points to shows that
not every one is a participant or a biospecimen:

```{r}
#| label: biospecimens
bs <- htan_tbl("biospecimen", con) |>
    select(HTAN_Biospecimen_ID, HTAN_Parent_ID, Biospecimen_Type,
        Acquisition_Method_Type, Preservation_Method, Tumor_Tissue_Type,
        Site_of_Resection_or_Biopsy) |>
    collect()
participants <- htan_tbl("clinical_tier1_demographics", con) |>
    distinct(HTAN_Participant_ID) |>
    pull()

bs |>
    mutate(parent_is = case_when(
        HTAN_Parent_ID == HTAN_Biospecimen_ID ~ "itself",
        HTAN_Parent_ID %in% participants ~ "a participant",
        HTAN_Parent_ID %in% bs$HTAN_Biospecimen_ID ~ "a biospecimen",
        .default = "an ID in neither table"
    )) |>
    count(parent_is)
```

A small function walks a biospecimen up to its participant. It stops at a
participant, an unknown ID, or a cycle:

```{r}
#| label: walk-biospecimen
biospecimen_path <- function(id, bs) {
    parent <- setNames(bs$HTAN_Parent_ID, bs$HTAN_Biospecimen_ID)
    path <- id
    while (id %in% names(parent) && !parent[[id]] %in% path) {
        id <- parent[[id]]
        path <- c(path, id)
    }
    path
}
biospecimen_path("HTA5_1430_1022", bs)
```

Participant HTA5_1430, from the Dana-Farber melanoma atlas, had a lymph
node metastasis resected (every biospecimen in the chain has
`Tumor_Tissue_Type` "Metastatic"). The fresh resection (`_1000`) was
divided, snap frozen and divided again before an assay was run on `_1022`.
Here is the path with each biospecimen's annotation:

```{r}
#| label: walk-biospecimen-table
bs |>
    filter(HTAN_Biospecimen_ID %in% biospecimen_path("HTA5_1430_1022", bs)) |>
    arrange(HTAN_Biospecimen_ID) |>
    select(HTAN_Biospecimen_ID, HTAN_Parent_ID, Preservation_Method,
        Site = Site_of_Resection_or_Biopsy)
```

Annotations such as tissue type and site are often repeated on every
biospecimen in a chain, but not always; when they differ, the one closest
to the assayed biospecimen describes what was measured.

## Files

Files connect to biospecimens and to each other through two columns of the
assay tables. Level 1 files, and other files measured directly on a
sample, have an `HTAN_Parent_Biospecimen_ID`: the biospecimen that went
into the instrument. Level 2 to 4 files have an
`HTAN_Parent_Data_File_ID`, naming the file or files they were computed
from.

So a Level 4 single-cell object points at its Level 3 count matrices,
which point at Level 2 alignments, which point at Level 1 FASTQs, which
point at a biospecimen. Reconstructing that chain from the assay tables
means joining across many tables and splitting comma-separated lists. The
DCC did it once, for every file, and published the result as
`id_provenance`.

# `id_provenance`, the lineage table

`id_provenance` has one row per combination of a file, one of its parent
files, and one of the biospecimens it was derived from, with the
participant attached. Its columns, with the descriptions cut to 70
characters:

```{r}
#| label: prov-columns
htan_columns("id_provenance", con) |>
    select(column, description) |>
    mutate(description = substr(description, 1, 70))
```

The biospecimen terms come from the HTAN manual
([relationships](https://docs.humantumoratlas.org/data_model/relationships/)).
The *originating biospecimen* is the biopsy or resection taken from the
participant, the *assayed biospecimen* is the one put into the instrument,
and the *biospecimen path* is the comma-separated chain from the first to
the second. In Release 7 the table holds:

```{r}
#| label: prov-size
prov <- htan_tbl("id_provenance", con)
prov |>
    summarise(
        rows = n(),
        files = n_distinct(HTAN_Data_File_ID),
        participants = n_distinct(HTAN_Participant_ID),
        biospecimens = n_distinct(HTAN_Assayed_Biospecimen_ID)
    ) |>
    collect()
```

A file has more than one row whenever it has several parents or several
biospecimens. Here are the counts per file, in bins:

```{r}
#| label: prov-multiplicity
per_file <- prov |>
    group_by(HTAN_Data_File_ID) |>
    summarise(
        parents = n_distinct(HTAN_Parent_Data_File_ID),
        biospecimens = n_distinct(HTAN_Assayed_Biospecimen_ID),
        participants = n_distinct(HTAN_Participant_ID)
    ) |>
    collect()
bucket <- function(x) {
    cut(x, c(-Inf, 0, 1, 2, 10, 100, Inf),
        labels = c("0", "1", "2", "3-10", "11-100", ">100"))
}
tibble::tibble(
    count = levels(bucket(1)),
    parents = as.vector(table(bucket(per_file$parents))),
    biospecimens = as.vector(table(bucket(per_file$biospecimens))),
    participants = as.vector(table(bucket(per_file$participants)))
)
```

Most files have one parent, one biospecimen and one participant. A file
with no parent is a Level 1 file, or a file whose parent was not
recorded. The long tail is merged objects: a Level 4 single-cell atlas
built from dozens of samples has dozens of parents, biospecimens and
participants.

`htan_lineage()` walks the parent links recursively in DuckDB and returns
one row per (query file, related file), with `depth` counting generations
(0 for the query, negative for ancestors, positive for descendants).

# Worked lineage walks

## A Visium spatial transcriptomics object, back to the patient

Start with a Level 4 Visium object from the Vanderbilt colorectal atlas:

```{r}
#| label: visium-file
visium_id <- "HTA11_7862_200000101134111"
htan_files(HTAN_Data_File_ID == visium_id, con = con) |>
    select(Component, Filename, File_Size, HTAN_Participant_ID,
        HTAN_Assayed_Biospecimen_ID, access) |>
    collect() |>
    glimpse()
```

Its ancestors:

```{r}
#| label: visium-lineage
htan_lineage(visium_id, con = con) |>
    arrange(depth, related_file_id) |>
    collect() |>
    transmute(depth, related_file_id, file = basename(Filename))
```

The chain is FASTQ (Level 1) to aligned BAM (Level 2) to the four pieces of
the Space Ranger output (Level 3: barcodes, features, the count matrix and
spot positions) to the processed object (Level 4). Every file in it was
measured on the same biospecimen. That biospecimen leads to the
participant:

```{r}
#| label: visium-biospecimen
biospecimen_path("HTA11_7862_2000001011", bs)
bs |>
    filter(HTAN_Biospecimen_ID == "HTA11_7862_2000001011") |>
    select(-HTAN_Parent_ID) |>
    glimpse()
```

The biospecimen is a fresh biopsy of a premalignant lesion of the sigmoid
colon. Its `Histologic_Morphology_Code` (not shown above) is `M82110`,
the ICD-O-3 code for tubular adenoma, the common precancerous colorectal
polyp. The participant's diagnosis record tells us almost nothing:

```{r}
#| label: visium-diagnosis
htan_tbl("clinical_tier1_diagnosis", con) |>
    filter(HTAN_Participant_ID == "HTA11_7862") |>
    select(Primary_Diagnosis, Tissue_or_Organ_of_Origin, Age_at_Diagnosis) |>
    collect()
```

This is common in the pre-cancer atlases: the biospecimen carries the
histology, and the diagnosis table, built around cancer diagnoses, says
"Not Reported". Look in both.

## Comma-separated parents

The assay table records the Visium object's parents in a single field:

```{r}
#| label: visium-assay-table
htan_tbl("10xvisium_spatialtranscriptomics_scRNAseq_level4_metadata",
    con) |>
    filter(HTAN_Data_File_ID == visium_id) |>
    pull(HTAN_Parent_Data_File_ID)
```

while `id_provenance` gives them one row each:

```{r}
#| label: visium-prov-rows
prov |>
    filter(HTAN_Data_File_ID == visium_id) |>
    select(HTAN_Parent_Data_File_ID, HTAN_Assayed_Biospecimen_ID) |>
    collect()
```

The comma-separated form is common. Here, for four assay tables, is the
number of files whose parent field holds a list:

```{r}
#| label: comma-parents
count_commas <- function(table) {
    htan_tbl(table, con) |>
        summarise(
            files = n(),
            several_parents = sum(as.integer(
                grepl(",", HTAN_Parent_Data_File_ID)), na.rm = TRUE)
        ) |>
        collect() |>
        mutate(table = table, .before = 1)
}
bind_rows(lapply(c("scRNAseq_level3_metadata", "scRNAseq_level4_metadata",
    "imaging_level4_metadata",
    "10xvisium_spatialtranscriptomics_scRNAseq_level4_metadata"),
    count_commas))
```

A filter such as `HTAN_Parent_Data_File_ID == "HTA11_7862_200000101133111"`
on the assay table misses the Visium object. Query `id_provenance`
instead, or split the field first; in DuckDB,
`unnest(string_split(x, ','))` does it.

## Descendants of a raw file

Walking down from a Level 1 file finds everything computed from it. Take
a single-cell RNA-seq FASTQ from Washington University:

```{r}
#| label: descendants
htan_lineage("HTA12_251_1000", direction = "descendants", con = con) |>
    arrange(depth, related_file_id) |>
    select(depth, related_file_id, Component) |>
    collect()
```

The descendants mix two assays. The scRNA-seq alignment is recorded as a
parent of scATAC-seq Level 3 files, and those are parents of both the RNA
and the ATAC Level 4 objects. This is presumably a multiome (joint RNA and
ATAC) experiment processed together, though the metadata do not say so
directly. Either way, lineage is a graph rather than a tree, and the assay
can change along a path.

## A multiplexed imaging cell table

Imaging follows the same pattern. Here is a Level 4 cell-by-marker table
from an Orion whole-slide image (Harvard Medical School):

```{r}
#| label: imaging-lineage
htan_lineage("HTA7_1000_1003", con = con) |>
    arrange(depth) |>
    collect() |>
    transmute(depth, related_file_id, Component, file = basename(Filename))
```

The table was computed from a segmentation mask (Level 3), and the mask
from the stitched, registered image (Level 2). For the channels (markers)
of an imaging file, see `imaging_channel_metadata`.

## A file from several participants

Some single-cell objects pool cells from several participants. This Level 4
AnnData file from the Dana-Farber melanoma atlas is named for one
participant, but `id_provenance` attributes it to four:

```{r}
#| label: pooled
pooled_id <- "HTA5_0000_1003255"
htan_files(HTAN_Data_File_ID == pooled_id, con = con) |>
    select(Filename, n_participants, HTAN_Participant_ID) |>
    collect() |>
    glimpse()
prov |>
    filter(HTAN_Data_File_ID == pooled_id) |>
    distinct(HTAN_Parent_Data_File_ID, HTAN_Participant_ID,
        HTAN_Assayed_Biospecimen_ID) |>
    collect()
```

The attribution comes from its parent, a Level 3 file that was itself
pooled from four participants' samples (the `0000` in its ID). The
metadata cannot tell you which cells in the object came from which
participant. If the object records that at all, it does so in its own
cell annotations. So when a file has more than one participant,
`htan_experiment()` asks for a `participant_col`: the column of `colData`
that names each cell's participant or biospecimen.

# Pitfalls

The data model is consistent; the data entered into it, by more than a
dozen centers over seven releases, are less so. These are the problems
most likely to give silently wrong answers.

**Comma-separated lists.** `HTAN_Parent_Data_File_ID` in the assay tables,
`Biospecimen_Path` in `id_provenance` and `Adjacent_Biospecimen_IDs` in
`biospecimen` hold comma-separated lists. In `htan_files()`,
`HTAN_Participant_ID` and `HTAN_Assayed_Biospecimen_ID` are comma-separated
for multi-participant files. Never compare these fields with `==`. Use
`id_provenance` or the `participants` argument of `htan_files()`, or
split them first.

**Several parents, several participants.** A file can have many parents
and many participants (see the multiplicity table above), so joining
`id_provenance` to anything multiplies rows. Use `distinct()` or
aggregate to one row per file before joining, as `htan_files()` does.

**Participants inferred from IDs.** As shown above,
`r pct(nrow(mismatch) / nrow(single))` of single-participant files carry
another participant's number in their ID. Take participants from
`id_provenance`.

**Biospecimen parents that lead nowhere.** Most biospecimens point to a
participant or another biospecimen. Some point to themselves, a cycle
that a naive recursive walk never leaves, and some to participant IDs
with no demographics record. The `biospecimen_path()` function above
stops at both.

**Duplicate records.** Some participants and biospecimens appear more than
once:

```{r}
#| label: duplicates
dup_count <- function(table, id) {
    htan_tbl(table, con) |>
        summarise(rows = n(), distinct_ids = n_distinct(.data[[id]])) |>
        collect() |>
        mutate(table = table, .before = 1)
}
bind_rows(
    dup_count("clinical_tier1_demographics", "HTAN_Participant_ID"),
    dup_count("clinical_tier1_diagnosis", "HTAN_Participant_ID"),
    dup_count("biospecimen", "HTAN_Biospecimen_ID")
)
```

A participant can legitimately have several diagnoses, but duplicated
demographics and biospecimen rows are most likely entry artefacts. Decide
which row to keep before joining; `htan_experiment()` keeps the first row
per participant, ordered by ID.

**Spelling and missingness.** Categorical values vary in case and form
(`Female` and `female`, `Primary` and `primary`, `Not Recorded` and
`Not recorded`), and missing values appear as `NA`, `"Not Reported"`,
`"unknown"`, `"Not Applicable"` and sometimes `"0"`. Before you tabulate,
normalise case and decide what counts as missing. Here are the values of
`Tumor_Tissue_Type`:

```{r}
#| label: missingness
htan_tbl("biospecimen", con) |>
    count(Tumor_Tissue_Type) |>
    arrange(desc(n)) |>
    collect()
```

**Numbers stored as text.** Clinical fields such as `Age_at_Diagnosis` and
`Days_to_Last_Follow_up` are character columns, because they mix numbers
with codes like `"Not Applicable"`. Convert them with `as.numeric()`, which
turns the codes into `NA`, and treat a value of `0` with suspicion. In the
Vanderbilt diagnosis record above, `Age_at_Diagnosis` is `"0"` while every
other field is "Not Reported"; that looks more like a placeholder than an
age.

```{r}
#| label: text-numbers
htan_tbl("clinical_tier1_diagnosis", con) |>
    head(1) |>
    select(Age_at_Diagnosis, Days_to_Last_Follow_up, Year_of_Diagnosis) |>
    collect() |>
    vapply(class, "")
```

**The same file in several tables.** A few files appear in more than one
assay table (for example, mass spectrometry files in both a Release 7
table and a legacy Release 2 table). `htan_files()` keeps one row per file,
from the newest table.

**Files without provenance.** Some files have no `id_provenance` rows,
and therefore no participant. By component:

```{r}
#| label: no-provenance
htan_files(n_participants == 0, con = con) |>
    count(Component) |>
    arrange(desc(n)) |>
    collect()
```

# Summary

- Identifiers start with a center prefix and, in Phase 1, do not distinguish
  biospecimens from files. The prefix identifies the center reliably; the
  participant number does not reliably identify the participant.
- The hierarchy is participant, then a chain of biospecimens linked by
  `HTAN_Parent_ID`, then a graph of files linked by
  `HTAN_Parent_Biospecimen_ID` (Level 1) and `HTAN_Parent_Data_File_ID`
  (Levels 2 to 4).
- `id_provenance` flattens all of that into one table, one row per file,
  parent and biospecimen; `htan_files()` and `htan_lineage()` are built on
  it.
- Expect lists in single fields, many-to-many links, duplicates and
  inconsistent spelling, and check before you join or count.

The [atlas overview](atlas-overview.html) uses these tables to describe the
whole release, and the [analysis vignette](analysis-workflow.html) follows
one file from query to analysis.

# Session information

```{r}
#| label: session-info
sessionInfo()
```