---
title: "Extending GenomicDataStream"
subtitle: "File and data types"
author: "Developed by [Gabriel Hoffman](http://gabrielhoffman.github.io/)"
date: "Run on `r format(Sys.time())`"
output:
  rmarkdown::html_document:
    highlight: pygments
    toc: true
    toc_depth: 3
    fig_width: 5
#bibliography: library.bib
vignette: >
  %\VignetteIndexEntry{Extending GenomicDataStream}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
  %\usepackage[utf8]{inputenc}
---

<!---


--->

```{r setup, echo=FALSE, results="hide"}
knitr::opts_chunk$set(
  tidy = FALSE,
  cache = TRUE,
  dev = "png",
  package.startup.message = FALSE,
  message = FALSE,
  error = FALSE,
  warning = FALSE
)
options(width = 100)
```

<style>
body {
text-align: justify}
</style>

# Additional file types
Consider extending the package to support to support a new file of type `X`.  First, create a new class `Xstream` that inherits from `GenomicDataStream` defined [here](../doxygen/html/classgds_1_1_genomic_data_stream.html).  `Xstream` must define a constructor taking `Param` defined [here](../doxygen/html/structgds_1_1_param.html), and implement interfaces for `getNextChunk()` for each data type.

A [`DataChunk`](../doxygen/html/classgds_1_1_data_chunk.html) stores information about each feature and sample in a class inheriting from [`DataInfo`](../doxygen/html/classgds_1_1_data_info.html).  Streams for genotype data store feature information in the [`VariantInfo`](../doxygen/html/classgds_1_1_variant_info.html) class, while `DelayedStream` uses [`MatrixInfo`](../doxygen/html/classgds_1_1_matrix_info.html).  These classes or a custom class can be used to store information for a new file type. 

While nothing else about the underlying data form at is assumed, `GenomicDataStream` designed to chunks of _features_ rather than chunks of _samples_. Features are stored as columns in the matrix returned by R/C++, <ins>independent of the underlying data storage format</ins>.    


# Additional matrix types
First, take a look a currently supported [data types](./GenomicDataStream.html#data-types), and note that each of these types wraps an array of `double` storing data in column-major order.  In fact, the constructors to the dense matrix types for Eigen and Armadillo return objects that point to the original `double` array, without allocating new memory.  So any new data type should have a constructor that takes a `double` array.

The simplest new type to implement would be a `float` version of an existing type.  This could reduce memory usage in downstream analyses and increase speed at the cost of numerical precision.   



# Importing into other projects

`GenomicDataStream` is written so that core functions are in C++17 with no dependence or R or Rcpp.  On top of that, there is a thin wrapper that uses Rcpp to interface between R and the lower-level library.  

The C++ code is divided into two sections:

  - `inst/include`: header-only C++17 code with no R dependencies
  
  - `src/export.cpp`: `Rcpp` layer between header-only library to R interface

`GenomicDataStream` provides flexability in terms of data input types and and matrix libraries.  This can useful in many cases, but the large number of dependencies can require installation of additional libraries and increase compile times.  Some of these dependencies can be avoided by removing support for some capabilities with compiler flags in `Makevars`:

 `-D DISABLE_DELAYED_STREAM`     
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Omit `DelayedStream` class, remove dependence on `Rcpp` and `beachmat`
 
 `-D DISABLE_EIGEN`   
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Omit support for Eigen matrix library, and remove dependence on `RcppEigen` and `Eigen`

 `-D DISABLE_RCPP`  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Omit support for `Rcpp` matrix library, and remove dependence on `Rcpp`

 `-D DISABLE_PLINK`  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Omit support for `PLINK` files (PGEN, BED), and remove dependence on `pgenlibr`




<!-- 
# Session info
<details>
```{r sessionInfo, echo=FALSE}
sessionInfo()
```
<\details> -->


<!-- # References -->