Aaron Quinlan and Ryan Layer's work on GQT is another nice example of compression facilitating big-data genomics. It's been presented<http://quinlanlab.org/publications.html#Lectures> and the code is open-development/open-source<https://github.com/ryanlayer/gqt> but as far as I know it's still unpublished.
-- Joshua J. Waterfall Senior Research Fellow Genetics Branch, CCR, NCI, NIH [email protected] @JoshWaterfall ________________________________ From: Davis, Sean Sent: Wednesday, July 08, 2015 4:18 PM To: List BIOINFO-GENERAL-NCI Subject: Re: interesting article on coming scale of genomic data Thanks, Tony. That article is nice in that it makes explicit comparisons to known big datasets; for many folks (myself included), talk of exabytes is simply hard to fathom out-of-context. There is a very active field of research working to address the data growth issue in genomics. A few recent papers give overviews and implementations of compression. Beyond compression, graph-based genomes offer incredible benefits for data processing as well as compression. http://www.ncbi.nlm.nih.gov/pubmed/24347576 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3932469/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4479802/ The latter link describes a compression strategy for genomics data of ~9500 fold. https://www.genomeweb.com/business-news/seven-bridges-use-195m-genomics-england-grant-commercialize-graph-based-genome-tool I saw a talk about this approach and the authors suggested that their testing of graph-based genome encoding could lead to 100k genomes being stored in 16GB (unpublished, as far as I know). Of course, the devil is in the details, but the data sizes doomsday articles quote do not often directly address the incredible opportunities for compression that exist for genomics data. The reason for slow adoption of these compression strategies is that the compressed data need to be ingested by other tools to be useful and tools that do so don't exist yet. If a compression standard emerges or we get a graph-based genomic standard, we may see storage issues become less concerning and will, instead, inherit a new set of analytical and data processing opportunities and challenges. Sean On Wed, Jul 8, 2015 at 2:49 PM, Kerlavage, Anthony (NIH/NCI) [E] <[email protected]<mailto:[email protected]>> wrote: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002195
