On 09/23/2010 07:44 AM, Thomas Down wrote:
Just getting in touch to let you know that we're very interested in getting
big piles of sequencing data onto genome browsers, and are certainly
interested in alternative formats. My current thoughts are that -- at least
for what we're doing -- a slightly more concise XML schema, plus richer
control of server-side summarization/binning (think maxbins++) might be the
sweet spot, but very interested to see other options as well.
Hello,
I've only just joined the list, so I think I'm entering this
conversation mid-way, and I apologize if I'm speaking out of turn. I do
have a few opinions about putting genomic data into JSON, though, and it
looks like this might be the time to express them, if you're discussing
DAS alternative content formats.
As you may know, JBrowse client-server communication is (currently) not
DAS. And that's on purpose, for multiple reasons that may be relevant
to this discussion. They are:
1. XML verbosity
2. cacheability
3. need for server-side code
JBrowse does something different from DAS in each of those areas. In
order of decreasing ease-of-adoption, they are:
1. XML verbosity:
It's true that if you gzip then this doesn't cost you too much in terms
of size-on-the-wire, but you still have to deal with it in memory on the
client and server.
Most of the genomics data formats I've seen have attributes that are
either *named* or *positional*. Positional data formats, like GFF2,
specify in advance the attributes (columns) that each record (line) can
have. This saves space, but makes the format awkward to extend (e.g.,
what happened when different people extended GFF in different ways).
With XML, by contrast, each attribute is named. That makes the tradeoff
the other way around; it takes up more space, but it's easier to extend,
because new attributes get new names and don't step on each other.
And then there are hybrids like GFF3, where some attributes are
specified positionally, and the rest are named.
But there's another middle ground--you can have positional attributes,
but have the *positions* be named. This is what JBrowse does; each
feature is represented in JSON as an array. Each position in the array
is named, in a separate array of strings. In other words, there are a
bunch of genomic features like this:
[
[10000,20000,-1,"foo"],
[50000,80000,1,"bar]
]
and a separate array (not sure what to call it, a "schema" array,
maybe?) that names each position, e.g.:
["start", "end", "strand", "name"]
or what have you. And, since it's JSON, attribute values can have their
own nested structure; for example, one of the attributes could be
"subfeatures" which could be an array of subfeatures. In JBrowse, each
track can have its own "schema"; attributes like "phase", "score", and
"subfeatures" can be left out of a track if it doesn't use them.
For genomic applications, where the space used by the schema array can
be amortized over large numbers of features, I think this is a pretty
substantial win. And if you need ad-hoc additional attributes, you can
always pull the GFF3 trick and have a set of named attributes at the end.
2. Cacheability
JBrowse partitions feature data by refseq, track, and genomic region.
And the boundaries between those chunks of data are defined statically.
That makes those chunks of data more cacheable; if you're viewing a
given region, and you move a little to the left or to the right, you
usually don't need to make a new HTTP request. And if you come back to
the same region the next day, your browser will likely have the same
data in its cache.
You could, potentially, provide useful caching-related HTTP headers from
current DAS servers, but my understanding is that implementations
typically don't. And I think that has something to do with dynamic
queries being harder to usefully cache.
Gregg has been working on proxying DAS sources for JBrowse clients; I
asked him if his proxy would provide useful caching headers, and he
said, "well, I could make them up" :) But it would be nice if we could
get them without having to do that.
3. Server-side code
The fact that the boundaries of the data chunks are statically defined
means that you can just pre-generate them and serve them with a plain
static HTTP server. Sometimes I meet people who disagree with this
choice, but if you have have a workload that's dominated by reads then I
think it's a pretty clear win, especially if you have a lot of users.
Plus, the HTTP server will generate appropriate caching-related headers
for free. And it's easier for less-technical users, or users with
limited rights on a server, to set up a JBrowse instance if they don't
need to set up CGI/servlets/whatever.
BAM and BigBed have made a similar choice to be statically-serveable;
you could view the JBrowse approach as doing something like what they're
doing, in a way that's easier to digest for web browser clients. In the
JBrowse case, range queries are done by the client using a lazily-loaded
nested containment list, described here:
http://biowiki.org/view/JBrowse/LazyFeatureLoading
It's possible to store the JSON gzipped on the server and then send it
out as-is; I compress the json files and give them a .jsonz extension,
and then add this to my apache config:
<Files *.jsonz>
ForceType text/javascript
Header set Content-Encoding: gzip
</Files>
which works in all the web browsers that JBrowse supports, including IE
6/7/8.
I hope this contributes to the discussion,
Mitch
_______________________________________________
DAS mailing list
[email protected]
http://lists.open-bio.org/mailman/listinfo/das
