For now I'm hand coding a big loop with transients and a bytebuffer. It is
an incredibly ugly imperative mess but, it is fast enough and so far it
fits nicely into ram.
It takes 2s (at this point gloss was taking upwards of 5 minutes) to decode
200MB all the way to tcp and uses 500MB of heap.
If you're interested in efficient binary decoding/encoding with a more
pleasant API than standard NIO ByteBuffers, check out Netty's buffer
package, io.netty.buffer (http://netty.io/5.0/api/index.html).
On Sunday, February 1, 2015 at 7:31:37 AM UTC-5, Milton Silva wrote:
For now I'm hand
Because of the way that Datomic stores it's data (5 tuples: entity,
attribute, value, transaction, and operation) it has some pretty simple
indexes that allow for powerful queries. EAVT allows for rapidly searching
by entity, VAET allows for quick reverse look ups, etc. The only index that
you
Since the data you are handing to the datomic query engine is un-indexed,
portions of the query will effectively be O(n). However if you do as Jan
suggests and put the data into Datomic the data will automatically indexed
several ways. Then when you query against the datomic db, the engine will
One trick I've used to speed up use of gloss is to use a lazy map so I only
actually parse a piece of data if I need to. In my particular application
this was an easy optimization because I was parsing a tagged data format,
and I could do minimal parsing just to determine the tag of the next
While using wireshark to analyse libpcap files (= 200 MB) I routinely
think that it would be great to preform relational queries but, wireshark
only supports search.
I thought I would decode the entire file, hold it in memory as clojure data
structures and use datomic's datalog.
Besides
Why not stream frames directly into the datomic db as they fall out of gloss?
This should be faster at query time anyhow due to indexes,
and let's datomic handle the memory management.
cheers Jan
On 31 Jan 2015, at 11:39, Milton Silva milton...@gmail.com wrote:
While using wireshark to
