One trick I've used to speed up use of gloss is to use a lazy map so I only
actually parse a piece of data if I need to. In my particular application
this was an easy optimization because I was parsing a tagged data format,
and I could do minimal parsing just to determine the tag of the next chunk
of data, and then build a lazy map entry that used a gloss codec specific
to that tag. I was also lucky that in most cases only a few tags are of
interest to anyone, so I was able to avoid lots of parsing overhead.
Without this optimization gloss might have been too slow for my purposes,
which is unfortunate.
On Sat, Jan 31, 2015 at 6:52 AM, Timothy Baldridge <tbaldri...@gmail.com>
wrote:
> Since the data you are handing to the datomic query engine is un-indexed,
> portions of the query will effectively be O(n). However if you do as Jan
> suggests and put the data into Datomic the data will automatically indexed
> several ways. Then when you query against the datomic db, the engine will
> pick up on these indexes and your queries could be much faster. In
> addition, Datomic has some rather advanced caching logic that should help
> with data usage if you are writing the data to a transactor (i.e. using the
> in-memory storage won't help much here, use free or dev storage).
>
> Timothy
>
>
>
> On Sat, Jan 31, 2015 at 5:53 AM, Jan-Paul Bultmann <
> janpaulbultm...@googlemail.com> wrote:
>
>> Why not stream frames directly into the datomic db as they fall out of
>> gloss?
>> This should be faster at query time anyhow due to indexes,
>> and let's datomic handle the memory management.
>>
>> cheers Jan
>>
>> On 31 Jan 2015, at 11:39, Milton Silva <milton...@gmail.com> wrote:
>>
>> While using wireshark to analyse libpcap files (>= 200 MB) I routinely
>> think that it would be great to preform relational queries but, wireshark
>> only supports search.
>>
>> I thought I would decode the entire file, hold it in memory as clojure
>> data structures and use datomic's datalog.
>>
>> Besides relational queries, a requirement is for the file to be decoded
>> (libpcap, ethernet-II, IP, TCP, diameter) in less then a minute(for a
>> 200MB) and the "typical" queries should also be less than a minute.
>>
>> I thought the frames could be represented like this:
>>
>> {:frame-id 1
>> :timestamp java's instant-object
>> :src-mac string
>> :dest-mac string
>> :src-ip
>> :dest-ip ...
>> ...}
>>
>> {:frame-ids [1 3]
>> :diameter-session-id ...}
>>
>> So, I started by using gloss to decode a 200MB file. Gloss is
>> fantastic to specify frames but, it is not meeting the time requirements.
>> It appear the problem has to do with the creation of a lot of objects. Even
>> with 3G of ram for the heap, it still crawls to a halt.
>>
>> I could try to perform some experiments to determine approximate answers
>> but, I think it is better to talk with people with more experience in order
>> to avoid common pitfalls..
>> My questions are:
>>
>> Will the JVM (with 3G) support a million hashmaps like the above?
>> Is Buffy able to do something like what I want?
>> Will datomic be able to handle this use case?
>> What would you suggest to solve this(e.g. don't use clojure data
>> structures.. but then datomic's datalog is not available to query?)?
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> “One of the main causes of the fall of the Roman Empire was that–lacking
> zero–they had no way to indicate successful termination of their C
> programs.”
> (Robert Firth)
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
