Why not stream frames directly into the datomic db as they fall out of gloss?
This should be faster at query time anyhow due to indexes,
and let's datomic handle the memory management.
cheers Jan
> On 31 Jan 2015, at 11:39, Milton Silva <milton...@gmail.com> wrote:
>
> While using wireshark to analyse libpcap files (>= 200 MB) I routinely think
> that it would be great to preform relational queries but, wireshark only
> supports search.
>
> I thought I would decode the entire file, hold it in memory as clojure data
> structures and use datomic's datalog.
>
> Besides relational queries, a requirement is for the file to be decoded
> (libpcap, ethernet-II, IP, TCP, diameter) in less then a minute(for a 200MB)
> and the "typical" queries should also be less than a minute.
>
> I thought the frames could be represented like this:
>
> {:frame-id 1
> :timestamp java's instant-object
> :src-mac string
> :dest-mac string
> :src-ip
> :dest-ip ...
> ...}
>
> {:frame-ids [1 3]
> :diameter-session-id ...}
>
> So, I started by using gloss to decode a 200MB file. Gloss is fantastic to
> specify frames but, it is not meeting the time requirements. It appear the
> problem has to do with the creation of a lot of objects. Even with 3G of ram
> for the heap, it still crawls to a halt.
>
> I could try to perform some experiments to determine approximate answers but,
> I think it is better to talk with people with more experience in order to
> avoid common pitfalls..
> My questions are:
>
> Will the JVM (with 3G) support a million hashmaps like the above?
> Is Buffy able to do something like what I want?
> Will datomic be able to handle this use case?
> What would you suggest to solve this(e.g. don't use clojure data structures..
> but then datomic's datalog is not available to query?)?
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
