While using wireshark to analyse libpcap files (>= 200 MB) I routinely
think that it would be great to preform relational queries but, wireshark
only supports search.
I thought I would decode the entire file, hold it in memory as clojure data
structures and use datomic's datalog.
Besides relational queries, a requirement is for the file to be decoded
(libpcap, ethernet-II, IP, TCP, diameter) in less then a minute(for a
200MB) and the "typical" queries should also be less than a minute.
I thought the frames could be represented like this:
{:frame-id 1
:timestamp java's instant-object
:src-mac string
:dest-mac string
:src-ip
:dest-ip ...
...}
{:frame-ids [1 3]
:diameter-session-id ...}
So, I started by using gloss to decode a 200MB file. Gloss is fantastic to
specify frames but, it is not meeting the time requirements. It appear the
problem has to do with the creation of a lot of objects. Even with 3G of
ram for the heap, it still crawls to a halt.
I could try to perform some experiments to determine approximate answers
but, I think it is better to talk with people with more experience in order
to avoid common pitfalls..
My questions are:
Will the JVM (with 3G) support a million hashmaps like the above?
Is Buffy able to do something like what I want?
Will datomic be able to handle this use case?
What would you suggest to solve this(e.g. don't use clojure data
structures.. but then datomic's datalog is not available to query?)?
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
