Re: QUESTION: Packet Parser for PCAP Plugin

2019-04-23 Thread Charles Givre
Hi Ted, I thought about that approach as well. My concern was cluttering up the plugin with lots of columns, especially as we add different protocols. However, if that's not a concern, I can have a go at it. I was thinking the same thing about the Kaitai struct. Would it be possible to

Re: QUESTION: Packet Parser for PCAP Plugin

2019-04-23 Thread Ted Dunning
Wow. Kaitai looks fabulous. It would be tempting to define a generic format that could use a kaitai spec to define the format of a file. Regarding the map output, I think we solved the same problem in the PCAP parser itself by simply putting all of the fields at the top level and making them

Re: QUESTION: Packet Parser for PCAP Plugin

2019-04-23 Thread Paul Rogers
Hi Charles, Two comments.  First, Drill "maps" are actually structs (nested tuples): every record must have the same set of columns within the "map." That is, though the Drill type is called a "map", and you might assume that, given that name, it would act like a JSON, Python of Java map, the

Re: QUESTION: Packet Parser for PCAP Plugin

2019-04-23 Thread Charles Givre
Hi Ted The library that gave me the idea is the Kaitai struct. The java library itself is released under the Apache or MIT license. It can parse a number of binary formats including DNS packets, ICMP and many others. It accepts a byte[] as input. I already wrote working code that reads it

Re: QUESTION: Packet Parser for PCAP Plugin

2019-04-23 Thread Ted Dunning
I think this would be very useful, particularly if it is easy to add additional parsing methods. When I started to pcap work, I couldn't find any libraries that combined what we needed in terms of function and license. On Tue, Apr 23, 2019, 9:34 AM Charles Givre wrote: > Hello all, > I saw a

QUESTION: Packet Parser for PCAP Plugin

2019-04-23 Thread Charles Givre
Hello all, I saw a few open source libraries that parse actual packet content and was interested in incorporating this into Drill's PCAP parser. I was thinking initially of writing this as a UDF, however, I think it would be much better to include this directly in Drill. What I was thinking