Here's a StackOverflow thread that got me started on the Python code. These days you could probably get Grok to write it for you 🙂
https://stackoverflow.com/questions/65010451/how-can-i-parse-all-the-packets-in-my-pcap-file-instead-of-one ________________________________ From: Mel Beckman via NANOG <[email protected]> Sent: Thursday, January 15, 2026 10:24 AM To: [email protected] <[email protected]> Cc: Jonas Muecke <[email protected]>; Mel Beckman <[email protected]> Subject: Re: Converting IPFIX flows to columnar format with all IEs - tool recommendations? I have used MapForce Pro edition, a generic commeercial tool for any-to-any data conversion, for similar tasks in the past. Specifically I was converting pcap logs of SMTP sessions to create a database of source and destination email for a custom security project. It worked pretty well, and the tool is reasonably priced. I did end up writing a Python pcap preprocessor but that was only to filter out stuff I didn't want. -mel ________________________________ From: Jonas Muecke via NANOG <[email protected]> Sent: Thursday, January 15, 2026 9:57 AM To: [email protected] <[email protected]> Cc: Jonas Muecke <[email protected]> Subject: Converting IPFIX flows to columnar format with all IEs - tool recommendations? Hi, I'm looking for recommendations on conversion and long-term storage of raw IPFIX flow data. Specifically, I need to convert IPFIX flows stored in pcap files into a columnar format like Parquet to enable easy and detailed historic analysis that isn't possible with aggregated data. Requirements: - Parse IPFIX from pcap files (including templates) - Preserve ALL information elements, including custom IEs with enterprise PENs - Output enterprise number + IE ID + data (detailed interpretation of the data not needed) - Handle IP fragment reassembly or large IP packets nfdump [1] gets close, but it skips custom IEs. Other tools require replaying the pcaps which risks overflowing buffers, so reading directly from pcap files would be preferred. Has anyone had similar requirements and found a solution? I'm open to multi-step conversions (e.g., via JSON). Long-term I'll capture directly to a better format, but need to process existing pcap archives first. Thanks, Jonas [1] https://github.com/phaag/nfdump -- Jonas Muecke Phd Student, TU Dresden _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/U7ZR5BJFNCBWI4EBLRDUVPVEV45GHID6/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/473NHRQNYUDTJ7BXBCWWNBOKCCWWJPAR/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/FBMRTUQ3O3DUUU7SYHHAYENYYKIGRU6T/
