Hello Benjamin,
Not knowing to much about the ASTERIX format, other than what you said and
what is in the links, I would say that this is a good fit for HDF5 and
PyTables. PyTables will certainly help you read in the data and manipulate
it.
However, before you abandon hachoir completely, I will say it is a lot
easier to write hdf5 files in PyTables than to use the HDF5 C API. If
hachoir is too slow, have you tried profiling the code to see what is
taking up the most time? Maybe you could just rewrite these parts in C?
Have you looked into Cythonizing it? Also, you don't seem to be using
numpy to read in the data... (there are some tricks given ASTERIX here, but
not insurmountable).
I ask the above, just so you don't have to completely rewrite everything.
You are correct though that pure python is probably not sufficient. Feel
free to ask more questions here.
Be Well
Anthony
On Wed, Jul 11, 2012 at 6:52 AM, <benjamin.bertr...@lfv.se> wrote:
> Hi,
>
> I'm working with Air Traffic Management and would like to perform checks /
> compute statistics on ASTERIX data.
> ASTERIX is an ATM Surveillance Data Binary Messaging Format (
> http://www.eurocontrol.int/asterix/public/standard_page/overview.html)
>
> The data consist of a concatenation of consecutive data blocks.
> Each data block consists of data category + length + records.
> Each record is of variable length and consists of several data items (that
> are well defined for each category).
> Some data items might be present or not depending on a field specification
> (bitfield).
>
> I started to write a parser using hachoir (
> https://bitbucket.org/haypo/hachoir/overview) a pure python library.
> But the parsing was really too slow and taking a lot of memory.
> That's not really useable.
>
> >From what I read, PyTables could really help to manipulate and analyze
> the data.
> So I've been thinking about writing a tool (probably in C) to convert my
> ASTERIX format to HDF5.
>
> Before I start, I'd like confirmation that this seems like a suitable
> application for PyTables.
> Is there another approach than writing a conversion tool to HDF5?
>
> Thanks in advance
>
> Benjamin
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
