Hi Tamas,
I'd say that there should be a layout in which you can store your data in HDF5 that is competitive with ROOT; it is just that finding it may require some more experimentation. Things like the compressor used, the chunksizes and the index level that you are using might be critical for achieving more performance. Could you send us some links to your codebases and perhaps elaborate more on the performance figures that you are getting on each of your approaches? Best, Francesc Alted ________________________________ From: Hdf-forum <[email protected]> on behalf of Tamas Gal <[email protected]> Sent: Friday, March 31, 2017 10:20:37 AM To: HDF Users Discussion List Subject: Re: [Hdf-forum] Optimising HDF5 data structure Dear Rafal, thanks for you reply. On 31. Mar 2017, at 09:52, Rafal Lichwala <[email protected]<mailto:[email protected]>> wrote: I see two solutions for your purposes. First - try to switch from Python to C++ - it's much faster. I am of course aware of the fact that Python is in general much slower than a statically typed compiled language, however pytables (http://www.pytables.org) and h5py (http://www.h5py.org) are thin wrappers and are tightly bound to the numpy library (http://www.numpy.org) which is totally competitive. I also use Julia to access HDF5 content and I did not notice a better performance. So I am not sure if this is a real bottleneck in our case... Second - I know this is HDF5 forum, but for such a huge but simple set of data, I would suggest to use some SQL engine as a backend. We definitely need a file based approach, so a centralised database engine is not an option. I also tried sqlite, however the performance is very poor compared to our HDF5 solution. So maybe our data structure is not that bad overall, yet our expectations might be a bit too high? Cheers, Tamas
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
