Dear Rafal,
thanks for you reply.
> On 31. Mar 2017, at 09:52, Rafal Lichwala <[email protected]> wrote:
>
> I see two solutions for your purposes.
> First - try to switch from Python to C++ - it's much faster.
I am of course aware of the fact that Python is in general much slower than a
statically typed compiled language, however pytables (http://www.pytables.org
<http://www.pytables.org/>) and h5py (http://www.h5py.org
<http://www.h5py.org/>) are thin wrappers and are tightly bound to the numpy
library (http://www.numpy.org <http://www.numpy.org/>) which is totally
competitive. I also use Julia to access HDF5 content and I did not notice a
better performance. So I am not sure if this is a real bottleneck in our case...
> Second - I know this is HDF5 forum, but for such a huge but simple set of
> data, I would suggest to use some SQL engine as a backend.
We definitely need a file based approach, so a centralised database engine is
not an option. I also tried sqlite, however the performance is very poor
compared to our HDF5 solution.
So maybe our data structure is not that bad overall, yet our expectations might
be a bit too high?
Cheers,
Tamas
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
