Hi Carlos,
use HDF5 1.10 . That one provides the feature to write to a file while
it always remains readable by another process and it ensures the file
will never be corrupted. That feature is called SWMR (single write,
multiple read) and was introduce with 1.10.
Also you may consider using the LZ4 filter for compression instead of
the internal deflate filter. LZ4 does not compress that strongly as
deflate, but it's faster by a magnitude, nearly like uncompressed read /
write, so it may be worth it, especially for time-constraint data I/O.
You may also want to optimize the chunked layout of the dataset
according to your data updates since each chunk is compressed on its own.
Cheers,
Werner
On 05.10.2016 02:08, Carlos Penedo Rocha wrote:
*Schlumberger-Private*
Hi,
I have a scenario in which my compressed h5 file needs to be updated
with new data that is coming in every, say, 5 seconds.
Approach #1: keep the file opened and just write data as they come, or
write a buffer at once.
Approach #2: open the file (RDWR), write the data (or a buffer) and
then close the file.
Approach #1 is not desirable for my case because if there’s any
problem (outage, etc), then the h5 file will likely get corrupted. Or
if I want to have a look at the file, I can’t because it’s still
writing (still opened).
Approach #2 is good to address the issue above, *BUT* I noticed that
if I open/write/close the file every 5 seconds, the file compression
gets really bad and the file size goes up big time. Approach 1 doesn’t
suffer from this problem.
So, my question is: is there an “Approach #3” that gives me the best
of the two worlds? Less likely to get me a corrupted h5 file and at
the same time, a good compression rate?
Thanks,
Carlos R.
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
