Hi all, we are planning on developing a web service that assembles potentially very large HDF5 files on the fly and sends them over HTTP. We cannot create the files and then send them once they are created - for one the required disc space might not be available on the server and the latency until the first byte is sent would also not be acceptable. Thus we would need to assemble the files piece by piece and send each piece once it is available.
Is this possible at all with the HDF5 format? We don't mind creating the binary files by hand but the final results must be readable by libhdf5. Initial tries with some simple scripts and a hex editor already failed at the superblock level, namely the "End of File Address". If its set to 0 (or some other wrong number that is not the file size) libhdf5 raises an error and refuses to read the file (note that I did adjust the superblock checksum every time). If we assemble the file piece by piece we don't know the final file size at the time the superblock is created and streamed over HTTP. We also don't know what ends up in the final file before we would have to send the first chunk - it might be possible to determine an upper bound ahead of time but it would add significant complexity to the point where we might not be able to execute the project. Is there any way around this? Some magic "End of File Address" number or some way to move the superblock to the end of the file? The HDF5 spec is really long and I did not yet read everything so maybe I am missing something. Assuming this can be dealt with: Are there any other potential roadblocks we might stumble into? If not: Any change on changing the HDF5 spec/libhdf5 so it interprets a "End of File Address" of 0 as "unspecified"? Looking forward to your thoughts and advice. Thanks a lot! Lion _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
