Hi,
I had similarly bad experience with many groups and many attributes as
well. Some things can be done (best do all of them):
1. Make sure you set
|H5Pset_libver_bounds(,||H5F_LIBVER_LATEST,||H5F_LIBVER_LATEST|) when
creating the file such that the newest library features are used
2. compress the group metadata, which may reduce disk reading times
(some high-speed compressor such as LZ4 is recommended)
3. Use the split file driver to place meta-data into different physical
files than the raw data, so meta-data are always compact on disk (might
be less of an issue for SSDs)
4. Reorganize your attributes into datasets instead; when appending data
it may be less inefficient to update the datasets, so during an
incremental data update you can still write attributes, then do some
postprocessing that builds a "cache" of the attributes into a dataset
instead, and during reading you read that dataset instead of the
attributes. This postprocessing to build the "attribute cache" would
take some time of course, but if it needs to be done once only while
reading happens frequently, it is worth the effort. Depends on your use
case scenario of course.
Cheers,
Werner
On 13.07.2017 18:33, Deepak 8 Kumar wrote:
Hello!
I have a HDF5 based application to read the hdf5 file which has almost
3200 groups and 56000 attributes. The application is using the
standard hdf5 api and it took almost 11 minutes to read only the
groups and attributes. I used StopWatch and observed that
H5Aopen_name() is taking almost 97 percent of the total time. I am
using HDF 1.10.1 Windows10 x64.
My question is that is this the expected behavior here with
H5Aopen_name() or I am not reading it properly?
What approach we should take in this kind of file with large number of
attributes?
Any insight is greatly appreciated.
Thanks,
Deepak Kumar
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
