Hmm….I am not entirely sure that what is behaving poorly is a 10000 row table
of *any* sizes for double array or any number of rows with a 10000 size double
array or both.
You mention chunking but then describe only *one* of the parameters of the
chunk, for the first column (the doubles) I think. But, isn’t this table a 2D
dataset in the HDF5 file? And, don’t you wind up having to chunk in both
dimensions? What are you using to chunk the second dimension (e.g. #rows)? Does
it have any impact on performance of hdfview?
Since you include that one of the chunk parameters you tried was ‘1’, it
suggests you might also be trying equally small values for the other chunk
dimension. And, I think that could be a potential source of performance
degradation. You could try writing a simple C or C++ program to open and
iterate through all the rows in the dataset and print the values (to /dev/null
or something). If that goes fast, then it would suggest the problem is in
hdfview itself.
That’s as far as my thinking goes.
Mark
"Hdf-forum on behalf of Daniel Rimmelspacher" wrote:
Dear Forum Members,
I am trying to store application data in a compound dataset (via Matlab
low-level API). The dataset looks like this:
Column1:
type1 = H5T.vlen_create('H5T_NATIVE_DOUBLE')
Column2:
type2= H5T.copy('H5T_C_S1')
H5T.set_size(type2, 'H5T_VARIABLE')
In order to try out the performance, I varied the length of type 1, i.e.
storing 10/100/1000/10000 doubles in 10000 rows. Storing though seems
pretty fast, but when I try to open the data set with the HDF Viewer I
encounter significant performance issues for 10000 values. I already
tried to vary the chunking parameter (1/10/100/1000) but without success.
Is this an issue with the HDF viewer or rather a problem with my data
model? Is there a trick in order to enhance dataset loading performance
inside the viewer?
Any hint is appreciated!
Thanks,
Daniel
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]<mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
