If Hdf5 reads in a big chunk and uncompresses it, it seems like a waste
to free that memory before someone else needs it - but I guess if you
hold onto it and the user needs a lot of memory you'll block them. I
wonder if there is a way to hook into requests for memory and free that
uncompressed chunk if someone else needs it, I googled and found
__malloc_hook for gnu C++, but I doubt that is robust or hooks into all
system requests for memory - looks like it is just for debugging your
gnu program.
An alternative to developing smart automatic chunk size defaults to find
an alternative to the current 1MB might be setting the value through an
option for configure when installing hdf5, and through environment
variables.
We apply a patch to our hdf5 installation to increase it to 32MB, if
this is useful to anyone, I put the patch below.
For us, the data a scientist looks at will be a stack of images, most of
the time they go though the stack one at a time, loading the whole
image, but I've also heard of analysis that is effectively random
access, and analysis that works with a region of interest.
Here's our patch:
--- src/H5Pfapl.c.orig 2015-05-28 09:01:47.000000000 -0700
+++ src/H5Pfapl.c 2015-06-05 17:16:43.075228397 -0700
@@ -64,7 +64,7 @@
#define H5F_ACS_DATA_CACHE_NUM_SLOTS_DEF 521
/* Definition for size of raw data chunk cache(bytes) */
#define H5F_ACS_DATA_CACHE_BYTE_SIZE_SIZE sizeof(size_t)
-#define H5F_ACS_DATA_CACHE_BYTE_SIZE_DEF (1024*1024)
+#define H5F_ACS_DATA_CACHE_BYTE_SIZE_DEF (32*1024*1024)
/* Definition for preemption read chunks first */
#define H5F_ACS_PREEMPT_READ_CHUNKS_SIZE sizeof(double)
#define H5F_ACS_PREEMPT_READ_CHUNKS_DEF 0.75f
best,
David
On 02/17/16 02:36, Filipe Maia wrote:
I also agree that the library should not enforce any set chunk size,
but that was not ever in question. The issue is finding the best chunk
cache size when the user has not defined one.
It seems we all agree that the current value of 1MB is outdated.
I also understand that we need to weigh the concerns of using too much
memory.
Taking the hyperslab size into account, together with the chunk size,
is a good idea and it would give us more valuable information to
calculate a better value for chunk cache (e.g. the maximum of chunk
size and hyperslab size for each dimension).
Another possibility would be to give compressions filters some more
information about the dataspace and allow them to set the chunk cache,
but that is a discussion for another thread.
The scenario of multiple user reads per chunk is not uncommon. For
example my datasets have many images and to be able to efficiently
compress it I need to chunk it with multiple images per chunk (as the
images share common features). The user usually looks at one image at
a time resulting in multiple reads per chunk. I don't think such
situations are atypical.
Cheers,
Filipe
On 17 February 2016 at 08:32, Ger van Diepen <[email protected]
<javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
I fully agree with Elena that in general you cannot and should not
set a predefined chunk cache size.
However, I do believe that HDF5 can guess the chunk cache size
based on the access pattern, provided the user has not already set
it. Usually the access pattern is regular, so based on the
hyperslab being accessed, it can assume that the next accesses
will be for the next similar hyperslabs. Probably a hint parameter
can be used to tell that the next hyperslabs will be accessed.
When the hyperslab shape changes, the user probably starts another
access pattern.
Of course, the system can never cater for fully random access, but
I believe that is not used very often. In such a case the user
should always set the cache size.
One can also think of some higher level functionality where the
user defines the cursor shape and access pattern making it
possible to size the cache automatically. Thereafter one can step
through the dataset using a simple next function. Maybe it also
makes optimizations in HDF5 possible since the cursor shape and
access pattern are known a priori (for instance if the cursor
shape is the chunk shape when finding, say, the peak value in a
dataset).
Cheers,
Ger
>>> "David A. Schneider" <[email protected]
<javascript:_e(%7B%7D,'cvml','[email protected]');>>
2/16/2016 9:15 PM >>>
Thanks Elena,
After reading the comments at the end, I think I should try to write a
bunch of small 1MB chunks and see what the read performance is.
However
suppose this leads to 100 times as many chunks, I had the
understanding
that too many chunks degrades read performance in other ways, but
maybe
it will still be a win.
Those are good points about leaving the parameters for optimal
performance to the applications, but it would be nice if there was a
mechanism to allow the writing applications to be responsible for
this,
or at least provide hints that the hdf5 library could decide if it can
support. Then if I am producing a h5 file that a scientist will use
through a high level h5 interface, the scientist can communicate the
reading access pattern, and I can translate it into a chunk layout for
writing, and dataset chunk cache parameters for reading.
best,
David
On 02/14/16 16:55, Elena Pourmal wrote:
> Hi David and Filipe,
>
> Chunking and compression is a powerful feature that boosts
performance and saves space, but if not used correctly (and as you
rightfully noted), leads to performance issues.
>
> We did discuss the solution you proposed and voted against it.
While it is reasonable to increase current default chunk cache
size from 1 MB to ???, it would be unwise for the HDF5 library to
use a chunk cache size equal to a dataset chunk size. We decided
to leave it to applications to determine the appropriate chunk
cache size and strategies (for example, use H5Pset_chunk_cache
instead of H5Pset_cache, or disable chunk cache completely!)
>
>
> Here are several reasons:
>
> 1. Chunk size can be pretty big because it worked well when data
was written, but it may not work well for reading applications. An
HDF5 application will use a lot of memory when working with such
files, especially, if many files and datasets are open. We see
this scenario very often when users work with the collections of
the HDF5 files (for example, NPP satellite data; the attached
paper discusses one of those use cases).
>
> 2. Making chunk cache size the same as chunk size will only
solve the performance problem when data that is written/or read
belongs to one chunk. This is not usually the case. Suppose you
have a row that spans among several chunks. When application reads
by one row at a time, it will not only use a lot of memory because
chunk cache is now big, but there will be the same performance
problem as you described in your email: the same chunk will be
read and discarded many times.
>
> The way to deal with the performance problem is to adjust access
pattern or have chunk cache that contains as many chunks as
possible for the I/O operation. The HDF5 library doesn’t know this
a priori and that is why we left it to applications. At this point
we don’t see how we can help except educating our users.
>
> I am attaching a white paper that will be posted on our Website;
see section 4. Comments are highly appreciated.
>
> Thank you!
>
> Elena
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Elena Pourmal The HDF Group http://hdfgroup.org
> 1800 So. Oak St., Suite 203, Champaign IL 61820
> 217.531.6112 <tel:217.531.6112>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
<javascript:_e(%7B%7D,'cvml','[email protected]');>
>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
<javascript:_e(%7B%7D,'cvml','[email protected]');>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
<javascript:_e(%7B%7D,'cvml','[email protected]');>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
