Bug found (in C++ api as usual)
The C++ API *should* take care of inc/dec ref appropriately although they
do this in each object class (may be higher in some class hierarchies like
datatypes) but something of a leaf otherwise, rather than through
inheritance of IdComponent. That strategy while working has left a few bugs
I've found / encountered both as leaks and dec'reffing references not
incref'd. As of 1.8.15, all that I was aware of though but this concern
should be warranted all the time based on past-burnings (this would be the
third time noticing something a shared_ptr like class/wrapper around HDF
resources (IdComponent...?) would completely eliminate.
dataset.getSpace() leaks a reference:
//create dataspace object using the existing id then return the object
DataSpace data_space; <--default constructor makes a valid hdf dataspace
for H5S_SCALAR
f_DataSpace_setId(&data_space, dataspace_id); <-- evil line, why didn't
we just use the ctor that takes the id parameter?
return( data_space );
//--------------------------------------------------------------------------
// Function: f_DataSpace_setId - friend
// Purpose: This function is friend to class H5::DataSpace so that it can
// can set DataSpace::id in order to work around a problem
// described in the JIRA issue HDFFV-7947.
// Applications shouldn't need to use it.
// param dspace - IN/OUT: DataSpace object to be changed
// param new_id - IN: New id to set
// Programmer Binh-Minh Ribler - 2015
//--------------------------------------------------------------------------
void f_DataSpace_setId(DataSpace* dspace, hid_t new_id) <--evil function
that shouldn't exist (as a friend no-less!)
{
dspace->id = new_id; <-- why not dspace->p_setId(new_id);? Just make
it public already as "reset" and get rid of the friend. Follow shared_ptr
semantics.. and bring all this stuff inside IdComponent.
.
}
-Jason
On Thu, Aug 13, 2015 at 9:37 AM, Miller, Mark C. <[email protected]> wrote:
> Hmm. Well I have no experience with HDF5's C++ interface.
>
> My first thought when reading your description was. . . I've seen that
> before. It happens when I forgot to H5Xclose() all the objects I H5Xopened
> (groups, datasets, types, dataspaces, etc.).
>
> However, with C++, I presume the interface is designed to close objects
> when they fall out of scope (e.g. deconstructor is called). So, in looking
> at your code, even though I don't see any explicit calls to close objects
> previously opened, I assume that *should* be happening when the objects
> fall out of scope. But, are you *certain* that *is* happening? Just before
> exiting main, you migth wanna make a call to H5Fget_obj_count() to get some
> idea how many objects HDF5 library thinks are still open in the file. If
> you get a large number, then that would suggest the problem is that the C++
> interface isn't somehow closing objects as they fall out of scope.
>
> Thats all I can think of. Sorry if no help.
>
> Mark
>
>
> From: Hdf-forum <[email protected]> on behalf of Jorj
> Pimm <[email protected]>
> Reply-To: HDF Users Discussion List <[email protected]>
> Date: Thursday, August 13, 2015 9:21 AM
> To: "[email protected]" <[email protected]>
> Subject: [Hdf-forum] Growing memory usage in small HDF program
>
> Hello,
>
> I am writing an application which writes large data sets to HDF5 files, in
> fixed size blocks, using the HDF C++ API (version 1.8.15, patch 1, built in
> msvc 2013 x64)
>
> I my application seems to quickly consume all the available memory on my
> system (win32 - around 5.9GB), and then crash whenever the system becomes
> stressed (windows kills it as it has no memory)
>
> I have also tested the application on a linux machine, where I saw similar
> results.
>
> I was under the impression that by using HDF5, the file would be brought
> in and out of memory in such a way that the library would only use a small
> working set - is this not true?
>
> I have experimented with HDF features such as flushing to disk, regularly
> closing and re opening, garbage collection and tuning chunking and caching
> settings and haven't managed to get a stable working set.
>
> I've attached a minimal example, can anyone point out my mistake?
>
> Thanks,
> - Jorj
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
