I haven't checked what all changes Binh-Minh has put forth but you should patch the 2 spots because that friend function f_DataSpace_setId still leaks a reference as is, no matter(?) where it's called.
-Jason On Thu, Aug 13, 2015 at 11:56 PM, Jorj Pimm <[email protected]> wrote: > Thanks Jason, > > I've just tried applying that to my local copy of HDF (using the Id > constructor of DataSpace), I can't say for sure - the memory usage of the > program still rose to 5GB+, but it seemed better... > > If I get a chance today I'll rewrite using the C api and see what that > changes. > > - Jorj > > On Fri, 14 Aug 2015 at 03:39 Jason Newton <[email protected]> wrote: > >> Bug found (in C++ api as usual) >> >> The C++ API *should* take care of inc/dec ref appropriately although they >> do this in each object class (may be higher in some class hierarchies like >> datatypes) but something of a leaf otherwise, rather than through >> inheritance of IdComponent. That strategy while working has left a few bugs >> I've found / encountered both as leaks and dec'reffing references not >> incref'd. As of 1.8.15, all that I was aware of though but this concern >> should be warranted all the time based on past-burnings (this would be the >> third time noticing something a shared_ptr like class/wrapper around HDF >> resources (IdComponent...?) would completely eliminate. >> >> >> dataset.getSpace() leaks a reference: >> >> //create dataspace object using the existing id then return the object >> DataSpace data_space; <--default constructor makes a valid hdf >> dataspace for H5S_SCALAR >> f_DataSpace_setId(&data_space, dataspace_id); <-- evil line, why >> didn't we just use the ctor that takes the id parameter? >> return( data_space ); >> >> >> >> //-------------------------------------------------------------------------- >> // Function: f_DataSpace_setId - friend >> // Purpose: This function is friend to class H5::DataSpace so that it >> can >> // can set DataSpace::id in order to work around a problem >> // described in the JIRA issue HDFFV-7947. >> // Applications shouldn't need to use it. >> // param dspace - IN/OUT: DataSpace object to be changed >> // param new_id - IN: New id to set >> // Programmer Binh-Minh Ribler - 2015 >> >> //-------------------------------------------------------------------------- >> void f_DataSpace_setId(DataSpace* dspace, hid_t new_id) <--evil function >> that shouldn't exist (as a friend no-less!) >> { >> dspace->id = new_id; <-- why not dspace->p_setId(new_id);? Just make >> it public already as "reset" and get rid of the friend. Follow shared_ptr >> semantics.. and bring all this stuff inside IdComponent. >> . >> } >> >> >> -Jason >> >> On Thu, Aug 13, 2015 at 9:37 AM, Miller, Mark C. <[email protected]> >> wrote: >> >>> Hmm. Well I have no experience with HDF5's C++ interface. >>> >>> My first thought when reading your description was. . . I've seen that >>> before. It happens when I forgot to H5Xclose() all the objects I H5Xopened >>> (groups, datasets, types, dataspaces, etc.). >>> >>> However, with C++, I presume the interface is designed to close objects >>> when they fall out of scope (e.g. deconstructor is called). So, in looking >>> at your code, even though I don't see any explicit calls to close objects >>> previously opened, I assume that *should* be happening when the objects >>> fall out of scope. But, are you *certain* that *is* happening? Just before >>> exiting main, you migth wanna make a call to H5Fget_obj_count() to get some >>> idea how many objects HDF5 library thinks are still open in the file. If >>> you get a large number, then that would suggest the problem is that the C++ >>> interface isn't somehow closing objects as they fall out of scope. >>> >>> Thats all I can think of. Sorry if no help. >>> >>> Mark >>> >>> >>> From: Hdf-forum <[email protected]> on behalf of >>> Jorj Pimm <[email protected]> >>> Reply-To: HDF Users Discussion List <[email protected]> >>> Date: Thursday, August 13, 2015 9:21 AM >>> To: "[email protected]" <[email protected]> >>> Subject: [Hdf-forum] Growing memory usage in small HDF program >>> >>> Hello, >>> >>> I am writing an application which writes large data sets to HDF5 files, >>> in fixed size blocks, using the HDF C++ API (version 1.8.15, patch 1, built >>> in msvc 2013 x64) >>> >>> I my application seems to quickly consume all the available memory on my >>> system (win32 - around 5.9GB), and then crash whenever the system becomes >>> stressed (windows kills it as it has no memory) >>> >>> I have also tested the application on a linux machine, where I saw >>> similar results. >>> >>> I was under the impression that by using HDF5, the file would be brought >>> in and out of memory in such a way that the library would only use a small >>> working set - is this not true? >>> >>> I have experimented with HDF features such as flushing to disk, >>> regularly closing and re opening, garbage collection and tuning chunking >>> and caching settings and haven't managed to get a stable working set. >>> >>> I've attached a minimal example, can anyone point out my mistake? >>> >>> Thanks, >>> - Jorj >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
