Well, ok, but…the *only* leak valgrind reported *was* the H5allocate_memory call.
I removed the H5allocate_memory() call entirely *and* kept the H5dont_atexit() call. When I do that, I do indeed get a huge *bloat* in HDF5. I have attached a log from valgrind What I mean by *bloat* is that the HDF5 library has NOT really *leaked* it. It still knows where all the memory is and is showing in the 'still reachable' category of stuff. The library is just not releasing it when it probably should. The *only* way I can get HDF5 library to free that memory is to call H5close() which is rather drastic!! Mark From: Hdf-forum <[email protected]<mailto:[email protected]>> on behalf of "Xavier, Matthew" <[email protected]<mailto:[email protected]>> Reply-To: HDF Users Discussion List <[email protected]<mailto:[email protected]>> Date: Tuesday, March 29, 2016 11:38 AM To: HDF Users Discussion List <[email protected]<mailto:[email protected]>> Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits. As you may note, I’m not allocating that memory to store something; I’m allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It’s the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn’t allocate a new conversion path. So if you ‘fix’ my call to H5allocate_memory(), you almost certainly will not see any errors. Matthew From: Hdf-forum [mailto:[email protected]] On Behalf Of Miller, Mark C. Sent: Tuesday, March 29, 2016 1:10 PM To: HDF Users Discussion List <[email protected]<mailto:[email protected]>> Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths Hi Matthew, Ok, I ran your code under valgrind and cannot confirm your leak report. I was doing this against 1.8.16 HDF5 library I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . . Valgrind run *WITH* H5free_memory()… gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5 ==135138== Memcheck, a memory error detector ==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info ==135138== Command: ./Hdf5TypePathLeak gorfo.h5 ==135138== got here ^C==135138== ==135138== HEAP SUMMARY: ==135138== in use at exit: 72 bytes in 1 blocks ==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated ==135138== ==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296) ==135138== by 0x5046907: H5MM_malloc (H5MM.c:66) ==135138== by 0x4E69930: H5allocate_memory (H5.c:900) ==135138== by 0x401310: main (Hdf5TypePathLeak.c:213) ==135138== ==135138== LEAK SUMMARY: ==135138== definitely lost: 72 bytes in 1 blocks ==135138== indirectly lost: 0 bytes in 0 blocks ==135138== possibly lost: 0 bytes in 0 blocks ==135138== still reachable: 0 bytes in 0 blocks ==135138== suppressed: 0 bytes in 0 blocks ==135138== ==135138== For counts of detected and suppressed errors, rerun with: -v ==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5) surface86{miller86}273: !vi vi Hdf5TypePathLeak.c valgrind run *WITHOUT* H5free_memory(). surface86{miller86}274: !g gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5 ==135256== Memcheck, a memory error detector ==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info ==135256== Command: ./Hdf5TypePathLeak gorfo.h5 ==135256== got here ^C==135256== ==135256== HEAP SUMMARY: ==135256== in use at exit: 36,000 bytes in 500 blocks ==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated ==135256== ==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1 ==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296) ==135256== by 0x5046907: H5MM_malloc (H5MM.c:66) ==135256== by 0x4E69930: H5allocate_memory (H5.c:900) ==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215) ==135256== ==135256== LEAK SUMMARY: ==135256== definitely lost: 36,000 bytes in 500 blocks ==135256== indirectly lost: 0 bytes in 0 blocks ==135256== possibly lost: 0 bytes in 0 blocks ==135256== still reachable: 0 bytes in 0 blocks ==135256== suppressed: 0 bytes in 0 blocks ==135256== ==135256== For counts of detected and suppressed errors, rerun with: -v ==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5) surface86{miller86}276: From: Hdf-forum <[email protected]<mailto:[email protected]>> on behalf of David <[email protected]<mailto:[email protected]>> Reply-To: HDF Users Discussion List <[email protected]<mailto:[email protected]>> Date: Tuesday, March 29, 2016 10:52 AM To: HDF Users Discussion List <[email protected]<mailto:[email protected]>> Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths Matthew, If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API. David On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <[email protected]<mailto:[email protected]>> wrote: I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.) I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different. I have not used committed datatypes. I will take a look at that function and see if it changes the behavior. Matthew From: Hdf-forum [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Miller, Mark C. Sent: Tuesday, March 29, 2016 12:02 PM To: HDF Users Discussion List <[email protected]<mailto:[email protected]>> Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths I use compound types a lot *but*not* with vlen fields. I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe. But, I also 'commit' the types to the files they are defined in. I wonder if commiting the types would make a difference in the memory behavior you are seeing? Mark From: Hdf-forum <[email protected]<mailto:[email protected]>> on behalf of "Xavier, Matthew" <[email protected]<mailto:[email protected]>> Reply-To: HDF Users Discussion List <[email protected]<mailto:[email protected]>> Date: Monday, March 28, 2016 2:15 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [Hdf-forum] H5T module 'leaks' conversion paths Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c). I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical. That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them. Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.) I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.) Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world. Thanks, Matthew Xavier _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected]<mailto:[email protected]> http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org> Twitter: https://twitter.com/hdf5
h5_leak.out.gz
Description: h5_leak.out.gz
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
