UNCLASSIFIED Hi all, In our application's use of HDF-5 we have a requirement for a slightly richer flavour of data type than HDF natively supports. We chose to make all the data types that we use for datasets in our HDF-5 files named (committed) data types, to which we attach a handful of metadata attributes.
However, this process causes a particular pattern of recursive use of named
data types that in our application causes HDF-5 to crash. Consider that to
create a dataset w/ a metadata-decorated data type for C++ type
"std::vector<double>" we first create a decorated named data type representing
the C++ "double" data type. This named HDF-5 data type is then used as the
super-type in declaring a variable length HDF-5 array data type, thus
representing the std::vector part. Hence, we have one named HDF-5 data type
whose super-type is another named HDF-5 data type.
This appears to function correctly so long as you hold the file open; if you
open the dataset and use H5Tget_type() to reveal the data type, H5Tcommitted()
returns 1 as it should. Further, if you get the data type's super-type, it also
reports H5Tcommitted() == 1. However, if you then close the file and re-open it
and repeat the process, it seems to corrupt the slack list associated w/ data
types. In our application we eventually dereference an invalid file pointer
held inside one of the slack list nodes, and it crashes. Fortunately I have
been able to reproduce the issue by creating a new test case (patch attached)
though in this case the test fails (as H5Tcommitted() reports 0 for the
super-type on re-opening of the file, not 1) and then the HDF library reset
causes an infinite loop to be reported. The output is:
Testing non-aligned conversions (ALIGNMENT=1)....
Testing H5Tget_class() PASSED
Testing H5Tcopy() PASSED
Testing H5Tdetect_class() PASSED
Testing compound datatypes PASSED
Testing query functions of compound and enumeration types PASSED
Testing transient datatypes PASSED
Testing named datatypes PASSED
Testing functions of encoding and decoding datatypes PASSED
Testing encoding datatypes with the 'use the latest format' flag PASSED
Testing exceptions for int <-> float conversions PASSED
Testing indirectly reopening committed datatypes PASSED
Testing indirectly reopening recursively committed datatypes including file
reopening*FAILED*
at ..\..\hdf5-1.8.14\test\dtypes.c:6635 in
test_named_indirect_reopen_file()...
Testing deleting objects that use named datatypes PASSED
Testing deleting objects that use named datatypes PASSED
Testing H5Tset/get_order for compound type PASSED
Testing string type creation using H5Tcreate PASSED
Testing deprected API routines for datatypes PASSED
HDF5: infinite loop closing library
D,A,S,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing string conversions PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing random string conversion speed PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing some type functions for string PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing compound element reordering PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing compound subset conversions PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing compound element shrinking & reordering PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing optimized struct converter PASSED
Testing compound element growing PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing compound element insertion PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing packing compound datatypes PASSED
Testing compound datatype with VL string PASSED
Testing array datatype of compound type with VL string PASSED
Testing registering type conversion routine with compound conversions PASSED
Testing adjust size of compound datatypes PASSED
Testing compound datatypes of boundary size with latest format PASSED
Testing unaligned VL strings in compound PASSED
Testing compound subset conversion with extra space in source PASSED
Testing visibility of internally registered type ids PASSED
Testing that H5Tpack removes trailing bytes PASSED
Testing accessing objects with compound datatypes that have no fields PASSED
Testing random enum conversion O(N) PASSED
Testing random enum conversion O(N log N) PASSED
HDF5: infinite loop closing library
D,G,A,S,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,
FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing non-native enumeration type conversion PASSED
Testing bitfield conversions PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing some type functions for bitfield PASSED
HDF5: infinite loop closing library
D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing opaque datatypes PASSED
Testing H5Tset/get_order PASSED
Testing string conversion between ASCII and UTF PASSED
***** 1 FAILURE! *****
HDF5: infinite loop closing library
D,S,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,F
L,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Press any key to continue . . .
The system is Microsoft Visual Studio 2008 on Windows 7 (32-bit build, 64-bit
operating system).
Interestingly, if you comment out just the one offending call to H5Tcommitted()
on the super-type after re-opening, the test passes and the library resets
fine. This is strange as this method only does a H5SL_SEARCH on the slack list
- which really shouldn't mutate any state (?). If you choose not to commit the
super-type, and instead assert H5Tcommitted() == 0, that also works. Despite
all this I suspect the content of the HDF-5 file is being setup incorrectly
when one named data type refers to another, but I can't figure out where. It
doesn't seem to require metadata attributes also attached to cause the failure.
Any ideas?
Mark
IMPORTANT: This email remains the property of the Department of Defence and is
subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have
received this email in error, you are requested to contact the sender and
delete the email.
dtypes.c.patch
Description: dtypes.c.patch
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
