Doubles don't seem to have changed much but I've collected some
working/broken cases for the same conceptual struct, with different layouts
and dump their HDF definition (in program, one line prior to table
instantiation), show ground truth on select fields prior to append, a
repeat of ground truth just after the append operation to the packet table
(remember, this is using the TB common append function internally so the
bug is most likely there or below), and corresponding HDF recorded value.
This issue seems to be dependent on alignments of types.
I've attached a text file of these runs.
-Jason
On Fri, Oct 16, 2015 at 1:59 PM, Jason Newton <[email protected]> wrote:
> As a clarification to hopefully avoid some knee jerk reactions:
>
> If I use uint32_t instead of uint64_t, everything works. I'll have to try
> double.
>
> In the aforementioned unix file packet logs, in a seperate h5 file, I
> commit an H5::CompType type computed in the same runtime so I can safely
> parse the binary file later while keeping most advantages of HDF5. So when
> I read the data in with python and convert it to HDF, I am using this named
> type as the src (from binary) and dst (to HDF) types. I hope this
> increases confidence that I'm not making a trivial mistake on the CompType
> construction.
>
> I do intend to look at the offsets of the fields at some point today to
> verify in the context where I have the problem, it is correct.
>
> -Jason
>
> On Fri, Oct 16, 2015 at 11:47 AM, Jason Newton <[email protected]> wrote:
>
>> Hi,
>>
>> I have structs like the following:
>>
>> struct record{
>> uint32_t count;
>> uint32_t event;
>> uint64_t timestamp_nsec;
>> uint64_t time_of_receipt_nsec;
>> uint32_t some_other_value;
>> uint8_t is_chunked;
>> };
>>
>>
>> I use the c++ api with 1.8.15 to make an H5::CompType containing those
>> fields, using the H5::PredType::Native's. The H5::CompType is constructed
>> as H5::CompType(sizeof(record)). Now when I write this to an append table
>> in online non-trivial programs (2 completely different programs so far),
>> things will be fairly deterministically corrupted. __attribute__((packed))
>> will make a difference how this corruption shows up, I've noted count and
>> is_chunked getting the same values before without it. With it, there tends
>> to only be corruption after the uint64_t's. Some fields are corrupted but
>> it's not easy to know what's up - other fields have the values of fields
>> somewhere else in the structure's memory which strongly implies there's
>> some incorrect offset in the struct used. I use HOFFSET to get the field
>> offsets in the above for H5::CompType::inesrtMember. I believe the
>> H5::CompType is correctly describing the structure's but the append process
>> is not being performed correctly - I use the same types of structs in
>> programs that write unix file packet logs and parse it with these memtypes
>> (in python) and it's fine I convert them to hdf5 datasets with h5py..
>>
>> I have this problem with HDF 1.8.13 and 1.8.15, with gcc 4.8.2 and clang
>> 4.5, on 64bit (the 64bit may be very relevant here, I'm suspecting
>> somewhere the uint64_t's are treated as uint32_t's or INT's or something).
>>
>> Any thoughts? I've tried producing it in some boiled down programs and
>> have yet failed to get a minimum case.
>>
>> -Jason
>>
>
>
failing case, packed sandwhiched uint64s :
struct frame_record_t{
uint32_t connection_id;
uint32_t pixel_format;
uint32_t payload_type;
uint64_t device_timestamp_nsec;
uint64_t time_of_sampling_nsec;
uint64_t time_of_receipt_nsec;
uint64_t time_of_receipt_end_nsec;
uint32_t event;
uint32_t framenumber;
uint32_t payload_length;
uint32_t image_size;
uint32_t exposure_time_usec;
uint8_t is_chunked;
std::array<uint8_t, 7> padding = {};
} __attribute__((packed));
[14:07:49.599101][info][] composition of type 72 struct size: 72
[14:07:49.599120][info][] member 0 name: connection_id class: H5T_INTEGER size:
4 offset 0
[14:07:49.599157][info][] member 1 name: pixel_format class: H5T_INTEGER size:
4 offset 4
[14:07:49.599168][info][] member 2 name: payload_type class: H5T_INTEGER size:
4 offset 8
[14:07:49.599176][info][] member 3 name: device_timestamp_nsec class:
H5T_INTEGER size: 8 offset 12
[14:07:49.599187][info][] member 4 name: time_of_sampling_nsec class:
H5T_INTEGER size: 8 offset 20
[14:07:49.599199][info][] member 5 name: time_of_receipt_nsec class:
H5T_INTEGER size: 8 offset 28
[14:07:49.599207][info][] member 6 name: time_of_receipt_end_nsec class:
H5T_INTEGER size: 8 offset 36
[14:07:49.599214][info][] member 7 name: event class: H5T_INTEGER size: 4
offset 44
[14:07:49.599222][info][] member 8 name: framenumber class: H5T_INTEGER size: 4
offset 48
[14:07:49.599230][info][] member 9 name: payload_length class: H5T_INTEGER
size: 4 offset 52
[14:07:49.599238][info][] member 10 name: image_size class: H5T_INTEGER size: 4
offset 56
[14:07:49.599246][info][] member 11 name: exposure_time_usec class: H5T_INTEGER
size: 4 offset 60
[14:07:49.599253][info][] member 12 name: is_chunked class: H5T_INTEGER size: 1
offset 64
[14:07:49.599263][info][] member 13 name: padding class: H5T_ARRAY size: 7
offset 65
ground truth:
[13:56:57.066407][debug][] image specifics: rxw: 2044x2044 cpadding: 0
device_timestamp: 24031897318440 tor: 13063791071972 framenumber: 197570
pixel_format: 0x01080009 event: 176742
[13:56:57.109150][debug][] post-append image specifics: device_timestamp:
24031897318440 tor: 13063791071972 framenumber: 197570 pixel_format: 0x01080009
event: 176742
hdf recorded:
3 17301513 4 10867718725897819763 10882667350981610617
11027944903236652153 841959618907257 216862 4178208 4177936 4000 1
0 0, 0, 0, 200, 255, 255, 255
failing case:
struct definition:
struct frame_record_t{
uint32_t connection_id;
uint32_t pixel_format;
uint32_t payload_type;
uint64_t device_timestamp_nsec;
uint64_t time_of_sampling_nsec;
uint64_t time_of_receipt_nsec;
uint64_t time_of_receipt_end_nsec;
uint32_t event;
uint32_t framenumber;
uint32_t payload_length;
uint32_t image_size;
uint32_t exposure_time_usec;
uint8_t is_chunked;
std::array<uint8_t, 7> padding = {};
};
[14:00:06.195356][info][] composition of type 80 struct size: 80
[14:00:06.195396][info][] member 0 name: connection_id class: H5T_INTEGER size:
4 offset 0
[14:00:06.195442][info][] member 1 name: pixel_format class: H5T_INTEGER size:
4 offset 4
[14:00:06.195459][info][] member 2 name: payload_type class: H5T_INTEGER size:
4 offset 8
[14:00:06.195485][info][] member 3 name: device_timestamp_nsec class:
H5T_INTEGER size: 8 offset 16
[14:00:06.195503][info][] member 4 name: time_of_sampling_nsec class:
H5T_INTEGER size: 8 offset 24
[14:00:06.195521][info][] member 5 name: time_of_receipt_nsec class:
H5T_INTEGER size: 8 offset 32
[14:00:06.195539][info][] member 6 name: time_of_receipt_end_nsec class:
H5T_INTEGER size: 8 offset 40
[14:00:06.195552][info][] member 7 name: event class: H5T_INTEGER size: 4
offset 48
[14:00:06.195563][info][] member 8 name: framenumber class: H5T_INTEGER size: 4
offset 52
[14:00:06.195591][info][] member 9 name: payload_length class: H5T_INTEGER
size: 4 offset 56
[14:00:06.195604][info][] member 10 name: image_size class: H5T_INTEGER size: 4
offset 60
[14:00:06.195615][info][] member 11 name: exposure_time_usec class: H5T_INTEGER
size: 4 offset 64
[14:00:06.195638][info][] member 12 name: is_chunked class: H5T_INTEGER size: 1
offset 68
[14:00:06.195649][info][] member 13 name: padding class: H5T_ARRAY size: 7
offset 69
gt
[14:13:02.018322][debug][] image specifics: rxw: 2044x2044 cpadding: 0
device_timestamp: 24996825147384 tor: 14028748527337 framenumber: 226097
pixel_format: 0x01080009 event: 205269
[14:13:02.060755][debug][] post-append image specifics: device_timestamp:
24996825147384 tor: 14028748527337 framenumber: 226097 pixel_format: 0x01080009
event: 205269
hdf recorded:
3 17301513 4 24996825147384 14028744980315 14028748527337
14028782353194 205269 226097 4178208 4177936 4000 1 0, 0, 0, 0, 0,
0, 0
passing case, struct definition:
struct frame_record_t{
uint32_t connection_id;
uint32_t pixel_format;
uint32_t payload_type;
uint64_t device_timestamp_nsec;
uint64_t time_of_sampling_nsec;
uint64_t time_of_receipt_nsec;
uint64_t time_of_receipt_end_nsec;
uint32_t event;
uint32_t framenumber;
uint32_t payload_length;
uint32_t image_size;
uint32_t exposure_time_usec;
uint8_t is_chunked;
};
[14:19:56.517912][info][] composition of type 72 struct size: 72
[14:19:56.517961][info][] member 0 name: connection_id class: H5T_INTEGER size:
4 offset 0
[14:19:56.518006][info][] member 1 name: pixel_format class: H5T_INTEGER size:
4 offset 4
[14:19:56.518033][info][] member 2 name: payload_type class: H5T_INTEGER size:
4 offset 8
[14:19:56.518050][info][] member 3 name: device_timestamp_nsec class:
H5T_INTEGER size: 8 offset 16
[14:19:56.518067][info][] member 4 name: time_of_sampling_nsec class:
H5T_INTEGER size: 8 offset 24
[14:19:56.518085][info][] member 5 name: time_of_receipt_nsec class:
H5T_INTEGER size: 8 offset 32
[14:19:56.518101][info][] member 6 name: time_of_receipt_end_nsec class:
H5T_INTEGER size: 8 offset 40
[14:19:56.518119][info][] member 7 name: event class: H5T_INTEGER size: 4
offset 48
[14:19:56.518130][info][] member 8 name: framenumber class: H5T_INTEGER size: 4
offset 52
[14:19:56.518144][info][] member 9 name: payload_length class: H5T_INTEGER
size: 4 offset 56
[14:19:56.518153][info][] member 10 name: image_size class: H5T_INTEGER size: 4
offset 60
[14:19:56.518163][info][] member 11 name: exposure_time_usec class: H5T_INTEGER
size: 4 offset 64
[14:19:56.518175][info][] member 12 name: is_chunked class: H5T_INTEGER size: 1
offset 68
gt:
[14:19:56.540535][debug][] image specifics: rxw: 2044x2044 cpadding: 0
device_timestamp: 25411351404744 tor: 14443287515879 framenumber: 238352
pixel_format: 0x01080009 event: 217524
[14:19:56.608426][debug][] post-append image specifics: device_timestamp:
25411351404744 tor: 14443287515879 framenumber: 238352 pixel_format: 0x01080009
event: 217524
hdf recorded:
3 17301513 4 25411351404744 14443283978084 14443287515879
14443321363169 217524 238352 4178208 4177936 4000 1
passing case
struct frame_record_t{
uint32_t connection_id;
uint32_t pixel_format;
uint32_t payload_type;
uint32_t event;
uint32_t framenumber;
uint32_t payload_length;
uint32_t image_size;
uint32_t exposure_time_usec;
uint8_t is_chunked;
uint64_t device_timestamp_nsec;
uint64_t time_of_sampling_nsec;
uint64_t time_of_receipt_nsec;
uint64_t time_of_receipt_end_nsec;
};
[14:59:31.397334][info][] composition of type 72 struct size: 72
[14:59:31.397358][info][] member 0 name: connection_id class: H5T_INTEGER size:
4 offset 0
[14:59:31.397392][info][] member 1 name: pixel_format class: H5T_INTEGER size:
4 offset 4
[14:59:31.397409][info][] member 2 name: payload_type class: H5T_INTEGER size:
4 offset 8
[14:59:31.397424][info][] member 3 name: event class: H5T_INTEGER size: 4
offset 12
[14:59:31.397440][info][] member 4 name: framenumber class: H5T_INTEGER size: 4
offset 16
[14:59:31.397451][info][] member 5 name: payload_length class: H5T_INTEGER
size: 4 offset 20
[14:59:31.397461][info][] member 6 name: image_size class: H5T_INTEGER size: 4
offset 24
[14:59:31.397471][info][] member 7 name: exposure_time_usec class: H5T_INTEGER
size: 4 offset 28
[14:59:31.397481][info][] member 8 name: is_chunked class: H5T_INTEGER size: 1
offset 32
[14:59:31.397491][info][] member 9 name: device_timestamp_nsec class:
H5T_INTEGER size: 8 offset 40
[14:59:31.397500][info][] member 10 name: time_of_sampling_nsec class:
H5T_INTEGER size: 8 offset 48
[14:59:31.397510][info][] member 11 name: time_of_receipt_nsec class:
H5T_INTEGER size: 8 offset 56
[14:59:31.397521][info][] member 12 name: time_of_receipt_end_nsec class:
H5T_INTEGER size: 8 offset 64
gt:
[14:59:32.283155][debug][] image specifics: rxw: 2044x2044 cpadding: 0
device_timestamp: 27786987686520 tor: 16818996900235 framenumber: 308585
pixel_format: 0x01080009 event: 287757
[14:59:32.325699][debug][] post-append image specifics: device_timestamp:
27786987686520 tor: 16818996900235 framenumber: 308585 pixel_format: 0x01080009
event: 287757
hdf recorded:
3 17301513 4 287757 308585 4178208 4177936 4000 1
27786987686520 16818993347832 16818996900235 16819030735934
failing case
struct frame_record_t{
uint32_t connection_id;
uint32_t pixel_format;
uint32_t payload_type;
uint32_t event;
uint32_t framenumber;
uint32_t payload_length;
uint32_t image_size;
uint32_t exposure_time_usec;
uint8_t is_chunked;
uint64_t device_timestamp_nsec;
uint64_t time_of_sampling_nsec;
uint64_t time_of_receipt_nsec;
uint64_t time_of_receipt_end_nsec;
} __attribute__((packed));
[15:07:22.878018][info][] composition of type 65 struct size: 65
[15:07:22.878052][info][] member 0 name: connection_id class: H5T_INTEGER size:
4 offset 0
[15:07:22.878090][info][] member 1 name: pixel_format class: H5T_INTEGER size:
4 offset 4
[15:07:22.878109][info][] member 2 name: payload_type class: H5T_INTEGER size:
4 offset 8
[15:07:22.878128][info][] member 3 name: event class: H5T_INTEGER size: 4
offset 12
[15:07:22.878152][info][] member 4 name: framenumber class: H5T_INTEGER size: 4
offset 16
[15:07:22.878172][info][] member 5 name: payload_length class: H5T_INTEGER
size: 4 offset 20
[15:07:22.878188][info][] member 6 name: image_size class: H5T_INTEGER size: 4
offset 24
[15:07:22.878204][info][] member 7 name: exposure_time_usec class: H5T_INTEGER
size: 4 offset 28
[15:07:22.878221][info][] member 8 name: is_chunked class: H5T_INTEGER size: 1
offset 32
[15:07:22.878237][info][] member 9 name: device_timestamp_nsec class:
H5T_INTEGER size: 8 offset 33
[15:07:22.878254][info][] member 10 name: time_of_sampling_nsec class:
H5T_INTEGER size: 8 offset 41
[15:07:22.878270][info][] member 11 name: time_of_receipt_nsec class:
H5T_INTEGER size: 8 offset 49
[15:07:22.878287][info][] member 12 name: time_of_receipt_end_nsec class:
H5T_INTEGER size: 8 offset 57
gt:
[15:07:23.843036][debug][] image specifics: rxw: 2044x2044 cpadding: 0
device_timestamp: 28258543015272 tor: 17290566736184 framenumber: 322526
pixel_format: 0x01080009 event: 301698
[15:07:23.885616][debug][] post-append image specifics: device_timestamp:
28258543015272 tor: 17290566736184 framenumber: 322526 pixel_format: 0x01080009
event: 301698
hdf recorded:
3 17301513 4 301698 322526 4178208 4177936 4000 1
4426384193724928 4426385084463104 4426393747783936 0
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
