What you want is run time type definitions - which is something HDF supports but C doesn't. Have a look at python/numpy's dtype for a good idea of the task you're in for, especially on the c side, which maps really well to hdf's type system. If you just want something really simple you dont' have to get too crazy.
Basically you need to know how big a type is (a runtime version of sizeof), how many fields are in it, the types of those fields, and to prefix sum ( a cumsum with a zero shifted in from the left) the size and arity of those fields to get the offsets of each field of the struct-blob-at-runtime. Then you have to work with those fields adaptively since their runtime type varies because someone's gotta work with bytes at the end of the day. It's still quite fast and what you see happening essentially with python's numpy api. The table/packet table apis will have no problem with these types as records. FYI for performance you should still buffer some of these up. The alternative approach is a storing structure of arrays - which is a very common technique - matlab uses it when it stores in hdf5 for instance and it is more .... available to other users of HDF5 who care not to venture into these things. I side on runtime structs but other software sometime isn't always so clever. This boils down to each column having it's own dataset/table/packet table and being whatever primative hdf5, such that you will likely not have to touch compounds. The drawback of this approach is that you always have to do work to reconstruct the object, which can be relatively slow in python for instance and involve lots of copying and metaprogramming. In matlab there is no generic solution for gluing it back together for instance - so you end up writing boilerplate and supporting that is difficult as the documents change. -Jason On Wed, Aug 23, 2017 at 5:10 AM, Rafal Lichwala <[email protected]> wrote: > Hi, > > I've read many examples from both H5TB high level API and low level API > for compound HDF data type, but I didn't find a good solution for my > special use case. All those examples have one problematic assumption: data > structure (which means number of fields and their types and values) must be > known a priori - that's the problem in my case, when I don't know this > structure and I need to create a table HDF dataset not only row-by-row, but > also field-by-field in the row. > > I need your advice how to achieve what I want using a proper sequence of > HDF API calls. > > Let's say my final HDF table will look like this: > ['a', 1, 3.14] > ['b', 2, 2.11] > ['c', 3, 1.89] > > So we simply have a HDF table with 3 columns of types: char, int, float > and 3 rows with some values. > > Creation of that table must be divided into some "steps". > After 1st "step" I should have a table: > ['a'] > > After 2nd step: > ['a', 1] > > After 3rd step: > ['a', 1, 3.14] > > After 4th step: > ['a', 1, 3.14] > ['b', x, x] > > where x after 4th step is undefined and can be some default values which > will be overwritten in the next steps. > > How to achieve that use case? > > Is it possible to create a table by calling H5TBmake_table(), but having > no fields and no records at the beginning and then just call > H5TBinsert_field() in the next steps? > > Is it possible to have "data" attribute of H5TBinsert_field() function a > NULL value when we insert a new field to a table dataset with no records > yet? > > What about 4th step - can I create just a first column value for a new > record in a table? > > I know it's maybe a strange use case, but the problem is that I could have > really huge structure model (a lot of columns and a lot of records) which > should be stored in the HDF table dataset, so I need to avoid "collecting" > required information (number of fields, their types, values) by initial > iterating over whole structure. > The second problem is that I have a vector of objects which need to be > stored as HDF table (where table row is the given object and columns are > its fields), but all examples I've seen just work on C struct. > > I would appreciate any advice! > > Regards, > Rafal > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
