OK but make sure you take a look at the documentation and contrast it with your use cases/requirements: https://www.hdfgroup.org/HDF5/doc/HL/H5PT_Intro.html . I know the learning curve is steep for doing what you want to do, having looked at it myself when I started using HDF5 for these same purposes a few years ago too (in C++/C). Let me assure you it is the right tool for the job though - both HDF5 and packet tables.
Another word of caution: C is the only first class citizen of HDF5 - C++, Java, and C# all have just subsets of it exposed for what they deemed was common to wrap and can have some some conveniences and issues the C-api doesn't. This is why I'd suggest just using the C-api for packet table as it itself is not a first class HDF citizen, nor are language wrappers. Documentation and knowledge of things like this suffers a bit also again because the C part is the only first class citizen.. and even then when there's no example or clarity on an issue, I just jump down into the library sourcecode to figure out what it's doing... not that I expect or hope you do :-) But the benefits are worth the pain: long term achievable and completely portable and good methods for compression, robust to changes of datatypes, FAST - and tools like hdfview and hdfdump. It solves the problem so well it is used by many other different api's these days including Matlab's version 7+ .mat format and NetCDF4. So please be encouraged to keep with it - I'm sure you'll eventually crack it. -Jason On Mon, Aug 10, 2015 at 1:53 PM, Petr KLAPKA <[email protected]> wrote: > Hi Jason, > > Unfortunately, Packet Tables aren't even mentioned in the HDF5DotNet > documentation. > > I've received advice from a few other folks to do this with expanding > datasets. I've been reading through the C API documentation on this, and > it looks like it will work as long as I can find the analogs to the > documented function in the C# wrapper... Getting burnt out on this today, > so I will pick it up again in the AM. > > Best regards, > > Petr Klapka > System Tools Engineer > *Valeo* Radar Systems > 46 River Rd > Hudson, NH 03051 > Mobile: (603) 921-4440 > Office: (603) 578-8045 > *"Festina lente."* > > On Mon, Aug 10, 2015 at 4:46 PM, Jason Newton <[email protected]> wrote: > >> Hi Petr, >> >> I know I've said this before and I'm not sure how first-class this is in >> C# (you probably will have to drop to the c-api) but packet tables do >> this. They keep the table api backend open so you can keep appending but >> otherwise are just another table (existing tools will treat it as a normal >> HDF5 table dataset). >> >> -Jason >> >> On Mon, Aug 10, 2015 at 12:09 PM, Petr KLAPKA <[email protected]> >> wrote: >> >>> Hello all, >>> >>> Now that I'm able to write types of pretty much unlimited complexity >>> into the HDF5 files, the next step is writing additional elements of data >>> to existing datasets. >>> >>> I'll go diving through the manuals again, but if anyone has a "quick >>> start guide" link for how to create a dataset and append to (stream to) it >>> throughout the life of the application, I'd be most grateful. >>> >>> My next goal is to add elements to the "myMajorCycles" dataset that is >>> pictured below with just a single element. I need to accomplish this in a >>> way where the dataset is kept open and can accept new data as it comes in, >>> but it must not remain completely in memory because the final dataset on >>> disk will be ~2GB. >>> >>> D:\Programs\HDF5Utils>h5dump c:\temp\myfirstfile.h5 >>> HDF5 "c:\temp\myfirstfile.h5" { >>> GROUP "/" { >>> DATASET "myMajorCycles" { >>> DATATYPE H5T_COMPOUND { >>> H5T_STD_U32LE "UniqueID"; >>> H5T_STD_U32LE "MsgCount"; >>> H5T_ARRAY { [5] H5T_COMPOUND { >>> H5T_STD_U32LE "MsgID"; >>> H5T_STD_U64LE "MsgHWTimestamp"; >>> H5T_STD_U32LE "MsgLengthBytes"; >>> H5T_ARRAY { [8] H5T_STD_U8LE } "MsgBody"; >>> } } "Messages"; >>> } >>> DATASPACE SIMPLE { ( 1 ) / ( 1 ) } >>> DATA { >>> (0): { >>> 23, >>> 5, >>> [ { >>> 10, >>> 18446744073709551615, >>> 8, >>> [ 0, 1, 2, 3, 4, 5, 6, 7 ] >>> }, >>> { >>> 11, >>> 18446744073709551615, >>> 8, >>> [ 10, 11, 12, 13, 14, 15, 16, 17 ] >>> }, >>> { >>> 12, >>> 18446744073709551615, >>> 8, >>> [ 20, 21, 22, 23, 24, 25, 26, 27 ] >>> }, >>> { >>> 13, >>> 18446744073709551615, >>> 8, >>> [ 30, 31, 32, 33, 34, 35, 36, 37 ] >>> }, >>> { >>> 14, >>> 18446744073709551615, >>> 8, >>> [ 40, 41, 42, 43, 44, 45, 46, 47 ] >>> } ] >>> } >>> } >>> } >>> } >>> } >>> >>> Many thanks! >>> >>> >>> Best regards, >>> >>> Petr Klapka >>> System Tools Engineer >>> *Valeo* Radar Systems >>> 46 River Rd >>> Hudson, NH 03051 >>> Mobile: (603) 921-4440 >>> Office: (603) 578-8045 >>> *"Festina lente."* >>> >>> >>> *This e-mail message is intended only for the use of the intended >>> recipient(s). >>> The information contained therein may be confidential or privileged, >>> and its disclosure or reproduction is strictly prohibited. >>> If you are not the intended recipient, please return it immediately to its >>> sender >>> at the above address and destroy it. * >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> > > > *This e-mail message is intended only for the use of the intended > recipient(s). > The information contained therein may be confidential or privileged, > and its disclosure or reproduction is strictly prohibited. > If you are not the intended recipient, please return it immediately to its > sender > at the above address and destroy it. * > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
