Thanks Mark, this is the path I'm on now. Best regards,
Petr Klapka System Tools Engineer *Valeo* Radar Systems 46 River Rd Hudson, NH 03051 Mobile: (603) 921-4440 Office: (603) 578-8045 *"Festina lente."* On Mon, Aug 10, 2015 at 8:17 AM, Koennecke Mark (PSI) <[email protected] > wrote: > Dear Petr Klapka, > > Am 06.08.2015 um 16:46 schrieb Petr KLAPKA <[email protected]>: > > Good morning! > > My name is Petr Klapka, My colleagues and I are in the process of > evaluating HDF5 as a potential file format for a data acquisition tool. > > > We use HDF5 in data acquisition at SINQ, PSI. Other places, like > synchrotron sources which generate much more data then we, too. > > > I have been working through the HDF5 tutorials and overcoming the API > learning curve. I was hoping you could offer some advice on the > suitability of HDF5 for our intended purpose and perhaps save me the time > of mis-using the format or API. > > The data being acquired are "samples" from four devices. Every ~50ms a > device provides a sample. The sample is an array of structs. The total > size of the array varies but will be on average around 8 kilobytes. (160k > per second per device). > > The data will need to be recorded over a period of about an hour, meaning > an uncompressed file size of around 2.3 Gigabytes. > > I will need to "play back" these samples, as well as jump around in the > file, seeking on sample meta data and time. > > My questions to you are: > > - Is HDF5 intended for data sets of this size and throughput given a > high performance Windows workstation? > > > Sure, HDF5 excels for this kind of data sizes. Only 2.3 GB…. But if you > use Windows you throw away most of the cababilities of your machine. > > > - What is the "correct" usage pattern for this scenario? > - Is it to use a "Group" for each device, and create a "Dataset" > for each sample? This would result in thousands of datasets in the file > per group, but I fully understand how to navigate this structure. > > This will not perform well. HDF does not like to have many small > structures generated. > > > - Or should there only be four "Datasets" that are extensible, and > each sensor "sample" be appended into the dataset? If this is the case, > can the dataset itself be searched for specific samples by time and > metadata? > > This is much better. Appending to a large array is good. You may have to > play with the chunking to get this working well. HDF5 does not support > explicit searching. > But if I understand your use case right, it would help to have another > array with time stamps running alongside. Then search for a specific time > interval > comes down finding the index range of the interesting time range in the > time stamp array and then retrieve the slabs of data from the arrays by the > indexes just > found. > > I know to little about your other meta data to suggest something. > > > - Or is this use case appropriate for the Table API? > > > Could be, I do not know that API well enough, > > Best Regards, > > Mark Könnecke > > I will begin with prototyping the first scenario, since it is the most > straight forward to understand and implement. Please let me know your > suggestions. Many thanks! > > Best regards, > > Petr Klapka > System Tools Engineer > *Valeo* Radar Systems > 46 River Rd > Hudson, NH 03051 > Mobile: (603) 921-4440 > Office: (603) 578-8045 > *"Festina lente."* > > *This e-mail message is intended only for the use of the intended > recipient(s). > The information contained therein may be confidential or privileged, > and its disclosure or reproduction is strictly prohibited. > If you are not the intended recipient, please return it immediately to its > sender > at the above address and destroy it. * > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 > > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 > -- *This e-mail message is intended only for the use of the intended recipient(s). The information contained therein may be confidential or privileged, and its disclosure or reproduction is strictly prohibited. If you are not the intended recipient, please return it immediately to its sender at the above address and destroy it. *
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
