Hi Nicolas, You are not very precise. What is the amount of data you have to write at the rates you mention? Is it all numeric, or also strings? Can the data be compressed?
Note that HDF5 does not support simultaneous access of readers and writers. It is expected in a future version. Also note that in case of a crash some internal HDF5 meta data may not be up-to-date, so data can be lost. This is worked on using journaling. For some of our astronomical applications we store the observational meta data directly in an HDF5 file, while the main (numeric) data are stored in an external file which is described in the HDF5 file, thus can be accessed as HDF5 data. The advantage is that no HDF5 overhead is involved and in case of a crash about all data are safe. However, beforehand you have to define the total data size. Furthermore, the data need to be regularly shaped. Cheers, Ger >>> Nicolas <nblouve...@gmail.com> 7/13/2012 2:31 PM >>> Dear list, I would like to know if hdf5 is suitable for real-time data logging or not ? More precisely: I work on a project in which we want to continuously (sampling rate ranging form 30 to 400Hz) mix a fair amount of data (several hours) of different natures (telemetry, signals, videos). Data have to be written in real-time (or with a small delay) in order to keep us from losing them on potential crash. Our first prototype is based on sqlite3, however we feel that some limitations could rise from a long run usage: speed, one database == one file, and difficulties for accessing database from several threads (Lock exception when reading and writing at the same time). So, I am considering the possibility to use hdf5 as a back-end for data storage on disk (and numpy/pytable for internal representation). Do you think it is possible to update hdf5 file on at a regular interval from such python binding ? Thank you very much ! Cheer, Nicolas _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@hdfgroup.org http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@hdfgroup.org http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
