Hello, I am no expert by any means in your field, but I found that in HFT (high-frequency trading and similar fields) similar pattern often come up.
Boost circular/ring buffer can handle the storage which one process writes and many reads from; upon tail eviction/flushing the pocket one can channel it to the write process --in case of fault tolerance needed you can have many ring buffers on different hosts. The above scheme can be done with various transportation mechanism ie: zeroMQ + protocol buffers/thrift. This is for production use. For running experiments where being fail-safe has little or no benefit one can access the HDF5 datasets locally on each cluster nodes; using MPI + Grid Engine + some_software_task + HDF5_serial pattern. If you find the above interesting, feel free to ask. best, steve On Sun, Dec 11, 2016 at 7:51 AM, Werner Benger <[email protected]> wrote: > Hi, > > there is a virtual file driver that operates on a memory image of an HDF5 > file. It should be no problem to have this one also operate on shared > memory. > > Werner > > > On 10.12.2016 05:50, Schneider, David A. wrote: > >> Is it possible to implement a in memory circular buffer using hdf5? This >> would be for multiple processes, sychronizing in a consumer publisher >> pattern, or maybe the server just writes to the circular buffer. Our data >> acquisition system is moving to hdf5 for offline files, but it also has to >> publish data to shared memory for some online analysis. It would be great >> if we could use the same schema and interface to the data for both offline >> and shared memory access. We're imaging that we will use SWMR to analysis >> the data acquisition data as it is being written to disk, but I'm not sure >> how this will work for shared memory. On disk, we just append to the >> datasets, in memory, we'd be overwriting old data. >> >> best, >> >> David Schneider >> SLAC/LCLS >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> > > -- > ____________________________________________________________ > _______________ > Dr. Werner Benger Visualization Research > Center for Computation & Technology at Louisiana State University (CCT/LSU) > 2019 Digital Media Center, Baton Rouge, Louisiana 70803 > Tel.: +1 225 578 4809 Fax.: +1 225 578-5362 > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
