Hi Peter, why do you do the compression on HDF5 side? This is something you can do more efficiently on business side.
Best Dimitris > Peter, there’s an API call that lets you write chunks directly > into the file including chunks which you have compressed outside > the HDF5 filter pipeline. Have a look at: > > http://www.hdfgroup.org/HDF5/doc/HL/RM_HDF5Optimized.html#H5DOwrite_chunk > > See how fast you can write with H5DOwrite_chunk and then do > a back-of-the-envelope calculation to see how elaborate > a queueing mechanism you want. > > G. > > From: Hdf-forum [mailto:[email protected]] On Behalf Of > Peter Majer > Sent: Monday, March 16, 2015 11:53 AM > To: [email protected] > Subject: [Hdf-forum] Queuing chunks for compression and writing > > Dear All > We have been experiencing and suffering from the fact that writing compressed > files with hdf is significantly slower than writing uncompressed. I have been > asking myself for a while whether there is a simple remedy. Would it be > possible to have two queues of chunks when writing a file, one for > compression and one for actual writing to achieve the following: > > 1) I enqueue N chunks for CompressionAndWriting. They initially enter > CompressQueue. > 2) The chunks from CompressQueue are concurrently compressed by multiple > compression threads and subsequently enqueued in a WriteQueue. > 3) A WriteThread sequentially writes all compressed chunks from > WriteQueue to the file system. > > This should allow to keep the WriteThread constantly busy and it should allow > compressed writing to be faster than uncompressed writing by a factor that is > more or less identical to the compression rate. > > Interfacewise it would be nice to have “StartWrite” and “FinishWrite” methods > where “Startwrite” simply copies the data into the CompressQueue and returns > immediately thereafter while FinishWrite would be blocking until the write > operation for the corresponding chunk has actually completed. > > Would this be possible? > Would it be feasible? > Would it be easy? > > Thanks, Peter > > > Dr. Peter Majer > Image Analysis Scientist and Software Architect > Bitplane AG > www.bitplane.com > > This message is intended only for the use of the addressee and may contain > information that is confidential and/or subject to copyright. If you are not > the intended recipient, you are hereby notified that any dissemination, > copying, or redistribution of this message is strictly prohibited. If you > have received this message in error please delete all copies immediately. Any > views or opinions presented in this email are solely those of the author and > do not necessarily represent those of Andor Technology Limited Companies. > Andor Technology Limited has taken reasonable precautions to ensure that no > viruses are contained in this email, but does not accept any responsibility > once this email has been transmitted. Andor Technology Limited is a > registered company in Northern Ireland, registration number: NI022466. > Registered Office: Andor Technology, 7 Millennium Way, Springvale Business > Park, Belfast, BT12 7AL. > ___________________________________________________________________________ > Please refer to www.oxinst.com/email-statement for regulatory information. > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
