All,
> On Oct 28, 2016, at 7:17 AM, Peter Steinbach <[email protected]> wrote: > > Hi Elvis, > > interesting I am mostly looking into 3D optical tomography images (which > exclusively use voxels represented by integers). > >> >> This may not be of interest, and does not include zstd, but I'm >> attaching an excerpt from some of the results I got when back when >> doing our basic benchmarking of some algorithms (all lossless). > > We've seen a rough factor of (2.+/-0.5) with lz4 r131 in compression as well > with unfiltered data. In my cases we are mostly interested in high > compression bandwidth and high compression ratio. lz4 so far gives > compression bandwidths up to 1GB/s depending on the quality aspired (of > course the compression ratios tend to be lower then). > >> >> It was based on those that we settled on Blosc_LZ4HC at level 4, since >> we were looking for very fast decompression times, while longer >> compression times and slightly larger file size was acceptable up to >> certain points. The gzip results are included mostly because that's >> what we were using at the time and I wanted them as a comparison, but >> we knew we wanted something else. The input for those benchmarks was a >> 500x300x300 float dataset containing a tomographic 3D image. > > to be honest, I am still surprised that hdf5 doesn't contain these > state-of-the-art encoders, but rather ships bzip2 et al. which are painfully > slow and don't make any account of computer architectures (lz4 is cache aware > AFAIK). But hey, coming up with a hdf5 compressor is straight forward after > one wrangled with the docs. I just don't know how contributing to hdf5 works. The HDF5 library has only two built-in "thrif-paty" compression methods - GZIP and SZIP. Support for those goes back to the first releases of HDF5. To take advantage of the new compression methods one has to use a filter plugin mechanism and the corresponding filter. The HDF Group maintains plugin functionality and, as Mark Miller mentioned in his email to the FORUM, we also support filters registration. We have been looking into how to make the registered filters available with the HDF5 releases without increasing our maintenance cost. Thank you! Elena ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elena Pourmal Client Management Director Interim Engineering Director The HDF Group 1800 So. Oak St., Suite 203, Champaign, IL 61820 www.hdfgroup.org (217)531-6112 (office) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> >> I might try to dig up the script I used for the benchmark and see if >> we still have the input I used, and do a test with lossy ZFP. It could >> be very interesting for creating 3D "thumbnails" in our application. > > indeed, that would be interesting to see. > Best, > Peter > >> >> Elvis >> >>> >>> P >>> >>> >>> On 10/28/2016 01:12 PM, Elvis Stansvik wrote: >>>> >>>> 2016-10-28 1:53 GMT+02:00 Miller, Mark C. <[email protected]>: >>>>> >>>>> Hi All, >>>>> >>>>> Just wanted to mention a new HDF5 floating point compression plugin >>>>> available on github... >>>>> >>>>> https://github.com/LLNL/H5Z-ZFP >>>>> >>>>> This plugin will come embedded in the next release of the Silo library as >>>>> well. >>>> >>>> >>>> Thanks for the pointer. That's very interesting. I had not heard about >>>> ZFP before. The ability to set a bound on the error in the lossless >>>> case seems very useful. >>>> >>>> Do you know if there has been any comparative benchmarks of ZFP >>>> against other compressors? >>>> >>>> After some basic benchmarking, we recently settled on Blosc_LZ4HC at >>>> level 4 for our datasets (3D float tomography data), but maybe it >>>> would be worthwhile to look at ZFP as well.. >>>> >>>> Best regards, >>>> Elvis >>>> >>>>> >>>>> -- >>>>> Mark C. Miller, LLNL >>>>> >>>>> _______________________________________________ >>>>> Hdf-forum is for HDF software users discussion. >>>>> [email protected] >>>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>>> Twitter: https://twitter.com/hdf5 >>>> >>>> >>>> _______________________________________________ >>>> Hdf-forum is for HDF software users discussion. >>>> [email protected] >>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>> Twitter: https://twitter.com/hdf5 >>>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
