2016-10-28 13:59 GMT+02:00 Elvis Stansvik <[email protected]>:
> 2016-10-28 13:23 GMT+02:00 Peter Steinbach <[email protected]>: > > I second this request big time and would add zstd, if we are already > trying > > out various encoders. ;) > > This may not be of interest, and does not include zstd, but I'm > attaching an excerpt from some of the results I got when back when > doing our basic benchmarking of some algorithms (all lossless). > > It was based on those that we settled on Blosc_LZ4HC at level 4, since > we were looking for very fast decompression times, while longer > compression times and slightly larger file size was acceptable up to > certain points. The gzip results are included mostly because that's > what we were using at the time and I wanted them as a comparison, but > we knew we wanted something else. The input for those benchmarks was a > 500x300x300 float dataset containing a tomographic 3D image. > Zstd was included in Blosc a while ago: http://blosc.org/blog/zstd-has-just-landed-in-blosc.html and its performance really shines, even on real data: http://alimanfoo.github.io/2016/09/21/genotype-compression-benchmark.html (although here, being only integers of 1 byte, only the BITSHUFFLE filter is used, but not the faster SHUFFLE). As Blosc offers the same API for a number of codecs, trying it in combination with Zstd should be really easy. > I might try to dig up the script I used for the benchmark and see if > we still have the input I used, and do a test with lossy ZFP. It could > be very interesting for creating 3D "thumbnails" in our application. > It would be nice if your benchmark code (and dataset) can be made publicly available so as to serve to others as a good comparison. > > Elvis > > > > > P > > > > > > On 10/28/2016 01:12 PM, Elvis Stansvik wrote: > >> > >> 2016-10-28 1:53 GMT+02:00 Miller, Mark C. <[email protected]>: > >>> > >>> Hi All, > >>> > >>> Just wanted to mention a new HDF5 floating point compression plugin > >>> available on github... > >>> > >>> https://github.com/LLNL/H5Z-ZFP > >>> > >>> This plugin will come embedded in the next release of the Silo library > as > >>> well. > >> > >> > >> Thanks for the pointer. That's very interesting. I had not heard about > >> ZFP before. The ability to set a bound on the error in the lossless > >> case seems very useful. > >> > >> Do you know if there has been any comparative benchmarks of ZFP > >> against other compressors? > >> > >> After some basic benchmarking, we recently settled on Blosc_LZ4HC at > >> level 4 for our datasets (3D float tomography data), but maybe it > >> would be worthwhile to look at ZFP as well.. > >> > >> Best regards, > >> Elvis > >> > >>> > >>> -- > >>> Mark C. Miller, LLNL > >>> > >>> _______________________________________________ > >>> Hdf-forum is for HDF software users discussion. > >>> [email protected] > >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists. > hdfgroup.org > >>> Twitter: https://twitter.com/hdf5 > >> > >> > >> _______________________________________________ > >> Hdf-forum is for HDF software users discussion. > >> [email protected] > >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > >> Twitter: https://twitter.com/hdf5 > >> > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > [email protected] > > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > Twitter: https://twitter.com/hdf5 > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 > -- Francesc Alted
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
