2016-10-28 16:33 GMT+02:00 Francesc Alted <[email protected]>: > 2016-10-28 13:59 GMT+02:00 Elvis Stansvik <[email protected]>: >> >> 2016-10-28 13:23 GMT+02:00 Peter Steinbach <[email protected]>: >> > I second this request big time and would add zstd, if we are already >> > trying >> > out various encoders. ;) >> >> This may not be of interest, and does not include zstd, but I'm >> attaching an excerpt from some of the results I got when back when >> doing our basic benchmarking of some algorithms (all lossless). >> >> It was based on those that we settled on Blosc_LZ4HC at level 4, since >> we were looking for very fast decompression times, while longer >> compression times and slightly larger file size was acceptable up to >> certain points. The gzip results are included mostly because that's >> what we were using at the time and I wanted them as a comparison, but >> we knew we wanted something else. The input for those benchmarks was a >> 500x300x300 float dataset containing a tomographic 3D image. > > > Zstd was included in Blosc a while ago: > > http://blosc.org/blog/zstd-has-just-landed-in-blosc.html > > and its performance really shines, even on real data: > > http://alimanfoo.github.io/2016/09/21/genotype-compression-benchmark.html > > (although here, being only integers of 1 byte, only the BITSHUFFLE filter is > used, but not the faster SHUFFLE). > > As Blosc offers the same API for a number of codecs, trying it in > combination with Zstd should be really easy.
Zstd indeed looks very well-balanced. The reason I didn't include it back when I did those benchmarks was that we were really focused on decompression speed in our application, compression speed was very much secondary. So I included mostly LZ4 codecs. > >> >> I might try to dig up the script I used for the benchmark and see if >> we still have the input I used, and do a test with lossy ZFP. It could >> be very interesting for creating 3D "thumbnails" in our application. > > > It would be nice if your benchmark code (and dataset) can be made publicly > available so as to serve to others as a good comparison. The dataset is unfortunately confidential and not something I can release. I'm attaching the script I used though, it's very simple. But, a disclaimer: The benchmarks I did were not really thorough. They were also internal and never really meant to be published. It was mostly a quick and dirty test to see which of these LZ4 codecs would be in the right ballpark for us. Elvis > >> >> >> Elvis >> >> > >> > P >> > >> > >> > On 10/28/2016 01:12 PM, Elvis Stansvik wrote: >> >> >> >> 2016-10-28 1:53 GMT+02:00 Miller, Mark C. <[email protected]>: >> >>> >> >>> Hi All, >> >>> >> >>> Just wanted to mention a new HDF5 floating point compression plugin >> >>> available on github... >> >>> >> >>> https://github.com/LLNL/H5Z-ZFP >> >>> >> >>> This plugin will come embedded in the next release of the Silo library >> >>> as >> >>> well. >> >> >> >> >> >> Thanks for the pointer. That's very interesting. I had not heard about >> >> ZFP before. The ability to set a bound on the error in the lossless >> >> case seems very useful. >> >> >> >> Do you know if there has been any comparative benchmarks of ZFP >> >> against other compressors? >> >> >> >> After some basic benchmarking, we recently settled on Blosc_LZ4HC at >> >> level 4 for our datasets (3D float tomography data), but maybe it >> >> would be worthwhile to look at ZFP as well.. >> >> >> >> Best regards, >> >> Elvis >> >> >> >>> >> >>> -- >> >>> Mark C. Miller, LLNL >> >>> >> >>> _______________________________________________ >> >>> Hdf-forum is for HDF software users discussion. >> >>> [email protected] >> >>> >> >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >>> Twitter: https://twitter.com/hdf5 >> >> >> >> >> >> _______________________________________________ >> >> Hdf-forum is for HDF software users discussion. >> >> [email protected] >> >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> Twitter: https://twitter.com/hdf5 >> >> >> > >> > _______________________________________________ >> > Hdf-forum is for HDF software users discussion. >> > [email protected] >> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> > Twitter: https://twitter.com/hdf5 >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 > > > > > -- > Francesc Alted > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5
compression-benchmark.py
Description: application/download
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
