2016-10-28 17:53 GMT+02:00 Francesc Alted <[email protected]>: > > > 2016-10-28 17:20 GMT+02:00 Elvis Stansvik <[email protected]>: >> >> 2016-10-28 16:33 GMT+02:00 Francesc Alted <[email protected]>: >> > 2016-10-28 13:59 GMT+02:00 Elvis Stansvik >> > <[email protected]>: >> >> >> >> 2016-10-28 13:23 GMT+02:00 Peter Steinbach <[email protected]>: >> >> > I second this request big time and would add zstd, if we are already >> >> > trying >> >> > out various encoders. ;) >> >> >> >> This may not be of interest, and does not include zstd, but I'm >> >> attaching an excerpt from some of the results I got when back when >> >> doing our basic benchmarking of some algorithms (all lossless). >> >> >> >> It was based on those that we settled on Blosc_LZ4HC at level 4, since >> >> we were looking for very fast decompression times, while longer >> >> compression times and slightly larger file size was acceptable up to >> >> certain points. The gzip results are included mostly because that's >> >> what we were using at the time and I wanted them as a comparison, but >> >> we knew we wanted something else. The input for those benchmarks was a >> >> 500x300x300 float dataset containing a tomographic 3D image. >> > >> > >> > Zstd was included in Blosc a while ago: >> > >> > http://blosc.org/blog/zstd-has-just-landed-in-blosc.html >> > >> > and its performance really shines, even on real data: >> > >> > >> > http://alimanfoo.github.io/2016/09/21/genotype-compression-benchmark.html >> > >> > (although here, being only integers of 1 byte, only the BITSHUFFLE >> > filter is >> > used, but not the faster SHUFFLE). >> > >> > As Blosc offers the same API for a number of codecs, trying it in >> > combination with Zstd should be really easy. >> >> Zstd indeed looks very well-balanced. The reason I didn't include it >> back when I did those benchmarks was that we were really focused on >> decompression speed in our application, compression speed was very >> much secondary. So I included mostly LZ4 codecs. > > > Yes, that makes sense, but I think you should give a try at least at the > lowest compression levels for Blosc+Zstd (1, 2 and probably 3 too). For > these low compression levels Blosc chooses a block size that comfortably > fits in L2. Also, note that the benchmarks above where for in-memory data, > so for a typical disk-based workflow using HDF5, Blosc+Zstd can still > perform well enough.
Alright, thanks for the tip. I read the benchmarks too fast and didn't realize it was all in-memory. I should definitely at Zstd. In our use case it's always from disk (or well, SSD), and sometimes even slow-ish network mounts. Elvis > > >> >> >> > >> >> >> >> I might try to dig up the script I used for the benchmark and see if >> >> we still have the input I used, and do a test with lossy ZFP. It could >> >> be very interesting for creating 3D "thumbnails" in our application. >> > >> > >> > It would be nice if your benchmark code (and dataset) can be made >> > publicly >> > available so as to serve to others as a good comparison. >> >> The dataset is unfortunately confidential and not something I can >> release. I'm attaching the script I used though, it's very simple. >> >> But, a disclaimer: The benchmarks I did were not really thorough. They >> were also internal and never really meant to be published. It was >> mostly a quick and dirty test to see which of these LZ4 codecs would >> be in the right ballpark for us. > > > Ok. Thanks anyway. > >> >> >> Elvis >> >> > >> >> >> >> >> >> Elvis >> >> >> >> > >> >> > P >> >> > >> >> > >> >> > On 10/28/2016 01:12 PM, Elvis Stansvik wrote: >> >> >> >> >> >> 2016-10-28 1:53 GMT+02:00 Miller, Mark C. <[email protected]>: >> >> >>> >> >> >>> Hi All, >> >> >>> >> >> >>> Just wanted to mention a new HDF5 floating point compression plugin >> >> >>> available on github... >> >> >>> >> >> >>> https://github.com/LLNL/H5Z-ZFP >> >> >>> >> >> >>> This plugin will come embedded in the next release of the Silo >> >> >>> library >> >> >>> as >> >> >>> well. >> >> >> >> >> >> >> >> >> Thanks for the pointer. That's very interesting. I had not heard >> >> >> about >> >> >> ZFP before. The ability to set a bound on the error in the lossless >> >> >> case seems very useful. >> >> >> >> >> >> Do you know if there has been any comparative benchmarks of ZFP >> >> >> against other compressors? >> >> >> >> >> >> After some basic benchmarking, we recently settled on Blosc_LZ4HC at >> >> >> level 4 for our datasets (3D float tomography data), but maybe it >> >> >> would be worthwhile to look at ZFP as well.. >> >> >> >> >> >> Best regards, >> >> >> Elvis >> >> >> >> >> >>> >> >> >>> -- >> >> >>> Mark C. Miller, LLNL >> >> >>> >> >> >>> _______________________________________________ >> >> >>> Hdf-forum is for HDF software users discussion. >> >> >>> [email protected] >> >> >>> >> >> >>> >> >> >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> >>> Twitter: https://twitter.com/hdf5 >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> Hdf-forum is for HDF software users discussion. >> >> >> [email protected] >> >> >> >> >> >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> >> Twitter: https://twitter.com/hdf5 >> >> >> >> >> > >> >> > _______________________________________________ >> >> > Hdf-forum is for HDF software users discussion. >> >> > [email protected] >> >> > >> >> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> > Twitter: https://twitter.com/hdf5 >> >> >> >> _______________________________________________ >> >> Hdf-forum is for HDF software users discussion. >> >> [email protected] >> >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> Twitter: https://twitter.com/hdf5 >> > >> > >> > >> > >> > -- >> > Francesc Alted >> > >> > _______________________________________________ >> > Hdf-forum is for HDF software users discussion. >> > [email protected] >> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> > Twitter: https://twitter.com/hdf5 > > > > > -- > Francesc Alted _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
