Hi Elvis,
interesting I am mostly looking into 3D optical tomography images (which
exclusively use voxels represented by integers).
This may not be of interest, and does not include zstd, but I'm
attaching an excerpt from some of the results I got when back when
doing our basic benchmarking of some algorithms (all lossless).
We've seen a rough factor of (2.+/-0.5) with lz4 r131 in compression as
well with unfiltered data. In my cases we are mostly interested in high
compression bandwidth and high compression ratio. lz4 so far gives
compression bandwidths up to 1GB/s depending on the quality aspired (of
course the compression ratios tend to be lower then).
It was based on those that we settled on Blosc_LZ4HC at level 4, since
we were looking for very fast decompression times, while longer
compression times and slightly larger file size was acceptable up to
certain points. The gzip results are included mostly because that's
what we were using at the time and I wanted them as a comparison, but
we knew we wanted something else. The input for those benchmarks was a
500x300x300 float dataset containing a tomographic 3D image.
to be honest, I am still surprised that hdf5 doesn't contain these
state-of-the-art encoders, but rather ships bzip2 et al. which are
painfully slow and don't make any account of computer architectures (lz4
is cache aware AFAIK). But hey, coming up with a hdf5 compressor is
straight forward after one wrangled with the docs. I just don't know how
contributing to hdf5 works.
I might try to dig up the script I used for the benchmark and see if
we still have the input I used, and do a test with lossy ZFP. It could
be very interesting for creating 3D "thumbnails" in our application.
indeed, that would be interesting to see.
Best,
Peter
Elvis
P
On 10/28/2016 01:12 PM, Elvis Stansvik wrote:
2016-10-28 1:53 GMT+02:00 Miller, Mark C. <[email protected]>:
Hi All,
Just wanted to mention a new HDF5 floating point compression plugin
available on github...
https://github.com/LLNL/H5Z-ZFP
This plugin will come embedded in the next release of the Silo library as
well.
Thanks for the pointer. That's very interesting. I had not heard about
ZFP before. The ability to set a bound on the error in the lossless
case seems very useful.
Do you know if there has been any comparative benchmarks of ZFP
against other compressors?
After some basic benchmarking, we recently settled on Blosc_LZ4HC at
level 4 for our datasets (3D float tomography data), but maybe it
would be worthwhile to look at ZFP as well..
Best regards,
Elvis
--
Mark C. Miller, LLNL
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
