Thanks! I'll discuss this with others and see what the best way to proceed
forward from this is. I think this has been a very productive discussion and
very useful feedback.
________________________________
From: Michael K. Edwards <[email protected]>
Sent: Thursday, November 9, 2017 5:01:33 PM
To: Jordan Henderson
Cc: HDF Users Discussion List
Subject: Re: [Hdf-forum] Collective IO and filters
And here's the change to H5Z-blosc (still using the private H5MM APIs):
diff --git a/src/blosc_filter.c b/src/blosc_filter.c
index bfd8c3e..9bc1a42 100644
--- a/src/blosc_filter.c
+++ b/src/blosc_filter.c
@@ -16,6 +16,7 @@
#include <string.h>
#include <errno.h>
#include "hdf5.h"
+#include "H5MMprivate.h"
#include "blosc_filter.h"
#if defined(__GNUC__)
@@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
/* We're compressing */
if (!(flags & H5Z_FLAG_REVERSE)) {
- /* Allocate an output buffer exactly as long as the input data; if
- the result is larger, we simply return 0. The filter is flagged
- as optional, so HDF5 marks the chunk as uncompressed and
- proceeds.
+ /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes
+ larger than the input data, to accommodate the BLOSC header.
+ If compression with the requested parameters causes the data itself
+ to grow (thereby causing the compressed data, with header, to exceed
+ the output buffer size), fall back to memcpy mode (clevel=0).
*/
- outbuf_size = (*buf_size);
+ outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;
#ifdef BLOSC_DEBUG
fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",
nbytes, outbuf_size);
#endif
- outbuf = malloc(outbuf_size);
+ outbuf = H5MM_malloc(outbuf_size);
if (outbuf == NULL) {
PUSH_ERR("blosc_filter", H5E_CALLBACK,
@@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
blosc_set_compressor(compname);
status = blosc_compress(clevel, doshuffle, typesize, nbytes,
- *buf, outbuf, nbytes);
+ *buf, outbuf, outbuf_size);
+ if (status < 0) {
+ status = blosc_compress(0, doshuffle, typesize, nbytes,
+ *buf, outbuf, outbuf_size);
+ }
if (status < 0) {
PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");
goto failed;
@@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
/* declare dummy variables */
size_t cbytes, blocksize;
- free(outbuf);
+ H5MM_xfree(outbuf);
/* Extract the exact outbuf_size from the buffer header.
*
@@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",
nbytes, outbuf_size);
#endif
- outbuf = malloc(outbuf_size);
+ if (outbuf_size == 0) {
+ H5MM_xfree(*buf);
+ *buf = NULL;
+ *buf_size = outbuf_size;
+ return 0; /* Size of compressed/decompressed data */
+ }
+
+ outbuf = H5MM_malloc(outbuf_size);
if (outbuf == NULL) {
PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate
decompression buffer");
@@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
} /* compressing vs decompressing */
if (status != 0) {
- free(*buf);
+ H5MM_xfree(*buf);
*buf = outbuf;
*buf_size = outbuf_size;
return status; /* Size of compressed/decompressed data */
}
failed:
- free(outbuf);
+ H5MM_xfree(outbuf);
return 0;
} /* End filter function */
On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson
<[email protected]> wrote:
> As the filtered collective path simply calls through the filter pipeline by
> way of the H5Z_pipeline() function, it would seem that either the filter
> pipeline itself is not handling this case correctly, or this is somewhat
> unexpected behavior for the pipeline to deal with.
>
>
> Either way, I think a pull request/diff file would be very useful for going
> over this. If you're able to generate a diff between what you have now and
> the current develop branch/H5Z-blosc code and put it here that would be
> useful. I don't think that there should be too much in the way of logistics
> for getting this code in, we just want to make sure that we approach the
> solution in the right way without breaking something else.
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
