[
https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749557#comment-15749557
]
Jason Lowe commented on HADOOP-13578:
-------------------------------------
Thanks for updating the patch! It's really close now.
We need to check for an error if ZSTD_endStream returns a failure, otherwise we
could end up looping forever on a persistent error being returned from
ZSTD_endStream. The outer compress logic will keep calling compress with
finish=true trying to wrap things up and the JNI code will keep avoiding
setting the finished flag because ZSTD_endStream isn't returning 0.
There's no return after the THROW macros here, so we can fall through and call
ZSTD_flushStream after ZSTD_compressStream returns an error. That means the
error being reported from compress might be clobbered by the error being
returned from flush if they are different. Also we can just move the error
check on "size" up inside the flush block since it only applies to that case,
or change "remaining_to_flush" to "size" and let the previous point be fixed by
falling through to the common error check on "size".
{code}
size_t size = dlsym_ZSTD_compressStream(stream, &output, &input);
if (dlsym_ZSTD_isError(size)) {
THROW(env, "java/lang/InternalError", dlsym_ZSTD_getErrorName(size));
}
if (finish && input.pos == input.size) {
// end the stream, flush and write the frame epilogue
size_t const remaining_to_flush = dlsym_ZSTD_endStream(stream, &output);
if (!remaining_to_flush) {
(*env)->SetBooleanField(env, this, ZStandardCompressor_finished,
JNI_TRUE);
}
}
else {
// need to flush the output buffer
// this also updates the output buffer position.
size = dlsym_ZSTD_flushStream(stream, &output);
}
if (dlsym_ZSTD_isError(size)) {
THROW(env, "java/lang/InternalError", dlsym_ZSTD_getErrorName(size));
}
{code}
> Add Codec for ZStandard Compression
> -----------------------------------
>
> Key: HADOOP-13578
> URL: https://issues.apache.org/jira/browse/HADOOP-13578
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: churro morales
> Assignee: churro morales
> Attachments: HADOOP-13578.patch, HADOOP-13578.v1.patch,
> HADOOP-13578.v2.patch, HADOOP-13578.v3.patch, HADOOP-13578.v4.patch,
> HADOOP-13578.v5.patch, HADOOP-13578.v6.patch, HADOOP-13578.v7.patch,
> HADOOP-13578.v8.patch
>
>
> ZStandard: https://github.com/facebook/zstd has been used in production for 6
> months by facebook now. v1.0 was recently released. Create a codec for this
> library.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
