[
https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668282#comment-15668282
]
Jason Lowe commented on HADOOP-13578:
-------------------------------------
Cool, thanks for the update and looking into things.
I was hoping with the discovery that the zstd streaming interface can seemingly
take arbitrary input and output buffer sizes that we could avoid the headaches
of copying leftover input data, etc. Instead we keep feeding input data to
ZSTD_decompressStream in a loop until either we run out of input data or the
output buffer is full. As long as there is still input data remaining (i.e.:
output buffer filled) then we don't need more input and therefore don't need to
shuffle data around in the input buffer. Hopefully that makes implementing the
Decompressor less clunky than before.
> Add Codec for ZStandard Compression
> -----------------------------------
>
> Key: HADOOP-13578
> URL: https://issues.apache.org/jira/browse/HADOOP-13578
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: churro morales
> Assignee: churro morales
> Attachments: HADOOP-13578.patch, HADOOP-13578.v1.patch,
> HADOOP-13578.v2.patch
>
>
> ZStandard: https://github.com/facebook/zstd has been used in production for 6
> months by facebook now. v1.0 was recently released. Create a codec for this
> library.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
