[
https://issues.apache.org/jira/browse/KAFKA-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753970#comment-15753970
]
ASF GitHub Bot commented on KAFKA-4514:
---------------------------------------
GitHub user dongjinleekr opened a pull request:
https://github.com/apache/kafka/pull/2267
KAFKA-4514: Add Codec for ZStandard Compression
Hello. This PR resolves [KAFKA-4514: Add Codec for ZStandard
Compression](https://issues.apache.org/jira/browse/KAFKA-4514). Please have a
look when you are free. Since I am a total newbie of Apache Kafka, feel free to
point out the deficiencies.
Add to the feature itself, I have a question: **Should we support an option
for ZStandard compression level?**
According to [ZStandard official
documentation](https://github.com/facebook/zstd/blob/dev/lib/zstd.h), it
supports compression level of 1 ~ 22. Because of that, [Hadoop added a new
configuration option named "io.compression.codec.zstd.level", whose default
value is 3](https://issues.apache.org/jira/browse/HADOOP-13578). In this PR, I
configured the compression level to 1 as a temporary one but wondering
following problems:
- Should we provide a configurable option?
- Would it better to change the default value, from 1 to another one?
I am looking forward to your advice. Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjinleekr/kafka feature/KAFKA-4514
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2267.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2267
----
commit aa751148b75cac9062b13b1115738a595556dd39
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T08:50:20Z
Add zstandard dependency
commit 22fe8d64ffbc2d2725bdfb2166f1ecdc75a96c8d
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T08:51:20Z
Add ZStdCompressionCodec
commit ab535e9f0a18132e35dcbef032dcde320204e12b
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T08:59:31Z
Add CompressionType.ZSTD and update ProducerConfig, MemoryRecordsBuilder
commit 87c3e803ab30344a9834c19075a3ebe617cf5787
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T09:00:55Z
Update CompressionCodec & CompressionFactory
commit 18af6439f1fad6811783da43d0a0f475164853c8
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T09:03:58Z
Add tests for ZStandard
commit 41ae0ac1ac809822712eb609726db00d00e0f36d
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T09:14:28Z
Add documentation comments
----
> Add Codec for ZStandard Compression
> -----------------------------------
>
> Key: KAFKA-4514
> URL: https://issues.apache.org/jira/browse/KAFKA-4514
> Project: Kafka
> Issue Type: Improvement
> Components: compression
> Reporter: Thomas Graves
>
> ZStandard: https://github.com/facebook/zstd and
> http://facebook.github.io/zstd/ has been in use for a while now. v1.0 was
> recently released. Hadoop
> (https://issues.apache.org/jira/browse/HADOOP-13578) and others are adopting
> it.
> We have done some initial trials and seen good results. Zstd seems to give
> great results => Gzip level Compression with Lz4 level CPU.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
