GitHub user dongjinleekr opened a pull request:
https://github.com/apache/kafka/pull/2267
KAFKA-4514: Add Codec for ZStandard Compression
Hello. This PR resolves [KAFKA-4514: Add Codec for ZStandard
Compression](https://issues.apache.org/jira/browse/KAFKA-4514). Please have a
look when you are free. Since I am a total newbie of Apache Kafka, feel free to
point out the deficiencies.
Add to the feature itself, I have a question: **Should we support an option
for ZStandard compression level?**
According to [ZStandard official
documentation](https://github.com/facebook/zstd/blob/dev/lib/zstd.h), it
supports compression level of 1 ~ 22. Because of that, [Hadoop added a new
configuration option named "io.compression.codec.zstd.level", whose default
value is 3](https://issues.apache.org/jira/browse/HADOOP-13578). In this PR, I
configured the compression level to 1 as a temporary one but wondering
following problems:
- Should we provide a configurable option?
- Would it better to change the default value, from 1 to another one?
I am looking forward to your advice. Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjinleekr/kafka feature/KAFKA-4514
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2267.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2267
----
commit aa751148b75cac9062b13b1115738a595556dd39
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T08:50:20Z
Add zstandard dependency
commit 22fe8d64ffbc2d2725bdfb2166f1ecdc75a96c8d
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T08:51:20Z
Add ZStdCompressionCodec
commit ab535e9f0a18132e35dcbef032dcde320204e12b
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T08:59:31Z
Add CompressionType.ZSTD and update ProducerConfig, MemoryRecordsBuilder
commit 87c3e803ab30344a9834c19075a3ebe617cf5787
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T09:00:55Z
Update CompressionCodec & CompressionFactory
commit 18af6439f1fad6811783da43d0a0f475164853c8
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T09:03:58Z
Add tests for ZStandard
commit 41ae0ac1ac809822712eb609726db00d00e0f36d
Author: Lee Dongjin <[email protected]>
Date: 2016-12-16T09:14:28Z
Add documentation comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
