[
https://issues.apache.org/jira/browse/KAFKA-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122310#comment-13122310
]
C. Scott Andreas commented on KAFKA-79:
---------------------------------------
Jay and Neha,
Fantastic - thanks for taking a look. Your GZip choice makes sense - for
inter-DC transport, totally agree with pushing the CPU a little harder to drive
down the wire size. Most of our current use is within the same rack, so we've
been optimizing for codec throughput. That said, I agree and like the idea of
pluggable compressors.
I'll clone, apply Neha's patch, and drop that in this weekend. I don't expect
that it will be too complicated (it's just an InputStream; our local use is
just a couple lines).
(Apologies for the slow reply; I had mistakenly disabled e-mail notifications).
– Scott
> Introduce the compression feature in Kafka
> ------------------------------------------
>
> Key: KAFKA-79
> URL: https://issues.apache.org/jira/browse/KAFKA-79
> Project: Kafka
> Issue Type: New Feature
> Affects Versions: 0.6
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.7
>
>
> With this feature, we can enable end-to-end block compression in Kafka. The
> idea is to enable compression on the producer for some or all topics, write
> the data in compressed format on the server and make the consumers
> compression aware. The data will be decompressed only on the consumer side.
> Ideally, there should be a choice of compression codecs to be used by the
> producer. That means a change to the message header as well as the network
> byte format. On the consumer side, the state maintenance behavior of the
> zookeeper consumer changes. For compressed data, the consumed offset will be
> advanced one compressed message at a time. For uncompressed data, consumed
> offset will be advanced one message at a time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
