[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037669#comment-14037669 ]
Joel Koshy commented on KAFKA-1374: ----------------------------------- I had started on this a while ago, but did not finish. Here's a WIP patch that doesn't quite work yet: https://gist.github.com/jjkoshy/4657a44e52e3f88be1c1 Another nuance with compression and compaction is what compression-codec do we use when writing out the compacted data? We could adopt a broker-side compression config that can be overridden on a per-topic basis. This would not only enable a consistent compression codec for each topic, but it will also make the above decision more straightforward - i.e., write out compacted messages in the configured compression codec for that topic (or broker-default if the topic does not have any override). Will file a separate jira for the above. > LogCleaner (compaction) does not support compressed topics > ---------------------------------------------------------- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug > Reporter: Joel Koshy > Fix For: 0.8.2 > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)