----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24214/#review50128 -----------------------------------------------------------
core/src/main/scala/kafka/log/LogCleaner.scala <https://reviews.apache.org/r/24214/#comment87704> Thinking about this a bit more. I am wondering if it would be better if we introduce a per-topic level log.compact.compress.codec property. During log compaction, we always write the retained data using the specified compress codec, independent of whether the original records are compressed or not. This provides the following benefits. 1. Whether the messages were compressed originally, they can be compressed on the broker side over time. Since compact topics preserve records much longer, enabling compression on the broker side will be beneficial in general. 2. As old records are removed, we still want to batch enough messages to do the compression. 3. The code can be a bit simpler. We can just (deep) iterate messages (using MemoryRecods.iterator) and append retained messages to an output MemoryRecords. The output MemoryRecords will be initialized with the configured compress codec and batch size. - Jun Rao On Aug. 9, 2014, 10:51 a.m., Manikumar Reddy O wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/24214/ > ----------------------------------------------------------- > > (Updated Aug. 9, 2014, 10:51 a.m.) > > > Review request for kafka. > > > Bugs: KAFKA-1374 > https://issues.apache.org/jira/browse/KAFKA-1374 > > > Repository: kafka > > > Description > ------- > > Addressed Jun's comments;Added few changes in LogCleaner stats for compressed > messages > > > Diffs > ----- > > core/src/main/scala/kafka/log/LogCleaner.scala > c20de4ad4734c0bd83c5954fdb29464a27b91dff > core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala > 5bfa764638e92f217d0ff7108ec8f53193c22978 > > Diff: https://reviews.apache.org/r/24214/diff/ > > > Testing > ------- > > > Thanks, > > Manikumar Reddy O > >
