> On May 12, 2015, 2:01 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/log/LogCleaner.scala, line 409 > > <https://reviews.apache.org/r/24214/diff/9/?file=824405#file824405line409> > > > > I would suggest one of two options over this (i.e., instead of two > > helper methods) > > - Inline both here and get rid of those > > - Have a single private helper (e.g., collectRetainedMessages)
removed the helper methods > On May 12, 2015, 2:01 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/log/LogCleaner.scala, line 479 > > <https://reviews.apache.org/r/24214/diff/9/?file=824405#file824405line479> > > > > We should now compress with the compression codec of the topic > > (KAFKA-1499) will do as separate JIRA > On May 12, 2015, 2:01 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/log/LogCleaner.scala, line 498 > > <https://reviews.apache.org/r/24214/diff/9/?file=824405#file824405line498> > > > > We should instead do a trivial refactor in ByteBufferMessageSet to > > compress messages in a preallocated buffer. It would be preferable to avoid > > having this compression logic in different places. moved the compresssMessages() method to ByteBufferMessageSet class. Pl let me know your thoughts.. - Manikumar Reddy ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24214/#review83392 ----------------------------------------------------------- On May 18, 2015, 5:29 p.m., Manikumar Reddy O wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/24214/ > ----------------------------------------------------------- > > (Updated May 18, 2015, 5:29 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-1374 > https://issues.apache.org/jira/browse/KAFKA-1374 > > > Repository: kafka > > > Description > ------- > > Addressing Joel's comments > > > Diffs > ----- > > core/src/main/scala/kafka/log/LogCleaner.scala > abea8b251895a5cc0788c6e25b112a2935a3f631 > core/src/main/scala/kafka/message/ByteBufferMessageSet.scala > 9dfe914991aaf82162e5e300c587c794555d5fd0 > core/src/main/scala/kafka/message/MessageSet.scala > 28b56e68cfdbbf107dd7cbd248ffa8fa6bbcd13f > core/src/test/scala/kafka/tools/TestLogCleaning.scala > 844589427cb9337acd89a5239a98b811ee58118e > core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala > 3b5aa9dc3b7ac5893c1d281ae1326be0e9ed8aad > core/src/test/scala/unit/kafka/log/LogTest.scala > 76d3bfd378f32fd2b216b3ebdec86e2070491924 > > Diff: https://reviews.apache.org/r/24214/diff/ > > > Testing > ------- > > /*TestLogCleaning stress test output for compressed messages/ > > Producing 100000 messages... > Logging produce requests to > /tmp/kafka-log-cleaner-produced-6014466306002699464.txt > Sleeping for 120 seconds... > Consuming messages... > Logging consumed messages to > /tmp/kafka-log-cleaner-consumed-177538909590644701.txt > 100000 rows of data produced, 13165 rows of data consumed (86.8% reduction). > De-duplicating and validating output files... > Validated 9005 values, 0 mismatches. > > Producing 1000000 messages... > Logging produce requests to > /tmp/kafka-log-cleaner-produced-3298578695475992991.txt > Sleeping for 120 seconds... > Consuming messages... > Logging consumed messages to > /tmp/kafka-log-cleaner-consumed-7192293977610206930.txt > 1000000 rows of data produced, 119926 rows of data consumed (88.0% reduction). > De-duplicating and validating output files... > Validated 89947 values, 0 mismatches. > > Producing 10000000 messages... > Logging produce requests to > /tmp/kafka-log-cleaner-produced-3336255463347572934.txt > Sleeping for 120 seconds... > Consuming messages... > Logging consumed messages to > /tmp/kafka-log-cleaner-consumed-9149188270705707725.txt > 10000000 rows of data produced, 1645281 rows of data consumed (83.5% > reduction). > De-duplicating and validating output files... > Validated 899853 values, 0 mismatches. > > > /*TestLogCleaning stress test output for non-compressed messages*/ > > Producing 100000 messages... > Logging produce requests to > /tmp/kafka-log-cleaner-produced-5174543709786189363.txt > Sleeping for 120 seconds... > Consuming messages... > Logging consumed messages to > /tmp/kafka-log-cleaner-consumed-5143455017777144701.txt > 100000 rows of data produced, 22775 rows of data consumed (77.2% reduction). > De-duplicating and validating output files... > Validated 17874 values, 0 mismatches. > > Producing 1000000 messages... > Logging produce requests to > /tmp/kafka-log-cleaner-produced-7814446915546169271.txt > Sleeping for 120 seconds... > Consuming messages... > Logging consumed messages to > /tmp/kafka-log-cleaner-consumed-5172557663160447626.txt > 1000000 rows of data produced, 129230 rows of data consumed (87.1% reduction). > De-duplicating and validating output files... > Validated 89947 values, 0 mismatches. > > Producing 10000000 messages... > Logging produce requests to > /tmp/kafka-log-cleaner-produced-6092986571905399164.txt > Sleeping for 120 seconds... > Consuming messages... > Logging consumed messages to > /tmp/kafka-log-cleaner-consumed-63626021421841220.txt > 10000000 rows of data produced, 1136608 rows of data consumed (88.6% > reduction). > De-duplicating and validating output files... > Validated 899853 values, 0 mismatches. > > > Thanks, > > Manikumar Reddy O > >
