GitHub user ZoneMayor opened a pull request: https://github.com/apache/kafka/pull/671
KAFKA-2977: Transient Failure in kafka.log.LogCleanerIntegrationTest.cleanerTest Hi @guozhangwang ----------------- Code is as below: val appends = writeDups(numKeys = 100, numDups = 3, log, CompressionCodec.getCompressionCodec(compressionCodec)) cleaner.startup() val firstDirty = log.activeSegment.baseOffset cleaner.awaitCleaned("log", 0, firstDirty) .... val appends2 = appends ++ writeDups(numKeys = 100, numDups = 3, log, CompressionCodec.getCompressionCodec(compressionCodec)) val firstDirty2 = log.activeSegment.baseOffset cleaner.awaitCleaned("log", 0, firstDirty2) ------------------ log cleaner and writeDups are two different threads; log cleaner do cleaning every 15s, timeout in "cleaner.awaitCleaned" is 60s; there is a filtering condition for a log to be chosen to become a cleaning target: cleanableRatio> 0.5(configured log.cleaner.min.cleanable.ratio) by default; It may happen that, during "val appends2 = appends ++ writeDups(numKeys = 100, numDups = 3, log, CompressionCodec.getCompressionCodec(compressionCodec))", log is also undergoing a cleaning process; Since the segment size configured in this test is quite small: 100, there is possibility that before the end of 'writeDups', some 'dirty segment' of the log is already cleaned; With tiny dirty part left, cleanableRatio> 0.5 cannot be satisfied; thus firstDirty2>lastCleaned2, which leads to this test failed; Does it make sense? You can merge this pull request into a Git repository by running: $ git pull https://github.com/ZoneMayor/kafka trunk-KAFKA-2977 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/671.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #671 ---- commit 95374147a28208d4850f6e73f714bf418935fc2d Author: ZoneMayor <jinxing6...@126.com> Date: 2015-11-27T03:49:34Z Merge pull request #1 from apache/trunk merge commit cec5b48b651a7efd3900cfa3c1fd0ab1eeeaa3ec Author: ZoneMayor <jinxing6...@126.com> Date: 2015-12-01T10:44:02Z Merge pull request #2 from apache/trunk 2015-12-1 commit a119d547bf1741625ce0627073c7909992a20f15 Author: ZoneMayor <jinxing6...@126.com> Date: 2015-12-04T13:42:27Z Merge pull request #3 from apache/trunk 2015-12-04#KAFKA-2893 commit b767a8dff85fc71c75d4cf5178c3f6f03ff81bfc Author: ZoneMayor <jinxing6...@126.com> Date: 2015-12-09T10:42:30Z Merge pull request #5 from apache/trunk 2015-12-9 commit 0070c2d71d06ee8baa1cddb3451cd5af6c6b1d4a Author: ZoneMayor <jinxing6...@126.com> Date: 2015-12-11T14:50:30Z Merge pull request #8 from apache/trunk 2015-12-11 commit 09908ac646d4c84f854dad63b8c99213b74a7063 Author: ZoneMayor <jinxing6...@126.com> Date: 2015-12-13T14:17:19Z Merge pull request #9 from apache/trunk 2015-12-13 commit ff1e68bb7101d12624c189174ef1dceb21ed9798 Author: jinxing <jinx...@fenbi.com> Date: 2015-12-13T14:31:34Z KAFKA-2054: Transient Failure in kafka.log.LogCleanerIntegrationTest.cleanerTest commit 6321ab6599cb7a981fac2a4eea64a5f2ea805dd6 Author: jinxing <jinx...@fenbi.com> Date: 2015-12-13T14:36:11Z removed unnecessary maven repo ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---