Hi Manikumar, No, we are not using compression on our topics. I will try out Todd Palino’s suggestion regarding the offsets topic.
Thanks Rakesh On 27/04/2016 23:34, "Manikumar Reddy" <[email protected]> wrote: >Hi, > > Are you enabling log compaction on a topic with compressed messages? > If yes, then that might be the reason for the exception. 0.8.2.2 Log >Compaction does > not support compressed messages. This got fixed in 0.9.0.0 (KAFKA-1641, >KAFKA-1374) > >Check below mail thread for some corrective actions >http://grokbase.com/t/kafka/users/159jbe18en/log-cleaner-thread-stops > > >On Thu, Apr 28, 2016 at 1:44 AM, Rakesh Vidyadharan < >[email protected]> wrote: > >> Hello, >> >> We enabled log compaction on a few topics, as we want to preserve >> permanently the latest versions of messages published to specific topics. >> After enabling compaction, the log cleaner thread dies with the same error >> for the topics we tried it on. It looks like kafka has starting offset >> that does not exist in the topic (at least that is how I am reading the >> error). Any ideas on how we can work around this error? >> >> Thanks >> Rakesh >> >> [2016-04-27 15:52:11,306] INFO [kafka-log-cleaner-thread-0], Starting >> (kafka.log.LogCleaner) >> [2016-04-27 15:52:11,322] INFO Cleaner 0: Beginning cleaning of log >> metamorphosis.lineup-0. (kafka.log.LogCleaner) >> [2016-04-27 15:52:11,323] INFO Cleaner 0: Building offset map for >> metamorphosis.lineup-0... (kafka.log.LogCleaner) >> [2016-04-27 15:52:11,415] INFO Cleaner 0: Building offset map for log >> metamorphosis.lineup-0 for 1 segments in offset range [1553258, 2138466). >> (kafka.log.LogCleaner) >> [2016-04-27 15:52:11,435] ERROR [kafka-log-cleaner-thread-0], Error due >> to (kafka.log.LogCleaner) >> java.lang.IllegalArgumentException: requirement failed: Last clean offset >> is 1553258 but segment base offset is 2125968 for log >> metamorphosis.lineup-0. >> at scala.Predef$.require(Predef.scala:233) >> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:509) >> at kafka.log.Cleaner.clean(LogCleaner.scala:307) >> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221) >> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199) >> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) >> [2016-04-27 15:52:11,436] INFO [kafka-log-cleaner-thread-0], Stopped >> (kafka.log.LogCleaner) >> >>
