Manikumar, How will that help? Increasing the number of log cleaner threads will lead to *less* memory for the buffer per thread, as it's divided up among available threads.
Lawrence, I'm reasonably sure you're hitting KAFKA-3587 here, and should upgrade to 0.10 ASAP. As far as I'm aware Kafka doesn't have any backporting or stable versions policy, so the only ways to get that patch are a) upgrade b) backport the patch yourself. b) seems extremely risky to me Thanks Tom On Tue, Jul 19, 2016 at 5:49 AM, Manikumar Reddy <[email protected]> wrote: > Try increasing log cleaner threads. > > On Tue, Jul 19, 2016 at 1:40 AM, Lawrence Weikum <[email protected]> > wrote: > > > It seems that the log-cleaner is still failing no matter what settings I > > give it. > > > > Here is the full output from one of our brokers: > > [2016-07-18 13:00:40,726] ERROR [kafka-log-cleaner-thread-0], Error due > > to (kafka.log.LogCleaner) > > java.lang.IllegalArgumentException: requirement failed: 192053210 > messages > > in segment __consumer_offsets-15/00000000000000000000.log but offset map > > can fit only 74999999. You can increase log.cleaner.dedupe.buffer.size or > > decrease log.cleaner.threads > > at scala.Predef$.require(Predef.scala:219) > > at > > kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:584) > > at > > kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:580) > > at > > > scala.collection.immutable.Stream$StreamWithFilter.foreach(Stream.scala:570) > > at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:580) > > at kafka.log.Cleaner.clean(LogCleaner.scala:322) > > at > > kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:230) > > at > kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:208) > > at > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) > > [2016-07-18 13:00:40,732] INFO [kafka-log-cleaner-thread-0], Stopped > > (kafka.log.LogCleaner) > > > > Currently, I have heap allocation up to 64GB, only one log-cleaning > thread > > is set to run, and log.cleaner.dedupe.buffer.size is 2GB. I get this > error > > if I try to increase it anymore: > > > > WARN [kafka-log-cleaner-thread-0], Cannot use more than 2G of cleaner > > buffer space per cleaner thread, ignoring excess buffer space... > > (kafka.log.LogCleaner) > > > > Is there something else I can do to help the broker compact the > > __consumer_offset topics? > > > > Thank you again for your help! > > > > Lawrence Weikum > > > > On 7/13/16, 1:06 PM, "Rakesh Vidyadharan" <[email protected]> > > wrote: > > > > We ran into this as well, and I ended up with the following that works > for > > us. > > > > log.cleaner.dedupe.buffer.size=536870912 > > log.cleaner.io.buffer.size=20000000 > > > > > > > > > > > > On 13/07/2016 14:01, "Lawrence Weikum" <[email protected]> wrote: > > > > >Apologies. Here is the full trace from a broker: > > > > > >[2016-06-24 09:57:39,881] ERROR [kafka-log-cleaner-thread-0], Error due > > to (kafka.log.LogCleaner) > > >java.lang.IllegalArgumentException: requirement failed: 9730197928 > > messages in segment __consumer_offsets-36/00000000000000000000.log but > > offset map can fit only 5033164. You can increase > > log.cleaner.dedupe.buffer.size or decrease log.cleaner.threads > > > at scala.Predef$.require(Predef.scala:219) > > > at > > kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:584) > > > at > > kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:580) > > > at > > > scala.collection.immutable.Stream$StreamWithFilter.foreach(Stream.scala:570) > > > at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:580) > > > at kafka.log.Cleaner.clean(LogCleaner.scala:322) > > > at > > kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:230) > > > at > kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:208) > > > at > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) > > >[2016-06-24 09:57:39,881] INFO [kafka-log-cleaner-thread-0], Stopped > > (kafka.log.LogCleaner) > > > > > > > > >Is log.cleaner.dedupe.buffer.size a broker setting? What is a good > > number to set it to? > > > > > > > > > > > >Lawrence Weikum > > > > > > > > >On 7/13/16, 11:18 AM, "Manikumar Reddy" <[email protected]> > > wrote: > > > > > >Can you post the complete error stack trace? Yes, you need to > > >restart the affected > > >brokers. > > >You can tweak log.cleaner.dedupe.buffer.size, log.cleaner.io.buffer.size > > >configs. > > > > > >Some related JIRAs: > > > > > >https://issues.apache.org/jira/browse/KAFKA-3587 > > >https://issues.apache.org/jira/browse/KAFKA-3894 > > >https://issues.apache.org/jira/browse/KAFKA-3915 > > > > > >On Wed, Jul 13, 2016 at 10:36 PM, Lawrence Weikum <[email protected]> > > >wrote: > > > > > >> Oh interesting. I didn’t know about that log file until now. > > >> > > >> The only error that has been populated among all brokers showing this > > >> behavior is: > > >> > > >> ERROR [kafka-log-cleaner-thread-0], Error due to > (kafka.log.LogCleaner) > > >> > > >> Then we see many messages like this: > > >> > > >> INFO Compaction for partition [__consumer_offsets,30] is resumed > > >> (kafka.log.LogCleaner) > > >> INFO The cleaning for partition [__consumer_offsets,30] is aborted > > >> (kafka.log.LogCleaner) > > >> > > >> Using Visual VM, I do not see any log-cleaner threads in those > > brokers. I > > >> do see it in the brokers not showing this behavior though. > > >> > > >> Any idea why the LogCleaner failed? > > >> > > >> As a temporary fix, should we restart the affected brokers? > > >> > > >> Thanks again! > > >> > > >> > > >> Lawrence Weikum > > >> > > >> On 7/13/16, 10:34 AM, "Manikumar Reddy" <[email protected]> > > wrote: > > >> > > >> Hi, > > >> > > >> Are you seeing any errors in log-cleaner.log? The log-cleaner thread > > can > > >> crash on certain errors. > > >> > > >> Thanks > > >> Manikumar > > >> > > >> On Wed, Jul 13, 2016 at 9:54 PM, Lawrence Weikum <[email protected] > > > > >> wrote: > > >> > > >> > Hello, > > >> > > > >> > We’re seeing a strange behavior in Kafka 0.9.0.1 which occurs about > > every > > >> > other week. I’m curious if others have seen it and know of a > > solution. > > >> > > > >> > Setup and Scenario: > > >> > > > >> > - Brokers initially setup with log compaction turned off > > >> > > > >> > - After 30 days, log compaction was turned on > > >> > > > >> > - At this time, the number of Open FDs was ~ 30K per > broker. > > >> > > > >> > - After 2 days, the __consumer_offsets topic was compacted > > >> > fully. Open FDs reduced to ~5K per broker. > > >> > > > >> > - Cluster has been under normal load for roughly 7 days. > > >> > > > >> > - At the 7 day mark, __consumer_offsets topic seems to have > > >> > stopped compacting on two of the brokers, and on those brokers, the > FD > > >> > count is up to ~25K. > > >> > > > >> > > > >> > We have tried rebalancing the partitions before. The first time, > the > > >> > destination broker had compacted the data fine and open FDs were > low. > > The > > >> > second time, the destination broker kept the FDs open. > > >> > > > >> > > > >> > In all the broker logs, we’re seeing this messages: > > >> > INFO [Group Metadata Manager on Broker 8]: Removed 0 expired offsets > > in 0 > > >> > milliseconds. (kafka.coordinator.GroupMetadataManager) > > >> > > > >> > There are only 4 consumers at the moment on the cluster; one topic > > with > > >> 92 > > >> > partitions. > > >> > > > >> > Is there a reason why log compaction may stop working or why the > > >> > __consumer_offsets topic would start holding thousands of FDs? > > >> > > > >> > Thank you all for your help! > > >> > > > >> > Lawrence Weikum > > >> > > > >> > > > >> > > >> > > >> > > > > > > > > > > > > >
