[ https://issues.apache.org/jira/browse/KAFKA-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ismael Juma resolved KAFKA-4496. -------------------------------- Resolution: Fixed The code changed a lot since 0.8.2.1 and it no longer contains an `Iterator.toSeq` call (which returns a `Stream` which is potentially problematic for a concurrently mutable collection), so I believe this has been fixed. > LogCleaner thread exits with NoSuchElementException > --------------------------------------------------- > > Key: KAFKA-4496 > URL: https://issues.apache.org/jira/browse/KAFKA-4496 > Project: Kafka > Issue Type: Bug > Components: core, offset manager > Affects Versions: 0.8.2.1 > Environment: SLES 11.3, 20 node cluster > Reporter: Mark Figura > > After running for some time (about 60 hours in this case), we are seeing the > LogCleaner thread exit with the messages below. Because of this, the > consumer_offsets topic grows very large over time. > [2016-12-01 20:00:44,050] INFO Cleaner 0: Building offset map for > __consumer_offsets-19... (kafka.log.LogCleaner) > [2016-12-01 20:00:44,183] INFO Cleaner 0: Building offset map for log > __consumer_offsets-19 for 0 segments in offset range [15698821, 0). > (kafka.log.LogCleaner) > [2016-12-01 20:00:44,220] ERROR [kafka-log-cleaner-thread-0], Error due to > (kafka.log.LogCleaner) > java.util.NoSuchElementException: head of empty stream > at scala.collection.immutable.Stream$Empty$.head(Stream.scala:1028) > at scala.collection.immutable.Stream$Empty$.head(Stream.scala:1026) > at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:508) > at kafka.log.Cleaner.clean(LogCleaner.scala:307) > at > kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221) > at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) > [2016-12-01 20:00:44,221] INFO [kafka-log-cleaner-thread-0], Stopped > (kafka.log.LogCleaner) > This looks similar to some other issues (KAFKA-1641, KAFKA-2235, etc.) but > does not match exactly. > Here is the server.properties: > delete.topic.enable=true > socket.send.buffer.bytes=102400 > socket.request.max.bytes=104857600 > log.cleaner.enable=true > log.retention.check.interval.ms=300000 > zookeeper.session.timeout.ms=30000 > num.io.threads=8 > broker.id=0 > port=9092 > log.dirs=/kafka/broker_0 > auto.create.topics.enable=false > num.network.threads=3 > socket.receive.buffer.bytes=102400 > log.retention.minutes=10 > log.cleaner.delete.retention.ms=1800000 > log.segment.bytes=1073741824 > num.recovery.threads.per.data.dir=1 > num.partitions=1 > zookeeper.connection.timeout.ms=30000 > zookeeper.connect=<zk servers> > ...and here's what one of the consumer-offsets data directories looks like: > kafka1:/kafka/broker_0/__consumer_offsets-19 # ls -lrt > total 2611332 > -rw-r--r-- 1 yarn yarn 104857433 Dec 1 23:17 00000000000000000000.log > -rw-r--r-- 1 yarn yarn 182272 Dec 1 23:17 00000000000000000000.index > -rw-r--r-- 1 yarn yarn 104857592 Dec 2 05:44 00000000000016820302.log > -rw-r--r-- 1 yarn yarn 201880 Dec 2 05:44 00000000000016820302.index > -rw-r--r-- 1 yarn yarn 104857497 Dec 2 10:18 00000000000017939487.log > -rw-r--r-- 1 yarn yarn 201464 Dec 2 10:18 00000000000017939487.index > -rw-r--r-- 1 yarn yarn 104857302 Dec 2 14:04 00000000000019061235.log > -rw-r--r-- 1 yarn yarn 201176 Dec 2 14:04 00000000000019061235.index > -rw-r--r-- 1 yarn yarn 104857594 Dec 2 17:43 00000000000020184726.log > -rw-r--r-- 1 yarn yarn 201120 Dec 2 17:43 00000000000020184726.index > -rw-r--r-- 1 yarn yarn 104857523 Dec 2 21:51 00000000000021308472.log > -rw-r--r-- 1 yarn yarn 201288 Dec 2 21:51 00000000000021308472.index > -rw-r--r-- 1 yarn yarn 104857509 Dec 3 03:20 00000000000022430910.log > -rw-r--r-- 1 yarn yarn 201600 Dec 3 03:20 00000000000022430910.index > -rw-r--r-- 1 yarn yarn 104857562 Dec 3 09:14 00000000000023551782.log > -rw-r--r-- 1 yarn yarn 201696 Dec 3 09:14 00000000000023551782.index > -rw-r--r-- 1 yarn yarn 104857526 Dec 3 13:12 00000000000024671856.log > -rw-r--r-- 1 yarn yarn 201216 Dec 3 13:12 00000000000024671856.index > -rw-r--r-- 1 yarn yarn 104857591 Dec 3 17:03 00000000000025794239.log > -rw-r--r-- 1 yarn yarn 201152 Dec 3 17:03 00000000000025794239.index > -rw-r--r-- 1 yarn yarn 104857522 Dec 3 21:17 00000000000026917212.log > -rw-r--r-- 1 yarn yarn 201336 Dec 3 21:17 00000000000026917212.index > -rw-r--r-- 1 yarn yarn 104857160 Dec 4 02:31 00000000000028039274.log > -rw-r--r-- 1 yarn yarn 201624 Dec 4 02:31 00000000000028039274.index > -rw-r--r-- 1 yarn yarn 104857524 Dec 4 08:49 00000000000029160221.log > -rw-r--r-- 1 yarn yarn 201728 Dec 4 08:49 00000000000029160221.index > -rw-r--r-- 1 yarn yarn 104857579 Dec 4 13:09 00000000000030279870.log > -rw-r--r-- 1 yarn yarn 201344 Dec 4 13:09 00000000000030279870.index > -rw-r--r-- 1 yarn yarn 104857581 Dec 4 17:10 00000000000031401545.log > -rw-r--r-- 1 yarn yarn 201320 Dec 4 17:10 00000000000031401545.index > -rw-r--r-- 1 yarn yarn 104857338 Dec 4 21:36 00000000000032523747.log > -rw-r--r-- 1 yarn yarn 201456 Dec 4 21:36 00000000000032523747.index > -rw-r--r-- 1 yarn yarn 104857579 Dec 5 03:38 00000000000033645679.log > -rw-r--r-- 1 yarn yarn 201744 Dec 5 03:38 00000000000033645679.index > -rw-r--r-- 1 yarn yarn 104857511 Dec 5 09:07 00000000000034765482.log > -rw-r--r-- 1 yarn yarn 201656 Dec 5 09:07 00000000000034765482.index > -rw-r--r-- 1 yarn yarn 104857537 Dec 5 13:00 00000000000035885509.log > -rw-r--r-- 1 yarn yarn 201224 Dec 5 13:00 00000000000035885509.index > -rw-r--r-- 1 yarn yarn 104857594 Dec 5 16:43 00000000000037008566.log > -rw-r--r-- 1 yarn yarn 201056 Dec 5 16:43 00000000000037008566.index > -rw-r--r-- 1 yarn yarn 104857407 Dec 5 20:49 00000000000038132175.log > -rw-r--r-- 1 yarn yarn 201192 Dec 5 20:49 00000000000038132175.index > -rw-r--r-- 1 yarn yarn 104857598 Dec 6 02:34 00000000000039255111.log > -rw-r--r-- 1 yarn yarn 201568 Dec 6 02:34 00000000000039255111.index > -rw-r--r-- 1 yarn yarn 104857531 Dec 6 08:30 00000000000040375848.log > -rw-r--r-- 1 yarn yarn 201736 Dec 6 08:30 00000000000040375848.index > -rw-r--r-- 1 yarn yarn 104857567 Dec 6 12:23 00000000000041495389.log > -rw-r--r-- 1 yarn yarn 201288 Dec 6 12:23 00000000000041495389.index > -rw-r--r-- 1 yarn yarn 104857542 Dec 6 16:07 00000000000042618234.log > -rw-r--r-- 1 yarn yarn 201088 Dec 6 16:07 00000000000042618234.index > -rw-r--r-- 1 yarn yarn 10485760 Dec 6 17:40 00000000000043741475.index > -rw-r--r-- 1 yarn yarn 44560223 Dec 6 17:40 00000000000043741475.log > Prior to the thread exiting, we see the compacted directories containing only > about 6 files or 100MB max (as expected). -- This message was sent by Atlassian JIRA (v6.3.4#6332)