[ https://issues.apache.org/jira/browse/KAFKA-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ismael Juma updated KAFKA-3917: ------------------------------- Affects Version/s: (was: 0.8.2.2) 0.10.2.0 > Some __consumer_offsets replicas grow way too big > ------------------------------------------------- > > Key: KAFKA-3917 > URL: https://issues.apache.org/jira/browse/KAFKA-3917 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 0.10.2.0 > Environment: Runs with Docker 1.10.1 in a container on > Linux 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 > Reporter: Maxim Vladimirskiy > Labels: reliability > > We noticed that some replicas of partitions of the __consumer_offsets topic > grow way too big. Looking inside respective folders it became apparent that > old segments had not been cleaned up. Please see below example of disk usage > data for both affected and not affected partitions: > Not affected partitions: > Partition: 0 Leader: 2 Replicas: 2,3,4 Isr: 2,4,3 > 2: 49M > 3: 49M > 4: 49M > Affected partitions: > Partition: 10 Leader: 2 Replicas: 2,0,1 Isr: 1,2,0 > 0: 86M > 1: 22G <<< too big! > 2: 86M > Partition: 38 Leader: 0 Replicas: 0,4,1 Isr: 1,0,4 > 0: 43M > 1: 26G <<< too big! > 4: 26G <<< too big! > As you can see sometimes only one replica is affected, sometimes both > replicas are affected. > When I try to restart a broker that has affected replicas it fails to start > with an exception that looks like this: > [2016-06-28 23:15:20,441] ERROR There was an error in one of the threads > during logs loading: java.lang.IllegalArgumentException: requirement failed: > Corrupt index found, index file > (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has non-zero > size but the last offset is -676703869 and the base offset is 0 > (kafka.log.LogManager) > [2016-06-28 23:15:20,442] FATAL [Kafka Server 1], Fatal error during > KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) > java.lang.IllegalArgumentException: requirement failed: Corrupt index found, > index file (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has > non-zero size but the last offset is -676703869 and the base offset is 0 > at scala.Predef$.require(Predef.scala:233) > at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) > at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) > at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:183) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at kafka.log.Log.loadSegments(Log.scala:183) > at kafka.log.Log.<init>(Log.scala:67) > at > kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142) > at kafka.utils.Utils$$anon$1.run(Utils.scala:54) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > After the content of the affected partition is deleted broker starts > successfully. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)