Todd Palino created KAFKA-2012:
----------------------------------
Summary: Broker should automatically handle corrupt index files
Key: KAFKA-2012
URL: https://issues.apache.org/jira/browse/KAFKA-2012
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Todd Palino
We had a bunch of unclean system shutdowns (power failure), which caused
corruption on our disks holding log segments in many cases. While the broker is
handling the log segment corruption properly (truncation), it is having
problems with corruption in the index files. Additionally, this only seems to
be happening on some startups (while we are upgrading).
The broker should just do what I do when I hit a corrupt index file - remove it
and rebuild it.
2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] []
Fatal error during KafkaServerStartable startup. Prepare to shutdown
java.lang.IllegalArgumentException: requirement failed: Corrupt index found,
index file
(/export/content/kafka/i001_caches/__consumer_offsets-39/00000000000000000000.index)
has non-zero size but the last offset is -2121629628 and the base offset is 0
at scala.Predef$.require(Predef.scala:233)
at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.log.Log.loadSegments(Log.scala:184)
at kafka.log.Log.<init>(Log.scala:82)
at
kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141)
at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
