Vamsi Subhash Achanta created KAFKA-1791:
--------------------------------------------
Summary: Corrupt index after safe shutdown and restart
Key: KAFKA-1791
URL: https://issues.apache.org/jira/browse/KAFKA-1791
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1
Environment: Debian6 with Sun-Java6
Reporter: Vamsi Subhash Achanta
Priority: Critical
We have 3 kafka brokers - all VMs. One of the broker was stopped for around 30
minutes to fix a problem with the bare metal. Upon restart, after some time,
the broker went out of file-descriptors (FDs) and started throwing errors. Upon
restart, it started throwing this corrupted index exceptions. I followed the
other JIRAs related to corrupted indices but the solutions mentioned there
(deleting the index and restart) didn't work - the index gets created again.
The other JIRAs solution of deleting those indexes which got wrongly compacted
(> 10MB) didn't work either.
What is the error? How can I fix this and bring back the broker? Thanks.
INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Found clean
shutdown file. Skipping recovery for all logs in data directory
'/var/lib/fk-3p-kafka/logs'
INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Loading log
'kf.production.b2b.return_order.status-25'
FATAL [2014-11-21 02:57:17,533] [main][] kafka.server.KafkaServerStartable -
Fatal error during KafkaServerStable startup. Prepare to shutdown
java.lang.IllegalArgumentException: requirement failed: Corrupt index found,
index file
(/var/lib/fk-3p-kafka/logs/kf.production.b2b.return_order.status-25/00000000000000000233.index)
has non-zero size but the last offset is 233 and the base offset is 233
at scala.Predef$.require(Predef.scala:145)
at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158)
at scala.collection.Iterator$class.foreach(Iterator.scala:631)
at
scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
at
scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495)
at kafka.log.Log.loadSegments(Log.scala:158)
at kafka.log.Log.<init>(Log.scala:64)
at
kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at
kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
INFO [2014-11-21 02:57:17,534] [main][] kafka.server.KafkaServer - [Kafka
Server 2], shutting down
INFO [2014-11-21 02:57:17,538] [main][] kafka.server.KafkaServer - [Kafka
Server 2], shut down completed
INFO [2014-11-21 02:57:17,539] [Thread-2][] kafka.server.KafkaServer - [Kafka
Server 2], shutting down
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
