Hi, We've had an issue on a live system (3 brokers, ~10 topics, some replicated, some partitioned) where a partition wasn't properly reassigned, causing several other partitions to go down.
First, this exception happened on broker 1 (we weren't doing anything particular on the system at the time): ERROR [AddPartitionsListener on 1]: Error while handling add partitions for data path /brokers/topics/topic1 (kafka.controller.PartitionStateMachine$AddPartitionsListener) java.util.NoSuchElementException: key not found: [topic1,0] at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:58) at scala.collection.mutable.HashMap.apply(HashMap.scala:64) at kafka.controller.ControllerContext$$anonfun$replicasForPartition$1.apply(KafkaController.scala:112) at kafka.controller.ControllerContext$$anonfun$replicasForPartition$1.apply(KafkaController.scala:111) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Set$Set1.foreach(Set.scala:74) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47) at scala.collection.SetLike$class.map(SetLike.scala:93) at scala.collection.AbstractSet.map(Set.scala:47) at kafka.controller.ControllerContext.replicasForPartition(KafkaController.scala:111) at kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:485) at kafka.controller.PartitionStateMachine$AddPartitionsListener$$anonfun$handleDataChange$1.apply$mcV$sp(PartitionStateMachine.scala:530) at kafka.controller.PartitionStateMachine$AddPartitionsListener$$anonfun$handleDataChange$1.apply(PartitionStateMachine.scala:519) at kafka.controller.PartitionStateMachine$AddPartitionsListener$$anonfun$handleDataChange$1.apply(PartitionStateMachine.scala:519) at kafka.utils.Utils$.inLock(Utils.scala:535) at kafka.controller.PartitionStateMachine$AddPartitionsListener.handleDataChange(PartitionStateMachine.scala:518) at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) At this point, broker 2 started continually spammed these messages (mentioning other topics, not just topic1): ERROR [ReplicaFetcherThread-0-1], Error for partition [othertopic1,2] to broker 1:class kafka.common.UnknownException (kafka.server.ReplicaFetcherThread) ERROR [ReplicaFetcherThread-0-1], Error for partition [othertopic2,0] to broker 1:class kafka.common.UnknownException (kafka.server.ReplicaFetcherThread) ERROR [ReplicaFetcherThread-0-1], Error for partition [othertopic3,0] to broker 1:class kafka.common.UnknownException (kafka.server.ReplicaFetcherThread) ERROR [ReplicaFetcherThread-0-1], Error for partition [topic1,0] to broker 1:class kafka.common.UnknownException (kafka.server.ReplicaFetcherThread) And broker 1 had these messages, but only for topic1: ERROR [KafkaApi-1] error when handling request Name: FetchRequest; Version: 0; CorrelationId: 41182755; ClientId: ReplicaFetcherThread-0-1; ReplicaId: 2; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [topic1,0] -> PartitionFetchInfo(0,1048576) (kafka.server.KafkaApis) kafka.common.NotAssignedReplicaException: Leader 1 failed to record follower 2's position 0 since the replica is not recognized to be one of the assigned replicas 1 for partition [topic1,0] at kafka.server.ReplicaManager.updateReplicaLEOAndPartitionHW(ReplicaManager.scala:574) at kafka.server.KafkaApis$$anonfun$recordFollowerLogEndOffsets$2.apply(KafkaApis.scala:388) at kafka.server.KafkaApis$$anonfun$recordFollowerLogEndOffsets$2.apply(KafkaApis.scala:386) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at scala.collection.MapLike$MappedValues.foreach(MapLike.scala:245) at kafka.server.KafkaApis.recordFollowerLogEndOffsets(KafkaApis.scala:386) at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:351) at kafka.server.KafkaApis.handle(KafkaApis.scala:60) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59) at java.lang.Thread.run(Thread.java:745) At this time, any topic that had broker 1 as a leader were not working. ZK thought that everything was ok and in sync. Restarting broker 1 fixed the broken topics for a bit, until broker 1 was reassigned as leader of some topics, at which point it broke again. Restarting broker 2 fixed it (!!!!). We're using kafka-2.10.0_0.8.2.0. Could anyone explain what happened, and (most importantly) how we stop it happening again in the future? Many thanks, SimonC