After reassigning a topic to 5 new brokers, the topic metadata returned by
the brokers for the topic is causing an exception when a producer tries to
read it:
java.util.NoSuchElementException: key not found: 7
...
at kafka.api.PartitionMetadata$.readFrom(TopicMetadata.scala:104)
The reassignment was from brokers 1, 4, 5, 7, 8 to 10, 11, 12, 13, 14. In
the topic metadata response, only the 5 new brokers (>= 10) are included,
but there is a partition whose ISR includes broker 7. It looks like this:
{:error-code 0, :partition-id 13, :leader-id 10, :num-replicas 4,
:replica-ids [10 12 13 14], :num-isr 5, :isr-ids [10 14 13 12 7]}
For all other partitions, the ISR and the replicas are the same brokers,
but for this partition, the ISR is larger. I stopped and started the
controller, and the and topic metadata was corrected.
Has anyone seen this before? Is there anything in particular to look for in
the logs?
Thanks,
Ryan
Here's the full exception:
java.util.NoSuchElementException: key not found: 7
at scala.collection.MapLike$class.default(MapLike.scala:225)
at scala.collection.immutable.HashMap.default(HashMap.scala:38)
at scala.collection.MapLike$class.apply(MapLike.scala:135)
at scala.collection.immutable.HashMap.apply(HashMap.scala:38)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.immutable.VectorIterator.foreach(Vector.scala:648)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
at scala.collection.immutable.Vector.foreach(Vector.scala:63)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.immutable.Vector.map(Vector.scala:63)
at kafka.api.PartitionMetadata$.readFrom(TopicMetadata.scala:104)
at kafka.api.TopicMetadata$$anonfun$readFrom$1.apply(TopicMetadata.scala:37)
at kafka.api.TopicMetadata$$anonfun$readFrom$1.apply(TopicMetadata.scala:36)
at scala.collection.immutable.Range.foreach(Range.scala:78)
at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
at
kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
at
kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.immutable.Range.foreach(Range.scala:81)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.immutable.Range.map(Range.scala:46)
at kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at
kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:49)
at
kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:186)
at
kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:150)
at
kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:149)
at
kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:95)
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:42)
