[jira] [Updated] (KAFKA-880) NoLeaderPartitionSet should be cleared before leader finder thread is started up
[ https://issues.apache.org/jira/browse/KAFKA-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-880: - Description: This was a recent regression. This could prevent the consumer from progressing because fetchers for the currently owned partitions may not be added (depending on the order that the map iterator yields). I think the fix should be simple - just clear the set after stopping the leader finder thread and stopping fetchers. [2013-04-25 17:06:38,377] WARN [sometopic-somehost-1366909575615-f801367d-leader-finder-thread] , Failed to find leader for Set([sometopic,11], [sometopic,25], [sometopic,24]) (kafka.consumer.ConsumerFetcherManager$Lead erFinderThread) java.util.NoSuchElementException: key not found: [sometopic,24] at scala.collection.MapLike$class.default(MapLike.scala:223) at scala.collection.immutable.Map$Map2.default(Map.scala:110) at scala.collection.MapLike$class.apply(MapLike.scala:134) at scala.collection.immutable.Map$Map2.apply(Map.scala:110) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread$$anonfun$doWork$4.apply(ConsumerFetcherManager.scala:81) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread$$anonfun$doWork$4.apply(ConsumerFetcherManager.scala:79) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:79) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) was: This was a recent regression. This could prevent the consumer from progressing because fetchers for the currently owned partitions may not be added (depending on the order that the map iterator yields). I think the fix should be simple - just clear the set after stopping the leader finder thread and stopping fetchers. {code} [2013-04-25 17:06:38,377] WARN [sometopic-somehost-1366909575615-f801367d-leader-finder-thread] , Failed to find leader for Set([sometopic,11], [sometopic,25], [sometopic,24]) (kafka.consumer.ConsumerFetcherManager$Lead erFinderThread) java.util.NoSuchElementException: key not found: [sometopic,24] at scala.collection.MapLike$class.default(MapLike.scala:223) at scala.collection.immutable.Map$Map2.default(Map.scala:110) at scala.collection.MapLike$class.apply(MapLike.scala:134) at scala.collection.immutable.Map$Map2.apply(Map.scala:110) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread$$anonfun$doWork$4.apply(ConsumerFetcherManager.scala:81) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread$$anonfun$doWork$4.apply(ConsumerFetcherManager.scala:79) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:79) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) {code} NoLeaderPartitionSet should be cleared before leader finder thread is started up Key: KAFKA-880 URL: https://issues.apache.org/jira/browse/KAFKA-880 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Joel Koshy Fix For: 0.8 This was a recent regression. This could prevent the consumer from progressing because fetchers for the currently owned partitions may not be added (depending on the order that the map iterator yields). I think the fix should be simple - just clear the set after stopping the leader finder thread and stopping fetchers. [2013-04-25 17:06:38,377] WARN [sometopic-somehost-1366909575615-f801367d-leader-finder-thread] , Failed to find leader for Set([sometopic,11], [sometopic,25], [sometopic,24])
[jira] [Updated] (KAFKA-880) NoLeaderPartitionSet should be cleared before leader finder thread is started up
[ https://issues.apache.org/jira/browse/KAFKA-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-880: -- Attachment: kafka-880.patch Good catch. Attach a patch. NoLeaderPartitionSet should be cleared before leader finder thread is started up Key: KAFKA-880 URL: https://issues.apache.org/jira/browse/KAFKA-880 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Joel Koshy Fix For: 0.8 Attachments: kafka-880.patch This was a recent regression. This could prevent the consumer from progressing because fetchers for the currently owned partitions may not be added (depending on the order that the map iterator yields). I think the fix should be simple - just clear the set after stopping the leader finder thread and stopping fetchers. [2013-04-25 17:06:38,377] WARN [sometopic-somehost-1366909575615-f801367d-leader-finder-thread] , Failed to find leader for Set([sometopic,11], [sometopic,25], [sometopic,24]) (kafka.consumer.ConsumerFetcherManager$Lead erFinderThread) java.util.NoSuchElementException: key not found: [sometopic,24] at scala.collection.MapLike$class.default(MapLike.scala:223) at scala.collection.immutable.Map$Map2.default(Map.scala:110) at scala.collection.MapLike$class.apply(MapLike.scala:134) at scala.collection.immutable.Map$Map2.apply(Map.scala:110) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread$$anonfun$doWork$4.apply(ConsumerFetcherManager.scala:81) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread$$anonfun$doWork$4.apply(ConsumerFetcherManager.scala:79) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:79) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-880) NoLeaderPartitionSet should be cleared before leader finder thread is started up
[ https://issues.apache.org/jira/browse/KAFKA-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-880: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review. Committed to 0.8. NoLeaderPartitionSet should be cleared before leader finder thread is started up Key: KAFKA-880 URL: https://issues.apache.org/jira/browse/KAFKA-880 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Joel Koshy Assignee: Jun Rao Fix For: 0.8 Attachments: kafka-880.patch This was a recent regression. This could prevent the consumer from progressing because fetchers for the currently owned partitions may not be added (depending on the order that the map iterator yields). I think the fix should be simple - just clear the set after stopping the leader finder thread and stopping fetchers. [2013-04-25 17:06:38,377] WARN [sometopic-somehost-1366909575615-f801367d-leader-finder-thread] , Failed to find leader for Set([sometopic,11], [sometopic,25], [sometopic,24]) (kafka.consumer.ConsumerFetcherManager$Lead erFinderThread) java.util.NoSuchElementException: key not found: [sometopic,24] at scala.collection.MapLike$class.default(MapLike.scala:223) at scala.collection.immutable.Map$Map2.default(Map.scala:110) at scala.collection.MapLike$class.apply(MapLike.scala:134) at scala.collection.immutable.Map$Map2.apply(Map.scala:110) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread$$anonfun$doWork$4.apply(ConsumerFetcherManager.scala:81) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread$$anonfun$doWork$4.apply(ConsumerFetcherManager.scala:79) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:79) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira