Patrick Lucas created KAFKA-2269:
------------------------------------
Summary: Partition imbalance does not converge to 0, even with
auto.leader.rebalance.enable=true
Key: KAFKA-2269
URL: https://issues.apache.org/jira/browse/KAFKA-2269
Project: Kafka
Issue Type: Bug
Components: controller
Affects Versions: 0.8.2.1
Reporter: Patrick Lucas
Assignee: Neha Narkhede
Attachments: graph.png
In the past four days I have replaced six brokers in a high-volume cluster. As
the new broker comes up and replicates the data it is responsible for, the
cluster-wide partition imbalance trends toward zero. But predictably, with
around 19 partitions to go, the partition imbalance levels off and never
reaches zero, even after all partitions have a full ISR. I waited as long as 24
hours to experiment with this, and in each case I had to manually run a
preferred replica election to correct the imbalance.
The state-change log file on the controller does have a traceback for each
partition contributing to the imbalance, as included below.
!graph.png!
{noformat}
[2015-06-15 09:43:03,034] ERROR Controller 172337636 epoch 113 encountered
error while electing leader for partition [my.topic,1] due to: Preferred
replica 172340284 for partition [my.topic,1] is either not alive or not in the
isr. Current leader and ISR:
[{"leader":172314088,"leader_epoch":923,"isr":[172314088,172322941]}].
(state.change.logger)
[2015-06-15 09:43:03,034] ERROR Controller 172337636 epoch 113 initiated state
change for partition [my.topic,1] from OnlinePartition to OnlinePartition
failed (state.change.logger)
kafka.common.StateChangeFailedException: encountered error while electing
leader for partition [my.topic,1] due to: Preferred replica 172340284 for
partition [my.topic,1] is either not alive or not in the isr. Current leader
and ISR: [{"leader":172314088,"leader_epoch":923,"isr":[172314088,172322941]}].
at
kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:380)
at
kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:208)
at
kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:146)
at
kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:145)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at
kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:145)
at
kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:631)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1158)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1153)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1153)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1150)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1148)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1148)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1127)
at
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
kafka.controller.KafkaController.kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance(KafkaController.scala:1127)
at
kafka.controller.KafkaController$$anonfun$onControllerFailover$1.apply$mcV$sp(KafkaController.scala:326)
at
kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:99)
at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: kafka.common.StateChangeFailedException: Preferred replica 172340284
for partition [my.topic,1] is either not alive or not in the isr. Current
leader and ISR:
[{"leader":172314088,"leader_epoch":923,"isr":[172314088,172322941]}]
at
kafka.controller.PreferredReplicaPartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:159)
at
kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:357)
... 32 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
