A few things here… 1) auto.leader.rebalance.enable can have serious performance impacts on larger clusters. It’s currently in need of some development work to enable it to batch leader elections into smaller groups and back off between them, as well as have a better backoff after broker startup. I don’t recommend using it.
2) auto.leader.rebalance.enable is not going to get you what you’re looking for. It only changes the leader for a partition to the “optimal” leader (I put that in quotes because it’s a pretty dumb algorithm. It’s whichever replica is listed first). It does not move partitions around to assure you have a balance of traffic across the cluster. If you want to rebalance partitions, you have a couple options right now: 1) Run kafka-reassign-partitions.sh. It will move all of the partitions around and try and assure an even count on each broker. It does not balance traffic, however, (if you have a really busy partition and a really slow partition, it considers them equal). 2) Use an external tool like https://github.com/linkedin/kafka-tools kafka-assigner. This is a script we developed at LinkedIn for doing operations that involve moving partitions around and provides a number of different ways to rebalance traffic. There are other tools available for doing this, but right now it requires something external to the Apache Kafka core. -Todd On Tue, Jun 13, 2017 at 5:30 PM, karan alang <karan.al...@gmail.com> wrote: > Hi All, > > Fpr Re-balancing Kafka partitions, we can set property -> > > > *auto.leader.rebalance.enable = true in server.properties file.* > > *Is that the recommended way or is it better to reBalance the kafka > partitions manually ?(using *scripts - *kafka-preferred-replica- > election.sh, > * > > *kafka-reassign-partition.sh)* > *One of the blogs mentioned that - it is preferable to Re-balance Kafka > topics manually, since setting * > > *auto.leader.rebalance.enable = true causes issues.* > > Pls let me know. > Any other best practices wrt. Re-balancing kafka topics ? > > thanks! > -- *Todd Palino* Senior Staff Engineer, Site Reliability Data Infrastructure Streaming linkedin.com/in/toddpalino
