Thanks Todd.. for the detailed reply. regds, Karan Alang
On Tue, Jun 13, 2017 at 3:19 PM, Todd Palino <[email protected]> wrote: > A few things here… > > 1) auto.leader.rebalance.enable can have serious performance impacts on > larger clusters. It’s currently in need of some development work to enable > it to batch leader elections into smaller groups and back off between them, > as well as have a better backoff after broker startup. I don’t recommend > using it. > > 2) auto.leader.rebalance.enable is not going to get you what you’re looking > for. It only changes the leader for a partition to the “optimal” leader (I > put that in quotes because it’s a pretty dumb algorithm. It’s whichever > replica is listed first). It does not move partitions around to assure you > have a balance of traffic across the cluster. > > If you want to rebalance partitions, you have a couple options right now: > 1) Run kafka-reassign-partitions.sh. It will move all of the partitions > around and try and assure an even count on each broker. It does not balance > traffic, however, (if you have a really busy partition and a really slow > partition, it considers them equal). > 2) Use an external tool like https://github.com/linkedin/kafka-tools > kafka-assigner. This is a script we developed at LinkedIn for doing > operations that involve moving partitions around and provides a number of > different ways to rebalance traffic. > > There are other tools available for doing this, but right now it requires > something external to the Apache Kafka core. > > -Todd > > > On Tue, Jun 13, 2017 at 5:30 PM, karan alang <[email protected]> > wrote: > > > Hi All, > > > > Fpr Re-balancing Kafka partitions, we can set property -> > > > > > > *auto.leader.rebalance.enable = true in server.properties file.* > > > > *Is that the recommended way or is it better to reBalance the kafka > > partitions manually ?(using *scripts - *kafka-preferred-replica- > > election.sh, > > * > > > > *kafka-reassign-partition.sh)* > > *One of the blogs mentioned that - it is preferable to Re-balance Kafka > > topics manually, since setting * > > > > *auto.leader.rebalance.enable = true causes issues.* > > > > Pls let me know. > > Any other best practices wrt. Re-balancing kafka topics ? > > > > thanks! > > > > > > -- > *Todd Palino* > Senior Staff Engineer, Site Reliability > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
