The command line tool that ships with Kafka is error prone. Our standard procedure is: 1. spin up the new broker 2. use `kafkat drain <old broker id> [--brokers <new broker id>] 3. shut down old broker
The `drain` command will generate and submit a partition assignment plan where the new broker id replaces the old one. It's pretty much a "gsub(old, new)". We do it regularly. It's almost a mundane operation. The only challenge is the volume of data being transferred over the network. Since there is no throttling mechanism, the network is sometime saturated which might impact other consumers/producers See https://github.com/airbnb/kafkat On Fri, Mar 4, 2016 at 7:28 AM Todd Palino <tpal...@gmail.com> wrote: > To answer your questions… > > 1 - Not in the way you want it to. There is a setting for automatic leader > election (which I do not recommend anyone use at this time), but all that > does is pick which of the currently assigned replicas should be the leader. > It does not reassign partitions from one broker to another. Kafka does not > have a facility for doing this automatically. > > 2 - No. The most you can do is move all the partitions off and then > immediately shut down the broker process. Any broker that is live in the > cluster can, and will, get partitions assigned to it by the controller. > > For what you want to do, you need you use the partition reassignment > command line tool that ships with Kafka to reassign partitions from the old > broker to the new one. Once that is complete, you can double check that the > old broker has no partitions left and shut it down. I have a tool that we > use internally to make this a lot easier, and I’m in the process of getting > a repository set up to make it available via open source. It allows for > more easily removing and adding brokers, and rebalancing partitions in a > cluster without having to craft the reassignments by hand. > > -Todd > > > On Fri, Mar 4, 2016 at 5:07 AM, Muqtafi Akhmad <muqt...@traveloka.com> > wrote: > > > dear Kafka users, > > > > I have some questions regarding decommissioning kafka broker node and > > replacing it with the new one. Lets say that we have three broker nodes > and > > each topic in Kafka has replication factor = 3, we upgrade one node with > > the following steps : > > 1. add one broker node to cluster > > 2. shutdown old broker node > > > > My questions are > > 1. When we add one new broker to the cluster will it trigger Kafka topic > / > > group leadership rebalance? > > 2. Is there any way to disable the to-be-decommissioned node to hold no > > topic/group leadership (acting as passive copy) so that it can be > > decommissioned with minimal effect to Kafka clients? > > > > Thank you, > > > > -- > > Muqtafi Akhmad > > Software Engineer > > Traveloka > > > > > > -- > *—-* > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
