Marcin, that is a typical task now. What version of Kafka are you running?
Take a look at https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion and https://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor Basically you can do a --generate to get existing JSON topology and with that take the results of "Current partition replica assignment" (the first JSON that outputs) and make whatever changes (like sed old node for new node and add more replica's which increase the replication factor, whatever you want) and then --execute. With lots of data this takes time so you will want to run --verify to see what is in progress... good thing do a node at a time (even topic at a time) however you want to manage and wait for it as such. The "preferred" replica is simply the first one in the list of replicas. The kafka-preferred-replica-election.sh just makes that replica the leader as this is not automatic yet. If you are running a version prior to 0.8.1.1 it might make sense to upgrade the old nodes first then run reassign to the new servers. /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Mon, Aug 25, 2014 at 8:59 PM, Marcin Michalski <mmichal...@tagged.com> wrote: > Hi, I would like to migrate my Kafka setup from old servers to new servers. > Let say I have 8 really old servers that have the kafka topics/partitions > replicated 4 ways and want to migrate the data to 4 brand new servers and > want the replication factor be 3. I wonder if anyone has ever performed > this type of migration? > > Will auto rebalancing take care of this automatically if I do the > following? > > Let say I bring down old broker id 1 down and startup new server broker id > 100 up, is there a way to migrate all of the data of the topic that had the > topic (where borker id 1 was the leader) over to the new broker 100? > > Or do I need to use *bin/kafka-preferred-replica-election.sh *to reassign > the topics/partitions from old broker 1 to broker 100? And then just keep > doing the same thing until all of the old brokers are decommissioned? > > Also, would kafka-preferred-replica-election.sh let me actually lower the > number of replicas as well, if I just simply make sure that given > topic/partition was only elected 3 times versus 4? > > Thanks for your insight, > Marcin >
