Yes, the replicas are stored in Zookeeper, so you can iterate over the information there to build a view of the cluster that you can use. If you want an example for this, take a look at the code for kafka-assigner in https://github.com/linkedin/kafka-tools. Or you can just use that tool to adjust replication factors and balance partitions.
-Todd On Fri, Dec 22, 2017 at 9:21 AM, Sagar <sagarmeansoc...@gmail.com> wrote: > Hi Todd, > > Thanks for the reply. Problem is I have about 160 topics(5 partitions for > each) for which I need to increase the replication factors for. So, I would > have to find the current leader for each of the partitions and hand code > the json which would become tedious. > > The partition leader info is stored in zookeeper? If yes, then can I > probably use that get the current json and then build the json which can be > fed to the tool. I tried to search using the zk shell but couldn't find... > > Sagar. > > On Fri, Dec 22, 2017 at 7:45 PM, Todd Palino <tpal...@gmail.com> wrote: > > > Preferred replica election is naive. It will always follow the order of > the > > replicas as they are set. So if you want to set the default leader, just > > make it the first replica in the list for the partition. We build the > JASON > > this way all the time. > > > > -Todd > > > > > > On Dec 22, 2017 6:46 AM, "Sagar" <sagarmeansoc...@gmail.com> wrote: > > > > Hi, > > > > Had a question on Kafka reassign partitions tool. > > > > We have a 3 node cluster but our replication factor is set to 1. So we > have > > been looking to increase it to 3 for HA. > > > > I tried the tool on a couple of topics and it increases the replication > > factor alright. Also it doesn't change the leader as the leader still is > in > > the RAR. > > > > This is how I run it: > > > > Json which is used: > > > > {"version":1,"partitions":[ > > > > {"topic":"cric-engine.engine.eng_fow","partition":3," > > replicas":[92,51,101]}]} > > > > Earlier config for the topic > > > > kafka-topics --describe --topic cric-engine.engine.eng_fow --zookeeper > > 10.0.4.165:2181,10.0.5.139:2181,10.0.6.106:2181 > > Topic:cric-engine.engine.eng_fow PartitionCount:5 ReplicationFactor:3 > > Configs: > > Topic: cric-engine.engine.eng_fow Partition: 0 Leader: 101 Replicas: > > 92,51,101 Isr: 101,51,92 > > Topic: cric-engine.engine.eng_fow Partition: 1 Leader: 51 Replicas: > > 92,51,101 Isr: 51,101,92 > > Topic: cric-engine.engine.eng_fow Partition: 2 Leader: 92 Replicas: 92 > Isr: > > 92 > > Topic: cric-engine.engine.eng_fow Partition: 3 Leader: 101 Replicas: 101 > > Isr: > > 101 > > Topic: cric-engine.engine.eng_fow Partition: 4 Leader: 51 Replicas: 51 > Isr: > > 51 > > > > After running: > > > > kafka-reassign-partitions --reassignment-json-file > > increase-replication-factor.json --execute --zookeeper 10.0.4.165:2181, > > 10.0.5.139:2181,10.0.6.106:2181 > > > > partitions 3 Replicas increase: > > > > kafka-topics --describe --topic cric-engine.engine.eng_fow --zookeeper > > 10.0.4.165:2181,10.0.5.139:2181,10.0.6.106:2181 > > Topic:cric-engine.engine.eng_fow PartitionCount:5 ReplicationFactor:3 > > Configs: > > Topic: cric-engine.engine.eng_fow Partition: 0 Leader: 101 Replicas: > > 92,51,101 Isr: 101,51,92 > > Topic: cric-engine.engine.eng_fow Partition: 1 Leader: 51 Replicas: > > 92,51,101 Isr: 51,101,92 > > Topic: cric-engine.engine.eng_fow Partition: 2 Leader: 92 Replicas: 92 > Isr: > > 92 > > Topic: cric-engine.engine.eng_fow Partition: 3 Leader: 101 Replicas: > > 92,51,101 Isr: 101,51,92 > > Topic: cric-engine.engine.eng_fow Partition: 4 Leader: 51 Replicas: 51 > Isr: > > 51 > > > > What I wanted to know is that does it affect the preferred replica? If > you > > see the Replicas, all of them are now 92,51,101 even though the leader > has > > remained the same from before. So, if any of the broker goes down or we > > run kafka-preferred-replica-election.sh, wouldn't it move all the > leaders > > to broker 92? Is my assesment correct? > > > > If yes, then is there a way I can still do this operation by getting > leader > > for a partition first, then adding it to the replica list and then > building > > the json dynamically? > > > > Thanks! > > Sagar. > > > -- *Todd Palino* Senior Staff Engineer, Site Reliability Data Infrastructure Streaming linkedin.com/in/toddpalino