That shouldn¹t do anything, however if you had the following:
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[1,2,3]}
}
And then submitted:
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas²:[2,1,3]} }
The leader would be moved from 1->2
-Clark
Cla
Thanks for all your pieces of advice!
I am working on my script to improve it.
I am trying to find a way to select the best partition to relocate it to
the newly added broker (taking in account the number of partition leaded by
the replica brokers).
I really want to avoid swaps to rebalance the
I have written such a script. It balances the cluster by the data size on
disk. It is written using lots of internal tools which is why its not
open-sourced. I plan to re-write it without the internal tooling.
In terms of leader balancing, when using the partition-reassignemnt
script, whichever br
Thanks for your answer,
Indeed, I have already worked on this kind of script. I ended up with 800
lines of groovy script that rebalance partitions across the cluster and
minimizing the number of partition moves. I also worked on the partition
leadership balancing.
I still have to work on my scri
What you are doing should work. Although you might find better results by
creating the partition assignment JSON file on your own rather than using
the move topics to brokers JSON file to generate it. Doing this yourself
will allow you to optimize the number of partition moves you are making
and ho
Yes exactly, I am just trying to rebalance partitions to the new node.
I just want to balance it evently across the all cluster.
Am I doing wrong?
On Thu, Jul 10, 2014 at 9:58 AM, Clark Haskins <
chask...@linkedin.com.invalid> wrote:
> I am confused as to exactly what you are trying to accompl
I am confused as to exactly what you are trying to accomplish. Are you
trying to evenly distribute partitions across your newly added nodes?
-Clark
Clark Elliott Haskins III
LinkedIn DDS Site Reliability Engineer
Kafka, Zookeeper, Samza SRE
Mobile: 505.385.1484
BlueJeans: https://www.bluejeans.co
Since you have replication factor 3 and only 3 brokers, you can't move data
around in the existing cluster since a partition can have at most replica
on each broker. You will need to add the new brokers in first before
running the reassignment tool.
Thanks,
Jun
On Tue, Jul 8, 2014 at 10:26 AM,
I realized that I did not respond to you Clark,
Here is the entire Json that I sliced in multiple pieces:
{"version":1,"partitions":[{"topic":"SLOTS","partition":20,"replicas":[101421743,105114483,101461702]},{"topic":"RTB","partition":12,"replicas":[101671664,101812541,101862816]},{"topic":"B_IM
Here is the entire logic to rebalance the cluster which is done by this
groovy script (
https://github.com/Lowess/Kafka/blob/master/KafkaPartitionRebalancer.groovy)
#1: Request the zookeeper and get the broker id list
#2: Request zookeeper and get the list of topic
#3: Generate the topic-to-move.j
Can you copy/paste the json you are passing to the reassignment tool? Plus
the commands. Also do a describe on your topics.
-Clark
Clark Elliott Haskins III
LinkedIn DDS Site Reliability Engineer
Kafka, Zookeeper, Samza SRE
Mobile: 505.385.1484
BlueJeans: https://www.bluejeans.com/chaskins
chas
I let the tool running for an entire weekend on the test cluster and on
Monday it was still saying "failed"...
I have 500 Go per Kafka node and it is a 8 nodes cluster.
I am also wondering if I am using the tool correctly. Currently I am
running the tool to rebalance everything across the entire
The failure could mean that the reassignment is still in progress. If you
have lots of data, it may take some time to move the data to new brokers.
You could observe the max lag in each broker to see how far behind new
replicas are (see http://kafka.apache.org/documentation.html#monitoring).
Thank
When I run the tool with the --verify option it says failed for the some
partitions.
The problem is I do not know if it is a zookeeper issue or if the tool
really failed.
I faced one time the zookeeper issue (
https://issues.apache.org/jira/browse/KAFKA-1382) and by killing the
responsible Kafka
How does it get stuck?
-Clark
Clark Elliott Haskins III
LinkedIn DDS Site Reliability Engineer
Kafka, Zookeeper, Samza SRE
Mobile: 505.385.1484
BlueJeans: https://www.bluejeans.com/chaskins
chask...@linkedin.com
https://www.linkedin.com/in/clarkhaskins
There is no place like 127.0.0.1
On 7/
Hi,
I am trying to add new brokers to an existing 8 nodes Kafka cluster. We
have around 10 topics and the number of partition is set to 50. In order to
test the reassgin-partitions scripts, I tried on a sandbox cluster the
following steps.
I developed a script which is able to parse the reassignm
16 matches
Mail list logo
