[
https://issues.apache.org/jira/browse/KUDU-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shenxingwuying updated KUDU-3390:
---------------------------------
Description:
The origin jira: https://issues.apache.org/jira/browse/KUDU-3061, and I create
a new Jira issus to record some infomations.
h1. Motivation
The number of leader replicas per tablet server can become imbalanced over
time, which lead to load skew on some nodes.
Two reasons of load skew:
* The main reason. Scan Requests has two modes: LeaderOnly(default) and
CLOSEST_REPLICA. For more accurate results, users will choose the
LeaderOnly(default) mode. Mostly, the scan load is positive correlation with
leader numbers.
* The other reason. Write requests, leaders receive write requests and
followers receive appendEntries(kudu is UpdateConsensus), the flow of
processing is a little different, which is hidden variables, maybe cause
imbalanced load. Leader rebalance will make leader and followers balanced and
eliminate hidden variables and make service more stable.
To deal with the situation, users use kudu CLI leader_step_down command and
write a script program to rebalance the leaders. That can be better. Leader
kudu-master can do c automatically.
h1. Solution
We can add an auto leader rebalance task to avoid leader replicas skew. Running
a periodic task do leader rebalance at kudu-master.
Leader rebalance only do leader transfer, do not copy replicas. The basic idea
is every tserver leaders' number : replicas' number = 1 : (replica_refactor -
1). This is the argrithms.
If we need leader rebalance, we'd better enable replicas rebalancer. If enable
leader rebalancer but disable auto rebalancer the algorithm work well but the
effect is not good. The algorithm can be convergence, and the algorithm's
target is every tserver' replicas, number of leader : number of follower is 1 :
(replica_refactor -1).
h1. Leader Rebalance results
I do some experiments for the effective. I have a cluster, 3 machines: 3 master
instances and 3 tserver instances.
I create a table with 40 tablets(partitions) and 3 replica_factor. And load a
lots of data (40000000 records).
I disabled the leader rebalance function, and manually leader transfer all
tablets to a tserver and run writes and scans.
Then I enabled the the leader rebalance function and runs scans. The workload
as below:
The Scan command: {{./kudu_tools/kudu perf table_scan $master_list Student
-columns=id,name,brief,age,score -num_threads=4 -nofill_cache
-replica_selection="LEADER"}}
40: 0: 0 means node1 : node2: node3
47%, 18%, 19% means node1 : node2: node3
|| ||leader ratio||scan cost||cpu usage||memory||io||
|before leader rebalance|40: 0: 0|811.586 s|47%, 18%, 19%|no changes|102MB/s
ioutil:55%, 8KB/s ioutil:2%, 64KB/s ioutil:3%|
|after leader rebalance|13: 14: 13|611.012 s|39%, 45%, 35%|no changes|53MB/s
ioutil:31%, 80MB/s ioutil:18%, 45MB/s ioutil:24%|
was:
The origin jira: https://issues.apache.org/jira/browse/KUDU-3061, and I create
a new Jira issus to record some infomations.
h1. Motivation
The number of leader replicas per tablet server can become imbalanced over
time, which lead to load skew on some nodes.
Two reasons of load skew:
* The main reason. Scan Requests has two modes: LeaderOnly(default) and
CLOSEST_REPLICA. For more accurate results, users will choose the
LeaderOnly(default) mode. Mostly, the scan load is positive correlation with
leader numbers.
* The other reason. Write requests, leaders receive write requests and
followers receive appendEntries(kudu is UpdateConsensus), the flow of
processing is a little different, which is hidden variables, maybe cause
imbalanced load. Leader rebalance will make leader and followers balanced and
eliminate hidden variables and make service more stable.
To deal with the situation, users use kudu CLI leader_step_down command and
write a script program to rebalance the leaders. That can be better. Leader
kudu-master can do c automatically.
h1. Solution
We can add an auto leader rebalance task to avoid leader replicas skew. Running
a periodic task do leader rebalance at kudu-master.
Leader rebalance only do leader transfer, do not copy replicas. The basic idea
is every tserver leaders' number : replicas' number = 1 : (replica_refactor -
1). This is the argrithms.
If we need leader rebalance, we'd better enable replicas rebalancer. If enable
leader rebalancer but disable auto rebalancer the algorithm work well but the
effect is not good. The algorithm can be convergence, and the algorithm's
target is every tserver' replicas, number of leader : number of follower is 1 :
(replica_refactor -1).
h1. Leader Rebalance results
I do some experiments for the effective. I have a cluster, 3 machines: 3 master
instances and 3 tserver instances.
I create a table with 40 tablets(partitions) and 3 replica_factor. And load a
lots of data (40000000 records).
I disabled the leader rebalance function, and manually leader transfer all
tablets to a tserver and run writes and scans.
Then I enabled the the leader rebalance function and runs scans. The workload
as below:
The Scan command: {{./kudu_tools/kudu perf table_scan $master_list Student
-columns=id,name,brief,age,score -num_threads=4 -nofill_cache
-replica_selection="LEADER"}}
|| ||leader ratio||scan cost||cpu usage||memory||io||
|before leader rebalance|40: 0: 0|811.586 s|47%, 18%, 19%|no changes|102MB/s
ioutil:55%, 8KB/s ioutil:2%, 64KB/s ioutil:3%|
|after leader rebalance|13: 14: 13|611.012 s|39%, 45%, 35%|no changes|53MB/s
ioutil:31%, 80MB/s ioutil:18%, node3: 45MB/s ioutil:24%|
> add new feature auto leader rebalancer
> --------------------------------------
>
> Key: KUDU-3390
> URL: https://issues.apache.org/jira/browse/KUDU-3390
> Project: Kudu
> Issue Type: New Feature
> Reporter: shenxingwuying
> Assignee: shenxingwuying
> Priority: Major
>
> The origin jira: https://issues.apache.org/jira/browse/KUDU-3061, and I
> create a new Jira issus to record some infomations.
>
>
> h1. Motivation
> The number of leader replicas per tablet server can become imbalanced over
> time, which lead to load skew on some nodes.
> Two reasons of load skew:
> * The main reason. Scan Requests has two modes: LeaderOnly(default) and
> CLOSEST_REPLICA. For more accurate results, users will choose the
> LeaderOnly(default) mode. Mostly, the scan load is positive correlation with
> leader numbers.
> * The other reason. Write requests, leaders receive write requests and
> followers receive appendEntries(kudu is UpdateConsensus), the flow of
> processing is a little different, which is hidden variables, maybe cause
> imbalanced load. Leader rebalance will make leader and followers balanced and
> eliminate hidden variables and make service more stable.
> To deal with the situation, users use kudu CLI leader_step_down command and
> write a script program to rebalance the leaders. That can be better. Leader
> kudu-master can do c automatically.
> h1. Solution
> We can add an auto leader rebalance task to avoid leader replicas skew.
> Running a periodic task do leader rebalance at kudu-master.
> Leader rebalance only do leader transfer, do not copy replicas. The basic
> idea is every tserver leaders' number : replicas' number = 1 :
> (replica_refactor - 1). This is the argrithms.
> If we need leader rebalance, we'd better enable replicas rebalancer. If
> enable leader rebalancer but disable auto rebalancer the algorithm work well
> but the effect is not good. The algorithm can be convergence, and the
> algorithm's target is every tserver' replicas, number of leader : number of
> follower is 1 : (replica_refactor -1).
> h1. Leader Rebalance results
> I do some experiments for the effective. I have a cluster, 3 machines: 3
> master instances and 3 tserver instances.
> I create a table with 40 tablets(partitions) and 3 replica_factor. And load a
> lots of data (40000000 records).
> I disabled the leader rebalance function, and manually leader transfer all
> tablets to a tserver and run writes and scans.
> Then I enabled the the leader rebalance function and runs scans. The workload
> as below:
> The Scan command: {{./kudu_tools/kudu perf table_scan $master_list Student
> -columns=id,name,brief,age,score -num_threads=4 -nofill_cache
> -replica_selection="LEADER"}}
>
> 40: 0: 0 means node1 : node2: node3
> 47%, 18%, 19% means node1 : node2: node3
>
> || ||leader ratio||scan cost||cpu usage||memory||io||
> |before leader rebalance|40: 0: 0|811.586 s|47%, 18%, 19%|no changes|102MB/s
> ioutil:55%, 8KB/s ioutil:2%, 64KB/s ioutil:3%|
> |after leader rebalance|13: 14: 13|611.012 s|39%, 45%, 35%|no changes|53MB/s
> ioutil:31%, 80MB/s ioutil:18%, 45MB/s ioutil:24%|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
