[ https://issues.apache.org/jira/browse/HBASE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Manning updated HBASE-22349: ---------------------------------- Component/s: Balancer > Stochastic Load Balancer skips balancing when node is replaced in cluster > ------------------------------------------------------------------------- > > Key: HBASE-22349 > URL: https://issues.apache.org/jira/browse/HBASE-22349 > Project: HBase > Issue Type: Bug > Components: Balancer > Affects Versions: 3.0.0-alpha-1, 1.3.0, 1.4.4, 2.0.0 > Reporter: Suthan Phillips > Assignee: David Manning > Priority: Major > Attachments: Hbase-22349.pdf > > > HBASE-24139 allows the load balancer to run when one server has 0 regions and > another server has more than 1 region. This is a special case of a more > generic problem, where one server has far too few or far too many regions. > The StochasticLoadBalancer defaults may decide the cluster is "balanced > enough" according to {{hbase.master.balancer.stochastic.minCostNeedBalance}}, > even though one server may have a far higher or lower number of regions > compared to the rest of the cluster. > One specific example of this we have seen is when we use {{RegionMover}} to > move regions back to a restarted RegionServer, if the > {{StochasticLoadBalancer}} happens to be running. The load balancer sees a > newly restarted RegionServer with 0 regions, and after HBASE-24139, it will > balance regions to this server. Simultaneously, {{RegionMover}} moves back > regions. The end result is that the newly restarted RegionServer has twice > the load of any other server in the cluster. Future iterations of the load > balancer do nothing, as the cluster cost does not exceed > {{minCostNeedBalance}}. > Another example is if the load balancer makes very slow progress on a > cluster, it may not move the average cluster load to a newly restarted > regionserver in one iteration. But after the first iteration, the balancer > may again not run due to cluster cost not exceeding {{minCostNeedBalance}}. > We can propose a solution where we reuse the {{slop}} concept in > {{SimpleLoadBalancer}} and use this to extend the HBASE-24139 logic for > deciding to run the balancer as long as there is a "sloppy" server in the > cluster. > +*Previous Description Notes Below, which are relevant, but as stated, were > already fixed by HBASE-24139*+ > In EMR cluster, whenever I replace one of the nodes, the regions never get > rebalanced. > The default minCostNeedBalance set to 0.05 is too high. > The region count on the servers were: 21, 21, 20, 20, 20, 20, 21, 20, 20, 20 > = 203 > Once a node(region server) got replaced with a new node (terminated and EMR > recreated a node), the region count on the servers became: 23, 0, 23, 22, 22, > 22, 22, 23, 23, 23 = 203 > From hbase-master-logs, I can see the below WARN which indicates that the > default minCostNeedBalance does not hold good for these scenarios. > ## > 2019-04-29 09:31:37,027 WARN > [ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1] > cleaner.CleanerChore: WALs outstanding under > hdfs://ip-172-31-35-122.ec2.internal:8020/user/hbase/oldWALs2019-04-29 > 09:31:42,920 INFO > [ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1] > balancer.StochasticLoadBalancer: Skipping load balancing because balanced > cluster; total cost is 52.041826194833405, sum multiplier is 1102.0 min cost > which need balance is 0.05 > ## > To mitigate this, I had to modify the default minCostNeedBalance to lower > value like 0.01f and restart Region Servers and Hbase Master. After modifying > this value to 0.01f I could see the regions getting re-balanced. > This has led me to the following questions which I would like to get it > answered from the HBase experts. > 1)What are the factors that affect the value of total cost and sum > multiplier? How could we determine the right minCostNeedBalance value for any > cluster? > 2)How did Hbase arrive at setting the default value to 0.05f? Is it optimal > value? If yes, then what is the recommended way to mitigate this scenario? > Attached: Steps to reproduce > > Note: HBase-17565 patch is already applied. -- This message was sent by Atlassian Jira (v8.20.7#820007)