sunhelly commented on pull request #3025: URL: https://github.com/apache/hbase/pull/3025#issuecomment-795406061
Thanks for reviewing, @Apache9. >Mind explaining a bit about the logic? And do you have any performance numbers? Yes, it's my pleasure. Currently, the balance for the whole cluster is, get all tables regions distribution map-> generate balance plans(by table or not) -> execute all plans by AM. As a result, for one RSGroup, how long the auto balance of it takes depending on how long the whole cluster plans generated and executed. For this patch, groups are balanced dependently, not only in generating plans, but also in executing plans. We propose this idea because our cluster was used a large multiplier of locality cost and before HBASE-18164 and HBASE-15515, some group balances were always stuck and blocked the other group balances, as well as the manually balances by shell commands, because they all used a whole cluster balancer lock. On our cluster with 20 groups, 1300+ tables, and 300000+ regions, the balancer consumes about 3 minutes for a relevant balanced cluster with a very small number of regions to move or swap. While using parallelize balance with 10 fixed thread pool, it consumes about 1 minutes. For executing region plans, the duration is restricted in the max balancing time, which is configed by `hbase.balancer.max.balancing`. I think the executing time for the plans is not the most important here, but separating group balance locks and make them work without mutual influence is meaningful. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
