Any plan backport HBASE-14309 into 0.98 ?
2015-10-31 10:26 GMT+08:00 Heng Chen <heng.chen.1...@gmail.com>:
> bq. Not running balancer because 3 region(s) in transition
>
> Yeah, but balancer runs every 5 minutes, 3 region(s) in transition log
> only show up only in a few places....
>
> 2015-10-31 10:23 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
>
>> bq. there are no logs to record why we not running balancer
>>
>> Here was the reason:
>>
>> bq. Not running balancer because 3 region(s) in transition:
>>
>> bq. Could we just balance regions not in transition?
>>
>> Yes. Please take a look at HBASE-14309
>>
>> Cheers
>>
>> On Fri, Oct 30, 2015 at 7:19 PM, Heng Chen <heng.chen.1...@gmail.com>
>> wrote:
>>
>> > My hbase cluster version is 0.98.6
>> >
>> > There are lots of regions on it, about 10000+
>> >
>> > Load is heavy, almost every time there are regions in split....
>> >
>> > So i found that the balancer not run for a long time.
>> >
>> > grep -i 'balancer' master.log, there are only logs like below
>> >
>> > 2015-09-30 11:29:13,994 DEBUG
>> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
>> master.HMaster:
>> > Not running balancer because 3 region(s) in transition:
>> > {30971a1ae707b9f5bbcd7b8802f32059={30971a1ae707b9f5bbcd7b8802f32059
>> > state=SPLITTING_NEW, ts=1443583753692,
>> > server=dx-ape-regionserver30-online,60020,1440183710528},
>> > 13eaacf6df912d0cb598067610c5a85f={13eaacf6df912d0cb598067610c5a85f
>> > state=SPLITTING_NEW, ...
>> > 2015-10-01 17:44:14,032 DEBUG
>> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
>> master.HMaster:
>> > Not running balancer because 3 region(s) in transition:
>> > {55fc1c408832233ee1dd01c70c61ae14={55fc1c408832233ee1dd01c70c61ae14
>> > state=SPLITTING, ts=1443692653425,
>> > server=dx-ape-regionserver27-online,60020,1440183264316},
>> > 07439db0ff1319d20b43aa4d2e43a4ae={07439db0ff1319d20b43aa4d2e43a4ae
>> > state=SPLITTING_NEW, ts=1...
>> > 2015-10-04 14:04:14,126 DEBUG
>> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
>> master.HMaster:
>> > Not running balancer because 3 region(s) in transition:
>> > {2bd0891dc9ca5fb15ea8b661127193b7={2bd0891dc9ca5fb15ea8b661127193b7
>> > state=SPLITTING, ts=1443938653837,
>> > server=dx-ape-regionserver9-online,60020,1440182448264},
>> > 76bbb47201c3958e3a9c1086bfb351c5={76bbb47201c3958e3a9c1086bfb351c5
>> > state=SPLITTING_NEW, ts=14...
>> > 2015-10-05 14:14:14,161 DEBUG
>> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
>> master.HMaster:
>> > Not running balancer because 3 region(s) in transition:
>> > {669719254f132476c6df0e0e9b1fc93f={669719254f132476c6df0e0e9b1fc93f
>> > state=SPLITTING_NEW, ts=1444025653911,
>> > server=dx-ape-regionserver1-online,60020,1440178926883},
>> > ec612addaabb22c8f46b2c903bd1158b={ec612addaabb22c8f46b2c903bd1158b
>> > state=SPLITTING_NEW, t...
>> > 2015-10-15 21:19:14,512 DEBUG
>> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
>> master.HMaster:
>> > Not running balancer because 3 region(s) in transition:
>> > {2b7a5c3ddc7ee919199c68611e6f6c96={2b7a5c3ddc7ee919199c68611e6f6c96
>> > state=SPLITTING, ts=1444915153714,
>> > server=dx-ape-regionserver12-online,60020,1440181883146},
>> > cda06b9ebd651c616361f73a469a1a52={cda06b9ebd651c616361f73a469a1a52
>> > state=SPLITTING_NEW, ts=1...
>> > 2015-10-15 23:39:14,513 DEBUG
>> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
>> master.HMaster:
>> > Not running balancer because 3 region(s) in transition:
>> > {b1d3429606407280e442d8ce3de873c4={b1d3429606407280e442d8ce3de873c4
>> > state=SPLITTING, ts=1444923553844,
>> > server=dx-ape-regionserver25-online,60020,1440183200463},
>> > ae7ba7ee139c7ba84ba707671b7959c4={ae7ba7ee139c7ba84ba707671b7959c4
>> > state=SPLITTING_NEW, ts=1...
>> > 2015-10-21 19:29:14,692 DEBUG
>> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
>> master.HMaster:
>> > Not running balancer because 3 region(s) in transition:
>> > {e677e41a383eb20429c9906bafc252bb={e677e41a383eb20429c9906bafc252bb
>> > state=SPLITTING_NEW, ts=1445426954437,
>> > server=dx-ape-regionserver11-online,60020,1440181972615},
>> > 0028b035271bdd6d30e7fb6f1ffb406d={0028b035271bdd6d30e7fb6f1ffb406d
>> > state=SPLITTING, ts=1...
>> > 2015-10-25 10:24:14,790 DEBUG
>> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
>> master.HMaster:
>> > Not running balancer because 3 region(s) in transition:
>> > {694912c058fcd0e6bff7b3eaed1b051b={694912c058fcd0e6bff7b3eaed1b051b
>> > state=SPLITTING_NEW, ts=1445739851757,
>> > server=dx-ape-regionserver27-online,60020,1440183264316},
>> > 7859193f7ca5ee2c98636cb812b549a7={7859193f7ca5ee2c98636cb812b549a7
>> > state=SPLITTING, ts=1...
>> >
>> >
>> > The balancer runs every 5 minutes, there are no logs to record why we
>> not
>> > running balancer, should we add some logs at least?
>> >
>> > As for the above logs, it seems we stop running balancer when regions
>> in
>> > transition
>> >
>> > This is the relates code
>> >
>> > // Only allow one balance run at at time.
>> > if (this.assignmentManager.getRegionStates().isRegionsInTransition()) {
>> > Map<String, RegionState> regionsInTransition =
>> > this.assignmentManager.getRegionStates().getRegionsInTransition();
>> > LOG.debug("Not running balancer because " +
>> regionsInTransition.size() +
>> > " region(s) in transition: " + org.apache.commons.lang.StringUtils.
>> > abbreviate(regionsInTransition.toString(), 256));
>> > return false;
>> > }
>> >
>> > And i have questions, why we use regions states to avoid more than
>> > one balancer running?
>> >
>> > Could we just balance regions not in transition?
>> >
>> >
>> > Thanks!
>> >
>>
>
>
