bq. Not running balancer because 3 region(s) in transition
Yeah, but balancer runs every 5 minutes, 3 region(s) in transition log only
show up only in a few places....
2015-10-31 10:23 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
> bq. there are no logs to record why we not running balancer
>
> Here was the reason:
>
> bq. Not running balancer because 3 region(s) in transition:
>
> bq. Could we just balance regions not in transition?
>
> Yes. Please take a look at HBASE-14309
>
> Cheers
>
> On Fri, Oct 30, 2015 at 7:19 PM, Heng Chen <heng.chen.1...@gmail.com>
> wrote:
>
> > My hbase cluster version is 0.98.6
> >
> > There are lots of regions on it, about 10000+
> >
> > Load is heavy, almost every time there are regions in split....
> >
> > So i found that the balancer not run for a long time.
> >
> > grep -i 'balancer' master.log, there are only logs like below
> >
> > 2015-09-30 11:29:13,994 DEBUG
> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
> master.HMaster:
> > Not running balancer because 3 region(s) in transition:
> > {30971a1ae707b9f5bbcd7b8802f32059={30971a1ae707b9f5bbcd7b8802f32059
> > state=SPLITTING_NEW, ts=1443583753692,
> > server=dx-ape-regionserver30-online,60020,1440183710528},
> > 13eaacf6df912d0cb598067610c5a85f={13eaacf6df912d0cb598067610c5a85f
> > state=SPLITTING_NEW, ...
> > 2015-10-01 17:44:14,032 DEBUG
> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
> master.HMaster:
> > Not running balancer because 3 region(s) in transition:
> > {55fc1c408832233ee1dd01c70c61ae14={55fc1c408832233ee1dd01c70c61ae14
> > state=SPLITTING, ts=1443692653425,
> > server=dx-ape-regionserver27-online,60020,1440183264316},
> > 07439db0ff1319d20b43aa4d2e43a4ae={07439db0ff1319d20b43aa4d2e43a4ae
> > state=SPLITTING_NEW, ts=1...
> > 2015-10-04 14:04:14,126 DEBUG
> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
> master.HMaster:
> > Not running balancer because 3 region(s) in transition:
> > {2bd0891dc9ca5fb15ea8b661127193b7={2bd0891dc9ca5fb15ea8b661127193b7
> > state=SPLITTING, ts=1443938653837,
> > server=dx-ape-regionserver9-online,60020,1440182448264},
> > 76bbb47201c3958e3a9c1086bfb351c5={76bbb47201c3958e3a9c1086bfb351c5
> > state=SPLITTING_NEW, ts=14...
> > 2015-10-05 14:14:14,161 DEBUG
> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
> master.HMaster:
> > Not running balancer because 3 region(s) in transition:
> > {669719254f132476c6df0e0e9b1fc93f={669719254f132476c6df0e0e9b1fc93f
> > state=SPLITTING_NEW, ts=1444025653911,
> > server=dx-ape-regionserver1-online,60020,1440178926883},
> > ec612addaabb22c8f46b2c903bd1158b={ec612addaabb22c8f46b2c903bd1158b
> > state=SPLITTING_NEW, t...
> > 2015-10-15 21:19:14,512 DEBUG
> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
> master.HMaster:
> > Not running balancer because 3 region(s) in transition:
> > {2b7a5c3ddc7ee919199c68611e6f6c96={2b7a5c3ddc7ee919199c68611e6f6c96
> > state=SPLITTING, ts=1444915153714,
> > server=dx-ape-regionserver12-online,60020,1440181883146},
> > cda06b9ebd651c616361f73a469a1a52={cda06b9ebd651c616361f73a469a1a52
> > state=SPLITTING_NEW, ts=1...
> > 2015-10-15 23:39:14,513 DEBUG
> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
> master.HMaster:
> > Not running balancer because 3 region(s) in transition:
> > {b1d3429606407280e442d8ce3de873c4={b1d3429606407280e442d8ce3de873c4
> > state=SPLITTING, ts=1444923553844,
> > server=dx-ape-regionserver25-online,60020,1440183200463},
> > ae7ba7ee139c7ba84ba707671b7959c4={ae7ba7ee139c7ba84ba707671b7959c4
> > state=SPLITTING_NEW, ts=1...
> > 2015-10-21 19:29:14,692 DEBUG
> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
> master.HMaster:
> > Not running balancer because 3 region(s) in transition:
> > {e677e41a383eb20429c9906bafc252bb={e677e41a383eb20429c9906bafc252bb
> > state=SPLITTING_NEW, ts=1445426954437,
> > server=dx-ape-regionserver11-online,60020,1440181972615},
> > 0028b035271bdd6d30e7fb6f1ffb406d={0028b035271bdd6d30e7fb6f1ffb406d
> > state=SPLITTING, ts=1...
> > 2015-10-25 10:24:14,790 DEBUG
> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore]
> master.HMaster:
> > Not running balancer because 3 region(s) in transition:
> > {694912c058fcd0e6bff7b3eaed1b051b={694912c058fcd0e6bff7b3eaed1b051b
> > state=SPLITTING_NEW, ts=1445739851757,
> > server=dx-ape-regionserver27-online,60020,1440183264316},
> > 7859193f7ca5ee2c98636cb812b549a7={7859193f7ca5ee2c98636cb812b549a7
> > state=SPLITTING, ts=1...
> >
> >
> > The balancer runs every 5 minutes, there are no logs to record why we
> not
> > running balancer, should we add some logs at least?
> >
> > As for the above logs, it seems we stop running balancer when regions in
> > transition
> >
> > This is the relates code
> >
> > // Only allow one balance run at at time.
> > if (this.assignmentManager.getRegionStates().isRegionsInTransition()) {
> > Map<String, RegionState> regionsInTransition =
> > this.assignmentManager.getRegionStates().getRegionsInTransition();
> > LOG.debug("Not running balancer because " + regionsInTransition.size()
> +
> > " region(s) in transition: " + org.apache.commons.lang.StringUtils.
> > abbreviate(regionsInTransition.toString(), 256));
> > return false;
> > }
> >
> > And i have questions, why we use regions states to avoid more than
> > one balancer running?
> >
> > Could we just balance regions not in transition?
> >
> >
> > Thanks!
> >
>
