Ok, I fixed this by manually reassign region servers to newly created ones.
def reassignRegionServer(admin: HBaseAdmin, regions: Seq[HRegionInfo], regionServers: Seq[ServerName]): Unit = { val rand = new Random() regions.foreach { r => val idx = rand.nextInt(regionServers.size) val server = regionServers(idx) // FIXME: what if selected region server is dead? admin.move(r.getEncodedNameAsBytes, server.getServerName.getBytes("UTF8")) } } er... Jianshi On Tue, Sep 23, 2014 at 12:24 AM, Jianshi Huang <jianshi.hu...@gmail.com> wrote: > Hmm...any workaround? I only want to do this: > > Rebalance the new regions *evenly* to all servers after manually adding > splits, so later bulk insertions won't cause contention. > > P.S. > Looks like two of the region servers which had majority of the regions > were down during Major compaction... I guess it had too much data. > > > Jianshi > > On Tue, Sep 23, 2014 at 12:13 AM, Jianshi Huang <jianshi.hu...@gmail.com> > wrote: > >> Yes, I have access to Master UI, however logs/*.log cannot be opened or >> downloaded, must be some security restrictions in the proxy... >> >> Jianshi >> >> On Tue, Sep 23, 2014 at 12:06 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Do you have access to Master UI ? >>> >>> <master-address>:60010/logs/ would show you list of log files. >>> >>> The you can view <master-address>:60010/logs/hbase-<user>-master-XXX.log >>> >>> Cheers >>> >>> On Mon, Sep 22, 2014 at 9:00 AM, Jianshi Huang <jianshi.hu...@gmail.com> >>> wrote: >>> >>> > Ah... I don't have access to HMaster logs... I need to ask the admin. >>> > >>> > Jianshi >>> > >>> > On Mon, Sep 22, 2014 at 11:49 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>> > >>> > > bq. assign per-table balancer class >>> > > >>> > > No that I know of. >>> > > Can you pastebin master log involving output from balancer ? >>> > > >>> > > Cheers >>> > > >>> > > On Mon, Sep 22, 2014 at 8:29 AM, Jianshi Huang < >>> jianshi.hu...@gmail.com> >>> > > wrote: >>> > > >>> > > > Hi Ted, >>> > > > >>> > > > I moved setBalancerRunning before balancer and run them twice. >>> However >>> > I >>> > > > still got highly skewed region distribution. >>> > > > >>> > > > I guess it's because of the StochasticLoadBalancer, can I assign >>> > > per-table >>> > > > balancer class in HBase? >>> > > > >>> > > > >>> > > > Jianshi >>> > > > >>> > > > On Mon, Sep 22, 2014 at 9:50 PM, Ted Yu <yuzhih...@gmail.com> >>> wrote: >>> > > > >>> > > > > admin.setBalancerRunning() call should precede the call to >>> > > > > admin.balancer(). >>> > > > > >>> > > > > You can inspect master log to see whether regions are being >>> moved off >>> > > the >>> > > > > heavily loaded server. >>> > > > > >>> > > > > Cheers >>> > > > > >>> > > > > On Mon, Sep 22, 2014 at 1:42 AM, Jianshi Huang < >>> > > jianshi.hu...@gmail.com> >>> > > > > wrote: >>> > > > > >>> > > > > > Hi Ted and others, >>> > > > > > >>> > > > > > I did the following after adding splits (without data) to my >>> table, >>> > > > > however >>> > > > > > the region is still very imbalanced (one region server has 221 >>> > > regions >>> > > > > and >>> > > > > > other 50 region servers have about 4~8 regions each). >>> > > > > > >>> > > > > > admin.balancer() >>> > > > > > admin.setBalancerRunning(true, true) >>> > > > > > >>> > > > > > The balancer class in my HBase cluster is >>> > > > > > >>> > > > > > org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer >>> > > > > > >>> > > > > > So, is this behavior expected? Can I assign different balancer >>> > class >>> > > to >>> > > > > my >>> > > > > > tables (I don't have HBase admin permission)? Which one should >>> I >>> > use? >>> > > > > > >>> > > > > > I just want HBase to evenly distribute the regions even they >>> don't >>> > > have >>> > > > > > data (that's the purpose of pre-split I think). >>> > > > > > >>> > > > > > >>> > > > > > Jianshi >>> > > > > > >>> > > > > > >>> > > > > > On Sat, Sep 6, 2014 at 12:45 AM, Ted Yu <yuzhih...@gmail.com> >>> > wrote: >>> > > > > > >>> > > > > > > Yes. See the following method in HBaseAdmin: >>> > > > > > > >>> > > > > > > public boolean balancer() >>> > > > > > > >>> > > > > > > >>> > > > > > > On Fri, Sep 5, 2014 at 9:38 AM, Jianshi Huang < >>> > > > jianshi.hu...@gmail.com >>> > > > > > >>> > > > > > > wrote: >>> > > > > > > >>> > > > > > > > Thanks Ted! >>> > > > > > > > >>> > > > > > > > Didn't know I still need to run the 'balancer' command. >>> > > > > > > > >>> > > > > > > > Is there a way to do it programmatically? >>> > > > > > > > >>> > > > > > > > Jianshi >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > On Sat, Sep 6, 2014 at 12:29 AM, Ted Yu < >>> yuzhih...@gmail.com> >>> > > > wrote: >>> > > > > > > > >>> > > > > > > > > After splitting the region, you may need to run balancer >>> to >>> > > > spread >>> > > > > > the >>> > > > > > > > new >>> > > > > > > > > regions out. >>> > > > > > > > > >>> > > > > > > > > Cheers >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > On Fri, Sep 5, 2014 at 9:25 AM, Jianshi Huang < >>> > > > > > jianshi.hu...@gmail.com >>> > > > > > > > >>> > > > > > > > > wrote: >>> > > > > > > > > >>> > > > > > > > > > Hi Shahab, >>> > > > > > > > > > >>> > > > > > > > > > I see, that seems to be the right way... >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > On Sat, Sep 6, 2014 at 12:21 AM, Shahab Yunus < >>> > > > > > > shahab.yu...@gmail.com> >>> > > > > > > > > > wrote: >>> > > > > > > > > > >>> > > > > > > > > > > Shahab >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > -- >>> > > > > > > > > > Jianshi Huang >>> > > > > > > > > > >>> > > > > > > > > > LinkedIn: jianshi >>> > > > > > > > > > Twitter: @jshuang >>> > > > > > > > > > Github & Blog: http://huangjs.github.com/ >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > -- >>> > > > > > > > Jianshi Huang >>> > > > > > > > >>> > > > > > > > LinkedIn: jianshi >>> > > > > > > > Twitter: @jshuang >>> > > > > > > > Github & Blog: http://huangjs.github.com/ >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > -- >>> > > > > > Jianshi Huang >>> > > > > > >>> > > > > > LinkedIn: jianshi >>> > > > > > Twitter: @jshuang >>> > > > > > Github & Blog: http://huangjs.github.com/ >>> > > > > > >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > Jianshi Huang >>> > > > >>> > > > LinkedIn: jianshi >>> > > > Twitter: @jshuang >>> > > > Github & Blog: http://huangjs.github.com/ >>> > > > >>> > > >>> > >>> > >>> > >>> > -- >>> > Jianshi Huang >>> > >>> > LinkedIn: jianshi >>> > Twitter: @jshuang >>> > Github & Blog: http://huangjs.github.com/ >>> > >>> >> >> >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > > > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/