As you noted in the FIXME, there're some factors which should be tackled by
balancer / assignment manager.
Please continue digging up master log so that we can find the cause for
balancer not fulfilling your goal.
Cheers
On Mon, Sep 22, 2014 at 10:09 AM, Jianshi Huang <jianshi.hu...@gmail.com>
wrote:
> Ok, I fixed this by manually reassign region servers to newly created ones.
>
> def reassignRegionServer(admin: HBaseAdmin, regions: Seq[HRegionInfo],
> regionServers: Seq[ServerName]): Unit = {
> val rand = new Random()
> regions.foreach { r =>
> val idx = rand.nextInt(regionServers.size)
> val server = regionServers(idx)
> // FIXME: what if selected region server is dead?
> admin.move(r.getEncodedNameAsBytes,
> server.getServerName.getBytes("UTF8"))
> }
> }
>
> er...
>
> Jianshi
>
> On Tue, Sep 23, 2014 at 12:24 AM, Jianshi Huang <jianshi.hu...@gmail.com>
> wrote:
>
> > Hmm...any workaround? I only want to do this:
> >
> > Rebalance the new regions *evenly* to all servers after manually adding
> > splits, so later bulk insertions won't cause contention.
> >
> > P.S.
> > Looks like two of the region servers which had majority of the regions
> > were down during Major compaction... I guess it had too much data.
> >
> >
> > Jianshi
> >
> > On Tue, Sep 23, 2014 at 12:13 AM, Jianshi Huang <jianshi.hu...@gmail.com
> >
> > wrote:
> >
> >> Yes, I have access to Master UI, however logs/*.log cannot be opened or
> >> downloaded, must be some security restrictions in the proxy...
> >>
> >> Jianshi
> >>
> >> On Tue, Sep 23, 2014 at 12:06 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>
> >>> Do you have access to Master UI ?
> >>>
> >>> <master-address>:60010/logs/ would show you list of log files.
> >>>
> >>> The you can view
> <master-address>:60010/logs/hbase-<user>-master-XXX.log
> >>>
> >>> Cheers
> >>>
> >>> On Mon, Sep 22, 2014 at 9:00 AM, Jianshi Huang <
> jianshi.hu...@gmail.com>
> >>> wrote:
> >>>
> >>> > Ah... I don't have access to HMaster logs... I need to ask the admin.
> >>> >
> >>> > Jianshi
> >>> >
> >>> > On Mon, Sep 22, 2014 at 11:49 PM, Ted Yu <yuzhih...@gmail.com>
> wrote:
> >>> >
> >>> > > bq. assign per-table balancer class
> >>> > >
> >>> > > No that I know of.
> >>> > > Can you pastebin master log involving output from balancer ?
> >>> > >
> >>> > > Cheers
> >>> > >
> >>> > > On Mon, Sep 22, 2014 at 8:29 AM, Jianshi Huang <
> >>> jianshi.hu...@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Hi Ted,
> >>> > > >
> >>> > > > I moved setBalancerRunning before balancer and run them twice.
> >>> However
> >>> > I
> >>> > > > still got highly skewed region distribution.
> >>> > > >
> >>> > > > I guess it's because of the StochasticLoadBalancer, can I assign
> >>> > > per-table
> >>> > > > balancer class in HBase?
> >>> > > >
> >>> > > >
> >>> > > > Jianshi
> >>> > > >
> >>> > > > On Mon, Sep 22, 2014 at 9:50 PM, Ted Yu <yuzhih...@gmail.com>
> >>> wrote:
> >>> > > >
> >>> > > > > admin.setBalancerRunning() call should precede the call to
> >>> > > > > admin.balancer().
> >>> > > > >
> >>> > > > > You can inspect master log to see whether regions are being
> >>> moved off
> >>> > > the
> >>> > > > > heavily loaded server.
> >>> > > > >
> >>> > > > > Cheers
> >>> > > > >
> >>> > > > > On Mon, Sep 22, 2014 at 1:42 AM, Jianshi Huang <
> >>> > > jianshi.hu...@gmail.com>
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > > Hi Ted and others,
> >>> > > > > >
> >>> > > > > > I did the following after adding splits (without data) to my
> >>> table,
> >>> > > > > however
> >>> > > > > > the region is still very imbalanced (one region server has
> 221
> >>> > > regions
> >>> > > > > and
> >>> > > > > > other 50 region servers have about 4~8 regions each).
> >>> > > > > >
> >>> > > > > > admin.balancer()
> >>> > > > > > admin.setBalancerRunning(true, true)
> >>> > > > > >
> >>> > > > > > The balancer class in my HBase cluster is
> >>> > > > > >
> >>> > > > > >
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
> >>> > > > > >
> >>> > > > > > So, is this behavior expected? Can I assign different
> balancer
> >>> > class
> >>> > > to
> >>> > > > > my
> >>> > > > > > tables (I don't have HBase admin permission)? Which one
> should
> >>> I
> >>> > use?
> >>> > > > > >
> >>> > > > > > I just want HBase to evenly distribute the regions even they
> >>> don't
> >>> > > have
> >>> > > > > > data (that's the purpose of pre-split I think).
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Jianshi
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > On Sat, Sep 6, 2014 at 12:45 AM, Ted Yu <yuzhih...@gmail.com
> >
> >>> > wrote:
> >>> > > > > >
> >>> > > > > > > Yes. See the following method in HBaseAdmin:
> >>> > > > > > >
> >>> > > > > > > public boolean balancer()
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Fri, Sep 5, 2014 at 9:38 AM, Jianshi Huang <
> >>> > > > jianshi.hu...@gmail.com
> >>> > > > > >
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Thanks Ted!
> >>> > > > > > > >
> >>> > > > > > > > Didn't know I still need to run the 'balancer' command.
> >>> > > > > > > >
> >>> > > > > > > > Is there a way to do it programmatically?
> >>> > > > > > > >
> >>> > > > > > > > Jianshi
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > On Sat, Sep 6, 2014 at 12:29 AM, Ted Yu <
> >>> yuzhih...@gmail.com>
> >>> > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > After splitting the region, you may need to run
> balancer
> >>> to
> >>> > > > spread
> >>> > > > > > the
> >>> > > > > > > > new
> >>> > > > > > > > > regions out.
> >>> > > > > > > > >
> >>> > > > > > > > > Cheers
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > On Fri, Sep 5, 2014 at 9:25 AM, Jianshi Huang <
> >>> > > > > > jianshi.hu...@gmail.com
> >>> > > > > > > >
> >>> > > > > > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > Hi Shahab,
> >>> > > > > > > > > >
> >>> > > > > > > > > > I see, that seems to be the right way...
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > On Sat, Sep 6, 2014 at 12:21 AM, Shahab Yunus <
> >>> > > > > > > shahab.yu...@gmail.com>
> >>> > > > > > > > > > wrote:
> >>> > > > > > > > > >
> >>> > > > > > > > > > > Shahab
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > --
> >>> > > > > > > > > > Jianshi Huang
> >>> > > > > > > > > >
> >>> > > > > > > > > > LinkedIn: jianshi
> >>> > > > > > > > > > Twitter: @jshuang
> >>> > > > > > > > > > Github & Blog: http://huangjs.github.com/
> >>> > > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > --
> >>> > > > > > > > Jianshi Huang
> >>> > > > > > > >
> >>> > > > > > > > LinkedIn: jianshi
> >>> > > > > > > > Twitter: @jshuang
> >>> > > > > > > > Github & Blog: http://huangjs.github.com/
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > --
> >>> > > > > > Jianshi Huang
> >>> > > > > >
> >>> > > > > > LinkedIn: jianshi
> >>> > > > > > Twitter: @jshuang
> >>> > > > > > Github & Blog: http://huangjs.github.com/
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > --
> >>> > > > Jianshi Huang
> >>> > > >
> >>> > > > LinkedIn: jianshi
> >>> > > > Twitter: @jshuang
> >>> > > > Github & Blog: http://huangjs.github.com/
> >>> > > >
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Jianshi Huang
> >>> >
> >>> > LinkedIn: jianshi
> >>> > Twitter: @jshuang
> >>> > Github & Blog: http://huangjs.github.com/
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Jianshi Huang
> >>
> >> LinkedIn: jianshi
> >> Twitter: @jshuang
> >> Github & Blog: http://huangjs.github.com/
> >>
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
