Ok, I fixed this by manually reassign region servers to newly created ones.
def reassignRegionServer(admin: HBaseAdmin, regions: Seq[HRegionInfo],
regionServers: Seq[ServerName]): Unit = {
val rand = new Random()
regions.foreach { r =>
val idx = rand.nextInt(regionServers.size)
val server = regionServers(idx)
// FIXME: what if selected region server is dead?
admin.move(r.getEncodedNameAsBytes,
server.getServerName.getBytes("UTF8"))
}
}
er...
Jianshi
On Tue, Sep 23, 2014 at 12:24 AM, Jianshi Huang <jianshi.hu...@gmail.com>
wrote:
> Hmm...any workaround? I only want to do this:
>
> Rebalance the new regions *evenly* to all servers after manually adding
> splits, so later bulk insertions won't cause contention.
>
> P.S.
> Looks like two of the region servers which had majority of the regions
> were down during Major compaction... I guess it had too much data.
>
>
> Jianshi
>
> On Tue, Sep 23, 2014 at 12:13 AM, Jianshi Huang <jianshi.hu...@gmail.com>
> wrote:
>
>> Yes, I have access to Master UI, however logs/*.log cannot be opened or
>> downloaded, must be some security restrictions in the proxy...
>>
>> Jianshi
>>
>> On Tue, Sep 23, 2014 at 12:06 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>>> Do you have access to Master UI ?
>>>
>>> <master-address>:60010/logs/ would show you list of log files.
>>>
>>> The you can view <master-address>:60010/logs/hbase-<user>-master-XXX.log
>>>
>>> Cheers
>>>
>>> On Mon, Sep 22, 2014 at 9:00 AM, Jianshi Huang <jianshi.hu...@gmail.com>
>>> wrote:
>>>
>>> > Ah... I don't have access to HMaster logs... I need to ask the admin.
>>> >
>>> > Jianshi
>>> >
>>> > On Mon, Sep 22, 2014 at 11:49 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>> >
>>> > > bq. assign per-table balancer class
>>> > >
>>> > > No that I know of.
>>> > > Can you pastebin master log involving output from balancer ?
>>> > >
>>> > > Cheers
>>> > >
>>> > > On Mon, Sep 22, 2014 at 8:29 AM, Jianshi Huang <
>>> jianshi.hu...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Hi Ted,
>>> > > >
>>> > > > I moved setBalancerRunning before balancer and run them twice.
>>> However
>>> > I
>>> > > > still got highly skewed region distribution.
>>> > > >
>>> > > > I guess it's because of the StochasticLoadBalancer, can I assign
>>> > > per-table
>>> > > > balancer class in HBase?
>>> > > >
>>> > > >
>>> > > > Jianshi
>>> > > >
>>> > > > On Mon, Sep 22, 2014 at 9:50 PM, Ted Yu <yuzhih...@gmail.com>
>>> wrote:
>>> > > >
>>> > > > > admin.setBalancerRunning() call should precede the call to
>>> > > > > admin.balancer().
>>> > > > >
>>> > > > > You can inspect master log to see whether regions are being
>>> moved off
>>> > > the
>>> > > > > heavily loaded server.
>>> > > > >
>>> > > > > Cheers
>>> > > > >
>>> > > > > On Mon, Sep 22, 2014 at 1:42 AM, Jianshi Huang <
>>> > > jianshi.hu...@gmail.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Hi Ted and others,
>>> > > > > >
>>> > > > > > I did the following after adding splits (without data) to my
>>> table,
>>> > > > > however
>>> > > > > > the region is still very imbalanced (one region server has 221
>>> > > regions
>>> > > > > and
>>> > > > > > other 50 region servers have about 4~8 regions each).
>>> > > > > >
>>> > > > > > admin.balancer()
>>> > > > > > admin.setBalancerRunning(true, true)
>>> > > > > >
>>> > > > > > The balancer class in my HBase cluster is
>>> > > > > >
>>> > > > > > org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
>>> > > > > >
>>> > > > > > So, is this behavior expected? Can I assign different balancer
>>> > class
>>> > > to
>>> > > > > my
>>> > > > > > tables (I don't have HBase admin permission)? Which one should
>>> I
>>> > use?
>>> > > > > >
>>> > > > > > I just want HBase to evenly distribute the regions even they
>>> don't
>>> > > have
>>> > > > > > data (that's the purpose of pre-split I think).
>>> > > > > >
>>> > > > > >
>>> > > > > > Jianshi
>>> > > > > >
>>> > > > > >
>>> > > > > > On Sat, Sep 6, 2014 at 12:45 AM, Ted Yu <yuzhih...@gmail.com>
>>> > wrote:
>>> > > > > >
>>> > > > > > > Yes. See the following method in HBaseAdmin:
>>> > > > > > >
>>> > > > > > > public boolean balancer()
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Fri, Sep 5, 2014 at 9:38 AM, Jianshi Huang <
>>> > > > jianshi.hu...@gmail.com
>>> > > > > >
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Thanks Ted!
>>> > > > > > > >
>>> > > > > > > > Didn't know I still need to run the 'balancer' command.
>>> > > > > > > >
>>> > > > > > > > Is there a way to do it programmatically?
>>> > > > > > > >
>>> > > > > > > > Jianshi
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Sat, Sep 6, 2014 at 12:29 AM, Ted Yu <
>>> yuzhih...@gmail.com>
>>> > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > After splitting the region, you may need to run balancer
>>> to
>>> > > > spread
>>> > > > > > the
>>> > > > > > > > new
>>> > > > > > > > > regions out.
>>> > > > > > > > >
>>> > > > > > > > > Cheers
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > On Fri, Sep 5, 2014 at 9:25 AM, Jianshi Huang <
>>> > > > > > jianshi.hu...@gmail.com
>>> > > > > > > >
>>> > > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > Hi Shahab,
>>> > > > > > > > > >
>>> > > > > > > > > > I see, that seems to be the right way...
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > On Sat, Sep 6, 2014 at 12:21 AM, Shahab Yunus <
>>> > > > > > > shahab.yu...@gmail.com>
>>> > > > > > > > > > wrote:
>>> > > > > > > > > >
>>> > > > > > > > > > > Shahab
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > --
>>> > > > > > > > > > Jianshi Huang
>>> > > > > > > > > >
>>> > > > > > > > > > LinkedIn: jianshi
>>> > > > > > > > > > Twitter: @jshuang
>>> > > > > > > > > > Github & Blog: http://huangjs.github.com/
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > --
>>> > > > > > > > Jianshi Huang
>>> > > > > > > >
>>> > > > > > > > LinkedIn: jianshi
>>> > > > > > > > Twitter: @jshuang
>>> > > > > > > > Github & Blog: http://huangjs.github.com/
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Jianshi Huang
>>> > > > > >
>>> > > > > > LinkedIn: jianshi
>>> > > > > > Twitter: @jshuang
>>> > > > > > Github & Blog: http://huangjs.github.com/
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Jianshi Huang
>>> > > >
>>> > > > LinkedIn: jianshi
>>> > > > Twitter: @jshuang
>>> > > > Github & Blog: http://huangjs.github.com/
>>> > > >
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Jianshi Huang
>>> >
>>> > LinkedIn: jianshi
>>> > Twitter: @jshuang
>>> > Github & Blog: http://huangjs.github.com/
>>> >
>>>
>>
>>
>>
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
