Great. Thanks for the clarification. In 0.98.1 I thought we saw a different behavior and regions were distributed when we split.
Thanks again for the help. Abe On Wed, Feb 18, 2015 at 12:01 PM, Ted Yu <yuzhih...@gmail.com> wrote: > When a region splits, the two daughter regions stay on the same host as the > parent region did. > > It is through balancing that the daughter regions (may) be assigned to > other server(s). > > Cheers > > On Wed, Feb 18, 2015 at 7:35 AM, Abe Weinograd <a...@flonet.com> wrote: > > > Another question, would this help when we split or only when we balance? > > > > Thanks, > > Abe > > > > On Mon, Feb 16, 2015 at 11:24 AM, Abe Weinograd <a...@flonet.com> wrote: > > > > > balancer said "true" and it is not disabled. Thanks again for your > help. > > > > > > Abe > > > > > > On Mon, Feb 16, 2015 at 11:23 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > >> What was the output from 'balancer' command ? > > >> > > >> Was is possible that balancer was disabled ? > > >> > > >> Cheers > > >> > > >> On Mon, Feb 16, 2015 at 8:04 AM, Abe Weinograd <a...@flonet.com> > wrote: > > >> > > >> > Ok. I forced the balancer run and am not seeing anything after a > few > > >> > minutes. Master logs isn't showing anything. Should I look at the > RS > > >> ones > > >> > instead? > > >> > > > >> > On Mon, Feb 16, 2015 at 11:03 AM, Ted Yu <yuzhih...@gmail.com> > wrote: > > >> > > > >> > > You should see effect in the next balancer run. > > >> > > > > >> > > Cheers > > >> > > > > >> > > On Mon, Feb 16, 2015 at 7:52 AM, Abe Weinograd <a...@flonet.com> > > >> wrote: > > >> > > > > >> > > > Excellent. If i trigger the balancer, should this start to help > or > > >> only > > >> > > for > > >> > > > future region creation? > > >> > > > > > >> > > > Thanks, > > >> > > > Abe > > >> > > > > > >> > > > On Mon, Feb 16, 2015 at 9:35 AM, Ted Yu <yuzhih...@gmail.com> > > >> wrote: > > >> > > > > > >> > > > > Yes. This setting should be modified on Master. > > >> > > > > > > >> > > > > Cheers > > >> > > > > > > >> > > > > On Mon, Feb 16, 2015 at 6:27 AM, Abe Weinograd < > a...@flonet.com> > > >> > wrote: > > >> > > > > > > >> > > > > > Thanks Ted. We are putting this in the hbase-site.xml for > the > > >> > > Master? > > >> > > > > > > > >> > > > > > Abe > > >> > > > > > > > >> > > > > > On Fri, Feb 13, 2015 at 5:03 PM, Shahab Yunus < > > >> > > shahab.yu...@gmail.com> > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > Thanks, we will try that and report back. > > >> > > > > > > > > >> > > > > > > Regards, > > >> > > > > > > Shahab > > >> > > > > > > > > >> > > > > > > On Fri, Feb 13, 2015 at 4:56 PM, Ted Yu < > > yuzhih...@gmail.com> > > >> > > wrote: > > >> > > > > > > > > >> > > > > > > > You can make TableSkewCostFunction more prominent by > > >> increasing > > >> > > the > > >> > > > > > value > > >> > > > > > > > for config parameter: > > >> > > > > > > > > > >> > > > > > > > hbase.master.balancer.stochastic.tableSkewCost > > >> > > > > > > > > > >> > > > > > > > Its default is 35. > > >> > > > > > > > > > >> > > > > > > > See if raising to 100 or 200 helps. > > >> > > > > > > > > > >> > > > > > > > On Fri, Feb 13, 2015 at 1:09 PM, Shahab Yunus < > > >> > > > > shahab.yu...@gmail.com> > > >> > > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > Yes, this sever hosts other regions from other tables > as > > >> > well. > > >> > > > > > > > > > > >> > > > > > > > > Regards > > >> > > > > > > > > Shahab > > >> > > > > > > > > > > >> > > > > > > > > On Fri, Feb 13, 2015 at 1:45 PM, Ted Yu < > > >> yuzhih...@gmail.com > > >> > > > > >> > > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > > > Interesting, > server7.ec3.internal,60020,1423845018628 > > >> was > > >> > > > > > > consistently > > >> > > > > > > > > > chosen as destination for the table. > > >> > > > > > > > > > Did server7.ec3.internal,60020,1423845018628 host > > >> regions > > >> > > from > > >> > > > > > other > > >> > > > > > > > > table > > >> > > > > > > > > > ? > > >> > > > > > > > > > > > >> > > > > > > > > > Cheers > > >> > > > > > > > > > > > >> > > > > > > > > > On Fri, Feb 13, 2015 at 10:27 AM, Shahab Yunus < > > >> > > > > > > shahab.yu...@gmail.com > > >> > > > > > > > > > > >> > > > > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > Table name is: > > >> > > > > > > > > > > MYTABLE_RECENT_4W_V2 > > >> > > > > > > > > > > > > >> > > > > > > > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP > > >> > > > > > > > > > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF > > >> > > > > > > > > > > > > >> > > > > > > > > > > This is the master log after invoking balancer > > command > > >> > from > > >> > > > > hbase > > >> > > > > > > > > shell. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Regards, > > >> > > > > > > > > > > Shahab > > >> > > > > > > > > > > > > >> > > > > > > > > > > On Fri, Feb 13, 2015 at 12:00 PM, Ted Yu < > > >> > > > yuzhih...@gmail.com> > > >> > > > > > > > wrote: > > >> > > > > > > > > > > > > >> > > > > > > > > > > > bq. all the regions of this table were back on > > this > > >> > same > > >> > > > RS! > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Interesting. Please check master log around the > > time > > >> > this > > >> > > > RS > > >> > > > > > was > > >> > > > > > > > > > brought > > >> > > > > > > > > > > > online. You can pastebin the relevant snippet. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:55 AM, Shahab Yunus < > > >> > > > > > > > > shahab.yu...@gmail.com> > > >> > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Hi Ted. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Yes, the cluster itself is balanced. On > average > > >> 300 > > >> > > > regions > > >> > > > > > per > > >> > > > > > > > > node > > >> > > > > > > > > > on > > >> > > > > > > > > > > > 10 > > >> > > > > > > > > > > > > nodes. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > # of tables is 53 of varying sizes. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Balancer was invoked and it didn't do anything > > >> (i.e. > > >> > no > > >> > > > > > > movement > > >> > > > > > > > of > > >> > > > > > > > > > > > > regions) but we didn't check the master's > logs. > > We > > >> > can > > >> > > do > > >> > > > > > that. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Interestingly, we restarted the RS which was > > >> holding > > >> > > all > > >> > > > > the > > >> > > > > > > > > regions > > >> > > > > > > > > > of > > >> > > > > > > > > > > > > this one table. The regions were nicely spread > > >> out to > > >> > > the > > >> > > > > > > > remaining > > >> > > > > > > > > > RS. > > >> > > > > > > > > > > > But > > >> > > > > > > > > > > > > when we brought back this RS, all the regions > of > > >> this > > >> > > > table > > >> > > > > > > were > > >> > > > > > > > > back > > >> > > > > > > > > > > on > > >> > > > > > > > > > > > > this same RS! > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Regards, > > >> > > > > > > > > > > > > Shahab > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, Feb 13, 2015 at 11:46 AM, Ted Yu < > > >> > > > > > yuzhih...@gmail.com> > > >> > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > How many tables are there in your cluster ? > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Is the cluster balanced overall (in terms of > > >> number > > >> > > of > > >> > > > > > > regions > > >> > > > > > > > > per > > >> > > > > > > > > > > > > server) > > >> > > > > > > > > > > > > > but this table is not ? > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > What happens (check master log) when you > issue > > >> > > > 'balancer' > > >> > > > > > > > command > > >> > > > > > > > > > > > through > > >> > > > > > > > > > > > > > shell ? > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Cheers > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:19 AM, Shahab > Yunus > > < > > >> > > > > > > > > > > shahab.yu...@gmail.com> > > >> > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > CDH 5.3 > > >> > > > > > > > > > > > > > > HBase 98.6 > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > We are writing data to an HBase table > > through > > >> a > > >> > M/R > > >> > > > > job. > > >> > > > > > We > > >> > > > > > > > pre > > >> > > > > > > > > > > split > > >> > > > > > > > > > > > > the > > >> > > > > > > > > > > > > > > table before each job run. The problem is > > that > > >> > most > > >> > > > of > > >> > > > > > the > > >> > > > > > > > > > regions > > >> > > > > > > > > > > > end > > >> > > > > > > > > > > > > up > > >> > > > > > > > > > > > > > > on the same RS. This results in that one > RS > > >> being > > >> > > > > > severely > > >> > > > > > > > > > > overloaded > > >> > > > > > > > > > > > > and > > >> > > > > > > > > > > > > > > subsequent M/R jobs failing trying to > write > > to > > >> > the > > >> > > > > > regions > > >> > > > > > > on > > >> > > > > > > > > > that > > >> > > > > > > > > > > > RS. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > The balancer is on and the split policy is > > >> > default. > > >> > > > No > > >> > > > > > > > changes > > >> > > > > > > > > > > there. > > >> > > > > > > > > > > > > It > > >> > > > > > > > > > > > > > is > > >> > > > > > > > > > > > > > > a 10 node cluster. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > All other related properties are defaults > > too. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Any idea, how can we force balancing of > the > > >> new > > >> > > > > regions? > > >> > > > > > Do > > >> > > > > > > > we > > >> > > > > > > > > > have > > >> > > > > > > > > > > > to > > >> > > > > > > > > > > > > > > consider compaction into the equation as > > well? > > >> > > > Thanks. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Regards, > > >> > > > > > > > > > > > > > > Shahab > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
