Re: Regions in transition

2013-02-07 Thread kiran
Also, I did try repairHoles, all fix options but in vain. I restarted master atleast thrice. On Fri, Feb 8, 2013 at 11:14 AM, kiran wrote: > > Dear All, > > How to move/reassign regions that are always in transition ? Because of > the cluster is not balancing and regionservers are going dow

Re: Regions in transition

2013-02-07 Thread samar kumar
Regions should never overlap Incase a region is in transition for a long long time, you could possibly force Unassign a region , if you are ok to lose the region. Regards, Samar On 08/02/13 11:14 AM, "kiran" wrote: > are always in transition ? Because of the >cluster is not balancing and regions

Re: Regions in transition

2013-02-07 Thread kiran
i issued unassign and close_region both but the region is still in transition also I deleted the .META entry for the region. Do I need to restart master ? On Fri, Feb 8, 2013 at 11:18 AM, samar kumar wrote: > Regions should never overlap > Incase a region is in transition for a long long time, y

Re: Regions in transition

2013-02-07 Thread samar kumar
Was the unassigning successful ? If not you can force it by - unassign 'REGIONNAME', true Regards, Samar On 08/02/13 12:01 PM, "kiran" wrote: >i issued unassign and close_region both but the region is still in >transition also I deleted the .META entry for the region. Do I need to >restart m

Re: Regions in transition

2013-02-07 Thread kiran
We ran the command unassign 'REGIONNAME',true the output is "completed in 57 sec", but still region is in transition. On Fri, Feb 8, 2013 at 12:05 PM, samar kumar wrote: > Was the unassigning successful ? > > If not you can force it by - unassign 'REGIONNAME', true > > Regards, > Samar > > > > O

Re: Regions in transition

2013-02-07 Thread samar kumar
Can you mention the state of the region.. You can find the details in you master status page On 08/02/13 12:09 PM, "kiran" wrote: >We ran the command unassign 'REGIONNAME',true the output is "completed in >57 sec", but still region is in transition. > > >On Fri, Feb 8, 2013 at 12:05 PM, samar ku

Re: Regions in transition

2013-02-07 Thread kiran
PENDING_OPEN On Fri, Feb 8, 2013 at 12:16 PM, samar kumar wrote: > Can you mention the state of the region.. You can find the details in you > master status page > > On 08/02/13 12:09 PM, "kiran" wrote: > > >We ran the command unassign 'REGIONNAME',true the output is "completed in > >57 sec", b

Re: Regions in transition

2013-02-07 Thread kiran
Hi, I tried to run hbase hbck -fixMeta after master restart. and in the output I got There is a hole in the region chain between You need to create a new .regioninfo and region dir in hdfs to plug the hole for about 10 odd regions What does this mean ? On Fri, Feb 8, 2013 at 12:20 PM, kiran

Re: Regions in transition

2013-02-07 Thread ramkrishna vasudevan
What is the version that you are using? You can run the tool called HBCK which should help you. But before running this what operation were you trying to make? DISABLED and then ENABLED a table or it was just normal region assignments while creating table or region balancing? Regard Ram On Fri,

Re: Regions in transition

2013-02-07 Thread Samir Ahmic
Hi, Kiran Welcome to beautiful world of HBase transition states :) . When i face RIT issue this are steps that i use to resolve it: 1. hbase hbck -fixAssignments (this depends on your version of hbase it can be also just -fix) If you don't have luck with 1. then you will need manual intervention

Re: Regions in transition

2013-02-07 Thread ramkrishna vasudevan
Yes, RIT states were bit confusing for the end user. In the latest version lot of efforts have gone into solve this problems. Hope for the best in the future :) Regards Ram On Fri, Feb 8, 2013 at 12:38 PM, Samir Ahmic wrote: > Hi, Kiran > > Welcome to beautiful world of HBase transition states

Re: Regions in transition

2013-02-07 Thread samar kumar
May be ./bin/hbase hbck -repairHoles could be helpful.. On 08/02/13 12:38 PM, "Samir Ahmic" wrote: >Hi, Kiran > >Welcome to beautiful world of HBase transition states :) . When i face >RIT issue this are steps that i use to resolve it: > >1. hbase hbck -fixAssignments (this depends on your ver

Re: Regions in transition

2013-02-07 Thread kiran
Thanks samar and ramakrishna I did use option 2 and deleted /hbase from the zkcli shell. I restarted the cluster now there are more regions in transition. Will it take some time to complete since it is a new assignment ? My hase version is 0.94.1, and I have a doubt with the property hbase.hregio

Re: Regions in transition

2013-02-08 Thread kiran
after some searching i found that main reason for regions with same keys is splitting and it is the culprit in our case for inconsistency. But I set my filesize to a very large size but i am unsure why still splitting is happening On Fri, Feb 8, 2013 at 1:20 PM, kiran wrote: > Thanks samar and

Re: Regions in transition

2013-02-09 Thread ramkrishna vasudevan
Okie. So if you don't mind can you attach your logs. What is the split policy and what is the size of the region? Its better we take a look at it and solve the problem if it is a kernel problem. Regards Ram On Fri, Feb 8, 2013 at 6:09 PM, kiran wrote: > after some searching i found that main

Re: Regions in transition

2013-02-10 Thread kiran
I think it might be reason for the splits. From 0.94.0, seems the default split policy has changes http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html On Sat, Feb 9, 2013 at 3:37 PM, ramkrishna vasudevan < ramkrishna.s.vasude...@gmail.com> wrot

Re: regions in transition

2015-12-21 Thread Ted Yu
Can you pick a few regions stuck in transition and check related region server logs to see why they couldn't be assigned ? Which release were you using previously ? Thanks On Mon, Dec 21, 2015 at 3:54 PM, Brian Jeltema wrote: > I am doing a cluster upgrade to the HDP 2.2 stack. For some reason

Re: regions in transition

2015-12-22 Thread Brian Jeltema
Following up, if I look at the MBase Master UI in the Ambari console I see links to all of the region servers. If I click on those links, the Region Server page comes up and in the Regions section, is displays ‘Not serving regions’. I’m not sure if that means something is disabled, or it just doe

Re: regions in transition

2015-12-22 Thread Brian Jeltema
> > Can you pick a few regions stuck in transition and check related region > server logs to see why they couldn't be assigned ? I don’t see anything in the region logs relating any regions. > > Which release were you using previously ? HDP 2.1 -> HDP 2.2 So is it safe to stop HBase and dele

Re: regions in transition

2015-12-22 Thread Ted Yu
>From RegionListTmpl.jamon : <%if (onlineRegions != null && onlineRegions.size() > 0) %> ... <%else> Not serving regions The message means that there was no region online on the underlying server. FYI On Tue, Dec 22, 2015 at 7:18 AM, Brian Jeltema wrote: > Following up, if I look at the

Re: regions in transition

2015-12-22 Thread Geovanie Marquez
check hmaster:60010 under TASKS (between Software Attributes and Tables) you will see if you have regions in transition. This will tell you which regions are transitioning and you can go to those region server logs and check them, I've run into a couple of these and every time they've talk to me ab

Re: regions in transition

2015-12-22 Thread Brian Jeltema
I’m running Ambari 2.0.2 and HPD 2.2. I don’t see any of this displayed at master:60010. I really think this problem is the result of cruft in ZooKeeper. Does anybody know if it’s safe to delete the node? > On Dec 22, 2015, at 11:40 AM, Geovanie Marquez > wrote: > > check hmaster:60010 und

Re: regions in transition

2015-12-22 Thread Brian Jeltema
Thanks. Can I safely delete the ZK node? > On Dec 22, 2015, at 11:29 AM, Ted Yu wrote: > > From RegionListTmpl.jamon : > > <%if (onlineRegions != null && onlineRegions.size() > 0) %> > ... > <%else> > Not serving regions > > > The message means that there was no region online on the underly

Re: regions in transition

2015-12-22 Thread Geovanie Marquez
You should really find out where you hmaster ui lives (there is a master UI for every node provided by the apache project) because it gives you information on the state of your system, but if you want to skip all that, here are the instructions for OfflineRepair, without knowing what is happening w

Re: regions in transition

2015-12-22 Thread Brian Jeltema
> > You should really find out where you hmaster ui lives (there is a master UI > for every node provided by the apache project) because it gives you > information on the state of your system, I’m familiar with the HMaster UI. I’m looking at it now. It does not contain the information you describ

Re: regions in transition

2015-12-23 Thread Brian Jeltema
Update on this: deleting the contents the /hbase-unsecure/region-in-transition node did fix my problem with HBase finding my table regions. I'm still have a problem though, possibly related. I’m seeing OutOfMemory errors in the region server logs (modified slightly): 2015-12-23 06:52:37,466 IN

Re: regions in transition

2015-12-23 Thread Brian Jeltema
I appear to have resolved the OOM error by greatly increasing the max process limit (to 64K). Using HDP 2.1 a limit of 1024 seemed to be working OK. I’m surprised I had to make a change of this magnitude. Brian > On Dec 23, 2015, at 7:20 AM, Brian Jeltema wrote: > > Update on this: > > delet

Re: Regions in Transition?

2012-01-20 Thread Stack
On Fri, Jan 20, 2012 at 10:41 AM, Mark wrote: > I recently bumped up the region size memory configuration on our HBase > cluster and after doing a rolling restart of our 5 nodes I saw the > following in the HBase status page: > > > Regions in Transition > > Region State > 32cb0e36cfa326d0a4317

Re: Regions in Transition?

2012-01-20 Thread lars hofhansl
, 2012 1:16 PM Subject: Re: Regions in Transition? On Fri, Jan 20, 2012 at 10:41 AM, Mark wrote: > I recently bumped up the region size memory configuration on our HBase > cluster and after doing a rolling restart of our 5 nodes I saw the > following in the HBase status page: > >

Re: Regions in Transition?

2012-01-20 Thread Harsh J
; HBase has to load (either HDFS or cache) and scan this amount of data for > each key value lookup. > > > -- Lars > > > > > From: Stack > To: user@hbase.apache.org > Sent: Friday, January 20, 2012 1:16 PM > Subject: Re: R

RE: Regions in Transition?

2012-01-20 Thread Ramakrishna s vasudevan
, 2012 11:47 AM To: user@hbase.apache.org Subject: Re: Regions in Transition? (Just to avoid confusion, Lars is talking about HBase Table CF's blocksize, unrelated to HDFS block sizes, which would be fine if set to 512m.) On 21-Jan-2012, at 9:49 AM, lars hofhansl wrote: > Also, you probabl

Re: Regions in Transition: FAILED_CLOSE status

2017-05-23 Thread jeff saremi
Our write code throws exceptions like the following: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 10331 actions: NotServingRegionException: 10331 times,at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:258) at org.apache.

Re: Regions in Transition: FAILED_CLOSE status

2017-05-23 Thread Vladimir Rodionov
You should check RS logs to see why regions can not be assigned. Get RS name from master log and check RS log -Vlad On Tue, May 23, 2017 at 11:47 AM, jeff saremi wrote: > Our write code throws exceptions like the following: > > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException

Re: Regions in Transition: FAILED_CLOSE status

2017-05-23 Thread jeff saremi
Are dead region servers to blame? Is this possibly stale information in the ZK? From: Vladimir Rodionov Sent: Tuesday, May 23, 2017 12:20:16 PM To: user@hbase.apache.org Subject: Re: Regions in Transition: FAILED_CLOSE status You should check RS logs to see why

Re: Regions in Transition: FAILED_CLOSE status

2017-05-23 Thread Vladimir Rodionov
wrote: > Are dead region servers to blame? Is this possibly stale information in > the ZK? > > > From: Vladimir Rodionov > Sent: Tuesday, May 23, 2017 12:20:16 PM > To: user@hbase.apache.org > Subject: Re: Regions in Transition: FAILED_CLO

Re: Regions in Transition: FAILED_CLOSE status

2017-05-23 Thread James Moore
t; > > > > > From: Vladimir Rodionov > > Sent: Tuesday, May 23, 2017 12:20:16 PM > > To: user@hbase.apache.org > > Subject: Re: Regions in Transition: FAILED_CLOSE status > > > > You should check RS logs to see why regions

Re: Regions in Transition: FAILED_CLOSE status

2017-05-23 Thread Vladimir Rodionov
> > > From: Vladimir Rodionov > > > Sent: Tuesday, May 23, 2017 12:20:16 PM > > > To: user@hbase.apache.org > > > Subject: Re: Regions in Transition: FAILED_CLOSE status > > > > > > You should check RS logs to see

Re: Regions in Transition: FAILED_CLOSE status

2017-05-23 Thread jeff saremi
to do this step) and now all regions show as fine Happy now! From: Vladimir Rodionov Sent: Tuesday, May 23, 2017 2:41:30 PM To: user@hbase.apache.org Subject: Re: Regions in Transition: FAILED_CLOSE status My bad, that is FAIL_CLOSE Anyway, start with Master log,