I did restart entire cluster and still that didn't help. Looks like once I get in this Race condition there is no way to come out of it?
On Thu, Sep 27, 2012 at 8:00 AM, rajesh babu chintaguntla < chrajeshbab...@gmail.com> wrote: > Hi Mohit, > > We should not delete znode's manually which will cause inconsistencies like > region may be shown as online on master, but it wont be on region server. > That's put is failing in your case. Master restart will bring back your > cluster to normal state(recovery any failures in enable/disable). Even hbck > also wont solve this problem. > > FYI, > Presently discussion is going on this issue. You can follow jira associated > with this issue at > https://issues.apache.org/jira/browse/HBASE-6469 > > On Thu, Sep 27, 2012 at 8:11 PM, Mohit Anchlia <mohitanch...@gmail.com > >wrote: > > > Thanks everyone for the input, it's helpful. I did remove the znode from > > /hbase/table/SESSIONID_TIMELINE and after that I was able to list the > > table. At that point I tried to do a "put" but when I did a put I got a > > message NoRegionServer online. I looked in the logs and it says the > Failed > > to open region server at nodexxx. When I went to nodexxx it complains > > something about unable to run testcompression. > > > > I setup SNAPPY compression on my table and I also ran SNAPPY compression > > test which was successful. Not sure what's going on in the cluster. > > On Thu, Sep 27, 2012 at 1:10 AM, Mohammad Tariq <donta...@gmail.com> > > wrote: > > > > > Hello Mohit, > > > > > > It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies for > the > > > typo. For rest of the things, I feel Ramkrishna sir has provided a good > > and > > > proper explanation. Please let us know if you still have any doubt or > > > question. > > > > > > Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to share > > > space with you people. > > > > > > Regards, > > > Mohammad Tariq > > > > > > > > > > > > On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan < > > > ramkrishna.vasude...@huawei.com> wrote: > > > > > > > Hi Mohith > > > > First of all thanks to Tariq for his replies. > > > > > > > > Just to add on, > > > > Basically HBase uses the Zookeeper to know the status of the cluster > > > like > > > > the no of tables enabled, disabled and deleted. > > > > Enabled and deleted states are handled bit different in the 0.94 > > version. > > > > > > > > ZK is used for various region assignments. > > > > > > > > Also the ZK is used to track the Active master and standby master. > > > > > > > > As you understand correctly that the master is responsible for the > > > overall > > > > maintenance of the no of tables and their respective states, it seeks > > the > > > > help of ZK to do it and that is where the states are persisted. > > > > > > > > Also there are few cases where the enable and disable table are > having > > > some > > > > issues due to some race conditions in the 0.92 versions, In the > latest > > > > version we are trying to resolve them. > > > > You can attach the master and RS logs to identify exactly what caused > > > this > > > > problem in your case which will be really help ful so that I can be > > fixed > > > > in > > > > the kernel. > > > > > > > > Regards > > > > Ram > > > > > > > > > -----Original Message----- > > > > > From: Mohit Anchlia [mailto:mohitanch...@gmail.com] > > > > > Sent: Thursday, September 27, 2012 5:09 AM > > > > > To: user@hbase.apache.org > > > > > Subject: Re: disable table > > > > > > > > > > I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll > > > > > restart > > > > > hbase and see if it works. > > > > > > > > > > One thing I don't understand is why is zookeeper holding > information > > > > > about > > > > > this table if it is enabled or disabled? Wouldn't this information > be > > > > > with > > > > > master? > > > > > > > > > > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia > > > > > <mohitanch...@gmail.com>wrote: > > > > > > > > > > > I don't see path like /hbase/SESSIONID_TIMELINE > > > > > > This is what I see > > > > > > > > > > > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table > > > > > > [SESSIONID_TIMELINE] > > > > > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table > > > > > > > > > > > > cZxid = 0x1000000fe > > > > > > ctime = Mon Sep 10 15:31:45 PDT 2012 > > > > > > mZxid = 0x1000000fe > > > > > > mtime = Mon Sep 10 15:31:45 PDT 2012 > > > > > > pZxid = 0x5000008f1 > > > > > > cversion = 3 > > > > > > dataVersion = 0 > > > > > > aclVersion = 0 > > > > > > ephemeralOwner = 0x0 > > > > > > dataLength = 0 > > > > > > numChildren = 1 > > > > > > > > > > > > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq > > > > > <donta...@gmail.com>wrote: > > > > > > > > > > > >> In order to delete a znode you have to go to the ZK shell and > > issue > > > > > the > > > > > >> "delete" command along with the required path. For example : > > > > > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you can > visit > > > > > the ZK > > > > > >> homepage at : zookeeper.apache.org > > > > > >> > > > > > >> Actually when we try to fetch data from an Hbase table, the > client > > > > > or app > > > > > >> first contacts the ZK to get the location of server holding the > > > > > >> -ROOT- table. From this we come to know about the server hosting > > the > > > > > >> .META. > > > > > >> table. This tells us the location of the server which actually > > holds > > > > > the > > > > > >> rows of interest. Because of some reasons the znode which was > > > > > holding this > > > > > >> info has either faced some catastrophe or lost the info > associated > > > > > with > > > > > >> this particular table. Or sometimes the znode remains unable to > > keep > > > > > >> itself > > > > > >> updated with the latest changes. That could also be a probable > > > > > reason. We > > > > > >> should always keep in mind that ZK is the centralized service > that > > > > > >> actually > > > > > >> coordinating everything behind the scene. As a result, any > problem > > > > > to the > > > > > >> ZK quorum means problem with Hbase custer. > > > > > >> > > > > > >> Regards, > > > > > >> Mohammad Tariq > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia > > > > > <mohitanch...@gmail.com > > > > > >> >wrote: > > > > > >> > > > > > >> > Thanks! I do see Inconsistency. How do I remove the znode. And > > > > > also > > > > > >> could > > > > > >> > you please help me understand how this might have happened? > > > > > >> > > > > > > >> > > > > > > >> > ERROR: Region > > > > > >> > > > > > > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. > > not > > > > > >> > deployed on any region server. > > > > > >> > > > > > > >> > > > > > > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq > > > > > <donta...@gmail.com> > > > > > >> > wrote: > > > > > >> > > > > > > >> > > A possible reason could be that the znode associated with > this > > > > > >> particular > > > > > >> > > table is not behaving properly. In such case, you can try > the > > > > > >> following: > > > > > >> > > > > > > > >> > > Stop Hbase > > > > > >> > > Stop ZK > > > > > >> > > Take a backup of ZK data > > > > > >> > > Restart ZK > > > > > >> > > Remove the znode > > > > > >> > > Start Hbase again > > > > > >> > > > > > > > >> > > After this hopefully your table would be enabled. > > > > > >> > > > > > > > >> > > Regards, > > > > > >> > > Mohammad Tariq > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq > > > > > <donta...@gmail.com> > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > > > Yes. Also have a look at the logs of the problematic > region > > if > > > > > hbck > > > > > >> > shows > > > > > >> > > > any inconsistency. > > > > > >> > > > > > > > > >> > > > Regards, > > > > > >> > > > Mohammad Tariq > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia < > > > > > >> mohitanch...@gmail.com > > > > > >> > > >wrote: > > > > > >> > > > > > > > > >> > > >> Which node should I look at for logs? Is this the master > > > > > node? I'll > > > > > >> > try > > > > > >> > > >> hbck. > > > > > >> > > >> > > > > > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq < > > > > > >> donta...@gmail.com> > > > > > >> > > >> wrote: > > > > > >> > > >> > > > > > >> > > >> > Hello Mohit, > > > > > >> > > >> > > > > > > >> > > >> > Try hbck once and see if it shows any > inconsistency. > > > > > Also, > > > > > >> you > > > > > >> > can > > > > > >> > > >> try > > > > > >> > > >> > restarting your cluster and deleting the table again. > > > > > Having a > > > > > >> look > > > > > >> > at > > > > > >> > > >> the > > > > > >> > > >> > logs could also be useful. > > > > > >> > > >> > > > > > > >> > > >> > Regards, > > > > > >> > > >> > Mohammad Tariq > > > > > >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia < > > > > > >> > > mohitanch...@gmail.com > > > > > >> > > >> > >wrote: > > > > > >> > > >> > > > > > > >> > > >> > > When I try to disable table I get: > > > > > >> > > >> > > > > > > > >> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE' > > > > > >> > > >> > > ERROR: > > org.apache.hadoop.hbase.TableNotEnabledException: > > > > > >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException: > > > > > >> > SESSIONID_TIMELINE > > > > > >> > > >> > > Here is some help for this command: > > > > > >> > > >> > > Start disable of named table: e.g. "hbase> disable > > 't1'" > > > > > >> > > >> > > > > > > > >> > > >> > > But then I try to enable I get: > > > > > >> > > >> > > > > > > > >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE' > > > > > >> > > >> > > ERROR: > > org.apache.hadoop.hbase.TableNotDisabledException: > > > > > >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException: > > > > > >> > > SESSIONID_TIMELINE > > > > > >> > > >> > > Here is some help for this command: > > > > > >> > > >> > > Start enable of named table: e.g. "hbase> enable > 't1'" > > > > > >> > > >> > > > > > > > >> > > >> > > I've tried flush, major_compaction also. I tseems > it's > > > > > stuck in > > > > > >> > > >> > > inconsistent state. Could someone point me to correct > > > > > >> direction? I > > > > > >> > > am > > > > > >> > > >> > using > > > > > >> > > >> > > 92.1 > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
