Hi Mohith Before restarting again just disable the compression means set it to default and restart the cluster. This is just to ensure that the cluster is able to come back from the enable/disable problem. The Snappy problem could be different. I am suggesting this so that we can isolate the problem is because of compression.
Regards Ram > -----Original Message----- > From: Mohit Anchlia [mailto:mohitanch...@gmail.com] > Sent: Thursday, September 27, 2012 9:55 PM > To: user@hbase.apache.org > Subject: Re: disable table > > I did restart entire cluster and still that didn't help. Looks like > once I > get in this Race condition there is no way to come out of it? > > On Thu, Sep 27, 2012 at 8:00 AM, rajesh babu chintaguntla < > chrajeshbab...@gmail.com> wrote: > > > Hi Mohit, > > > > We should not delete znode's manually which will cause > inconsistencies like > > region may be shown as online on master, but it wont be on region > server. > > That's put is failing in your case. Master restart will bring back > your > > cluster to normal state(recovery any failures in enable/disable). > Even hbck > > also wont solve this problem. > > > > FYI, > > Presently discussion is going on this issue. You can follow jira > associated > > with this issue at > > https://issues.apache.org/jira/browse/HBASE-6469 > > > > On Thu, Sep 27, 2012 at 8:11 PM, Mohit Anchlia > <mohitanch...@gmail.com > > >wrote: > > > > > Thanks everyone for the input, it's helpful. I did remove the znode > from > > > /hbase/table/SESSIONID_TIMELINE and after that I was able to list > the > > > table. At that point I tried to do a "put" but when I did a put I > got a > > > message NoRegionServer online. I looked in the logs and it says the > > Failed > > > to open region server at nodexxx. When I went to nodexxx it > complains > > > something about unable to run testcompression. > > > > > > I setup SNAPPY compression on my table and I also ran SNAPPY > compression > > > test which was successful. Not sure what's going on in the cluster. > > > On Thu, Sep 27, 2012 at 1:10 AM, Mohammad Tariq > <donta...@gmail.com> > > > wrote: > > > > > > > Hello Mohit, > > > > > > > > It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies > for > > the > > > > typo. For rest of the things, I feel Ramkrishna sir has provided > a good > > > and > > > > proper explanation. Please let us know if you still have any > doubt or > > > > question. > > > > > > > > Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to > share > > > > space with you people. > > > > > > > > Regards, > > > > Mohammad Tariq > > > > > > > > > > > > > > > > On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan < > > > > ramkrishna.vasude...@huawei.com> wrote: > > > > > > > > > Hi Mohith > > > > > First of all thanks to Tariq for his replies. > > > > > > > > > > Just to add on, > > > > > Basically HBase uses the Zookeeper to know the status of the > cluster > > > > like > > > > > the no of tables enabled, disabled and deleted. > > > > > Enabled and deleted states are handled bit different in the > 0.94 > > > version. > > > > > > > > > > ZK is used for various region assignments. > > > > > > > > > > Also the ZK is used to track the Active master and standby > master. > > > > > > > > > > As you understand correctly that the master is responsible for > the > > > > overall > > > > > maintenance of the no of tables and their respective states, it > seeks > > > the > > > > > help of ZK to do it and that is where the states are persisted. > > > > > > > > > > Also there are few cases where the enable and disable table are > > having > > > > some > > > > > issues due to some race conditions in the 0.92 versions, In the > > latest > > > > > version we are trying to resolve them. > > > > > You can attach the master and RS logs to identify exactly what > caused > > > > this > > > > > problem in your case which will be really help ful so that I > can be > > > fixed > > > > > in > > > > > the kernel. > > > > > > > > > > Regards > > > > > Ram > > > > > > > > > > > -----Original Message----- > > > > > > From: Mohit Anchlia [mailto:mohitanch...@gmail.com] > > > > > > Sent: Thursday, September 27, 2012 5:09 AM > > > > > > To: user@hbase.apache.org > > > > > > Subject: Re: disable table > > > > > > > > > > > > I did /hbase/table/SESSIONID_TIMELINE and that seem to work. > I'll > > > > > > restart > > > > > > hbase and see if it works. > > > > > > > > > > > > One thing I don't understand is why is zookeeper holding > > information > > > > > > about > > > > > > this table if it is enabled or disabled? Wouldn't this > information > > be > > > > > > with > > > > > > master? > > > > > > > > > > > > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia > > > > > > <mohitanch...@gmail.com>wrote: > > > > > > > > > > > > > I don't see path like /hbase/SESSIONID_TIMELINE > > > > > > > This is what I see > > > > > > > > > > > > > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table > > > > > > > [SESSIONID_TIMELINE] > > > > > > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table > > > > > > > > > > > > > > cZxid = 0x1000000fe > > > > > > > ctime = Mon Sep 10 15:31:45 PDT 2012 > > > > > > > mZxid = 0x1000000fe > > > > > > > mtime = Mon Sep 10 15:31:45 PDT 2012 > > > > > > > pZxid = 0x5000008f1 > > > > > > > cversion = 3 > > > > > > > dataVersion = 0 > > > > > > > aclVersion = 0 > > > > > > > ephemeralOwner = 0x0 > > > > > > > dataLength = 0 > > > > > > > numChildren = 1 > > > > > > > > > > > > > > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq > > > > > > <donta...@gmail.com>wrote: > > > > > > > > > > > > > >> In order to delete a znode you have to go to the ZK shell > and > > > issue > > > > > > the > > > > > > >> "delete" command along with the required path. For example > : > > > > > > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you > can > > visit > > > > > > the ZK > > > > > > >> homepage at : zookeeper.apache.org > > > > > > >> > > > > > > >> Actually when we try to fetch data from an Hbase table, > the > > client > > > > > > or app > > > > > > >> first contacts the ZK to get the location of server > holding the > > > > > > >> -ROOT- table. From this we come to know about the server > hosting > > > the > > > > > > >> .META. > > > > > > >> table. This tells us the location of the server which > actually > > > holds > > > > > > the > > > > > > >> rows of interest. Because of some reasons the znode which > was > > > > > > holding this > > > > > > >> info has either faced some catastrophe or lost the info > > associated > > > > > > with > > > > > > >> this particular table. Or sometimes the znode remains > unable to > > > keep > > > > > > >> itself > > > > > > >> updated with the latest changes. That could also be a > probable > > > > > > reason. We > > > > > > >> should always keep in mind that ZK is the centralized > service > > that > > > > > > >> actually > > > > > > >> coordinating everything behind the scene. As a result, any > > problem > > > > > > to the > > > > > > >> ZK quorum means problem with Hbase custer. > > > > > > >> > > > > > > >> Regards, > > > > > > >> Mohammad Tariq > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia > > > > > > <mohitanch...@gmail.com > > > > > > >> >wrote: > > > > > > >> > > > > > > >> > Thanks! I do see Inconsistency. How do I remove the > znode. And > > > > > > also > > > > > > >> could > > > > > > >> > you please help me understand how this might have > happened? > > > > > > >> > > > > > > > >> > > > > > > > >> > ERROR: Region > > > > > > >> > > > > > > > > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. > > > not > > > > > > >> > deployed on any region server. > > > > > > >> > > > > > > > >> > > > > > > > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq > > > > > > <donta...@gmail.com> > > > > > > >> > wrote: > > > > > > >> > > > > > > > >> > > A possible reason could be that the znode associated > with > > this > > > > > > >> particular > > > > > > >> > > table is not behaving properly. In such case, you can > try > > the > > > > > > >> following: > > > > > > >> > > > > > > > > >> > > Stop Hbase > > > > > > >> > > Stop ZK > > > > > > >> > > Take a backup of ZK data > > > > > > >> > > Restart ZK > > > > > > >> > > Remove the znode > > > > > > >> > > Start Hbase again > > > > > > >> > > > > > > > > >> > > After this hopefully your table would be enabled. > > > > > > >> > > > > > > > > >> > > Regards, > > > > > > >> > > Mohammad Tariq > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq > > > > > > <donta...@gmail.com> > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > >> > > > Yes. Also have a look at the logs of the problematic > > region > > > if > > > > > > hbck > > > > > > >> > shows > > > > > > >> > > > any inconsistency. > > > > > > >> > > > > > > > > > >> > > > Regards, > > > > > > >> > > > Mohammad Tariq > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia < > > > > > > >> mohitanch...@gmail.com > > > > > > >> > > >wrote: > > > > > > >> > > > > > > > > > >> > > >> Which node should I look at for logs? Is this the > master > > > > > > node? I'll > > > > > > >> > try > > > > > > >> > > >> hbck. > > > > > > >> > > >> > > > > > > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq < > > > > > > >> donta...@gmail.com> > > > > > > >> > > >> wrote: > > > > > > >> > > >> > > > > > > >> > > >> > Hello Mohit, > > > > > > >> > > >> > > > > > > > >> > > >> > Try hbck once and see if it shows any > > inconsistency. > > > > > > Also, > > > > > > >> you > > > > > > >> > can > > > > > > >> > > >> try > > > > > > >> > > >> > restarting your cluster and deleting the table > again. > > > > > > Having a > > > > > > >> look > > > > > > >> > at > > > > > > >> > > >> the > > > > > > >> > > >> > logs could also be useful. > > > > > > >> > > >> > > > > > > > >> > > >> > Regards, > > > > > > >> > > >> > Mohammad Tariq > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia < > > > > > > >> > > mohitanch...@gmail.com > > > > > > >> > > >> > >wrote: > > > > > > >> > > >> > > > > > > > >> > > >> > > When I try to disable table I get: > > > > > > >> > > >> > > > > > > > > >> > > >> > > hbase(main):011:0> disable > 'SESSIONID_TIMELINE' > > > > > > >> > > >> > > ERROR: > > > org.apache.hadoop.hbase.TableNotEnabledException: > > > > > > >> > > >> > > > org.apache.hadoop.hbase.TableNotEnabledException: > > > > > > >> > SESSIONID_TIMELINE > > > > > > >> > > >> > > Here is some help for this command: > > > > > > >> > > >> > > Start disable of named table: e.g. "hbase> > disable > > > 't1'" > > > > > > >> > > >> > > > > > > > > >> > > >> > > But then I try to enable I get: > > > > > > >> > > >> > > > > > > > > >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE' > > > > > > >> > > >> > > ERROR: > > > org.apache.hadoop.hbase.TableNotDisabledException: > > > > > > >> > > >> > > > org.apache.hadoop.hbase.TableNotDisabledException: > > > > > > >> > > SESSIONID_TIMELINE > > > > > > >> > > >> > > Here is some help for this command: > > > > > > >> > > >> > > Start enable of named table: e.g. "hbase> > enable > > 't1'" > > > > > > >> > > >> > > > > > > > > >> > > >> > > I've tried flush, major_compaction also. I > tseems > > it's > > > > > > stuck in > > > > > > >> > > >> > > inconsistent state. Could someone point me to > correct > > > > > > >> direction? I > > > > > > >> > > am > > > > > > >> > > >> > using > > > > > > >> > > >> > > 92.1 > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
