Hello, Thanks a lot, the table is back online. One last question ? Can you provide a log pattern to spot this ? Just in case it occurs again ;-)
Regards, Adam. ________________________________________ De : Matteo Bertozzi <[email protected]> Envoyé : mardi 22 novembre 2016 19:48 À : [email protected] Objet : Re: Table is disabled an no way to get it back online hadoop fs -rmr /hbase/MasterProcWALs Matteo On Tue, Nov 22, 2016 at 10:42 AM, Cecile, Adam <[email protected]> wrote: > Can you explain me how to delete masterwals directory? > > > > Sent from my Samsung device > > > -------- Original message -------- > From: Matteo Bertozzi <[email protected]> > Date: 22/11/2016 19:12 (GMT+01:00) > To: [email protected] > Subject: Re: Table is disabled an no way to get it back online > > I don't think this has anything to do with HBASE-13415 or the bugfix it > related to it i'm working on. > > this is probably the usual case of mismatch state with zk. Disable is > saying that the table is already not disabled. > so, enableTable() is the one that should give any exception in case. but I > don't see any. > I suggest to just drop the MasterWALs directory, drop the znode > /hbase/table/sentinel-meta, restart the master and try disable and then > enable. that should bring you back with the table online > > Matteo > > > On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <[email protected]> > wrote: > > > Thanks for everything. As you said, this bug is supposed to be fixed in > > 1.2.0. Matteo is reading this list as well ? > > > > Regards, Adam. > > ________________________________________ > > De : Ted Yu <[email protected]> > > Envoyé : mardi 22 novembre 2016 19:00 > > À : [email protected] > > Objet : Re: Table is disabled an no way to get it back online > > > > Please take a look at HBASE-13415 > > > > From the log, you're using hbase 1.2.0 already. But I heard there is a > > subtle bug which is being fixed. > > > > Matteo is the person with best knowledge in this regard. > > > > On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <[email protected]> > > wrote: > > > > > Another one, because I'm not sure the log is overwritten when > restarting. > > > This one has been cleared before service start. > > > ________________________________________ > > > De : Cecile, Adam <[email protected]> > > > Envoyé : mardi 22 novembre 2016 18:42 > > > À : [email protected] > > > Objet : RE: Table is disabled an no way to get it back online > > > > > > Hello, > > > > > > Sadly I could not use the webui, it killed my firefox (probably way too > > > much time). Here is the debug log... (11Mb uncompressed for maybe two > > > minutes running !!) > > > > > > Best regards, Adam. > > > ________________________________________ > > > De : Ted Yu <[email protected]> > > > Envoyé : mardi 22 novembre 2016 17:05 > > > À : [email protected] > > > Objet : Re: Table is disabled an no way to get it back online > > > > > > In log4j.properties : > > > > > > log4j.logger.org.apache.hadoop.hbase=DEBUG > > > > > > On master UI, you can select the Procedures tab. Pastebin what you see > > > (text is enough). > > > > > > Thanks > > > > > > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <[email protected]> > > > wrote: > > > > > > > Hey Ted, > > > > > > > > Thank you. Heading home right now but I'll start the laptop again. > Not > > > > sure exactly how I should turn debug log on so if you have the > > > information > > > > it'd be appreciated, otherwise I'll look at the xml files. > > > > > > > > Regards, Adam. > > > > ________________________________________ > > > > De : Ted Yu <[email protected]> > > > > Envoyé : mardi 22 novembre 2016 15:46 > > > > À : [email protected] > > > > Objet : Re: Table is disabled an no way to get it back online > > > > > > > > Master log contained entries in the following form: > > > > > > > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3] > > > > procedure2.ProcedureExecutor: Rolledback procedure > > DisableTableProcedure > > > > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK > > > > exec-time=242hrs, 10mins, 28.896sec > > > > exception=org.apache.hadoop.hbase.TableNotEnabledException: > > > sentinel-meta > > > > > > > > Note the procedure Id was around 43000, far lower than 147464. > > > > > > > > Can you turn debug log on and repost master log ? > > > > > > > > Thanks > > > > > > > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <[email protected]> > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > > > > > > > We're having a table stuck in disabled state. First I'd like to > start > > > > with > > > > > what I tried already: > > > > > > > > > > > > > > > * Restart all machines involved in HBase cluster > > > > > > > > > > * hbase hbck with varios arguments > > > > > > > > > > * hdfs fsck > > > > > > > > > > * Purge ZK /hbase and restart masters > > > > > > > > > > > > > > > Now more details anout what happens: > > > > > > > > > > * When enabling from hbase shell: > > > > > > > > > > > > > > > hbase(main):002:0> enable "sentinel-meta" > > > > > ERROR: The procedure 147464 is still running > > > > > > > > > > > > > > > The task ID changes every time I run the command so I think it's > > > talking > > > > > about itself (and it gets stuck for a while before saying anything) > > > > > > > > > > > > > > > In the log, all I can see is: > > > > > > > > > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] > > > > > procedure2.ProcedureExecutor: Rolledback procedure > > > DisableTableProcedure > > > > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK > > > > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop. > > hbase. > > > > TableNotEnabledException: > > > > > sentinel-meta > > > > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure. > > > > DisableTableProcedure: > > > > > Table sentinel-meta isn't enabled; skipping disable > > > > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] > > > > > procedure2.ProcedureExecutor: Rolledback procedure > > > DisableTableProcedure > > > > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK > > > > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop. > > hbase. > > > > TableNotEnabledException: > > > > > sentinel-meta > > > > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure. > > > > DisableTableProcedure: > > > > > Table sentinel-meta isn't enabled; skipping disable > > > > > > > > > > > > > > > Please also find attached a complete log from startup to shutdown > on > > a > > > > > single active master. You'll see the table is found as well as the > > > > regions > > > > > but it gets deactivated with no reason. > > > > > > > > > > > > > > > Thanks a lot for your help, we're kinda running out if ideas here. > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > Adam. > > > > > > > > > > > > > > > > > > > > > > > > >
