In log4j.properties : log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see (text is enough). Thanks On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <[email protected]> wrote: > Hey Ted, > > Thank you. Heading home right now but I'll start the laptop again. Not > sure exactly how I should turn debug log on so if you have the information > it'd be appreciated, otherwise I'll look at the xml files. > > Regards, Adam. > ________________________________________ > De : Ted Yu <[email protected]> > Envoyé : mardi 22 novembre 2016 15:46 > À : [email protected] > Objet : Re: Table is disabled an no way to get it back online > > Master log contained entries in the following form: > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3] > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK > exec-time=242hrs, 10mins, 28.896sec > exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta > > Note the procedure Id was around 43000, far lower than 147464. > > Can you turn debug log on and repost master log ? > > Thanks > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <[email protected]> > wrote: > > > Hello, > > > > > > We're having a table stuck in disabled state. First I'd like to start > with > > what I tried already: > > > > > > * Restart all machines involved in HBase cluster > > > > * hbase hbck with varios arguments > > > > * hdfs fsck > > > > * Purge ZK /hbase and restart masters > > > > > > Now more details anout what happens: > > > > * When enabling from hbase shell: > > > > > > hbase(main):002:0> enable "sentinel-meta" > > ERROR: The procedure 147464 is still running > > > > > > The task ID changes every time I run the command so I think it's talking > > about itself (and it gets stuck for a while before saying anything) > > > > > > In the log, all I can see is: > > > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] > > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase. > TableNotEnabledException: > > sentinel-meta > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure. > DisableTableProcedure: > > Table sentinel-meta isn't enabled; skipping disable > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] > > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase. > TableNotEnabledException: > > sentinel-meta > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure. > DisableTableProcedure: > > Table sentinel-meta isn't enabled; skipping disable > > > > > > Please also find attached a complete log from startup to shutdown on a > > single active master. You'll see the table is found as well as the > regions > > but it gets deactivated with no reason. > > > > > > Thanks a lot for your help, we're kinda running out if ideas here. > > > > > > Best regards, > > > > > > Adam. > > > > > > >
