[ https://issues.apache.org/jira/browse/HBASE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405945#comment-13405945 ]
stack commented on HBASE-6294: ------------------------------ Thats a bug Lars. Aim is to just stop/start to move 0.94 to 0.96. Most of the znodes are automigrated. I must have missed some. I made HBASE-6316 to make sure this not necessary going to 0.96. > Detect leftover data in ZK after a user delete all its HBase data > ----------------------------------------------------------------- > > Key: HBASE-6294 > URL: https://issues.apache.org/jira/browse/HBASE-6294 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.94.0 > Reporter: Jean-Daniel Cryans > Priority: Critical > Fix For: 0.96.0, 0.94.1 > > > It seems we have a new failure mode when a user deletes the hbase root.dir > but doesn't delete the ZK data. For example a user on IRC came with this log: > {noformat} > 2012-06-30 09:07:48,017 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open > region: kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02. > 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: > The following folder is in HBase's root directory and doesn't contain a table > descriptor, do consider deleting it: kw > 2012-06-30 09:07:48,018 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > regionserver:34193-0x1383bfe01b70001 Attempting to transition node > 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to > RS_ZK_REGION_OPENING > 2012-06-30 09:07:48,018 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694, > region=2e8a318837602c9c9961e9d690b7fd02 > 2012-06-30 09:07:48,020 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_FAILED_OPEN, server=localhost,34193,1341036300138, > region=b254af24c9127b8bb22cb6d24e523dad > 2012-06-30 09:07:48,020 DEBUG > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED > event for b254af24c9127b8bb22cb6d24e523dad > 2012-06-30 09:07:48,020 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; > was=kw_r,,1340981822374.b254af24c9127b8bb22cb6d24e523dad. state=CLOSED, > ts=1341036467998, server=localhost,34193,1341036300138 > 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > master:50890-0x1383bfe01b70000 Creating (or updating) unassigned node for > b254af24c9127b8bb22cb6d24e523dad with OFFLINE state > 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > regionserver:34193-0x1383bfe01b70001 Successfully transitioned node > 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to > RS_ZK_REGION_OPENING > 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Opening region: {NAME => > 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.', STARTKEY => '', ENDKEY > => '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,} > 2012-06-30 09:07:48,029 ERROR > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open > of region=kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02., starting to > roll back the global memstore size. > java.lang.IllegalStateException: Could not instantiate a region instance. > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3490) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3628) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:532) > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3487) > ... 7 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:133) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:125) > at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:411) > ... 11 more > 2012-06-30 09:07:48,031 INFO > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of > region {NAME => 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.', > STARTKEY => '', ENDKEY => '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,} > failed, marking as FAILED_OPEN in ZK > 2012-06-30 09:07:48,032 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > regionserver:34193-0x1383bfe01b70001 Attempting to transition node > 2e8a318837602c9c9961e9d690b7fd02 from RS_ZK_REGION_OPENING to > RS_ZK_REGION_FAILED_OPEN > 2012-06-30 09:07:48,031 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_OPENING, server=localhost,34193,1341036300138, > region=2e8a318837602c9c9961e9d690b7fd02 > 2012-06-30 09:07:48,043 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694, > region=b254af24c9127b8bb22cb6d24e523dad > {noformat} > The exception itself is not very useful, nor is the NPE deep in the coproc > stack. What was really useful was this: > {noformat} > 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: > The following folder is in HBase's root directory and doesn't contain a table > descriptor, do consider deleting it: kw > {noformat} > So the HBase wants to assign a region from a table that doesn't exist and we > fail in an obscure way. I told the user to shut down HBase, nuke > /tmp/hbase-user as it will contain both the HBase data and the ZK data, and > restart. It worked. > This situation is new in 0.94, we need to detect it so our users have a > better experience getting started with HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira