[ https://issues.apache.org/jira/browse/HBASE-26391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
anonymous updated HBASE-26391: ------------------------------ Description: There is a table {{mytable}} in a HBase cluster with two HMasters and two RegionServers. 1. Current active HMaster is HMaster1; 2. Client request to delete the table {{mytable}}; 3. HMaster1 deletes znode in ZooKeeper: /hbase/table/mytable {code:java} [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1367), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1359), org.apache.hadoop.hbase.zookeeper.ZKTableStateManager.setDeletedTable(ZKTableStateManager.java:244), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteAssignmentState(DeleteTableProcedure.java:383), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:136), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:57), org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139), org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1166), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:954), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)] {code} 4. HMaster1 crashes before deleting the znode: /hbase/table-lock/mytable {code:java} [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1341), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1330), org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.tableDeleted(TableLockManager.java:442), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.markTableAsDeleted(MasterProcedureScheduler.java:895), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.completionCleanup(MasterProcedureScheduler.java:261), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.procedureFinished(ProcedureExecutor.java:1415), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:974), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)] {code} 5. HMaster2 becomes active 6. The table {{mytable}} was deleted, but the znode "/hbase/table-lock/mytable" was left on ZooKeeper was: There is a table {{mytable}} in a HBase cluster with two HMasters and two RegionServers. 1. Current active HMaster is HMaster1; 2. Client request to delete the table {{mytable}}; 3. HMaster1 deletes znode in ZooKeeper: /hbase/table/mytable {code:java} [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1367), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1359), org.apache.hadoop.hbase.zookeeper.ZKTableStateManager.setDeletedTable(ZKTableStateManager.java:244), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteAssignmentState(DeleteTableProcedure.java:383), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:136), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:57), org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139), org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1166), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:954), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)] {code} 4. HMaster1 crashes before deleting the znode: /hbase/table-lock/mytable {code:java} [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1341), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1330), org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.tableDeleted(TableLockManager.java:442), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.markTableAsDeleted(MasterProcedureScheduler.java:895), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.completionCleanup(MasterProcedureScheduler.java:261), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.procedureFinished(ProcedureExecutor.java:1415), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:974), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)] {code} 5. HMaster2 becomes active 6. The table {{mytablle}} was deleted, but the znode "/hbase/table-lock/mytable" was left on ZooKeeper > Unexpected crash in the process of deleting a table causes stale data in > ZooKeeper > ---------------------------------------------------------------------------------- > > Key: HBASE-26391 > URL: https://issues.apache.org/jira/browse/HBASE-26391 > Project: HBase > Issue Type: Bug > Affects Versions: 1.7.1 > Reporter: anonymous > Priority: Major > > There is a table {{mytable}} in a HBase cluster with two HMasters and two > RegionServers. > 1. Current active HMaster is HMaster1; > 2. Client request to delete the table {{mytable}}; > 3. HMaster1 deletes znode in ZooKeeper: /hbase/table/mytable > {code:java} > [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), > > org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1367), > > org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1359), > > org.apache.hadoop.hbase.zookeeper.ZKTableStateManager.setDeletedTable(ZKTableStateManager.java:244), > > org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteAssignmentState(DeleteTableProcedure.java:383), > > org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:136), > > org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:57), > > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139), > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499), > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1166), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:954), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)] > {code} > 4. HMaster1 crashes before deleting the znode: /hbase/table-lock/mytable > {code:java} > [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), > org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1341), > org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1330), > org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.tableDeleted(TableLockManager.java:442), > > org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.markTableAsDeleted(MasterProcedureScheduler.java:895), > > org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.completionCleanup(MasterProcedureScheduler.java:261), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.procedureFinished(ProcedureExecutor.java:1415), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:974), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)] > {code} > 5. HMaster2 becomes active > 6. The table {{mytable}} was deleted, but the znode > "/hbase/table-lock/mytable" was left on ZooKeeper -- This message was sent by Atlassian Jira (v8.3.4#803005)