[ https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16560863#comment-16560863 ]
stack commented on HBASE-19121: ------------------------------- I think I've asked for this above but here is more detail. A corrupt Master proc WAL file was responsible for two regions being stuck in OPENING. It looks like this in Master log: {code} 2018-07-28 12:33:49,724 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=ve0530.halxg.cloudera.com,16020,1532716446468, table=IntegrationTestBigLinkedList, region=8198218a4532a0ee544cb069970f9a77 2018-07-28 12:33:49,724 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=ve0530.halxg.cloudera.com,16020,1532716446468, table=IntegrationTestBigLinkedList, region=4459746bcff48c116337e732ac4df705 2018-07-28 12:34:17,532 WARN [PEWorker-2] assignment.RegionTransitionProcedure: Failed transition, suspend 3600secs pid=14482, ppid=14168, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=IntegrationTestBigLinkedList, region=4459746bcff48c116337e732ac4df705, server=ve0530.halxg.cloudera.com,16020,1532716446468; rit=OPENING, location=ve0530.halxg.cloudera.com,16020,1532716446468; waiting on rectified condition fixed by other Procedure or operator intervention org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Expected [SPLITTING, SPLIT, MERGING, OPEN, CLOSING] so could move to CLOSING but current state=OPENING at org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode.transitionState(RegionStates.java:164) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1542) at org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:204) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:345) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:95) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1474) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1249) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1763) 2018-07-28 12:34:17,533 INFO [PEWorker-2] procedure2.TimeoutExecutorThread: ADDED pid=14482, ppid=14168, state=WAITING_TIMEOUT:REGION_TRANSITION_DISPATCH; UnassignProcedure table=IntegrationTestBigLinkedList, region=4459746bcff48c116337e732ac4df705, server=ve0530.halxg.cloudera.com,16020,1532716446468; timeout=3600000, timestamp=1532810057533 2018-07-28 12:34:19,078 WARN [PEWorker-12] assignment.RegionTransitionProcedure: Failed transition, suspend 3600secs pid=14373, ppid=14168, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=IntegrationTestBigLinkedList, region=8198218a4532a0ee544cb069970f9a77, server=ve0530.halxg.cloudera.com,16020,1532716446468; rit=OPENING, location=ve0530.halxg.cloudera.com,16020,1532716446468; waiting on rectified condition fixed by other Procedure or operator intervention org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Expected [SPLITTING, SPLIT, MERGING, OPEN, CLOSING] so could move to CLOSING but current state=OPENING at org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode.transitionState(RegionStates.java:164) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1542) at org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:204) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:345) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:95) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1474) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1249) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1763) {code} At least the log is clear no what has to be done. We are seeing the STUCK messages.. and then out comes the prescription on a period. The Locks&Procedures UI shows that there is an exclusive lock on the two regions making it so no other Procedure can run to do fixup: {code} Locks REGION: 8198218a4532a0ee544cb069970f9a77 Lock type: EXCLUSIVE Owner procedure: { ID => '14373', PARENT_ID => '14168', STATE => 'WAITING_TIMEOUT', OWNER => 'stack', TYPE => 'UnassignProcedure table=IntegrationTestBigLinkedList, region=8198218a4532a0ee544cb069970f9a77, server=ve0530.halxg.cloudera.com,16020,1532716446468', START_TIME => 'Fri Jul 27 14:25:57 PDT 2018', LAST_UPDATE => 'Sat Jul 28 12:34:19 PDT 2018', PARAMETERS => [ { transitionState => 'REGION_TRANSITION_DISPATCH', regionInfo => { regionId => '1532507299562', tableName => { namespace => 'ZGVmYXVsdA==', qualifier => 'SW50ZWdyYXRpb25UZXN0QmlnTGlua2VkTGlzdA==' }, startKey => 'WHJsjHk=', endKey => 'WOReaw==', offline => 'false', split => 'false', replicaId => '0' }, hostingServer => { hostName => 've0530.halxg.cloudera.com', port => '16020', startCode => '1532716446468' }, attempt => '34' } ] } REGION: 4459746bcff48c116337e732ac4df705 Lock type: EXCLUSIVE Owner procedure: { ID => '14482', PARENT_ID => '14168', STATE => 'WAITING_TIMEOUT', OWNER => 'stack', TYPE => 'UnassignProcedure table=IntegrationTestBigLinkedList, region=4459746bcff48c116337e732ac4df705, server=ve0530.halxg.cloudera.com,16020,1532716446468', START_TIME => 'Fri Jul 27 14:25:57 PDT 2018', LAST_UPDATE => 'Sat Jul 28 12:34:17 PDT 2018', PARAMETERS => [ { transitionState => 'REGION_TRANSITION_DISPATCH', regionInfo => { regionId => '1532510055516', tableName => { namespace => 'ZGVmYXVsdA==', qualifier => 'SW50ZWdyYXRpb25UZXN0QmlnTGlua2VkTGlzdA==' }, startKey => 'h43UQpjowgpeGc6LsEkspQ==', endKey => 'h/+PcQ==', offline => 'false', split => 'false', replicaId => '0' }, hostingServer => { hostName => 've0530.halxg.cloudera.com', port => '16020', startCode => '1532716446468' }, attempt => '34' } ] } {code} So we need some way of cleaning up the lock and then assigning. > HBCK for AMv2 (A.K.A HBCK2) > --------------------------- > > Key: HBASE-19121 > URL: https://issues.apache.org/jira/browse/HBASE-19121 > Project: HBase > Issue Type: Bug > Components: hbck > Reporter: stack > Assignee: Umesh Agashe > Priority: Major > Attachments: hbase-19121.master.001.patch > > > We don't have an hbck for the new AM. Old hbck may actually do damage going > against AMv2. > Fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)