[ https://issues.apache.org/jira/browse/HBASE-26885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Wang updated HBASE-26885: ------------------------------- Description: Currently it will submit lots of unnecessary OpenRegionProcedure by retry. Related log looks like below, 'localhost,1,1' is the bogus server: {code:java} 2022-03-22 10:17:48,301 WARN [PEWorker-8] assignment.RegionRemoteProcedureBase: Can not add remote operation pid=17952, ppid=17951, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region {ENCODED => 490391c232c7aa13f7e0d50bfe1f7235, NAME => 'TestTable1,0000000000000000002497747,1647568640784.490391c232c7aa13f7e0d50bfe1f7235.', STARTKEY => '0000000000000000002497747', ENDKEY => ''} to server localhost,1,1, this usually because the server is alread dead, give up and mark the procedure as complete, the parent procedure will take care of this. org.apache.hadoop.hbase.procedure2.NoServerDispatchException: localhost,1,1; pid=17952, ppid=17951, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure at org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:168) at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:285) at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:58) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965) 2022-03-22 10:17:48,301 DEBUG [PEWorker-8] procedure2.RootProcedureState: Add procedure pid=17952, ppid=17951, state=SUCCESS, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 8th rollback step {code} was: Currently it will submit lots of unnecessary OpenRegionProcedure by retry. Related log looks like below, "localhost,1,1" is the bogus server: {code:java} 2022-03-22 10:17:48,301 WARN [PEWorker-8] assignment.RegionRemoteProcedureBase: Can not add remote operation pid=17952, ppid=17951, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region {ENCODED => 490391c232c7aa13f7e0d50bfe1f7235, NAME => 'TestTable1,0000000000000000002497747,1647568640784.490391c232c7aa13f7e0d50bfe1f7235.', STARTKEY => '0000000000000000002497747', ENDKEY => ''} to server localhost,1,1, this usually because the server is alread dead, give up and mark the procedure as complete, the parent procedure will take care of this. org.apache.hadoop.hbase.procedure2.NoServerDispatchException: localhost,1,1; pid=17952, ppid=17951, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure at org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:168) at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:285) at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:58) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965) 2022-03-22 10:17:48,301 DEBUG [PEWorker-8] procedure2.RootProcedureState: Add procedure pid=17952, ppid=17951, state=SUCCESS, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 8th rollback step {code} > The TRSP should not go on when it get a bogus server name from AM > ----------------------------------------------------------------- > > Key: HBASE-26885 > URL: https://issues.apache.org/jira/browse/HBASE-26885 > Project: HBase > Issue Type: Improvement > Components: proc-v2 > Reporter: Zheng Wang > Assignee: Zheng Wang > Priority: Major > > Currently it will submit lots of unnecessary OpenRegionProcedure by retry. > Related log looks like below, 'localhost,1,1' is the bogus server: > {code:java} > 2022-03-22 10:17:48,301 WARN [PEWorker-8] > assignment.RegionRemoteProcedureBase: Can not add remote operation pid=17952, > ppid=17951, state=RUNNABLE, locked=true; > org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region > {ENCODED => 490391c232c7aa13f7e0d50bfe1f7235, NAME => > 'TestTable1,0000000000000000002497747,1647568640784.490391c232c7aa13f7e0d50bfe1f7235.', > STARTKEY => '0000000000000000002497747', ENDKEY => ''} to server > localhost,1,1, this usually because the server is alread dead, give up and > mark the procedure as complete, the parent procedure will take care of this. > org.apache.hadoop.hbase.procedure2.NoServerDispatchException: localhost,1,1; > pid=17952, ppid=17951, state=RUNNABLE, locked=true; > org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure > at > org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:168) > at > org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:285) > at > org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:58) > at > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965) > 2022-03-22 10:17:48,301 DEBUG [PEWorker-8] procedure2.RootProcedureState: Add > procedure pid=17952, ppid=17951, state=SUCCESS, locked=true; > org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 8th > rollback step {code} -- This message was sent by Atlassian Jira (v8.20.1#820001)