[ https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648739#comment-16648739 ]
Jingyun Tian commented on HBASE-21291: -------------------------------------- {code} IdLock.Entry lockEntry = procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait); if (lockEntry == null && !force) { LOG.debug("Waited {} ms, but {} is still running, skipping bypass with force={}", lockWait, procedure, force); return false; } else if (lockEntry == null) { LOG.debug("Waited {} ms, but {} is still running, begin bypass with force={}", lockWait, procedure, force); } {code} [~stack] For a stuck procedure, set the force flag to true will skip grabbing the lock. But lockWait must be > 0 otherwise tryLockEntry will wait for the lock forever. Thus my simple fix is add a condition check that lockWait should be positive. > Add a test for bypassing stuck state-machine procedures > ------------------------------------------------------- > > Key: HBASE-21291 > URL: https://issues.apache.org/jira/browse/HBASE-21291 > Project: HBase > Issue Type: Bug > Affects Versions: 2.2.0 > Reporter: Jingyun Tian > Assignee: Jingyun Tian > Priority: Major > Attachments: HBASE-21291.master.001.patch, > HBASE-21291.master.002.patch, HBASE-21291.master.003.patch, > HBASE-21291.master.004.patch > > > {code} > if (!procedure.isFailed()) { > if (subprocs != null) { > if (subprocs.length == 1 && subprocs[0] == procedure) { > // Procedure returned itself. Quick-shortcut for a state > machine-like procedure; > // i.e. we go around this loop again rather than go back out on > the scheduler queue. > subprocs = null; > reExecute = true; > LOG.trace("Short-circuit to next step on pid={}", > procedure.getProcId()); > } else { > // Yield the current procedure, and make the subprocedure runnable > // subprocs may come back 'null'. > subprocs = initializeChildren(procStack, procedure, subprocs); > LOG.info("Initialized subprocedures=" + > (subprocs == null? null: > Stream.of(subprocs).map(e -> "{" + e.toString() + "}"). > collect(Collectors.toList()).toString())); > } > } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) { > LOG.debug("Added to timeoutExecutor {}", procedure); > timeoutExecutor.add(procedure); > } else if (!suspended) { > // No subtask, so we are done > procedure.setState(ProcedureState.SUCCESS); > } > } > {code} > Currently implementation of ProcedureExecutor will set the reExcecute to true > for state machine like procedure. Then if this procedure is stuck at one > certain state, it will loop forever. > {code} > IdLock.Entry lockEntry = > procExecutionLock.getLockEntry(proc.getProcId()); > try { > executeProcedure(proc); > } catch (AssertionError e) { > LOG.info("ASSERT pid=" + proc.getProcId(), e); > throw e; > } finally { > procExecutionLock.releaseLockEntry(lockEntry); > {code} > Since procedure will get the IdLock and release it after execution done, > state machine procedure will never release IdLock until it is finished. > Then bypassProcedure doesn't work because is will try to grab the IdLock at > first. > {code} > IdLock.Entry lockEntry = > procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)