hellodml commented on issue #18570:
URL:
https://github.com/apache/dolphinscheduler/issues/18570#issuecomment-5353064303
@ruanwenjun One more piece of evidence that ties the two problems together.
`MasterServerLoadProtection` reads the count from memory, not from the DB:
```java
int currentWorkflowInstanceCount = workflowRepository.getAll().size();
```
That is why our master never recovered on its own:
```
1. command 520266 bootstraps successfully
workflowRepository.put(execution) <- registered, count +1
registerWorkflowEventBus(execution) <- starts executing normally
2. 11 ms later the same command is handled again
deleteCommandOrThrow -> CommandDuplicateHandleException
wrapped by CompletableFuture into CompletionException
3. bootstrapError: if (throwable instanceof CommandDuplicateHandleException)
the direct instanceof cannot see through the wrapper -> falls through
4. forceUpdateWorkflowInstanceState(516982, FAILURE)
this updates the DB only. It does NOT remove the execution from
workflowRepository and does NOT deregister its event bus.
5. the still-running FIRST execution now has a FAILURE row underneath it:
UnsupportedOperationException: The WorkflowInstance: 516982 state is
FAILURE, no need to notify
6. that execution can never complete -> never leaves workflowRepository
-> getAll().size() never decreases -> master stays overloaded forever
```
This matches the logs exactly. During the outage the logged count sat at
exactly
25 and never moved — `25 ×2665` in hour 09, `25 ×7194` in hour 10 (nothing
else
at all), `25 ×2236` in hour 11, against a limit of 20. Those were 25
registered
executions whose DB rows had been force-failed underneath them.
It also answers your point that the master should report NORMAL once the
active
workflow count drops: it cannot drop, because the counter *is* the in-memory
registry and the wedged executions never leave it.
So `ExceptionUtils.throwableOfType(...)` in `bootstrapError` looks like the
root
fix rather than a cosmetic one — with step 4 skipped, the healthy first
execution
is never corrupted, completes normally, deregisters itself, and the count
falls
back on its own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
