SEPURI-SAI-KRISHNA opened a new issue, #18538: URL: https://github.com/apache/dolphinscheduler/issues/18538
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened When a task fails and still has retry attempts left, the master schedules a delayed `TaskRetryLifecycleEvent`. The delay is computed in `TaskRetryLifecycleEvent#of`: ```java final long remainingTime = TimeUnit.MINUTES.toMillis(delayTime) + System.currentTimeMillis() - taskInstance.getEndTime().getTime(); ``` The intent is "retry the task at `endTime + retryInterval`", so the remaining delay should be ``` retryInterval - (now - endTime) ``` but the expression actually evaluates to ``` retryInterval + (now - endTime) ``` The elapsed time since the task ended is **added** to the delay instead of being **subtracted** from it. The signs of `now` and `endTime` are swapped. ### What you expected to happen The retry should fire `retryInterval` after the task ended. If the retry interval has already elapsed by the time the event is created, the task should be retried immediately. ## Impact In the common path (`onFailedEvent` fires milliseconds after the task ended) the error is only a few milliseconds and is invisible. It becomes significant whenever the retry event is created **long after** the task actually ended, because the delay then grows without bound: * **Master failover.** `WorkflowFailoverCommandHandler#assembleWorkflowExecutionGraph` rebuilds the execution graph from the *existing* task instances, so a task instance left in `FAILURE` with retries remaining is preserved as-is. When the workflow is re-triggered, `TaskFailureStateAction#onStartEvent` republishes a `TaskFailedLifecycleEvent` carrying the **stale** `endTime` read from the database. `TaskRetryLifecycleEvent#of` then computes `retryInterval + (now - endTime)`, so the retry is postponed by the whole outage duration on top of the configured interval. A task that failed an hour before the failover, with a 1-minute retry interval, waits ~61 minutes instead of being retried immediately. * Any other path where the failure event is processed after a delay (event backlog, workflow paused/resumed) is skewed the same way. The workflow simply looks stuck in `RUNNING_EXECUTION` while the retry sits in the delay queue. ### How to reproduce 1. Define a workflow with one task that fails, with `failRetryTimes >= 1` and `failRetryInterval = 1` (minute). 2. Run it and let the task fail so it enters `FAILURE` while waiting to retry. 3. Kill the master that owns the workflow instance before the retry fires. 4. Wait a few minutes, then let another master pick the instance up by failover. 5. The retry is scheduled `1 minute + <outage duration>` in the future instead of firing right away. A deterministic unit-level reproduction: call `TaskRetryLifecycleEvent.of(taskExecution)` with a task instance whose `endTime` is two hours in the past and whose `retryInterval` is 5 minutes. Expected delay: `0`. Actual delay: `~2h05m`. ### Anything else The same expression is the only place the retry delay is computed, so the fix is confined to `TaskRetryLifecycleEvent#of`. ### Version dev ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
