zhangshenghang opened a new issue, #13314:
URL: https://github.com/apache/seatunnel/issues/13314
### What happened
`PhysicalVertex.resetExecutionState` swallows an IMap write failure but
still resets the in-memory state, producing a permanent divergence between
memory and IMap that blocks the restored pipeline at DEPLOYING forever (until
the next master failover).
Chain (dev @ 89f91a89af):
1. `PhysicalVertex.resetExecutionState` (`PhysicalVertex.java:553-562`):
when the `runningJobStateIMap.set(..., CREATED)` write fails after all retries,
the catch only logs ("If master/worker node done, The job will restore and fix
the state from TaskExecutionService") — **but line 562 still executes
`this.currExecutionState = ExecutionState.CREATED`**. No code path ever
performs the promised "fix from TaskExecutionService".
2. All transitions go through `updateTaskState`, which reads the state back
**from the IMap** (`PhysicalVertex.java:387`).
3. After the pipeline restore redeploys, `makeTaskGroupDeploy` →
`updateTaskState(DEPLOYING)` is rejected by the end-state guard
(`PhysicalVertex.java:410-413`, "Task is trying to leave terminal state
FAILED").
4. The vertex never leaves CREATED,
`SubPlan.areAllVerticesRunningOrFinished` stays false, and the pipeline hangs
at DEPLOYING with only a log line. It only gets unstuck by a later master
failover, whose `restoreExecutionState` walks the FAILED branch and triggers
yet another restore.
### Trigger
A transient IMap write failure (cluster restart window, Hazelcast
unavailability) during `reset()` of any vertex in a pipeline restore.
### Impact
Pipeline permanently stuck at DEPLOYING; job RUNNING forever; requires
master failover to unstick (at the cost of another full pipeline restore).
### Suggested fix
If the IMap write fails after retries, abort the reset (throw) so
`prepareRestorePipeline` fails the restore attempt and the retry path takes
over — instead of continuing with memory/IMap divergence.
### SeaTunnel Version
dev (89f91a89af), Zeta engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]