zhangshenghang opened a new issue, #13314:
URL: https://github.com/apache/seatunnel/issues/13314

   ### What happened
   
   `PhysicalVertex.resetExecutionState` swallows an IMap write failure but 
still resets the in-memory state, producing a permanent divergence between 
memory and IMap that blocks the restored pipeline at DEPLOYING forever (until 
the next master failover).
   
   Chain (dev @ 89f91a89af):
   
   1. `PhysicalVertex.resetExecutionState` (`PhysicalVertex.java:553-562`): 
when the `runningJobStateIMap.set(..., CREATED)` write fails after all retries, 
the catch only logs ("If master/worker node done, The job will restore and fix 
the state from TaskExecutionService") — **but line 562 still executes 
`this.currExecutionState = ExecutionState.CREATED`**. No code path ever 
performs the promised "fix from TaskExecutionService".
   2. All transitions go through `updateTaskState`, which reads the state back 
**from the IMap** (`PhysicalVertex.java:387`).
   3. After the pipeline restore redeploys, `makeTaskGroupDeploy` → 
`updateTaskState(DEPLOYING)` is rejected by the end-state guard 
(`PhysicalVertex.java:410-413`, "Task is trying to leave terminal state 
FAILED").
   4. The vertex never leaves CREATED, 
`SubPlan.areAllVerticesRunningOrFinished` stays false, and the pipeline hangs 
at DEPLOYING with only a log line. It only gets unstuck by a later master 
failover, whose `restoreExecutionState` walks the FAILED branch and triggers 
yet another restore.
   
   ### Trigger
   
   A transient IMap write failure (cluster restart window, Hazelcast 
unavailability) during `reset()` of any vertex in a pipeline restore.
   
   ### Impact
   
   Pipeline permanently stuck at DEPLOYING; job RUNNING forever; requires 
master failover to unstick (at the cost of another full pipeline restore).
   
   ### Suggested fix
   
   If the IMap write fails after retries, abort the reset (throw) so 
`prepareRestorePipeline` fails the restore attempt and the retry path takes 
over — instead of continuing with memory/IMap divergence.
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to