zhangshenghang opened a new issue, #13320:
URL: https://github.com/apache/seatunnel/issues/13320

   ### What happened
   
   During master failover the per-vertex liveness probe treats an *invocation 
exception* as "task not executing" and fails healthy running tasks, forcing a 
full pipeline restore (data replay for streaming jobs) out of a transient blip.
   
   Chain (dev @ 89f91a89af):
   
   1. `PhysicalVertex.initStateFuture` (`PhysicalVertex.java:197-205`) probes 
RUNNING/DEPLOYING vertices with `checkTaskGroupIsExecuting`, and calls 
`updateTaskState(ExecutionState.FAILING)` when it returns false.
   2. `checkTaskGroupIsExecuting` (`PhysicalVertex.java:240-248`) returns false 
not only for a definitive "not executing" answer, but also when `invoke.get()` 
throws `InterruptedException`/`ExecutionException` (network blip, worker GC 
pause, busy worker, membership still settling after the switch) — the two 
semantics are conflated (the log even says "checkTaskGroupIsExecuting return 
false" on the error path).
   3. Master failover rebuilds every JobMaster and runs this probe for 
**every** RUNNING vertex (`SubPlan.restorePipelineState` → 
`restoreExecutionState` → `initStateFuture`), so one transient invocation 
failure anywhere fails a task that is actually alive on the worker → the whole 
pipeline is cancelled and restored.
   
   ### Impact
   
   Unnecessary pipeline restores (checkpoint replay / source re-read) triggered 
by transient probe failures during failover — the recovery mechanism itself 
becomes a source of instability for streaming jobs.
   
   ### Suggested fix
   
   Distinguish "check failed" from "confirmed not executing": on an invocation 
exception, retry the probe (bounded) before concluding; only a definitive false 
should fail the vertex.
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to