zhangshenghang opened a new issue, #13320: URL: https://github.com/apache/seatunnel/issues/13320
### What happened During master failover the per-vertex liveness probe treats an *invocation exception* as "task not executing" and fails healthy running tasks, forcing a full pipeline restore (data replay for streaming jobs) out of a transient blip. Chain (dev @ 89f91a89af): 1. `PhysicalVertex.initStateFuture` (`PhysicalVertex.java:197-205`) probes RUNNING/DEPLOYING vertices with `checkTaskGroupIsExecuting`, and calls `updateTaskState(ExecutionState.FAILING)` when it returns false. 2. `checkTaskGroupIsExecuting` (`PhysicalVertex.java:240-248`) returns false not only for a definitive "not executing" answer, but also when `invoke.get()` throws `InterruptedException`/`ExecutionException` (network blip, worker GC pause, busy worker, membership still settling after the switch) — the two semantics are conflated (the log even says "checkTaskGroupIsExecuting return false" on the error path). 3. Master failover rebuilds every JobMaster and runs this probe for **every** RUNNING vertex (`SubPlan.restorePipelineState` → `restoreExecutionState` → `initStateFuture`), so one transient invocation failure anywhere fails a task that is actually alive on the worker → the whole pipeline is cancelled and restored. ### Impact Unnecessary pipeline restores (checkpoint replay / source re-read) triggered by transient probe failures during failover — the recovery mechanism itself becomes a source of instability for streaming jobs. ### Suggested fix Distinguish "check failed" from "confirmed not executing": on an invocation exception, retry the probe (bounded) before concluding; only a definitive false should fail the vertex. ### SeaTunnel Version dev (89f91a89af), Zeta engine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
