zhangshenghang opened a new issue, #13315:
URL: https://github.com/apache/seatunnel/issues/13315

   Related to #12314 (same fire-and-forget reporting family, different 
operation and failure mode).
   
   ### What happened
   
   If a task's `restoreState` fails and the error report to the master is lost, 
the task spins in WAITING_RESTORE forever and nothing ever times the phase out: 
the pipeline stays RUNNING with no progress and no error.
   
   Chain (dev @ 89f91a89af):
   
   1. Task side: `SeaTunnelTask.stateProcess` spins in `WAITING_RESTORE` every 
100 ms until `restoreComplete` is done (`SeaTunnelTask.java:267-276`). 
`restoreComplete` only completes inside `restoreState` on success.
   2. Worker side: `NotifyTaskRestoreOperation` runs `task.restoreState(...)` 
asynchronously and, on failure, only does `sendToMaster(new 
CheckpointErrorReportOperation(...))` — **no join, no retry** 
(`NotifyTaskRestoreOperation.java:111-115`).
   3. Master side: `CheckpointCoordinator.restoreTaskState` only `.join()`s the 
operation *delivery* (`CheckpointCoordinator.java:430-433`); `runInternal` 
returns right after submitting the async function, so delivery success says 
nothing about restore success.
   4. If the report is lost (invocation failure during a partition or master 
switchover), `allTaskReady` never satisfies the size check 
(`CheckpointCoordinator.java:459-461`), and **no scheduler exists before 
`allTaskReady`** (`CheckpointCoordinator.java:476-478`) — there is no timeout 
covering this phase at all (checkpoint timeout only guards already-triggered 
pending checkpoints).
   
   Contrast: terminal task-status reporting retries indefinitely 
(`TaskExecutionService.notifyTaskStatusToMaster`, incl. a dedicated 
`JobRestoreInProgressException` retry path), so the two reporting channels have 
deliberately different reliability — and the restore phase uses the unreliable 
one.
   
   ### Impact
   
   Pipeline RUNNING forever with zero indication; only workaround is manual 
restart.
   
   ### Suggested fix
   
   - Retry the `CheckpointErrorReportOperation` the way 
`notifyTaskStatusToMaster` retries terminal states, and/or
   - Add a configurable timeout for the WAITING_RESTORE phase (restore start → 
READY_START per task) that fails the task and triggers the normal pipeline 
recovery.
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to