zhangshenghang opened a new issue, #13313:
URL: https://github.com/apache/seatunnel/issues/13313

   ### What happened
   
   When `restoreCoordinator` fails midway, `prepareRestorePipeline` swallows 
the exception and continues the restore, leaving the coordinator with `shutdown 
== true`. The restored pipeline then runs normally but **never takes another 
checkpoint**, with no error anywhere.
   
   Chain (all on dev @ 89f91a89af):
   
   1. `SubPlan.prepareRestorePipeline` (dev `SubPlan.java:609-638`) calls 
`reset()` (pipeline state → CREATED) and then 
`jobMaster.getCheckpointManager().reportedPipelineRunning(pipelineId, false)`.
   2. `reportedPipelineRunning` → `CheckpointCoordinator.restoreCoordinator` 
(`CheckpointCoordinator.java:706-763`): line 709 replaces 
`checkpointCoordinatorFuture` with a new future, line 710 writes `RUNNING` to 
the IMap, then line 712 calls `loadReadyToCloseStartingTask()`.
   3. `loadReadyToCloseStartingTask` (`CheckpointCoordinator.java:593-617`) 
**throws `RuntimeException` when the IMap read fails after retries** 
(deliberate fail-fast to avoid a stuck master failover).
   4. Because `cleanPendingCheckpoint(CHECKPOINT_COORDINATOR_RESET)` (line 714) 
and `shutdown = false` (line 715) come *after* the throwing call, they never 
execute. `shutdown` keeps the `true` value set by the failed generation's 
`cleanPendingCheckpoint`.
   5. The exception reaches `prepareRestorePipeline`'s catch (dev 
`SubPlan.java:630-637`): `currPipelineStatus` is `CREATED` (not an end state) → 
**`return true`**, and the restore proceeds.
   6. Tasks deploy, report `WAITING_RESTORE`/`READY_START`, data flows — but 
`tryTriggerPendingCheckpoint` rejects every trigger at the `isShutdown()` guard 
(`CheckpointCoordinator.java:837-847`) forever. The only statement that ever 
resets `shutdown` is line 715, which was skipped.
   
   ### Trigger
   
   A single transient `runningJobStateIMap` read failure inside 
`restoreCoordinator` during a pipeline restore (the exact condition the code 
itself handles by throwing).
   
   ### Impact
   
   A RUNNING streaming pipeline silently loses checkpointing from that restore 
onward. A later crash replays from the last good checkpoint — potentially hours 
of data replay (or loss, depending on sink semantics). Nothing in the logs 
indicates checkpoints stopped.
   
   ### Suggested fix
   
   Either rethrow out of `prepareRestorePipeline` when 
`reportedPipelineRunning` fails (fail this restore attempt → next retry), or 
reorder `restoreCoordinator` so `shutdown = false` / 
`cleanPendingCheckpoint(RESET)` are not skipped when the ready-to-close load 
fails, and fail the pipeline explicitly instead of continuing.
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to