zhangshenghang opened a new issue, #13313: URL: https://github.com/apache/seatunnel/issues/13313
### What happened When `restoreCoordinator` fails midway, `prepareRestorePipeline` swallows the exception and continues the restore, leaving the coordinator with `shutdown == true`. The restored pipeline then runs normally but **never takes another checkpoint**, with no error anywhere. Chain (all on dev @ 89f91a89af): 1. `SubPlan.prepareRestorePipeline` (dev `SubPlan.java:609-638`) calls `reset()` (pipeline state → CREATED) and then `jobMaster.getCheckpointManager().reportedPipelineRunning(pipelineId, false)`. 2. `reportedPipelineRunning` → `CheckpointCoordinator.restoreCoordinator` (`CheckpointCoordinator.java:706-763`): line 709 replaces `checkpointCoordinatorFuture` with a new future, line 710 writes `RUNNING` to the IMap, then line 712 calls `loadReadyToCloseStartingTask()`. 3. `loadReadyToCloseStartingTask` (`CheckpointCoordinator.java:593-617`) **throws `RuntimeException` when the IMap read fails after retries** (deliberate fail-fast to avoid a stuck master failover). 4. Because `cleanPendingCheckpoint(CHECKPOINT_COORDINATOR_RESET)` (line 714) and `shutdown = false` (line 715) come *after* the throwing call, they never execute. `shutdown` keeps the `true` value set by the failed generation's `cleanPendingCheckpoint`. 5. The exception reaches `prepareRestorePipeline`'s catch (dev `SubPlan.java:630-637`): `currPipelineStatus` is `CREATED` (not an end state) → **`return true`**, and the restore proceeds. 6. Tasks deploy, report `WAITING_RESTORE`/`READY_START`, data flows — but `tryTriggerPendingCheckpoint` rejects every trigger at the `isShutdown()` guard (`CheckpointCoordinator.java:837-847`) forever. The only statement that ever resets `shutdown` is line 715, which was skipped. ### Trigger A single transient `runningJobStateIMap` read failure inside `restoreCoordinator` during a pipeline restore (the exact condition the code itself handles by throwing). ### Impact A RUNNING streaming pipeline silently loses checkpointing from that restore onward. A later crash replays from the last good checkpoint — potentially hours of data replay (or loss, depending on sink semantics). Nothing in the logs indicates checkpoints stopped. ### Suggested fix Either rethrow out of `prepareRestorePipeline` when `reportedPipelineRunning` fails (fail this restore attempt → next retry), or reorder `restoreCoordinator` so `shutdown = false` / `cleanPendingCheckpoint(RESET)` are not skipped when the ready-to-close load fails, and fail the pipeline explicitly instead of continuing. ### SeaTunnel Version dev (89f91a89af), Zeta engine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
