zhangshenghang opened a new issue, #12558:
URL: https://github.com/apache/seatunnel/issues/12558
### Description
Since #10687 ("Guard state cleanup races after node failure"),
`CheckpointCoordinator#updateStatus` skips persisting a coordinator status
transition when the `checkpoint_state_<job>_<pipeline>` IMap key is absent, and
`CoordinatorService` removes the job state keys without any coordination with
that write. Two problems follow:
1. **Late callbacks can recreate cleaned state (TOCTOU).** `updateStatus`
does `get(checkpointStateImapKey)` and, if non-null, `set(...)` — with no lock.
If the job state cleanup removes the key between the two calls, a late
coordinator callback re-creates the `checkpoint_state_*` key after cleanup,
leaving orphaned state in `runningJobStateIMap` that no one will ever remove.
The cleanup also has no way to wait for or retry against a concurrent
transition.
2. **Live jobs never persist their coordinator status.** The first
transition of a running pipeline (e.g. `reportedPipelineRunning ->
restoreCoordinator -> updateStatus(RUNNING)`) now finds the key absent and
skips, and since nothing else ever creates the key, **no transition is ever
persisted for jobs that started fresh**. Consequences: on master switch /
restore, the coordinator status is always missing (`checkpointCoordinatorStatus
== null`), so coordinator restore cannot detect the previous status, and
`isNoErrorCompleted()` (the "pipeline already finished without error, do not
re-run" path) can never return true for these jobs.
Expected:
- The gate should be the owning job state (the job key in
`runningJobStateIMap`), not the checkpoint state key itself: transitions are
persisted only while the job is alive — a cleaned job has no job key anymore —
while live jobs can still create their key on the first transition.
- The check-and-write of `updateStatus` and the removal of the job key +
`checkpoint_state_*` keys in the state cleanup should run under the same
per-job lock of `runningJobStateIMap` (with a bounded wait), so a late callback
can never recreate a key that cleanup already removed, and the cleanup retries
later when the lock is unavailable.
### Impact
Orphaned `checkpoint_state_*` IMap entries after job cleanup, and the
coordinator-status based restore/no-error-completed optimizations silently stop
working for jobs started after #10687.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
