zhangshenghang opened a new issue, #13324:
URL: https://github.com/apache/seatunnel/issues/13324

   ### What happened
   
   Terminal coordinator status writes to the IMap are best-effort (failures 
swallowed with a warning), while correctness decisions on master failover trust 
exactly that IMap entry (dev @ 89f91a89af):
   
   1. `CheckpointCoordinator.updateStatus` 
(`CheckpointCoordinator.java:1547-1578`) retries the 
`runningJobStateIMap.set(...)` write, and when retries are exhausted it only 
logs `"Set {} state {} to IMap failed, skip do it"` and continues — the 
terminal status (e.g. FINISHED at `completePendingCheckpoint`, 
`CheckpointCoordinator.java:1469`) is never persisted.
   2. `isNoErrorCompleted` (`CheckpointCoordinator.java:1510-1520`) decides 
"this pipeline already completed" purely from the IMap status: 
`latestCompletedCheckpoint.getCheckpointType().isFinalCheckpoint() && (FINISHED 
|| SUSPEND equals status)`.
   
   If the FINISHED write was swallowed, a later master failover concludes the 
pipeline never completed → `restoreCoordinator(alreadyStarted=true)` 
re-triggers the final COMPLETED_POINT checkpoint and re-runs completion work 
for a pipeline whose tasks already finished (duplicate final checkpoint; 
re-execution side effects). Symmetrically, a swallowed FAILED write can make a 
failed pipeline look recoverable.
   
   ### Trigger
   
   Transient IMap unavailability exactly when a coordinator terminal status is 
written (job finishing during cluster degradation).
   
   ### Impact
   
   Failover can re-run or re-finish already-completed pipelines; terminal 
pipeline states are not durable.
   
   ### Suggested fix
   
   Terminal status writes (FINISHED/SUSPEND/FAILED/CANCELED) must not be 
best-effort: retry to success with backoff, or fail loudly so the pipeline 
end-state computation knows the state could not be persisted, instead of 
swallowing and letting `isNoErrorCompleted` read stale data.
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to