zhangshenghang opened a new issue, #13322:
URL: https://github.com/apache/seatunnel/issues/13322

   Related to #12677 (which tracks retention reaching ~2x max-retained within a 
single run). This issue tracks two adjacent behaviors not covered there.
   
   ### What happened
   
   `completedCheckpointIds` is not reset when the coordinator is restored, and 
ids are recorded before storage succeeds (dev @ 89f91a89af):
   
   1. **Not cleared across restores**: `cleanPendingCheckpoint` 
(`CheckpointCoordinator.java:1246-1302`) resets `pendingCheckpoints`, 
`pipelineTaskStatus`, ready-to-close sets, `pendingCounter` and 
`schemaChanging` — but not `completedCheckpointIds`. After a pipeline restore, 
ids from the old generation stay in the queue, so the physical storage can hold 
up to ~2x `max-retained` across the restore boundary, beyond what the user 
configured.
   2. **Enqueued before stored**: `completePendingCheckpoint` appends the id at 
`CheckpointCoordinator.java:1410` and only then runs `storeCheckPoint` in the 
try block. On a storage failure the exception propagates (sneakyThrow at 
1439-1442) but **the id stays in the queue**; a later rotation (1422-1437) will 
call `deleteCheckpoint` with an id that was never persisted (LocalFileStorage 
logs errors for the missing file), and the orphaned file (if partially written) 
is never tracked for deletion again.
   
   ### Impact
   
   Storage usage exceeding the configured retention after restores; retention 
deletions targeting non-existent ids; untracked orphan files after store 
failures.
   
   ### Suggested fix
   
   Clear `completedCheckpointIds` in 
`cleanPendingCheckpoint(CHECKPOINT_COORDINATOR_RESET)` (or otherwise reset it 
per generation), and only enqueue an id after `storeCheckPoint` succeeds 
(remove it on failure).
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to