DanielLeens commented on PR #12096:
URL: https://github.com/apache/seatunnel/pull/12096#issuecomment-6607597653

   Thanks @SEZ9, agreed on the priorities, with F1 first. Here is the shape I 
would like to settle before writing code, so you can push back early.
   
   **1. ID-only enumeration (F5).** `restoreCompletedCheckpointRetentionState` 
(`CheckpointCoordinator.java:1682-1708` on `aa30f22e`) only uses 
`PipelineState::getCheckpointId` from what it loads, yet 
`getCheckpointsByJobIdAndPipelineId` reads and deserializes every file. Both 
in-repo plugins (local-file and HDFS) already encode the pipeline id and 
checkpoint id in the file name (`AbstractCheckpointStorage#getCheckPointName`, 
parsed by `getPipelineIdByFileName` / `getCheckpointIdByFileName`), so the IDs 
can come from a directory listing without opening any payload. Proposal: a new 
`default` method on the `CheckpointStorage` SPI that returns the 
retention-eligible checkpoint IDs for a job/pipeline, overridden in both 
in-repo plugins via file-name parsing. The `default` returns an empty list, so 
an out-of-tree plugin that does not implement it keeps today's `dev` behavior 
(no restart-time pruning, no payload read) instead of breaking compilation or 
paying for deserialization.
   
   **2. Savepoint signal (F2 / F4).** The type only lives inside the serialized 
`CompletedCheckpoint`, so the listing needs a marker that does not require 
reading payloads. My preference is a type segment in the file name for every 
file written from now on, with a fail-safe restart scan: only files positively 
identified as regular checkpoints are eligible. Savepoints, and any file the 
scan cannot classify (which includes everything written before the upgrade), 
are never queued by the restart path. Pre-upgrade checkpoints are then not 
pruned after a restart, which is exactly `dev` behavior today, so nothing 
regresses and this path can never delete a user savepoint. The detail I would 
like your view on is the name format: it must stay parseable by the existing 
`getCheckpointIdByFileName` (segment 3, split on `.`), so that an older node 
reading a newer file during a rolling upgrade still resolves the right ID. I 
would therefore append the marker as a trailing segment after the checkpoin
 t id and leave every existing segment unchanged. The alternative, a sidecar 
metadata record, avoids a name change but adds a second write per checkpoint 
and a consistency problem, so I lean toward the name.
   
   **3. Independent of the storage change:**
   - **F1:** exclude the restore checkpoint ID from the cleanup candidates, 
with a test that restores an older same-job checkpoint while `max-retained` is 
already exceeded.
   - **F3:** per-ID progress with a bounded retry, with a test for a 
permanently undeletable file followed by deletable ones.
   - **F7:** catch and log any runtime failure in the constructor-time scan and 
cleanup (today only `CheckpointStorageException` at `:1688` and `:1713`), so 
coordinator recreation during failover cannot be aborted by retention.
   
   **4. F6 / F8:** en/zh docs for `checkpoint.storage.max-retained` (exactly 
`max-retained` kept, one `deleteCheckpoint` call per completed checkpoint, 
restart-time pruning of identifiable checkpoints only), plus a compatibility 
note that also tells storage-plugin implementers about the new `default` SPI 
method.
   
   I plan to push all of this as one revision once you are comfortable with 
point 2. Nothing is implemented yet, so the current head stays not ready.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to