joeyutong commented on PR #885: URL: https://github.com/apache/flink-agents/pull/885#issuecomment-4976551399
Thanks for raising the question of whether cleanup can be safe by default. A checkpoint-aligned, user-controlled cleanup path may be a safer general solution: 1. Persist and expose the mapping from each checkpoint to its effective Kafka recovery marker—the per-partition minimum across all subtasks. 2. Leave the irreversible cleanup decision to users. Based on their recovery requirements, they can choose the oldest checkpoint to retain and delete records before its marker offsets through Kafka's `deleteRecords` API. 3. During `rebuildState()`, validate the requested offsets against Kafka's current beginning offsets. If a recovery offset is older than the available data, fail fast with a clear error instead of silently rebuilding incomplete state. Since `rebuildState()` already starts from the recovery marker, the same marker is a natural boundary for physical cleanup. This aligns cleanup with the actual recovery semantics without automatically invalidating older checkpoints. Once users advance that boundary, fail-fast validation makes any incompatible recovery attempt explicit. This could be explored as a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
