SEZ9 commented on issue #13322: URL: https://github.com/apache/seatunnel/issues/13322#issuecomment-6995356469
Thanks for the detailed report, and thanks @DanielLeens for the reproduction. The fault-injection test on `89f91a89af` confirms both queue behaviors: the id is enqueued before `storeCheckPoint` succeeds (the queue is `[9]` after the injected failure), and `cleanPendingCheckpoint(CHECKPOINT_COORDINATOR_RESET)` leaves `completedCheckpointIds` unchanged. As noted in that comment, the following are still unverified: the ~2x on-disk retention across a restore, partially written files, and the subsequent deletion behavior. @JeremyXin thanks for volunteering, please go ahead. Your plan (record ids only after successful storage, reconcile the retention queue during restore, check the interaction with #12677, and add tests for storage failures and retention across restores) sounds like the right scope. Two things would help in the PR: - When reconciling on restore, how will the old generation's checkpoints be handled? If their ids are simply dropped from the queue, would retention ever delete those files, and is the checkpoint the restore itself depends on kept? - Could the tests also cover the unverified points above, so we know whether they actually occur? <!-- streview-comment:2446 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
