SEZ9 commented on issue #13322:
URL: https://github.com/apache/seatunnel/issues/13322#issuecomment-6995356469

   Thanks for the detailed report, and thanks @DanielLeens for the 
reproduction. The fault-injection test on `89f91a89af` confirms both queue 
behaviors: the id is enqueued before `storeCheckPoint` succeeds (the queue is 
`[9]` after the injected failure), and 
`cleanPendingCheckpoint(CHECKPOINT_COORDINATOR_RESET)` leaves 
`completedCheckpointIds` unchanged.
   
   As noted in that comment, the following are still unverified: the ~2x 
on-disk retention across a restore, partially written files, and the subsequent 
deletion behavior.
   
   @JeremyXin thanks for volunteering, please go ahead. Your plan (record ids 
only after successful storage, reconcile the retention queue during restore, 
check the interaction with #12677, and add tests for storage failures and 
retention across restores) sounds like the right scope. Two things would help 
in the PR:
   
   - When reconciling on restore, how will the old generation's checkpoints be 
handled? If their ids are simply dropped from the queue, would retention ever 
delete those files, and is the checkpoint the restore itself depends on kept?
   - Could the tests also cover the unverified points above, so we know whether 
they actually occur?
   
   <!-- streview-comment:2446 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to