DanielLeens commented on issue #13224:
URL: https://github.com/apache/seatunnel/issues/13224#issuecomment-6795352958

   Thanks for recording these as explicit follow-ups rather than widening 
#12440. The linked exactly-once PR is still open and its Build is in progress, 
so this Issue should remain a follow-up lane and must not rework the in-flight 
implementation.
   
   The two follow-ups are both necessary. #12440 currently carries 
`List<WriteStatus>` in its checkpoint commit information, and its E2E verifies 
visibility after a successful job but does not exercise a restart between 
checkpoint completion and aggregated commit.
   
   Please begin this work only on the merged base, in two focused changes:
   
   1. Replace the Hudi-client `WriteStatus` checkpoint payload with a 
connector-owned, versioned state representation that contains exactly the 
information required by the Hudi commit call. Add serialization and restore 
tests that prove a completed checkpoint can still be committed after recovery; 
do not silently drop any commit information during a state-format transition.
   2. Add a fault-injection E2E that stops the job after the checkpoint is 
completed and handed to the aggregated committer but before the Hudi instant is 
committed. Restart from that checkpoint and assert the final table contents and 
committed instant set: no missing rows, no duplicate rows, no publication of 
the uncompleted attempt, and the expected reclaim/rollback behavior for 
orphaned instants.
   
   Please keep the default `AT_LEAST_ONCE` path outside this follow-up unless a 
regression is demonstrated, and report the exact engine coverage and failure 
hook used by the E2E in the PR description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to