wombatu-kun opened a new pull request, #20389: URL: https://github.com/apache/hudi/pull/20389
### Describe the issue this Pull Request addresses Closes #20388 With blocking instant generation (Flink COPY_ON_WRITE or CDC tables written with upsert), `StreamWriteOperatorCoordinator` could start the next instant while the previous one was still being committed, and the EAGER failed-writes rollback in `startCommit` rolled the previous instant back. It shows up as `ITTestHoodieDataSource#testLookupJoin[5]` failing on unrelated PRs, most recently on #20366: https://github.com/apache/hudi/actions/runs/39052223751/job/117452975075 ### Summary and Changelog In blocking mode a new instant now waits until the instants of all earlier checkpoints are committed, and a request that no writer can use any more never creates one. - `EventBuffers#awaitAllInstantsToCompleteIfNecessary` waits while a buffer of an earlier checkpoint exists, whether its write metadata events were applied yet or not, and re-checks that under the `CommitGuard` lock after every wakeup, so the reset of another buffer no longer releases it early, and a reset that lands between the check and the wait is no longer lost. The single-await `CommitGuard#blockFor(String)` is removed. - `StreamWriteOperatorCoordinator` drops a waiting instant request once it is stale: after a failover reset (`subtaskReset`, `resetToCheckpoint`) or `close()`, or once a later checkpoint completes, which a live writer cannot do before it gets its instant. Otherwise a request from a failed attempt would keep the `instant-request` thread blocked and could then start an instant that no writer is going to use. - Tests in `TestStreamWriteOperatorCoordinator` cover the race, regional and global failover, a request arriving after the failover and closing the coordinator; `TestEventBuffers` covers the wait predicate. ### Impact Flink COPY_ON_WRITE and CDC upsert jobs no longer roll back the instant they are committing, which failed that commit and could leave metadata table entries for its deleted files. Writes without blocking instant generation, such as MERGE_ON_READ without CDC, are unchanged, and there is no config or user-facing API change. ### Risk Level low The change is limited to instant creation in blocking mode, which now waits in more cases; liveness, the Flink 1.20 and 2.2 failover callbacks and memory visibility were reviewed against the code. Verified: the new tests fail on master ("The instant creation of checkpoint 2 must not roll back the pending instant of checkpoint 1 ==> expected: <0> but was: <1>"), each part of the fix is pinned by a test that fails without it, and the `hudi-flink` unit tests and `ITTestHoodieDataSource` pass. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
