nsivabalan opened a new pull request, #20431:
URL: https://github.com/apache/hudi/pull/20431
> **WIP** — stacked draft for early alignment.
### Describe the issue this Pull Request addresses
Stacked on #20415 (sink re-creates its write client on a schema change) and
#20429 (async table services retry a run that failed because their write client
was replaced). **Review only the last commit**; the earlier commits are those
PRs.
With #20415, `HoodieStreamingSink` re-creates its write client when the
writer schema changes and closes the previous one, as `HoodieStreamer` does. An
async compaction or clustering still running on the previous client may then
fail. Before #20429 that stopped the async service, and the sink then failed
its next micro-batch with `Async Compactor shutdown unexpectedly` — without the
cause — ending the query.
Related: #20400.
### Summary and Changelog
- No sink-side change is needed for the retry itself: the sink already
switches its async services to the new client before closing the previous one
(`retireWriteClient`), so with #20429 a run that fails on the closed client is
retried by its service with the new client and the query keeps running.
- `HoodieStreamingSink`: when an async table service stops with a failure it
could not recover from, the micro-batch that fails because of it (`Async
Compactor shutdown unexpectedly` / `Async clustering service shutdown
unexpectedly`) now carries that failure as its cause.
- Tests (`TestStructuredStreaming`), using a Spark record merger that
controls merges inside async compaction tasks:
- `testStructuredStreamingRetriesAsyncCompactionFailedByWriteClientSwap`:
an async compaction is held mid-run, a batch writer evolves the table schema so
that the next micro-batch re-creates the sink's write client, then the held
compaction fails; it is retried with the new client and completes, and the
query keeps running.
- `testStructuredStreamingFailsWithCauseWhenAsyncCompactionFails`: an
async compaction that keeps failing fails the query, with the compaction
failure in the cause chain.
### Impact
Streaming queries survive an async compaction/clustering that fails because
a schema change replaced the sink's write client; other async table service
failures still fail the query, now with their cause.
### Risk Level
low. One sink-side change (failure cause propagation); the retry behaviour
comes from #20429. Verified with `TestStructuredStreaming` (27/27).
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
