smtwilio opened a new pull request, #19696:
URL: https://github.com/apache/hudi/pull/19696
### Describe the issue this Pull Request addresses
In continuous mode each table's sync() blocks until shutdown, so the
existing sequential loop would only ever sync the first table.
### Summary and Changelog
Sync the tables concurrently (one thread per table) when continuous mode is
on, while keeping the sequential path for run-once mode.
**Add a `--fail-fast-on-continuous` flag:** when enabled, the first table
failure shuts down the sibling streamers and throws a `HoodieException` so the
job exits non-zero; when disabled (default), each table is synced independently
and a single failure does not affect the others.
Use thread-safe collections for the success/failed table tracking now that
syncs run in parallel.
### Impact
<!-- Describe any public API or user-facing feature change or any
performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain
the risk.
If medium or high, explain what verification was done to mitigate the
risks. -->
### Documentation Update
1. Multi table streamer now supports continuous mode.
2. `--fail-fast-on-continuous` flag added
### Contributor's checklist
- [X] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [X] Enough context is provided in the sections above
- [X] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
