danny0405 opened a new pull request, #20679: URL: https://github.com/apache/hudi/pull/20679
### Describe the issue this Pull Request addresses Closes #20674; part of #20401 and #20407. Hudi's Spark streaming source currently consumes the entire commit backlog in one micro-batch. Add an optional commit cap for standalone readStream and SDP chaining, including native AvailableNow support so Spark does not bypass the cap with its fallback wrapper. ### Summary and Changelog - Add hoodie.datasource.read.streaming.max.commits.per.trigger with no default; configured values must be positive integers. - Share admission-control and AvailableNow handling across both source offset versions. Count completed write commits, excluding clean/rollback offsets; preserve V1 hollow-commit handling and sort V2 boundaries by completion time. - Capture the initial offset before pinning AvailableNow's end, drain capped batches, and leave later arrivals for the next run. Respect Spark's all-available limit for Trigger.Once. - Add COW/MOR acceptance for both offset versions: five commits drain as 2/2/1 batches; a sixth arriving during execution waits for restart; another unchanged restart emits no batch. - Add unit coverage for invalid/unbounded limits, service offsets, completion-time ties, all-available override and an initially empty pinned run. - Document configuration, restart behavior and limits. No external code was copied. ### Impact New opt-in source configuration; reads remain unbounded when unset. Both source implementations now implement Spark admission control and native AvailableNow. No checkpoint or storage-format changes. This caps completed active-timeline write commits, not rows/bytes. Equal completion timestamps cannot be split; initial snapshot fallback and explicit snapshot recovery remain unbounded. ### Risk Level medium. Changes shared streaming offset selection. Clean/build/style/license validation and full selected suites passed on JDK 17: - Spark 4.1 / Scala 2.13: TestHoodieSourceAdmissionControl, 3 passed; TestStreamingSource plus TestStreamSourceReadByStateTransitionTime, 21 passed. - Spark 3.5 / Scala 2.12: the same unit suite, 3 passed; source suites, 19 passed and 2 expected Spark-4.1-only variant skips. - git diff --check passed. The four new real Spark backlog/restart combinations passed on both versions. This is based on master independently of #20516; the combined source-recovery branch and CLI/Connect throttling acceptance were not run. Both PRs touch source classes and will need integration/rebase if either lands first. Snapshot recovery chunking is expressly outside this issue. ### Documentation Update Added the configuration's built-in documentation and hudi-integ-test/spark-pipelines/SOURCE_THROTTLING.md with a reader example and operating limits. Website publication remains follow-up before release; this PR does not claim it is published. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
