danny0405 opened a new pull request, #20679:
URL: https://github.com/apache/hudi/pull/20679

   ### Describe the issue this Pull Request addresses
   
   Closes #20674; part of #20401 and #20407. Hudi's Spark streaming source 
currently consumes the entire commit backlog in one micro-batch. Add an 
optional commit cap for standalone readStream and SDP chaining, including 
native AvailableNow support so Spark does not bypass the cap with its fallback 
wrapper.
   
   ### Summary and Changelog
   
   - Add hoodie.datasource.read.streaming.max.commits.per.trigger with no 
default; configured values must be positive integers.
   - Share admission-control and AvailableNow handling across both source 
offset versions. Count completed write commits, excluding clean/rollback 
offsets; preserve V1 hollow-commit handling and sort V2 boundaries by 
completion time.
   - Capture the initial offset before pinning AvailableNow's end, drain capped 
batches, and leave later arrivals for the next run. Respect Spark's 
all-available limit for Trigger.Once.
   - Add COW/MOR acceptance for both offset versions: five commits drain as 
2/2/1 batches; a sixth arriving during execution waits for restart; another 
unchanged restart emits no batch.
   - Add unit coverage for invalid/unbounded limits, service offsets, 
completion-time ties, all-available override and an initially empty pinned run.
   - Document configuration, restart behavior and limits. No external code was 
copied.
   
   ### Impact
   
   New opt-in source configuration; reads remain unbounded when unset. Both 
source implementations now implement Spark admission control and native 
AvailableNow. No checkpoint or storage-format changes. This caps completed 
active-timeline write commits, not rows/bytes. Equal completion timestamps 
cannot be split; initial snapshot fallback and explicit snapshot recovery 
remain unbounded.
   
   ### Risk Level
   
   medium. Changes shared streaming offset selection. Clean/build/style/license 
validation and full selected suites passed on JDK 17:
   - Spark 4.1 / Scala 2.13: TestHoodieSourceAdmissionControl, 3 passed; 
TestStreamingSource plus TestStreamSourceReadByStateTransitionTime, 21 passed.
   - Spark 3.5 / Scala 2.12: the same unit suite, 3 passed; source suites, 19 
passed and 2 expected Spark-4.1-only variant skips.
   - git diff --check passed.
   
   The four new real Spark backlog/restart combinations passed on both 
versions. This is based on master independently of #20516; the combined 
source-recovery branch and CLI/Connect throttling acceptance were not run. Both 
PRs touch source classes and will need integration/rebase if either lands 
first. Snapshot recovery chunking is expressly outside this issue.
   
   ### Documentation Update
   
   Added the configuration's built-in documentation and 
hudi-integ-test/spark-pipelines/SOURCE_THROTTLING.md with a reader example and 
operating limits. Website publication remains follow-up before release; this PR 
does not claim it is published.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to