nsivabalan opened a new issue, #20674: URL: https://github.com/apache/hudi/issues/20674
### Task Description Add a cap on how many commits one micro-batch of Hudi's Spark structured-streaming source consumes. This applies to plain `readStream` and to SDP streaming-table chaining (#20407). Target: this release. Follow-up from the #20516 review ([review](https://github.com/apache/hudi/pull/20516#pullrequestreview-5930942678)). **Today.** `HoodieStreamSourceV1` / `HoodieStreamSourceV2` implement only Spark's `Source`, and `getOffset` always returns the latest completed instant. Each micro-batch therefore reads everything from the checkpoint to the newest commit: - A consumer that was stopped for a while catches up in one unbounded batch. - With #20516's `FAIL` default, a consumer that can't keep up fails once archival or cleaning overtakes its checkpoint. **Precedent.** - Hudi's other incremental readers already cap batches: HoodieIncrSource's `hoodie.streamer.source.hoodieincr.num_instants` (default 5) and Flink's `read.commits.limit`. - Delta Lake has `maxFilesPerTrigger` (default 1000) and `maxBytesPerTrigger`. - Iceberg has `streaming-max-files-per-micro-batch`. **Proposed shape** - **Config:** e.g. `hoodie.datasource.read.streaming.max.commits.per.trigger`. No default, so the source stays unbounded as today. It counts completed write commits after the start offset. - **Admission control:** implement `SupportsAdmissionControl`. `latestOffset(start, limit)` returns the offset of the Nth completed commit after `start`, and `getDefaultReadLimit` comes from the config. - **`Trigger.AvailableNow`:** implement `SupportsTriggerAvailableNow`, so AvailableNow (SDP triggered updates) pins its end offset once and drains it in capped batches. Without it, Spark wraps the source and ignores the limit. - **Offset versions:** count only write commits, not cleans or rollbacks, which V1 offsets can land on. In V2, count by completion time. - **Tests:** COW and MOR, both offset versions. A backlog of N commits with a limit of k should give ceil(N/k) batches with exact rows. Add an AvailableNow case that stops at the pinned end. **Out of scope:** chunking `READ_UPTO_LATEST_COMMIT` snapshot recovery (#20516). It stays a single batch for now, but this admission-control hook is where chunked recovery would plug in later. ### Task Type Code improvement/refactoring ### Related Issues **Parent feature issue:** #20401 **Related:** #20516, #20407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
