lokeshj1703 opened a new issue, #19756: URL: https://github.com/apache/hudi/issues/19756
### Problem The S3/GCS cloud-object incremental sources (`S3EventsHoodieIncrSource` / `GcsEventsHoodieIncrSource`) can silently drop records when a previous batch persisted a `commit#fileKey` mid-commit-pagination checkpoint (i.e. the prior batch hit `sourceLimit` before finishing the start commit's files). Files in the start commit after the checkpoint key become unreachable, and the checkpoint advances past them. Root causes: 1. `QueryRunner.runIncrementalQuery` passes `queryInfo.getStartInstant()` as the Spark `START_COMMIT`. The V1 incremental relation's range is start-exclusive, so the start commit is dropped from the scan; the downstream `(commit_time || object_key) > 'commit#fileKey'` filter then matches nothing in the start commit and the checkpoint is emitted as a bare instant, skipping the remaining files. 2. The file-group-reader based read path drops rows when a filter references a column not in the required schema (the required filters from the relation factory are not visible to the Spark planner, so the required schema can miss filter columns). Triggers: cold-start backfills against a source table with a large initial commit, bursty writers, or a low `hoodie.deltastreamer.read.source.limit`. ### Expected The incremental query includes the start commit so mid-commit pagination resumes correctly and no records are dropped. ### Related apache/hudi #18949 proposed this fix against an older master and is stale/conflicting. This is a fresh version rebased onto current master. ### Environment - Hudi version: master (1.x) - S3/GCS incremental sources with source-limit based batching. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
