lokeshj1703 opened a new pull request, #19757: URL: https://github.com/apache/hudi/pull/19757
### Describe the issue this Pull Request addresses closes #19756 ### Summary and Changelog S3/GCS cloud-object incremental sources could silently drop records when a prior batch persisted a `commit#fileKey` mid-commit-pagination checkpoint. Two fixes: - `QueryRunner.runIncrementalQuery` now passes `queryInfo.getPreviousInstant()` as `START_COMMIT` (the incremental range is start-exclusive, so using the previous instant keeps the start commit in the scan), forces `INCREMENTAL_READ_TABLE_VERSION` to 6 for these V1-checkpoint sources, and sets `INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN` to false. - `HoodieFileGroupReaderBasedFileFormat` reads a `readRequiredSchema` that adds columns referenced only by required filters (which the Spark planner does not see) back to the required schema, and projects back to the output schema only when the schema was augmented, preserving columnar batches otherwise. Adds `TestIncrementalReadWithFileGroupReader` plus regression coverage in `TestS3EventsHoodieIncrSource` / `TestGcsEventsHoodieIncrSource`. This is a fresh version of the stale #18949 rebased onto current master (reconciled with the vectorized file-format code). ### Impact Fixes silent record loss for S3/GCS incremental sources using source-limit batching. No public API change. ### Risk Level medium. Touches the cloud incremental source query path and the file-group-reader based read path. Covered by new functional tests and existing incr-source regression tests. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
