prashantwason opened a new pull request, #19914: URL: https://github.com/apache/hudi/pull/19914
### Describe the issue this Pull Request addresses COW incremental reads go through `HoodieFileGroupReaderBasedFileFormat` but have no log files to merge — they read only base Parquet files. Three guards written for MOR incremental (which needs row-level log merging) were applied unconditionally to all incremental reads, penalizing the COW path with unnecessary overhead. ### Summary and Changelog Three changes to `HoodieFileGroupReaderBasedFileFormat`, all scoped to COW incremental reads (no log files to merge): - **Enable file splitting**: `isSplitable` returned `false` for all incremental reads (`!isIncremental`). COW incremental has no log files, so splitting is safe. Removed the `!isIncremental` guard; MOR and bootstrap reads still disable splitting. - **Enable vectorized reading**: `supportVectorizedRead` was gated on `!isIncremental`, forcing row-by-row deserialization even when the schema supports columnar batches. Changed to `!(isMOR && isIncremental)` so only MOR incremental disables vectorization. - **Bypass HoodieFileGroupReader for base-file-only reads**: When a file slice has no log files, delegate directly to Spark's stock `ParquetFileFormat` reader, avoiding the overhead of `HoodieFileGroupReader` (schema handler, record merger, row copy/seal). Changelog: - `HoodieFileGroupReaderBasedFileFormat.isSplitable`: allow splitting for COW incremental - `HoodieFileGroupReaderBasedFileFormat.supportBatch`: allow vectorized reading for COW incremental - `HoodieFileGroupReaderBasedFileFormat.buildReaderWithPartitionValues`: add stock `ParquetFileFormat` bypass for base-file-only file slices ### Benchmark Tested on a production COW table (22,432 files, 1.2M records, 7 incremental commits): | Test | Rate | |------|------| | Hudi incremental (before) | 4,552 r/s | | Hudi incremental (after) | 10,220 r/s | | Plain parquet baseline | 11,081 r/s | Overhead reduced from ~34% to ~8% vs plain parquet. The remaining gap is the expected `HoodieIncrementalFileIndex` setup cost (~7-10s). ### Risk level Low — all three changes are guarded by `!isMOR` or log-file presence checks. MOR reads, bootstrap reads, and snapshot reads are unaffected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
