prashantwason opened a new pull request, #20211: URL: https://github.com/apache/hudi/pull/20211
### Describe the issue this Pull Request addresses With `hoodie.datasource.read.incr.fallback.fulltablescan.enable=true`, the `fullTableScan` lazy val in `HoodieIncrementalRelationV1Trait` and `HoodieIncrementalRelationV2Trait` checks whether any file touched by the incremental commits has been deleted. It does this by calling `storage.exists()` on each file, one at a time, on the driver: ```scala affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)) ``` These traits back the MOR incremental relations, and also COW tables with table version < 8, which are read through `MergeOnReadIncrementalRelationV1` via `HoodieCopyOnWriteIncrementalHadoopFsRelationFactory`. In a production pipeline reading 10 commits that touched about 51K files on GCS, this made about 51K sequential HEAD requests (about 20 ms each), so the driver stalled for about 17 minutes with no log output. `IncrementalRelationV1` / `IncrementalRelationV2` already run this check in parallel with `sparkContext.parallelize`, but the MOR traits never got that change. ### Summary and Changelog The existence check in the MOR incremental fallback scan now runs across executors instead of one file at a time on the driver. - Add `HoodieBaseRelation.checkIfAnyFilesMissing`, which spreads the `exists()` calls over `min(fileCount, defaultParallelism)` partitions and gives the Spark job a description (`Checking existence of N files for incremental fallback scan`) so it is easy to find in the Spark UI. The previous job description is restored afterwards. - `HoodieIncrementalRelationV1Trait.fullTableScan` and `HoodieIncrementalRelationV2Trait.fullTableScan` use it. They skip the check when fallback is disabled or the start/end instant is archived, and log when they fall back because files are missing. ### Impact Performance only, with no API or behavior change. The fallback check now takes about as long as a single Spark job with `defaultParallelism` tasks, instead of growing with the number of affected files on the driver. ### Risk Level low. Only how the existence check is computed changes; when the check runs and how its result is used stay the same. Verified against the production pipeline above: the check now shows up as one Spark job and the 17-minute driver stall is gone. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
