danny0405 commented on issue #10334: URL: https://github.com/apache/hudi/issues/10334#issuecomment-5739192506
@Hans-Raintree I checked the 1.1.0 source, and your concern is valid for the incremental path used by the file-group reader. In [`HoodieIncrementalRelationV2Trait.fullTableScan`](https://github.com/apache/hudi/blob/release-1.1.0/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala#L193-L198), `affectedFilesInCommits.asScala.exists(...)` performs sequential storage existence checks on the driver. The path glob is applied only afterward, when filtering file slices. A query for one partition can therefore probe files from other partitions touched by the selected commits, and a missing file outside the requested partition can trigger fallback. There is also a distinction between affected files and files actually needed by the query: [`listAffectedFilesForCommits`](https://github.com/apache/hudi/blob/release-1.1.0/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java#L488-L507) deduplicates by full path, whereas the reader subsequently selects the latest merged file slices. That leaves room for unnecessary checks of older file versions. I would prioritize selecting the query-relevant slices first, preserving the existing glob and base/log-file semantics, then checking their required files with bounded concurrency. Cleaner metadata could help identify known deletions, but using it to replace existence checks would need guarantees about metadata coverage and concurrent cleaning. Two qualifications for diagnosing the 105s vs. 2.7s difference: - Enabling fallback can incur these checks even when no full scan ultimately happens. If the selected range includes archived instants, however, the condition short-circuits directly to fallback and skips the existence checks. The retention settings make this distinction worth checking. - The sequential behavior depends on the reader path. The file-group reader reuses this relation for both COW and MOR, while the [legacy COW reader](https://github.com/apache/hudi/blob/release-1.1.0/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelationV2.scala#L174-L245) already filters by glob and distributes existence checks through Spark. Could you share the table type/version, `hoodie.file.group.reader.enabled` setting, and whether the queried range included archived instants? A driver profile or storage-request metrics from the slow run would help distinguish existence-check overhead from an actual fallback scan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
