Sagar Sumit created HUDI-7841: --------------------------------- Summary: RLI should consider only pruned partitions for file skipping Key: HUDI-7841 URL: https://issues.apache.org/jira/browse/HUDI-7841 Project: Apache Hudi Issue Type: Improvement Reporter: Sagar Sumit Fix For: 1.0.0
Even though RLI scans only matching files, it tries to get those candidate files by iterating over all files from file index. See - [https://github.com/apache/hudi/blob/f4be74c29471fbd6afff472f8db292e6b1f16f05/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala#L47] Instead, it can use the `prunedPartitionsAndFileSlices` to only consider pruned partitions whenever there is a partition predicate. -- This message was sent by Atlassian Jira (v8.20.10#820010)