hudi-agent commented on code in PR #20328:
URL: https://github.com/apache/hudi/pull/20328#discussion_r4278358448
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -113,14 +114,44 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
}
val validIndexedColumns = getValidIndexedColumnsFunc.apply(indexDefinition)
- val indexFilter =
queryFilters.map(translateIntoColumnStatsIndexFilterExpr(_, isExpressionIndex,
validIndexedColumns)).reduce(And)
- if (indexFilter.equals(TrueLiteral)) {
- // if there are any non indexed cols or we can't translate source expr,
we have to read all files and may not benefit from col stats lookup.
- fileNamesFromPrunedPartitions
+ // Only split top-level AND predicates; keep OR and NOT subtrees intact.
+ val indexFilters =
queryFilters.flatMap(SparkBaseIndexSupport.splitConjuncts)
+ .map(translateIntoColumnStatsIndexFilterExpr(_, isExpressionIndex,
validIndexedColumns))
+ .filterNot(_.equals(TrueLiteral))
+ if (indexFilters.isEmpty) {
+ // No translated predicate can prune files, so avoid scanning the index.
+ fileNamesFromPrunedPartitions
} else {
- // only lookup in col stats if all filters are eligible to be looked up
in col stats index in MDT
+ val projectedColumns = indexDf.columns.toSet
+ val statsToNullCount = validIndexedColumns.flatMap { column =>
Review Comment:
🤖 nit: would it make sense to pull the stats-to-nullCount mapping and the
per-conjunct guard into a small private helper (e.g.
`buildMissingStatsGuardedFilter`)? It would keep `getCandidateFiles` focused on
lookup.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
