rangareddy commented on issue #17267:
URL: https://github.com/apache/hudi/issues/17267#issuecomment-5365808989
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-8164).
**Findings: confirmed, unchanged on `master`.**
`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:439-448`,
inside `lookupCandidateFilesInMetadataTable`:
```scala
if (isDataSkippingEnabled) {
for(indexSupport: SparkBaseIndexSupport <- indicesSupport) {
if (indexSupport.isIndexAvailable &&
indexSupport.supportsQueryType(options)) {
val prunedFileNames = indexSupport.computeCandidateIsStrict(spark,
this, queryFilters, queryReferencedColumns,
prunedPartitionsAndFileSlices, shouldPushDownFilesFilter)
if (prunedFileNames.nonEmpty) {
return Try(prunedFileNames)
}
}
}
}
```
The first index that prunes anything returns immediately, so no index ever
sees another index's pruned set - exactly the behaviour described.
One thing worth adding to the ticket while designing the chaining: because
of this `return`, index precedence today is decided implicitly by the iteration
order of `indicesSupport`. Any chaining design has to make that ordering
explicit and deliberate, and the "shared contract/param among these index
supports" the description calls for is where that belongs.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
