aokolnychyi commented on code in PR #57340:
URL: https://github.com/apache/spark/pull/57340#discussion_r3618873312
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileIndex.scala:
##########
@@ -133,6 +133,35 @@ trait FileIndex {
def listFiles(
partitionFilters: Seq[Expression], dataFilters: Seq[Expression]):
Seq[PartitionDirectory]
+ /**
+ * Given the `partitionKeyFilters` and remaining `dataFilters` Spark will
use to prune
+ * files for this index, returns the filters that this index guarantees are
fully applied for all
+ * returned rows. Only these are dropped from the row-level `FilterExec`
above the
+ * scan; any filter not returned is still evaluated after the scan for
correctness. By default
+ * only partition-key filters are considered fully pushed, preserving the
standard file-source
+ * behavior.
+ *
+ * For example:
+ * - An index that fully applies additional predicates (e.g. one that
prunes on non-partition
+ * columns) can override this to also return a subset of `dataFilters`.
+ * - Conversely, an index must omit a filter that it only partially
applies. With partition
+ * evolution, not all data files are partitioned by a given partition
column (e.g. older files
+ * written under a previous partition spec). Such files may contain both
matching and
+ * non-matching rows for that column, so the partition filter is not
fully pushed and must be
+ * omitted.
+ *
+ * Note these filters are the same predicates that [[listFiles]] prunes
with, but not necessarily
Review Comment:
This statement worries me. Like I don't know if this is any better that
having a simple boolean flag to indicate if FileIndex enforces / pushes down
all partition predicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
