szehon-ho opened a new pull request, #58145:
URL: https://github.com/apache/spark/pull/58145
### What changes were proposed in this pull request?
Adds two optional methods to the `SupportsPushDownCatalystFilters` mix-in,
both defaulting to `Nil`:
- `additionalCatalystFilters`: filters the source inferred that the query
did not state. Spark
resolves their column names against the scan output and adds them to the
logical `Filter`.
- `fullyPushedFilters`: filters the source evaluates itself. An additional
filter listed here is
recorded on `DataSourceV2ScanRelation.pushedFilters` and dropped from
`FilterExec`, so it stays
available to the optimizer without being evaluated twice.
`V2ScanRelationPushDown` resolves and appends the additional filters, and
`DataSourceV2Strategy` is
where a fully pushed filter is dropped from the physical `Filter`.
### Why are the changes needed?
A source often knows a predicate the query did not state, and today it has
no way to tell Spark.
The common case is partitioning derived from a column: a table partitioned
by a transform of a
column (Iceberg partition transforms) or by a generated column derived from
one (Delta generated
columns) can translate `ts = '2026-01-01 10:00:00'` into a filter on the
`days(ts)` partition
column, and then read a single partition.
Spark cannot see that. It estimates the scan from whole-table statistics, so
a query reading one
partition is costed as if it read everything, which affects join selection
and other cost-based
decisions. Reporting the inferred filter lets the optimizer size the scan
from the partitions
actually read. Keeping the filter on the logical `Filter` is what makes it
visible to statistics
estimation, while `fullyPushedFilters` avoids paying to evaluate it again.
### Does this PR introduce _any_ user-facing change?
No. `SupportsPushDownCatalystFilters` lives in
`org.apache.spark.sql.internal.connector`, both new
methods have default implementations, and behavior is unchanged unless a
source overrides them.
### How was this patch tested?
New tests in `DataSourceV2Suite` cover the end-to-end behavior: an
additional filter that is not
fully pushed stays in `FilterExec`, and one that is fully pushed stays on
the logical `Filter`, is
recorded on the scan, and is dropped from `FilterExec`, with `checkAnswer`
in both cases. The test
source evaluates the filter it reports as fully pushed. New tests in
`DataSourceV2StrategySuite`
cover name resolution for nested columns (`s.tz`) and quoted dotted parts
(`` `a.b`.`c.d` ``).
Both suites pass: 101 tests, 0 failures.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
