Yan Yan created SPARK-56385:
-------------------------------
Summary: Track pushed filter expressions on
DataSourceV2ScanRelation
Key: SPARK-56385
URL: https://issues.apache.org/jira/browse/SPARK-56385
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Yan Yan
Introducing an additional field in `DataSourceV2ScanRelation` to record which
filter expressions were fully pushed down to the data source during logical
planning.
Once a filter is pushed into a DSv2 scan, the logical plan loses track of it —
the scan node contributes no constraints to the optimizer. This prevents
constraint propagation (e.g., inferring filters across joins) and
identification of redundant post-scan filters.
This change stores the pushed filter information on the scan relation. Wiring
it into the constraint propagation framework is deferred to a follow-up as it
requires plan stability testing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
