Yang Jie created SPARK-61336:
--------------------------------
Summary: CleanupDynamicPruningFilters should count only top-level
equality conjuncts as fixing a partition key
Key: SPARK-61336
URL: https://issues.apache.org/jira/browse/SPARK-61336
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
`CleanupDynamicPruningFilters` removes the DPP filter on a partition key that
an equality already fixes (SPARK-38148). It collects an equality anywhere in a
conjunct of the `Filter` above the scan, including one below `NOT` or `OR`, so
`part <> 3` or `(part = 3 OR id > 50)` also drops the DPP filter, although
neither fixes `part`. The results stay the same, but the scan reads partitions
the DPP filter could have pruned.
This holds for V1 file sources, Hive tables and DSv2 scans other than the
built-in V2 file scans, which count only top-level `=` and `<=>` conjuncts
since SPARK-30628. The rule could count only top-level conjuncts for every
relation, so the two paths agree and V1 and V2 behave the same.
Found in the review of #55579 (SPARK-30628).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]