Yang Jie created SPARK-61336:
--------------------------------

             Summary: CleanupDynamicPruningFilters should count only top-level 
equality conjuncts as fixing a partition key
                 Key: SPARK-61336
                 URL: https://issues.apache.org/jira/browse/SPARK-61336
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Yang Jie


`CleanupDynamicPruningFilters` removes the DPP filter on a partition key that 
an equality already fixes (SPARK-38148). It collects an equality anywhere in a 
conjunct of the `Filter` above the scan, including one below `NOT` or `OR`, so 
`part <> 3` or `(part = 3 OR id > 50)` also drops the DPP filter, although 
neither fixes `part`. The results stay the same, but the scan reads partitions 
the DPP filter could have pruned.

This holds for V1 file sources, Hive tables and DSv2 scans other than the 
built-in V2 file scans, which count only top-level `=` and `<=>` conjuncts 
since SPARK-30628. The rule could count only top-level conjuncts for every 
relation, so the two paths agree and V1 and V2 behave the same.

Found in the review of #55579 (SPARK-30628).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to