pan3793 opened a new pull request, #58735: URL: https://github.com/apache/spark/pull/58735
### What changes were proposed in this pull request? `PushdownPredicatesAndPruneColumnsForCTEDef` collects the predicates at each CTE reference and pushes their OR-merged combination into the shared CTE definition. Each reference keeps its own predicates, so this only works for deterministic predicates -- a non-deterministic one is then evaluated twice, in the definition and again at each reference. This PR filters the collected predicates to deterministic ones before push-down. ### Why are the changes needed? A non-deterministic predicate evaluated a second time in the CTE definition can drop rows, which is a wrong-results bug: ```sql create or replace temp view t as select * from values (0), (1), (2) as t(c1); with v as (select c1, rand(1) r from t) select c1 from v where rand(2) < 0.5 union all select c1 from v where rand(3) < 0.5; ``` The definition is non-deterministic and referenced twice, so it survives `InlineCTE`. `(rand(2) < 0.5) OR (rand(3) < 0.5)` is pushed into the definition while both references keep their own filter, so the optimized plan holds four `rand` filters instead of two and rows are filtered twice. The rule's scaladoc claimed determinism was taken care of by `ScanOperation`, but SPARK-39764 (3.4.0) replaced that with `PhysicalOperation`, which returns a single filter even when it is non-deterministic (its `filters.length > 1` assert only binds when more than one filter was collected). The comment is corrected along with the fix. ### Does this PR introduce _any_ user-facing change? Yes, it fixes wrong results for the query shape above. Affects 3.4.0 and later. ### How was this patch tested? New test in `CTEInlineSuite`, which fails without the fix (four `rand` filters instead of two). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
