xumingming commented on PR #58661: URL: https://github.com/apache/spark/pull/58661#issuecomment-5632256002
> PushdownPredicatesAndPruneColumnsForCTEDef now pushes only deterministic reference predicates into a CTE that is not inlined. A non-deterministic predicate such as rand() < 0.5 was applied both in the definition and at the reference, halving the rows: pre-existing for non-deterministic multiply-referenced CTEs, and reachable for any CTE once it is MATERIALIZED. Column pruning still accounts for every predicate. The double application seen in the explain golden is fine for deterministic predicates only. @pan3793 Good catch on this! How about extract the fix into dedicated PR, so earlier versions can backport/cherry-picked easily. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
