pan3793 commented on PR #58661: URL: https://github.com/apache/spark/pull/58661#issuecomment-5621052029
Thanks @peter-toth, all five addressed in 1c6d2a9d8a6. Two more things went into the same commit from a further pass over the tree: - `PushdownPredicatesAndPruneColumnsForCTEDef` pushed non-deterministic reference predicates into a definition that is not inlined while the reference kept them, so `WITH v AS MATERIALIZED (SELECT c1 FROM t) SELECT count(*) FROM v WHERE rand() < 0.5` filtered twice. Pre-existing for non-deterministic multiply-referenced CTEs, reachable for any CTE once it is `MATERIALIZED`. It now pushes only deterministic predicates; pruning still accounts for every predicate. The explain golden's double application is fine for deterministic predicates only. - A materialized CTE inside a correlated subquery hit `_LEGACY_ERROR_TEMP_2168` at optimization when the query of the WITH clause references the outer query, since `DecorrelateInnerQuery` cannot pass through a `WithCTE` that is not inlined. That is now rejected at analysis with `UNSUPPORTED_FEATURE.MATERIALIZED_CTE_IN_CORRELATED_SUBQUERY`, exactly on the correlated path: a correlation above a derived table holding the WITH clause and an uncorrelated subquery stay accepted. Documented on the syntax page. Also: `CTERelationDef` prints the option as `MATERIALIZED` / `NOT MATERIALIZED`, so it cannot be read as `forceSkipInline`, and `cte.sql` gains the unreferenced correlated case. PR description updated to the final mechanism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
