AnhTtis opened a new pull request, #58377: URL: https://github.com/apache/spark/pull/58377
### What changes were proposed in this pull request? In Spark Catalyst, `With` promises that common expressions are evaluated only once even when referenced multiple times. `RewriteWithExpression` keeps that promise by hoisting multi-referenced definitions into a child `Project`. However, inside conditional branches (such as `CASE WHEN`, `If`, `Coalesce`) or join conditions spanning both join sides, eager pre-evaluation cannot be unconditionally placed into a child `Project` without risking premature evaluation of expressions that can throw exceptions. Consequently, `RewriteWithExpression` inlines the common expressions into each reference site. When the inlined expression is nondeterministic (such as `randstr(...)`, `rand()`, `uuid()`, `uniform(...)`, `shuffle(...)`, `reflect(...)`), inlining causes each reference to evaluate independently. For example, `CASE WHEN a > 0 THEN randstr(3, 0) BETWEEN 'a' AND 'b' END` expands `BETWEEN` to two references, causing two different random strings to be generated for the `>=` and `<=` checks. This PR addresses the issue by introducing **lazy per-row evaluation** for common expressions: 1. Multi-referenced common expressions inside conditional branches and join conditions are evaluated lazily upon first reference in the active control flow path. 2. The evaluated result and nullability are cached per row, ensuring that subsequent references within the same row reuse the cached value. 3. Eliminates inaccurate multiple evaluations of nondeterministic expressions in conditional branches and resolves the join condition inlining TODO. Fixes [SPARK-58902](https://issues.apache.org/jira/browse/SPARK-58902). ### Why are the changes needed? To guarantee single-evaluation semantics for common expressions inside conditional branches and join conditions, preventing silent data corruption for nondeterministic functions. ### Does this PR introduce _any_ user-facing change? Yes. Nondeterministic expressions inside conditional branches (such as `BETWEEN` on `randstr`) now evaluate exactly once per row instead of evaluating distinct values for each reference. ### How was this patch tested? - Added optimizer unit tests in `RewriteWithExpressionSuite.scala`. - Added end-to-end SQL regression tests for conditional nondeterministic expressions in `DataFrameSuite.scala` and `SQLQueryTestSuite`. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
