marcuslin123 opened a new pull request, #57980: URL: https://github.com/apache/spark/pull/57980
### What changes were proposed in this pull request? This PR deep-copies stateful expressions before they are evaluated by the optimizer. Specifically, it: - clones the logical plan and replaces stateful expressions with fresh copies before `QueryExecution.optimizedPlan` invokes the optimizer; - uses fresh stateful expressions when `ConvertToLocalRelation` evaluates project and filter expressions; and - adds regression coverage for both paths using `NamedLambdaVariable`. ### Why are the changes needed? Logical plans can be shared by DataFrames derived from the same base DataFrame. Although `QueryExecution` cloned the plan before optimization, stateful expressions such as `NamedLambdaVariable` could retain shared mutable state across those clones. Concurrent optimization could therefore mutate the same expression state from multiple threads and silently corrupt query results. `ConvertToLocalRelation` had the same issue because it evaluated expressions from the shared plan directly. ### Does this PR introduce _any_ user-facing change? Yes. Concurrent actions on DataFrames derived from a shared logical plan no longer risk silent result corruption caused by shared mutable expression state. ### How was this patch tested? Added regression tests to `ConvertToLocalRelationSuite` and `QueryExecutionSuite`. ``` build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelationSuite' build/sbt 'sql/testOnly org.apache.spark.sql.execution.QueryExecutionSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5). Codex was used for code assistance and review; the patch was not entirely generated by Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
