anshulbaliga7 commented on PR #58077: URL: https://github.com/apache/spark/pull/58077#issuecomment-5550833437
Thanks @cloud-fan for the review, have addressed the architectural gap: the evaluator's captured child is now transformed via `transformUp` to replace every prepared `ScalarSubquery` with its already-evaluated Literal before entering `ctx.references`, so no nested `BaseSubqueryExec` (from a scalar subquery embedded in the multi-column LHS) crosses the task-closure serialization boundary. Verified the ordering is safe by tracing `SparkPlan.executeQuery` - `waitForSubqueries()` (which resolves all nested `ScalarSubquery` results) always completes before `doExecute()/doGenCode()` can force the evaluator's lazy construction. Added a regression that serializes the evaluator via `JavaSerializer` and confirms no `ScalarSubquery/BaseSubqueryExec` remains reachable in the deserialized child while `eval()` still produces the correct result. Also folded in the three P3 fixes (accurate `CreateNamedStruct` comment, accurate legacy/ANSI wording, avoided the extra array copy in `multiColEvaluator`). 12/12 tests pass locally. Can you PTAL again? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
