L. C. Hsieh created SPARK-61761:
-----------------------------------

             Summary: Add a generic test that whole-stage codegen stages do not 
serialize plan nodes into tasks
                 Key: SPARK-61761
                 URL: https://issues.apache.org/jira/browse/SPARK-61761
             Project: Spark
          Issue Type: Test
          Components: SQL
    Affects Versions: 4.4.0
            Reporter: L. C. Hsieh


SPARK-61661, SPARK-61666 and SPARK-61668 removed ctx.addReferenceObj("plan", 
this) from the generated code of HashAggregateExec, SortExec, 
SortMergeJoinExec, SortMergeAsOfJoinExec and ShuffledHashJoinExec, so tasks no 
longer deserialize the plan subtree under those operators. Each added a test 
for its own operators only.

Add a generic test in WholeStageCodegenSuite that runs queries covering the 
operators that generate code, with AQE and 
spark.sql.execution.usePartitionEvaluator on and off, and asserts that no 
WholeStageCodegenExec in the final plan (including adaptive query stages, 
cached plans and subqueries) serializes a SparkPlan node with the RDD it adds 
on top of its input RDDs.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to