L. C. Hsieh created SPARK-61761:
-----------------------------------
Summary: Add a generic test that whole-stage codegen stages do not
serialize plan nodes into tasks
Key: SPARK-61761
URL: https://issues.apache.org/jira/browse/SPARK-61761
Project: Spark
Issue Type: Test
Components: SQL
Affects Versions: 4.4.0
Reporter: L. C. Hsieh
SPARK-61661, SPARK-61666 and SPARK-61668 removed ctx.addReferenceObj("plan",
this) from the generated code of HashAggregateExec, SortExec,
SortMergeJoinExec, SortMergeAsOfJoinExec and ShuffledHashJoinExec, so tasks no
longer deserialize the plan subtree under those operators. Each added a test
for its own operators only.
Add a generic test in WholeStageCodegenSuite that runs queries covering the
operators that generate code, with AQE and
spark.sql.execution.usePartitionEvaluator on and off, and asserts that no
WholeStageCodegenExec in the final plan (including adaptive query stages,
cached plans and subqueries) serializes a SparkPlan node with the RDD it adds
on top of its input RDDs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]