L. C. Hsieh created SPARK-61668:
-----------------------------------
Summary: Avoid serializing the child plans into every task from
ShuffledHashJoinExec codegen
Key: SPARK-61668
URL: https://issues.apache.org/jira/browse/SPARK-61668
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen
from referencing the plan node itself, so its child subtree is no longer
serialized into every task binary.
ShuffledHashJoinExec codegen still calls ctx.addReferenceObj("plan", this) to
call buildHashedRelation from the generated code, so the operator and both of
its child subtrees are serialized into every task of the stage.
buildHashedRelation needs an audit of what it reads (build keys, build plan
output and metrics) so that the generated code can reference a small helper
instead.
The SortExec / SortMergeJoinExec / SortMergeAsOfJoinExec instances of the same
pattern are tracked in SPARK-61666, which also covers adding a test-time guard
against SparkPlan objects in what codegen stages ship to tasks once both are
fixed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]