L. C. Hsieh created SPARK-61668:
-----------------------------------

             Summary: Avoid serializing the child plans into every task from 
ShuffledHashJoinExec codegen
                 Key: SPARK-61668
                 URL: https://issues.apache.org/jira/browse/SPARK-61668
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen 
from referencing the plan node itself, so its child subtree is no longer 
serialized into every task binary.

ShuffledHashJoinExec codegen still calls ctx.addReferenceObj("plan", this) to 
call buildHashedRelation from the generated code, so the operator and both of 
its child subtrees are serialized into every task of the stage. 
buildHashedRelation needs an audit of what it reads (build keys, build plan 
output and metrics) so that the generated code can reference a small helper 
instead.

The SortExec / SortMergeJoinExec / SortMergeAsOfJoinExec instances of the same 
pattern are tracked in SPARK-61666, which also covers adding a test-time guard 
against SparkPlan objects in what codegen stages ship to tasks once both are 
fixed.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to