[ 
https://issues.apache.org/jira/browse/SPARK-61666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

L. C. Hsieh updated SPARK-61666:
--------------------------------
    Description: 
Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen 
from referencing the plan node itself, so its child subtree is no longer 
serialized into every task binary.

The same ctx.addReferenceObj("plan", this) pattern remains in:
- SortExec (createSorter)
- SortMergeJoinExec (cleanupResources)
- SortMergeAsOfJoinExec (cleanupResources)

Each of these serializes the operator and its whole child subtree into every 
task of the stage. The three have to be changed together: SortExec.createSorter 
writes the mutable rowSorter field on the node, and 
SortMergeJoinExec/SortMergeAsOfJoinExec.cleanupResources reach that same field 
through their children. That works today only because both "plan" references 
deserialize to the same object within one task binary, so the shared sorter 
state has to move together with the helpers.

ShuffledHashJoinExec has the same pattern and is tracked separately in 
SPARK-61668.

Once both are fixed, add a test-time guard that rejects SparkPlan objects in 
what any WholeStageCodegenExec stage ships to tasks (its closure / evaluator 
factory and the generated code references), so the pattern does not come back.


  was:
Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen 
from referencing the plan node itself, so its child subtree is no longer 
serialized into every task binary.

The same ctx.addReferenceObj("plan", this) pattern remains in:
- SortExec (createSorter)
- SortMergeJoinExec (cleanupResources)
- SortMergeAsOfJoinExec (cleanupResources)
- ShuffledHashJoinExec (buildHashedRelation)

Each of these serializes the operator and its whole child subtree into every 
task of the stage. This is not a mechanical port of SPARK-61661: 
SortExec.createSorter writes the mutable rowSorter field on the node, and 
SortMergeJoinExec/SortMergeAsOfJoinExec.cleanupResources reach that same field 
through their children. That works today only because both "plan" references 
deserialize to the same object within one task binary, so the shared sorter 
state has to move together with the helpers. 
ShuffledHashJoinExec.buildHashedRelation needs its own audit.

Once these are fixed, add a test-time guard that rejects SparkPlan objects in 
what any WholeStageCodegenExec stage ships to tasks (its closure / evaluator 
factory and the generated code references), so the pattern does not come back.


        Summary: Avoid serializing the child plan into every task from SortExec 
and sort-merge join codegen  (was: Avoid serializing the child plan into every 
task from SortExec and join operators codegen)

> Avoid serializing the child plan into every task from SortExec and sort-merge 
> join codegen
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61666
>                 URL: https://issues.apache.org/jira/browse/SPARK-61666
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>
> Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen 
> from referencing the plan node itself, so its child subtree is no longer 
> serialized into every task binary.
> The same ctx.addReferenceObj("plan", this) pattern remains in:
> - SortExec (createSorter)
> - SortMergeJoinExec (cleanupResources)
> - SortMergeAsOfJoinExec (cleanupResources)
> Each of these serializes the operator and its whole child subtree into every 
> task of the stage. The three have to be changed together: 
> SortExec.createSorter writes the mutable rowSorter field on the node, and 
> SortMergeJoinExec/SortMergeAsOfJoinExec.cleanupResources reach that same 
> field through their children. That works today only because both "plan" 
> references deserialize to the same object within one task binary, so the 
> shared sorter state has to move together with the helpers.
> ShuffledHashJoinExec has the same pattern and is tracked separately in 
> SPARK-61668.
> Once both are fixed, add a test-time guard that rejects SparkPlan objects in 
> what any WholeStageCodegenExec stage ships to tasks (its closure / evaluator 
> factory and the generated code references), so the pattern does not come back.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to