[
https://issues.apache.org/jira/browse/SPARK-61666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-61666:
--------------------------------
Description:
Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen
from referencing the plan node itself, so its child subtree is no longer
serialized into every task binary.
The same ctx.addReferenceObj("plan", this) pattern remains in:
- SortExec (createSorter)
- SortMergeJoinExec (cleanupResources)
- SortMergeAsOfJoinExec (cleanupResources)
Each of these serializes the operator and its whole child subtree into every
task of the stage. The three have to be changed together: SortExec.createSorter
writes the mutable rowSorter field on the node, and
SortMergeJoinExec/SortMergeAsOfJoinExec.cleanupResources reach that same field
through their children. That works today only because both "plan" references
deserialize to the same object within one task binary, so the shared sorter
state has to move together with the helpers.
ShuffledHashJoinExec has the same pattern and is tracked separately in
SPARK-61668.
Once both are fixed, add a test-time guard that rejects SparkPlan objects in
what any WholeStageCodegenExec stage ships to tasks (its closure / evaluator
factory and the generated code references), so the pattern does not come back.
was:
Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen
from referencing the plan node itself, so its child subtree is no longer
serialized into every task binary.
The same ctx.addReferenceObj("plan", this) pattern remains in:
- SortExec (createSorter)
- SortMergeJoinExec (cleanupResources)
- SortMergeAsOfJoinExec (cleanupResources)
- ShuffledHashJoinExec (buildHashedRelation)
Each of these serializes the operator and its whole child subtree into every
task of the stage. This is not a mechanical port of SPARK-61661:
SortExec.createSorter writes the mutable rowSorter field on the node, and
SortMergeJoinExec/SortMergeAsOfJoinExec.cleanupResources reach that same field
through their children. That works today only because both "plan" references
deserialize to the same object within one task binary, so the shared sorter
state has to move together with the helpers.
ShuffledHashJoinExec.buildHashedRelation needs its own audit.
Once these are fixed, add a test-time guard that rejects SparkPlan objects in
what any WholeStageCodegenExec stage ships to tasks (its closure / evaluator
factory and the generated code references), so the pattern does not come back.
Summary: Avoid serializing the child plan into every task from SortExec
and sort-merge join codegen (was: Avoid serializing the child plan into every
task from SortExec and join operators codegen)
> Avoid serializing the child plan into every task from SortExec and sort-merge
> join codegen
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-61666
> URL: https://issues.apache.org/jira/browse/SPARK-61666
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
>
> Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen
> from referencing the plan node itself, so its child subtree is no longer
> serialized into every task binary.
> The same ctx.addReferenceObj("plan", this) pattern remains in:
> - SortExec (createSorter)
> - SortMergeJoinExec (cleanupResources)
> - SortMergeAsOfJoinExec (cleanupResources)
> Each of these serializes the operator and its whole child subtree into every
> task of the stage. The three have to be changed together:
> SortExec.createSorter writes the mutable rowSorter field on the node, and
> SortMergeJoinExec/SortMergeAsOfJoinExec.cleanupResources reach that same
> field through their children. That works today only because both "plan"
> references deserialize to the same object within one task binary, so the
> shared sorter state has to move together with the helpers.
> ShuffledHashJoinExec has the same pattern and is tracked separately in
> SPARK-61668.
> Once both are fixed, add a test-time guard that rejects SparkPlan objects in
> what any WholeStageCodegenExec stage ships to tasks (its closure / evaluator
> factory and the generated code references), so the pattern does not come back.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]