L. C. Hsieh created SPARK-61661:
-----------------------------------

             Summary: Avoid serializing the child plan into every task from 
HashAggregateExec codegen
                 Key: SPARK-61661
                 URL: https://issues.apache.org/jira/browse/SPARK-61661
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


The code generated by HashAggregateExec (grouping-key path) references the plan 
node itself via ctx.addReferenceObj("plan", this) to call a few runtime helpers 
(createHashMap, getEmptyAggregationBuffer, getTaskContext, 
addFastHashMapCloseHook, finishAggregate). Code references are serialized into 
every task binary and deserialized by every task, so each task deserializes the 
aggregate's whole child subtree, and for a final aggregate the whole map-side 
plan through the exchange, although none of the helpers reads the child.

On a plan with 150 CASE columns and 40 sums (local[8], 264 tasks), per-task 
task binary deserialization drops from ~1.8 ms to ~0.6 ms (258 KB -> 152 KB) 
when the generated code references a small helper holding only the schemas and 
aggregate buffer expressions instead of the plan node.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to