L. C. Hsieh created SPARK-61661:
-----------------------------------
Summary: Avoid serializing the child plan into every task from
HashAggregateExec codegen
Key: SPARK-61661
URL: https://issues.apache.org/jira/browse/SPARK-61661
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
The code generated by HashAggregateExec (grouping-key path) references the plan
node itself via ctx.addReferenceObj("plan", this) to call a few runtime helpers
(createHashMap, getEmptyAggregationBuffer, getTaskContext,
addFastHashMapCloseHook, finishAggregate). Code references are serialized into
every task binary and deserialized by every task, so each task deserializes the
aggregate's whole child subtree, and for a final aggregate the whole map-side
plan through the exchange, although none of the helpers reads the child.
On a plan with 150 CASE columns and 40 sums (local[8], 264 tasks), per-task
task binary deserialization drops from ~1.8 ms to ~0.6 ms (258 KB -> 152 KB)
when the generated code references a small helper holding only the schemas and
aggregate buffer expressions instead of the plan node.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]