[ 
https://issues.apache.org/jira/browse/SPARK-61647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Kumar updated SPARK-61647:
---------------------------------
    Description: 
A partial hash aggregate with the codegen fast hash map registers a 
task-completion hook that closes the map, in case its output is not fully 
consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees the 
map's memory page, but the hook keeps the map, and with it its bucket array, 
reachable until the task ends: 512 KiB at the default 
{{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is on 
the heap and not tracked by the memory manager.

A task that runs many partitions of such an aggregate creates one map per 
partition, also for an empty one, and holds all of their bucket arrays until it 
ends.

On master, this happens where {{GroupPartitionsExec}} groups a keyed scan's 
splits above a partial hash aggregate. {{CombineAdjacentAggregation}} folds a 
partial and a final aggregate across {{GroupPartitionsExec}}, but not the first 
partial aggregate of a distinct aggregation, which sits below a 
{{PartialMerge}} one. In a local test, a DSv2 scan whose splits all report the 
same partition key, with {{groupBy(key).agg(countDistinct(j), sum(j))}}, ran 
every split's partial aggregate in one task. With 168 splits, the task held 84 
MiB of bucket arrays until it ended. With 2,048 splits, it held 1 GiB.

SPARK-60265 (https://github.com/apache/spark/pull/59522) would make this more 
common: its packing node coalesces a scan's partitions below a shuffle, above 
any partial aggregate.

{{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the 
generated vectorized map. Nothing probes a map after it is closed.

  was:
A partial hash aggregate with the codegen fast hash map registers a 
task-completion hook that closes the map, in case its output is not fully 
consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees the 
map's memory page, but the hook keeps the map, and with it its bucket array, 
reachable until the task ends: 512 KiB at the default 
{{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is on 
the heap and not tracked by the memory manager.

A task that coalesces many partitions of such an aggregate creates one map per 
partition, also for an empty one, and holds all of their bucket arrays until it 
ends. Nothing on master coalesces below a partial aggregate today, but 
SPARK-60265 (https://github.com/apache/spark/pull/59522) does: its packing node 
coalesces a scan's partitions below a shuffle, above any partial aggregate. 
With 2,048 input splits in one task, the task held 1 GiB of bucket arrays.

{{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the 
generated vectorized map. Nothing probes a map after it is closed.


> Drop a fast hash map's buckets when it closes
> ---------------------------------------------
>
>                 Key: SPARK-61647
>                 URL: https://issues.apache.org/jira/browse/SPARK-61647
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Rakesh Kumar
>            Priority: Major
>              Labels: pull-request-available
>
> A partial hash aggregate with the codegen fast hash map registers a 
> task-completion hook that closes the map, in case its output is not fully 
> consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees 
> the map's memory page, but the hook keeps the map, and with it its bucket 
> array, reachable until the task ends: 512 KiB at the default 
> {{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is 
> on the heap and not tracked by the memory manager.
> A task that runs many partitions of such an aggregate creates one map per 
> partition, also for an empty one, and holds all of their bucket arrays until 
> it ends.
> On master, this happens where {{GroupPartitionsExec}} groups a keyed scan's 
> splits above a partial hash aggregate. {{CombineAdjacentAggregation}} folds a 
> partial and a final aggregate across {{GroupPartitionsExec}}, but not the 
> first partial aggregate of a distinct aggregation, which sits below a 
> {{PartialMerge}} one. In a local test, a DSv2 scan whose splits all report 
> the same partition key, with {{groupBy(key).agg(countDistinct(j), sum(j))}}, 
> ran every split's partial aggregate in one task. With 168 splits, the task 
> held 84 MiB of bucket arrays until it ended. With 2,048 splits, it held 1 GiB.
> SPARK-60265 (https://github.com/apache/spark/pull/59522) would make this more 
> common: its packing node coalesces a scan's partitions below a shuffle, above 
> any partial aggregate.
> {{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the 
> generated vectorized map. Nothing probes a map after it is closed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to