[ 
https://issues.apache.org/jira/browse/SPARK-61647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Kumar updated SPARK-61647:
---------------------------------
    Description: 
A partial hash aggregate with the codegen fast hash map registers a 
task-completion hook that closes the map, in case its output is not fully 
consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees the 
map's memory page, but the hook keeps the map reachable until the task ends, 
and with it the map's bucket array and its batch: 512 KiB of buckets at the 
default {{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16, and with 
a variable-length key or value, a {{VariableLengthRowBasedKeyValueBatch}} whose 
array of key offsets is the same size. The generated aggregation code also 
keeps the batch reachable. These arrays are on the heap and not tracked by the 
memory manager.

A task that runs many partitions of such an aggregate creates one map per 
partition, also for an empty one, and holds all of these arrays until it ends.

On master, this happens where {{GroupPartitionsExec}} groups a keyed scan's 
splits above a partial hash aggregate. {{CombineAdjacentAggregation}} folds a 
partial and a final aggregate across {{GroupPartitionsExec}}, but not the first 
partial aggregate of a distinct aggregation, which sits below a 
{{PartialMerge}} one. In a local test, a DSv2 scan whose splits all report the 
same partition key, with {{groupBy(key).agg(countDistinct(j), sum(j))}}, ran 
every split's partial aggregate in one task. With 168 splits, the task held 84 
MiB of bucket arrays until it ended. With 2,048 splits, it held 1 GiB.

SPARK-60265 (https://github.com/apache/spark/pull/59522) would make this more 
common: its packing node coalesces a scan's partitions below a shuffle, above 
any partial aggregate. With 2,048 splits packed into one task and a string 
grouping key, the task held 1 GiB of buckets and 1 GiB of key offsets.

{{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the 
generated vectorized map, and {{VariableLengthRowBasedKeyValueBatch.close()}} 
can drop its key offsets. Nothing reads them after a close.

  was:
A partial hash aggregate with the codegen fast hash map registers a 
task-completion hook that closes the map, in case its output is not fully 
consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees the 
map's memory page, but the hook keeps the map, and with it its bucket array, 
reachable until the task ends: 512 KiB at the default 
{{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is on 
the heap and not tracked by the memory manager.

A task that runs many partitions of such an aggregate creates one map per 
partition, also for an empty one, and holds all of their bucket arrays until it 
ends.

On master, this happens where {{GroupPartitionsExec}} groups a keyed scan's 
splits above a partial hash aggregate. {{CombineAdjacentAggregation}} folds a 
partial and a final aggregate across {{GroupPartitionsExec}}, but not the first 
partial aggregate of a distinct aggregation, which sits below a 
{{PartialMerge}} one. In a local test, a DSv2 scan whose splits all report the 
same partition key, with {{groupBy(key).agg(countDistinct(j), sum(j))}}, ran 
every split's partial aggregate in one task. With 168 splits, the task held 84 
MiB of bucket arrays until it ended. With 2,048 splits, it held 1 GiB.

SPARK-60265 (https://github.com/apache/spark/pull/59522) would make this more 
common: its packing node coalesces a scan's partitions below a shuffle, above 
any partial aggregate.

{{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the 
generated vectorized map. Nothing probes a map after it is closed.

        Summary: Drop a closed fast hash map's buckets and key offsets  (was: 
Drop a fast hash map's buckets when it closes)

> Drop a closed fast hash map's buckets and key offsets
> -----------------------------------------------------
>
>                 Key: SPARK-61647
>                 URL: https://issues.apache.org/jira/browse/SPARK-61647
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Rakesh Kumar
>            Priority: Major
>              Labels: pull-request-available
>
> A partial hash aggregate with the codegen fast hash map registers a 
> task-completion hook that closes the map, in case its output is not fully 
> consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees 
> the map's memory page, but the hook keeps the map reachable until the task 
> ends, and with it the map's bucket array and its batch: 512 KiB of buckets at 
> the default {{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16, 
> and with a variable-length key or value, a 
> {{VariableLengthRowBasedKeyValueBatch}} whose array of key offsets is the 
> same size. The generated aggregation code also keeps the batch reachable. 
> These arrays are on the heap and not tracked by the memory manager.
> A task that runs many partitions of such an aggregate creates one map per 
> partition, also for an empty one, and holds all of these arrays until it ends.
> On master, this happens where {{GroupPartitionsExec}} groups a keyed scan's 
> splits above a partial hash aggregate. {{CombineAdjacentAggregation}} folds a 
> partial and a final aggregate across {{GroupPartitionsExec}}, but not the 
> first partial aggregate of a distinct aggregation, which sits below a 
> {{PartialMerge}} one. In a local test, a DSv2 scan whose splits all report 
> the same partition key, with {{groupBy(key).agg(countDistinct(j), sum(j))}}, 
> ran every split's partial aggregate in one task. With 168 splits, the task 
> held 84 MiB of bucket arrays until it ended. With 2,048 splits, it held 1 GiB.
> SPARK-60265 (https://github.com/apache/spark/pull/59522) would make this more 
> common: its packing node coalesces a scan's partitions below a shuffle, above 
> any partial aggregate. With 2,048 splits packed into one task and a string 
> grouping key, the task held 1 GiB of buckets and 1 GiB of key offsets.
> {{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the 
> generated vectorized map, and {{VariableLengthRowBasedKeyValueBatch.close()}} 
> can drop its key offsets. Nothing reads them after a close.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to