[
https://issues.apache.org/jira/browse/SPARK-54089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongze Zhang updated SPARK-54089:
---------------------------------
Description:
There are a few memory consumers that only support on-heap mode. Including:
# LongToUnsafeRowMap (for long key hash join)
# ExternalSorter (for non-serializable sort-based shuffle)
It's ideal to make them support off-heap when Spark is running with off-heap
memory mode. For 3rd accelerator plugins that leverage Spark off-heap memory,
having the large allocations in vanilla Spark supporting off-heap mode will
significantly ease the memory settings for running the mixed query plan that
contains both vanilla Spark and offloaded computations.
was:
There are a few memory consumers that only support on-heap mode. Including:
# LongToUnsafeRowMap (for long key broadcast hash join)
# ExternalSorter (for non-serializable sort-based shuffle)
It's ideal to make them support off-heap when Spark is running with off-heap
memory mode. For 3rd accelerator plugins that leverage Spark off-heap memory,
having the large allocations in vanilla Spark supporting off-heap mode will
significantly ease the memory settings for running the mixed query plan that
contains both vanilla Spark and offloaded computations.
> Add off-heap mode support for on-heap-only memory consumers
> -----------------------------------------------------------
>
> Key: SPARK-54089
> URL: https://issues.apache.org/jira/browse/SPARK-54089
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.0.1
> Reporter: Hongze Zhang
> Priority: Major
>
> There are a few memory consumers that only support on-heap mode. Including:
> # LongToUnsafeRowMap (for long key hash join)
> # ExternalSorter (for non-serializable sort-based shuffle)
> It's ideal to make them support off-heap when Spark is running with off-heap
> memory mode. For 3rd accelerator plugins that leverage Spark off-heap memory,
> having the large allocations in vanilla Spark supporting off-heap mode will
> significantly ease the memory settings for running the mixed query plan that
> contains both vanilla Spark and offloaded computations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
