Hongze Zhang created SPARK-54089:
------------------------------------
Summary: Add off-heap mode support for on-heap-only memory
consumers
Key: SPARK-54089
URL: https://issues.apache.org/jira/browse/SPARK-54089
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.1
Reporter: Hongze Zhang
There are a few memory consumers that only support on-heap mode. Including:
# LongToUnsafeRowMap (for long key broadcast hash join)
# ExternalSorter (for non-serializable sort-based shuffle)
It's ideal to make them support off-heap when Spark is running with off-heap
memory mode. For 3rd accelerator plugins that leverage Spark off-heap memory,
having all the large allocations supporting off-heap mode will significantly
ease the memory settings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
