Hongze Zhang created SPARK-54502:
------------------------------------
Summary: Support off-heap memory mode for hash relations in
broadcast hash join
Key: SPARK-54502
URL: https://issues.apache.org/jira/browse/SPARK-54502
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.1
Reporter: Hongze Zhang
Currently Spark is hard coded to use on-heap memory mode for broadcast hash
join. It was because the broadcasted hash tables don't have a certain lifecycle
so need to be managed by GC to avoid memory leak. Though this will cause waste
of JVM heap memory when off-heap memory mode is enabled.
This will aim to improve the design and code to add off-heap support for
broadcast hash join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
