[
https://issues.apache.org/jira/browse/SPARK-27431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-27431.
----------------------------------
Resolution: Invalid
> move HashedRelation to global UnifiedMemoryManager and enable offheap
> ---------------------------------------------------------------------
>
> Key: SPARK-27431
> URL: https://issues.apache.org/jira/browse/SPARK-27431
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Xiaoju Wu
> Priority: Major
>
> Why is HashedRelation currently managed by a newly created MemoryManager and
> disabled with offheap? Can we improve this part?
> /**
> * Create a HashedRelation from an Iterator of InternalRow.
> */
> def apply(
> input: Iterator[InternalRow],
> key: Seq[Expression],
> sizeEstimate: Int = 64,
> taskMemoryManager: TaskMemoryManager = null): HashedRelation = {
> val mm = Option(taskMemoryManager).getOrElse {
> new TaskMemoryManager(
> new UnifiedMemoryManager(
> new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
> Long.MaxValue,
> Long.MaxValue / 2,
> 1),
> 0)
> }
> if (key.length == 1 && key.head.dataType == LongType) {
> LongHashedRelation(input, key, sizeEstimate, mm)
> } else {
> UnsafeHashedRelation(input, key, sizeEstimate, mm)
> }
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
