[
https://issues.apache.org/jira/browse/SPARK-29657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dzcxzl updated SPARK-29657:
---------------------------
Issue Type: Bug (was: Improvement)
> Iterator spill supporting radix sort with null prefix
> -----------------------------------------------------
>
> Key: SPARK-29657
> URL: https://issues.apache.org/jira/browse/SPARK-29657
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: dzcxzl
> Priority: Trivial
>
> In the case of radix sort, when the insertRecord part of the keyPrefix is
> null, the iterator type returned by getSortedIterator is ChainedIterator.
> Currently ChainedIterator does not support spill, causing
> UnsafeExternalSorter to take up a lot of execution memory, allocatePage
> fails, throw SparkOutOfMemoryError Unable to acquire xxx bytes of memory, got > 0
> The following is a log of an error we encountered in the production
> environment.
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: Memory
> used in task 66055
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired
> by
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@39dd866e:
> 64.0 KB
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired
> by
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@74d17927:
> 4.6 GB
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired
> by
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@31478f9c:
> 61.0 MB
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: 0 bytes
> of memory were used by task 66055 but are not associated with specific
> consumers
> [Executor task launch worker for task 66055] INFO TaskMemoryManager:
> 4962998749 bytes of memory are used for execution and 2218326 bytes of memory
> are used for storage
> [Executor task launch worker for task 66055] ERROR Executor: Exception in
> task 42.3 in stage 29.0 (TID 66055)
> SparkOutOfMemoryError: Unable to acquire 3436 bytes of memory, got 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
