[
https://issues.apache.org/jira/browse/SPARK-26684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-26684.
-----------------------------------
Resolution: Invalid
> Add logs when allocating large memory for PooledByteBufAllocator
> ----------------------------------------------------------------
>
> Key: SPARK-26684
> URL: https://issues.apache.org/jira/browse/SPARK-26684
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.4.0
> Reporter: liupengcheng
> Priority: Trivial
>
> Currently, Spark use `PooledByteBufAllocator` to allocate memory for channel
> reading. However, the allocated heap/offheap memory size is not tracked.
> Sometimes, this make it difficult to find out the cause of OOM failures(for
> instance, direct memory oom). we have to dump the heap and use more advanced
> tools like MAT to locate the cause.
> Actually, we can add some logs for `PooledByteBufAllocator` when allocating
> large memory, which can facilitate the debugging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
