liupengcheng created SPARK-26684:
------------------------------------
Summary: Add logs when allocating large memory for
PooledByteBufAllocator
Key: SPARK-26684
URL: https://issues.apache.org/jira/browse/SPARK-26684
Project: Spark
Issue Type: Improvement
Components: Shuffle
Affects Versions: 2.4.0
Reporter: liupengcheng
Currently, Spark use `PooledByteBufAllocator` to allocate memory for channel
reading. However, the allocated heap/offheap memory size is not tracked.
Sometimes, this make it difficult to find out the cause of OOM failures(for
instance, direct memory oom). we have to dump the heap and use more advanced
tools like MAT to locate the cause.
Actually, we can add some logs for `PooledByteBufAllocator` when allocating
large memory, which can facilitate the debugging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
