xihuan created SPARK-54753:
------------------------------
Summary: possible memory leak in spark 4.0.1
Key: SPARK-54753
URL: https://issues.apache.org/jira/browse/SPARK-54753
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.1
Reporter: xihuan
for spark 4.0.1 local mode, the memory is not released after long time running,
while downgrade the spark version v3.5.6, there is no issue.
we got the eclipse memory analyzer to check the heap dump.
h3. Problem Suspect 1
9,109 instances of {*}org.apache.hadoop.conf.Configuration{*}, loaded by
*org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888*
occupy *972,232,520 (61.58%)* bytes.
Most of these instances are referenced from one instance of
{*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*}, loaded by {*}<system
class loader>{*}, which occupies *48 (0.00%)* bytes. The instance is referenced
by {*}org.apache.spark.sql.artifact.ArtifactManager @ 0x80a14948{*}, loaded by
{*}org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888{*}.
Thread *jdk.internal.misc.InnocuousThread @ 0x866c1710 Cleaner-1* has a local
variable or reference to *jdk.internal.ref.CleanerImpl @ 0x866c1898* which is
on the shortest path to {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef @
0x861b7638{*}. The thread *jdk.internal.misc.InnocuousThread @ 0x866c1710
Cleaner-1* keeps local variables with total size *184 (0.00%)* bytes.
Significant stack frames and local variables
* jdk.internal.ref.CleanerImpl.run()V (CleanerImpl.java:140)
** jdk.internal.ref.CleanerImpl @ 0x866c1898 retains 24 (0.00%) bytes
The stacktrace of this Thread is available. [See
stacktrace|file:///C:/Users/xihuan/AppData/Local/Temp/2/report8526993352425331840/pages/27.html].
[See stacktrace with involved local
variables|file:///C:/Users/xihuan/AppData/Local/Temp/2/report8526993352425331840/pages/28.html].
*Keywords*
* org.apache.hadoop.conf.Configuration
* org.springframework.boot.loader.launch.LaunchedClassLoader
* jdk.internal.ref.CleanerImpl$PhantomCleanableRef
* org.apache.spark.sql.artifact.ArtifactManager
* jdk.internal.ref.CleanerImpl.run()V
* CleanerImpl.java:140
h3.
h3.
h3. Problem Suspect 2
12,291 instances of {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*},
loaded by *<system class loader>* occupy *432,466,008 (27.39%)* bytes.
*Keywords*
* jdk.internal.ref.CleanerImpl$PhantomCleanableRef
Could anyone help provide any insights on the issue? Thanks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
