[
https://issues.apache.org/jira/browse/SPARK-54753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xihuan updated SPARK-54753:
---------------------------
Summary: memory leak in Apache Spark 4.0.1 (was: possible memory leak in
spark 4.0.1)
> memory leak in Apache Spark 4.0.1
> ---------------------------------
>
> Key: SPARK-54753
> URL: https://issues.apache.org/jira/browse/SPARK-54753
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.0.1
> Reporter: xihuan
> Priority: Major
>
> For Apache Spark *4.0.1* local mode, the memory is not released after long
> time running, while downgrade the spark version {*}v3.5.6{*}, there is no
> issue.
> We got the eclipse memory analyzer to check the heap dump.
> h3. Problem Suspect 1
> 9,109 instances of {*}org.apache.hadoop.conf.Configuration{*}, loaded by
> *org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888*
> occupy *972,232,520 (61.58%)* bytes.
> Most of these instances are referenced from one instance of
> {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*}, loaded by {*}<system
> class loader>{*}, which occupies *48 (0.00%)* bytes. The instance is
> referenced by {*}org.apache.spark.sql.artifact.ArtifactManager @
> 0x80a14948{*}, loaded by
> {*}org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888{*}.
> Thread *jdk.internal.misc.InnocuousThread @ 0x866c1710 Cleaner-1* has a local
> variable or reference to *jdk.internal.ref.CleanerImpl @ 0x866c1898* which is
> on the shortest path to {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef @
> 0x861b7638{*}. The thread *jdk.internal.misc.InnocuousThread @ 0x866c1710
> Cleaner-1* keeps local variables with total size *184 (0.00%)* bytes.
> Significant stack frames and local variables
> * jdk.internal.ref.CleanerImpl.run()V (CleanerImpl.java:140)
> ** jdk.internal.ref.CleanerImpl @ 0x866c1898 retains 24 (0.00%) bytes
> The stacktrace of this Thread is available.
> *Keywords*
> * org.apache.hadoop.conf.Configuration
> * org.springframework.boot.loader.launch.LaunchedClassLoader
> * jdk.internal.ref.CleanerImpl$PhantomCleanableRef
> * org.apache.spark.sql.artifact.ArtifactManager
> * jdk.internal.ref.CleanerImpl.run()V
> * CleanerImpl.java:140
> h3. Problem Suspect 2
> 12,291 instances of {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*},
> loaded by *<system class loader>* occupy *432,466,008 (27.39%)* bytes.
> *Keywords*
> * jdk.internal.ref.CleanerImpl$PhantomCleanableRef
>
> Could anyone help provide any insights on the issue? Thanks.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
