This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 00571c1ed02c [SPARK-57873][CORE] Support disabling ContextCleaner
periodic GC via non-positive interval
00571c1ed02c is described below
commit 00571c1ed02c0988217c56e86576a0f8fb8c8ea0
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Jul 1 23:00:57 2026 -0700
[SPARK-57873][CORE] Support disabling ContextCleaner periodic GC via
non-positive interval
### What changes were proposed in this pull request?
This PR proposes to skip scheduling the periodic GC task in
`ContextCleaner.start()` when `spark.cleaner.periodicGC.interval` is
non-positive, and documents that such a value disables the periodic GC.
### Why are the changes needed?
Previously, `spark.cleaner.periodicGC.interval=0s` causes `SparkContext`
creation failures because the value was passed directly to
`scheduleAtFixedRate`, which throws `IllegalArgumentException` when `period <=
0`. This change makes a non-positive value a way to disable the periodic
`System.gc()` according to the user's intention instead of naive underlying
Java error.
```
$ bin/spark-shell -c spark.cleaner.periodicGC.interval=0s
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.2.0
/_/
Using Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 25.0.3)
...
26/07/01 21:19:41 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:610)
at org.apache.spark.ContextCleaner.start(ContextCleaner.scala:133)
```
### Does this PR introduce _any_ user-facing change?
No because this is a new feature. Previously, it fails with
`IllegalArgumentException` at the start-up.
### How was this patch tested?
Pass the CIs with the newly added test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #56953 from dongjoon-hyun/SPARK-57873.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/ContextCleaner.scala | 6 ++++--
core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala | 10 ++++++++++
docs/configuration.md | 3 ++-
3 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/ContextCleaner.scala
b/core/src/main/scala/org/apache/spark/ContextCleaner.scala
index 54ea8c94daac..0b3c22a22cb4 100644
--- a/core/src/main/scala/org/apache/spark/ContextCleaner.scala
+++ b/core/src/main/scala/org/apache/spark/ContextCleaner.scala
@@ -129,8 +129,10 @@ private[spark] class ContextCleaner(
cleaningThread.setDaemon(true)
cleaningThread.setName("Spark Context Cleaner")
cleaningThread.start()
- periodicGCService.scheduleAtFixedRate(() => System.gc(),
- periodicGCInterval, periodicGCInterval, TimeUnit.SECONDS)
+ if (periodicGCInterval > 0) {
+ periodicGCService.scheduleAtFixedRate(() => System.gc(),
+ periodicGCInterval, periodicGCInterval, TimeUnit.SECONDS)
+ }
}
/**
diff --git a/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
b/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
index 813de4132ab2..bb2d7d5c4d8e 100644
--- a/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
@@ -336,6 +336,16 @@ class ContextCleanerSuite extends ContextCleanerSuiteBase {
case _ => false
}, askStorageEndpoints = true).isEmpty)
}
+
+ test("SPARK-57873: non-positive spark.cleaner.periodicGC.interval disables
periodic GC") {
+ sc.stop()
+ val conf = new SparkConf()
+ .setMaster("local[2]")
+ .setAppName("periodicGCDisabled")
+ .set(CLEANER_PERIODIC_GC_INTERVAL.key, "0s")
+ sc = new SparkContext(conf)
+ assert(sc.cleaner.isDefined)
+ }
}
diff --git a/docs/configuration.md b/docs/configuration.md
index 821e674589a8..3eeee90d0778 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -2224,7 +2224,8 @@ Apart from these, the following properties are also
available, and may be useful
<td><code>spark.cleaner.periodicGC.interval</code></td>
<td>30min</td>
<td>
- Controls how often to trigger a garbage collection.<br><br>
+ Controls how often to trigger a garbage collection. Setting this to 0 or a
negative
+ value disables the periodic garbage collection.<br><br>
This context cleaner triggers cleanups only when weak references are
garbage collected.
In long-running applications with large driver JVMs, where there is little
memory pressure
on the driver, this may happen very occasionally or not at all. Not
cleaning at all may
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
