This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0ac3189c1375 [SPARK-57723][CORE] Enable `spark.rdd.compress` by
default like PySpark
0ac3189c1375 is described below
commit 0ac3189c13753728db662dcec06a7c5444573f35
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jun 26 18:27:40 2026 -0700
[SPARK-57723][CORE] Enable `spark.rdd.compress` by default like PySpark
### What changes were proposed in this pull request?
This PR aims to enable `spark.rdd.compress` by default for **non-Python**
languages because we are supporting more and more languages via `Spark
Connect`, we had better match the behavior with PySpark for the feature parity
across all languages. Currently, only `Python` language has this advantage
unfairly in the Apache Spark language eco-system.
https://github.com/apache/spark/blob/c3e9ea2299c0fb064e913b2c8463ec777ef6cdb4/python/pyspark/core/context.py#L88
- Scala/Java (main repository)
- https://github.com/apache/spark-connect-swift
- https://github.com/apache/spark-connect-rust
- https://github.com/apache/spark-connect-go
### Why are the changes needed?
Since Apache Spark 2.0, PySpark has been taking advantage of
`spark.rdd.compres` by default stably to compress serialized RDD partitions and
has been saving memory and disk at a small CPU cost. Since `PySpark` has long
defaulted this to `true`; making it the core default applies the same proven
default across all languages.
- https://github.com/apache/spark/pull/1568
For the record, Python is the first user-facing language in the Apache
Spark website and its behavior is also the same.
- https://spark.apache.org
<img width="1071" height="347" alt="Screenshot 2026-06-26 at 13 37 43"
src="https://github.com/user-attachments/assets/cc92c785-9763-44cc-9277-c1b40bf2a0c3";
/>
### Does this PR introduce _any_ user-facing change?
- For PySpark users, no.
- For non-PySpark users, yes. Serialized RDD partitions are now compressed
by default. To restore the previous behavior, set `spark.rdd.compress` to
`false`.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #56824 from dongjoon-hyun/SPARK-57723.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/internal/config/package.scala | 2 +-
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala | 1 +
docs/configuration.md | 2 +-
docs/core-migration-guide.md | 2 ++
python/pyspark/core/context.py | 1 -
5 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 47620b2406ec..79ac503e69be 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -2313,7 +2313,7 @@ package object config {
"Compression will use spark.io.compression.codec")
.version("0.6.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
private[spark] val RDD_PARALLEL_LISTING_THRESHOLD =
ConfigBuilder("spark.rdd.parallelListingThreshold")
diff --git
a/core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala
b/core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala
index 8dcc1b798181..933fd0b8c0e7 100644
--- a/core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala
@@ -1416,6 +1416,7 @@ class BlockManagerSuite extends SparkFunSuite with
Matchers with PrivateMethodTe
"caching on disk" -> StorageLevel.DISK_ONLY
).foreach { case (name, storageLevel) =>
encryptionTest(s"test putBlockDataAsStream with $name") { conf =>
+ conf.set(RDD_COMPRESS, false)
init(conf)
val ioEncryptionKey =
if (conf.get(IO_ENCRYPTION_ENABLED))
Some(CryptoStreamUtils.createKey(conf)) else None
diff --git a/docs/configuration.md b/docs/configuration.md
index 3e1077b6ab79..821e674589a8 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -2091,7 +2091,7 @@ Apart from these, the following properties are also
available, and may be useful
</tr>
<tr>
<td><code>spark.rdd.compress</code></td>
- <td>false</td>
+ <td>true</td>
<td>
Whether to compress serialized RDD partitions (e.g. for
<code>StorageLevel.MEMORY_ONLY_SER</code> in Java
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index 2d3df8a038f4..f42d34dfeffe 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -24,6 +24,8 @@ license: |
## Upgrading from Core 4.2 to 4.3
+- Since Spark 4.3, Spark compresses serialized RDD partitions by default. To
restore the legacy behavior, you can set `spark.rdd.compress` to `false`.
+
- Since Spark 4.3, Spark executor pods connect to the driver via the driver
pod IP directly instead of the driver's Kubernetes Service. To restore the
legacy behavior, you can set `spark.kubernetes.executor.useDriverPodIP` to
`false`.
- Since Spark 4.3, Spark sets `allowPrivilegeEscalation` to `false` on the
driver and executor containers' security context by default. To restore the
legacy behavior, you can set
`spark.kubernetes.securityContext.allowPrivilegeEscalation` to `true`.
diff --git a/python/pyspark/core/context.py b/python/pyspark/core/context.py
index 8bdea00dad4e..2a6d8ab68e5f 100644
--- a/python/pyspark/core/context.py
+++ b/python/pyspark/core/context.py
@@ -85,7 +85,6 @@ __all__ = ["SparkContext"]
# the default ones for Spark if they are not configured by user.
DEFAULT_CONFIGS: Dict[str, Any] = {
"spark.serializer.objectStreamReset": 100,
- "spark.rdd.compress": True,
# Disable artifact isolation in PySpark, or user-added .py file won't work
"spark.sql.artifact.isolation.enabled": "false",
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
