This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 8cbf3fa4d [CELEBORN-2409] Fix outdated Spark version descriptions in
docs
8cbf3fa4d is described below
commit 8cbf3fa4dd142ce04e7ed88d342eb64051fb90dc
Author: yew1eb <[email protected]>
AuthorDate: Thu Aug 20 22:54:24 2026 +0800
[CELEBORN-2409] Fix outdated Spark version descriptions in docs
### What changes were proposed in this pull request?
Fix outdated Spark version descriptions in docs:
- `README.md`: add Spark 4.1/4.2 to the supported Spark versions list
- `docs/README.md`, `docs/deploy.md`: "clients for Spark 2.x and Spark 3.x"
-> "clients for Spark 2.x, Spark 3.x and Spark 4.x"
- `docs/developers/sbt.md`: update the shaded client publish example from
`-Pspark-4.0` to `-Pspark-4.2`, aligning with `build/release/release.sh`
### Why are the changes needed?
Spark 4.1/4.2 support was added in CELEBORN-2239 and CELEBORN-2383, but
these doc descriptions were not updated accordingly.
### Does this PR resolve a correctness bug?
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
- [ ] Yes
### How was this patch tested?
Doc-only changes, no code changes.
Closes #3788 from yew1eb/spark-4.2-docs.
Authored-by: yew1eb <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
README.md | 3 +--
docs/README.md | 2 +-
docs/deploy.md | 2 +-
docs/developers/overview.md | 2 +-
docs/developers/sbt.md | 4 ++--
5 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/README.md b/README.md
index 0d5fcfaf6..2f14f5b66 100644
--- a/README.md
+++ b/README.md
@@ -41,7 +41,7 @@ Celeborn Worker's slot count is decided by `total usable disk
size / average shu
Celeborn worker's slot count decreases when a partition is allocated and
increments when a partition is freed.
## Build
-1. Celeborn supports Spark 3.0/3.1/3.2/3.3/3.4/3.5/4.0, Flink
1.18/1.19/1.20/2.0/2.1/2.2/2.3 and Hadoop MapReduce 3.
+1. Celeborn supports Spark 3.0/3.1/3.2/3.3/3.4/3.5/4.0/4.1/4.2, Flink
1.18/1.19/1.20/2.0/2.1/2.2/2.3 and Hadoop MapReduce 3.
2. Celeborn tested under Scala 2.11/2.12/2.13 and Java 8/11/17 environment.
Build Celeborn via `make-distribution.sh`:
@@ -436,7 +436,6 @@ Check
[SPARK-42689](https://issues.apache.org/jira/browse/SPARK-42689) and [CELE
for more details.
For Spark versions < 3.5.0, we provide a patch to enable users to use Spark
with DRA and Celeborn.
-- For Spark 2.x check [Spark2
Patch](assets/spark-patch/Celeborn_Dynamic_Allocation_spark2.patch).
- For Spark 3.0 check [Spark3.0
Patch](assets/spark-patch/Celeborn_Dynamic_Allocation_spark3_0.patch).
- For Spark 3.1 check [Spark3.1
Patch](assets/spark-patch/Celeborn_Dynamic_Allocation_spark3_1.patch).
- For Spark 3.2 check [Spark3.2
Patch](assets/spark-patch/Celeborn_Dynamic_Allocation_spark3_2.patch).
diff --git a/docs/README.md b/docs/README.md
index ae144bcc7..f4cf901c2 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -84,7 +84,7 @@ WorkerRef: null
## Start Spark with Celeborn
#### Copy Celeborn Client to Spark's jars
-Celeborn release binary contains clients for Spark 2.x and Spark 3.x, copy the
corresponding client jar into Spark's
+Celeborn release binary contains clients for Spark 3.x and Spark 4.x, copy the
corresponding client jar into Spark's
`jars/` directory:
```shell
cp
$CELEBORN_HOME/spark/celeborn-client-spark-<spark.major.version>-shaded_<scala.binary.version>-<celeborn.version>.jar
$SPARK_HOME/jars/
diff --git a/docs/deploy.md b/docs/deploy.md
index 25f1f78ce..5a1524df1 100644
--- a/docs/deploy.md
+++ b/docs/deploy.md
@@ -155,7 +155,7 @@ WorkerRef: null
```
## Deploy Spark client
-Celeborn release binary contains clients for Spark 2.x and Spark 3.x, copy the
corresponding client jar into Spark's
+Celeborn release binary contains clients for Spark 3.x and Spark 4.x, copy the
corresponding client jar into Spark's
`jars/` directory:
Copy
`$CELEBORN_HOME/spark/celeborn-client-spark-<spark.major.version>-shaded_<scala.binary.version>-<celeborn.version>.jar`
to `$SPARK_HOME/jars/`.
diff --git a/docs/developers/overview.md b/docs/developers/overview.md
index 950dbb49a..9404507d1 100644
--- a/docs/developers/overview.md
+++ b/docs/developers/overview.md
@@ -88,7 +88,7 @@ Currently, Celeborn only supports `Local Disks` and `HDFS`.
Supporting for other
Celeborn's primary components(i.e. Master, Worker, Client) are engine
irrelevant. The Client APIs are extensible
and easy to implement plugins for various engines.
-Currently, Celeborn officially supports
[Spark](https://spark.apache.org/)(both Spark 2.x, Spark 3.x and Spark 4.x),
+Currently, Celeborn officially supports
[Spark](https://spark.apache.org/)(both Spark 3.x and Spark 4.x),
[Flink](https://flink.apache.org/)(1.18/1.19/1.20/2.0/2.1/2.2/2.3),
[Gluten](https://github.com/apache/gluten) and
[Auron](https://github.com/apache/auron). Also, developers are integrating
Celeborn with other engines,
for example [MR3](https://mr3docs.datamonad.com/docs/mr3/).
diff --git a/docs/developers/sbt.md b/docs/developers/sbt.md
index 1529a5591..63506e6f2 100644
--- a/docs/developers/sbt.md
+++ b/docs/developers/sbt.md
@@ -314,9 +314,9 @@ Publish the shade client for Spark 3.5:
$ ./build/sbt -Pspark-3.5 celeborn-client-spark-3-shaded/publish
```
-Publish the shade client for Spark 4.0:
+Publish the shade client for Spark 4.2:
```shell
-$ ./build/sbt -Pspark-4.0 celeborn-client-spark-4-shaded/publish
+$ ./build/sbt -Pspark-4.2 celeborn-client-spark-4-shaded/publish
```
Publish the shade client for Flink 1.18:
