This is an automated email from the ASF dual-hosted git repository.
philo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 91162e1079 [DOC] Update documentation for Spark version support and
TLP graduation (#11851)
91162e1079 is described below
commit 91162e107981afad9e5cd25a24eb6be25166482a
Author: Ankita Victor <[email protected]>
AuthorDate: Mon Mar 30 21:21:40 2026 +0530
[DOC] Update documentation for Spark version support and TLP graduation
(#11851)
---
README.md | 4 ++--
docs/developers/HowTo.md | 4 ++--
docs/developers/MicroBenchmarks.md | 4 ++--
docs/developers/NewToGluten.md | 2 +-
docs/get-started/Velox.md | 10 ++++++----
docs/get-started/build-guide.md | 6 +++++-
docs/get-started/getting-started.md | 2 +-
docs/index.md | 6 +++---
8 files changed, 22 insertions(+), 16 deletions(-)
diff --git a/README.md b/README.md
index 3071db9c48..ea54568d33 100644
--- a/README.md
+++ b/README.md
@@ -56,7 +56,7 @@ Gluten's key components:
* **Columnar Shuffle**: Handles shuffling of Gluten's columnar data. The
shuffle service of Spark core is reused, while a columnar exchange operator is
implemented to support Gluten's columnar data format.
* **Fallback Mechanism**: Provides fallback to vanilla Spark for unsupported
operators. Gluten's ColumnarToRow (C2R) and RowToColumnar (R2C) convert data
between Gluten's columnar format and Spark's internal row format to support
fallback transitions.
* **Metrics**: Collected from Gluten native engine to help monitor execution,
identify bugs, and diagnose performance bottlenecks. The metrics are displayed
in Spark UI.
-* **Shim Layer**: Ensures compatibility with multiple Spark versions. Gluten
supports the latest 3–4 Spark releases during its development cycle, and
currently supports Spark 3.2, 3.3, 3.4, and 3.5.
+* **Shim Layer**: Ensures compatibility with multiple Spark versions. Gluten
supports the latest 3–4 Spark releases during its development cycle, and
currently supports Spark 3.3, 3.4, 3.5, 4.0, and 4.1.
## 3. User Guide
@@ -113,7 +113,7 @@ Welcome to contribute to the Gluten project! See
[CONTRIBUTING.md](CONTRIBUTING.
## 6. Community
-Gluten successfully became an Apache Incubator project in March 2024. Here are
several ways to connect with the community.
+Gluten successfully became an Apache Incubator project in March 2024 and
graduated as an Apache Top-Level Project in March 2026. Here are several ways
to connect with the community.
### GitHub
diff --git a/docs/developers/HowTo.md b/docs/developers/HowTo.md
index 256473a65c..2ac2d9f44c 100644
--- a/docs/developers/HowTo.md
+++ b/docs/developers/HowTo.md
@@ -53,7 +53,7 @@ ${GLUTEN_HOME}/dev/builddeps-veloxbe.sh --build_tests=ON
--build_benchmarks=ON -
```
cd ${GLUTEN_HOME}
-mvn test -Pspark-3.2 -Pbackends-velox -pl backends-velox \
+mvn test -Pspark-3.5 -Pbackends-velox -pl backends-velox \
-am -DtagsToInclude="org.apache.gluten.tags.GenerateExample" \
-Dtest=none -DfailIfNoTests=false \
-Dexec.skip
@@ -61,7 +61,7 @@ mvn test -Pspark-3.2 -Pbackends-velox -pl backends-velox \
- After the above operations, the example files are generated under
`${GLUTEN_HOME}/backends-velox`
- You can check it by the command `tree
${GLUTEN_HOME}/backends-velox/generated-native-benchmark/`
-- You may replace `-Pspark-3.2` with `-Pspark-3.3` if your spark's version is
3.3
+- You may replace `-Pspark-3.5` with `-Pspark-3.3` or `-Pspark-3.4` for
earlier Spark versions
```shell
$ tree ${GLUTEN_HOME}/backends-velox/generated-native-benchmark/
diff --git a/docs/developers/MicroBenchmarks.md
b/docs/developers/MicroBenchmarks.md
index 2121d2a1de..ce0996a6cc 100644
--- a/docs/developers/MicroBenchmarks.md
+++ b/docs/developers/MicroBenchmarks.md
@@ -36,8 +36,8 @@ generate example input files:
cd /path/to/gluten/
./dev/buildbundle-veloxbe.sh --build_tests=ON --build_benchmarks=ON
-# Run test to generate input data files. If you are using spark 3.3, replace
-Pspark-3.2 with -Pspark-3.3.
-mvn test -Pspark-3.2 -Pbackends-velox -pl backends-velox -am \
+# Run test to generate input data files. If you are using spark 3.3, replace
-Pspark-3.5 with -Pspark-3.3.
+mvn test -Pspark-3.5 -Pbackends-velox -pl backends-velox -am \
-DtagsToInclude="org.apache.gluten.tags.GenerateExample" -Dtest=none
-DfailIfNoTests=false -Dexec.skip
```
diff --git a/docs/developers/NewToGluten.md b/docs/developers/NewToGluten.md
index c957785687..6c14363b2c 100644
--- a/docs/developers/NewToGluten.md
+++ b/docs/developers/NewToGluten.md
@@ -13,7 +13,7 @@ Gluten supports Ubuntu 20.04/22.04, CentOS 7/8, and MacOS.
### JDK
-Gluten supports JDK 8 for Spark 3.2, 3.3, 3.4, and 3.5. For Spark 3.3 and
later versions, Gluten
+Gluten supports JDK 8 for Spark 3.3, 3.4, and 3.5. For Spark 3.3 and later
versions, Gluten
also supports JDK 11 and 17.
Note: Starting with Spark 4.0, the minimum required JDK version is 17.
diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md
index 27ff7d7456..593a4ef880 100644
--- a/docs/get-started/Velox.md
+++ b/docs/get-started/Velox.md
@@ -92,14 +92,16 @@ Currently, Gluten is using a [IBM
Velox](https://github.com/IBM/velox) which is
## compile Gluten java module and create package jar
cd /path/to/gluten
-# For spark3.2.x
-mvn clean package -Pbackends-velox -Pspark-3.2 -DskipTests
# For spark3.3.x
mvn clean package -Pbackends-velox -Pspark-3.3 -DskipTests
# For spark3.4.x
mvn clean package -Pbackends-velox -Pspark-3.4 -DskipTests
-# For spark3.5.x
+# For spark3.5.x (default)
mvn clean package -Pbackends-velox -Pspark-3.5 -DskipTests
+# For spark4.0.x (requires JDK 17+ and Scala 2.13)
+mvn clean package -Pbackends-velox -Pspark-4.0 -Pjava-17 -Pscala-2.13
-DskipTests
+# For spark4.1.x (requires JDK 17+ and Scala 2.13)
+mvn clean package -Pbackends-velox -Pspark-4.1 -Pjava-17 -Pscala-2.13
-DskipTests
```
Notes： Building Velox may fail caused by OOM. You can prevent this failure by
adjusting `NUM_THREADS` (e.g., `export NUM_THREADS=4`) before building
Gluten/Velox. The recommended minimal memory size is 64G.
@@ -405,7 +407,7 @@ Once built successfully, hudi features will be included in
gluten-velox-bundle-X
# Coverage
-Spark3.3 has 387 functions in total. ~240 are commonly used. To get the
support status of all Spark built-in functions, please refer to [Velox
Backend's Supported Operators &
Functions](../velox-backend-support-progress.md).
+Spark3.5 has 400+ functions in total. ~240 are commonly used. To get the
support status of all Spark built-in functions, please refer to [Velox
Backend's Supported Operators &
Functions](../velox-backend-support-progress.md).
> Velox doesn't support [ANSI
> mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html)),
> so as Gluten. Once ANSI mode is enabled in Spark config, Gluten will
> fallback to Vanilla Spark.
diff --git a/docs/get-started/build-guide.md b/docs/get-started/build-guide.md
index 99bf656074..cdfad12f67 100644
--- a/docs/get-started/build-guide.md
+++ b/docs/get-started/build-guide.md
@@ -28,7 +28,7 @@ Please set them via `--`, e.g. `--build_type=Release`.
| build_velox_tests | Build Velox tests.
| OFF |
| build_velox_benchmarks | Build Velox benchmarks (velox_tests and connectors
will be disabled if ON) | OFF |
| build_arrow | Build arrow java/cpp and install the libs in local.
Can turn it OFF after first build. | ON |
-| spark_version | Build for specified version of Spark(3.3, 3.4, 3.5,
ALL). `ALL` means build for all versions. | ALL |
+| spark_version | Build for specified version of Spark(3.3, 3.4, 3.5,
4.0, 4.1, ALL). `ALL` means build for all versions. | ALL |
### Velox build parameters for build-velox.sh
Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.
@@ -61,6 +61,8 @@ The below parameters can be set via `-P` for mvn.
| spark-3.3 | Build Gluten for Spark 3.3. | disabled
|
| spark-3.4 | Build Gluten for Spark 3.4. | disabled |
| spark-3.5 | Build Gluten for Spark 3.5. | enabled |
+| spark-4.0 | Build Gluten for Spark 4.0. Requires JDK 17+ and Scala
2.13. | disabled |
+| spark-4.1 | Build Gluten for Spark 4.1. Requires JDK 17+ and Scala
2.13. | disabled |
## Gluten Jar for Deployment
The gluten jar built out is under `GLUTEN_SRC/package/target/`.
@@ -71,3 +73,5 @@ It's name pattern is
`gluten-<backend_type>-bundle-spark<spark.bundle.version>_<
| 3.3.1 | 3.3 | 2.12 |
| 3.4.4 | 3.4 | 2.12 |
| 3.5.5 | 3.5 | 2.12 |
+| 4.0.1 | 4.0 | 2.13 |
+| 4.1.1 | 4.1 | 2.13 |
diff --git a/docs/get-started/getting-started.md
b/docs/get-started/getting-started.md
index ea143e9300..34ad08a78f 100644
--- a/docs/get-started/getting-started.md
+++ b/docs/get-started/getting-started.md
@@ -24,7 +24,7 @@ Gluten supports two native backends:
- **OS**: Ubuntu 20.04/22.04 or CentOS 7/8 (other Linux distros may work with
static build but are not officially tested)
- **JDK**: OpenJDK 8 or 17 (Spark 4.0 requires JDK 17+)
-- **Spark**: 3.2.2, 3.3.1, 3.4.4, 3.5.5, or 4.0.0-preview
+- **Spark**: 3.3.1, 3.4.4, 3.5.5, 4.0.1, or 4.1.1
- **Scala**: 2.12 (Spark 4.0 requires Scala 2.13)
### 2. Build
diff --git a/docs/index.md b/docs/index.md
index 76b76ef6b7..e5b769d1a4 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -3,10 +3,10 @@ layout: default
title: Home
nav_order: 1
permalink: /
-description: This site serves as a collection of documentation about the
Gluten, a plugin to Double SparkSQL's Performance
+description: This site serves as a collection of documentation about Gluten, a
middle layer for offloading JVM-based SQL engines' execution to native engines
---
# Overview
-Gluten: Plugin to Double SparkSQL's Performance.
+Gluten: A Middle Layer for Offloading JVM-based SQL Engines' Execution to
Native Engines.
# 1 Introduction
@@ -61,4 +61,4 @@ There are several key components in Gluten:
* **Columnar Shuffle**: shuffles Gluten columnar data. The shuffle service
still reuses the one in Spark core. A kind of columnar exchange operator is
implemented to support Gluten columnar data format.
* **Fallback Mechanism**: supports falling back to Vanilla spark for
unsupported operators. Gluten ColumnarToRow (C2R) and RowToColumnar (R2C) will
convert Gluten columnar data and Spark's internal row data if needed. Both C2R
and R2C are implemented in native code as well
* **Metrics**: collected from Gluten native engine to help identify bugs,
performance bottlenecks, etc. The metrics are displayed in Spark UI.
-* **Shim Layer**: supports multiple Spark versions. We plan to only support
Spark's latest 2 or 3 releases. Currently, Spark-3.2, Spark-3.3 & Spark-3.4
(experimental) are supported.
+* **Shim Layer**: supports multiple Spark versions. We plan to only support
Spark's latest 3-4 releases. Currently, Spark 3.3, 3.4, 3.5, 4.0, and 4.1 are
supported.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
