This is an automated email from the ASF dual-hosted git repository.
philo-he pushed a commit to branch branch-1.7
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/branch-1.7 by this push:
new 147ee1762d [1.7] Update release document and script (#12733)
147ee1762d is described below
commit 147ee1762d21fdfaa84d8c8ff3ae18a795f696a0
Author: Philo He <[email protected]>
AuthorDate: Tue Aug 11 02:25:11 2026 +0800
[1.7] Update release document and script (#12733)
---
LICENSE | 9 ----
dev/release/package-release.sh | 31 +++++++++++---
docs/developers/HowToRelease.md | 93 ++++++++++++++++++++++++++++++++++-------
3 files changed, 105 insertions(+), 28 deletions(-)
diff --git a/LICENSE b/LICENSE
index b4f98a86e3..d668b5a8b4 100644
--- a/LICENSE
+++ b/LICENSE
@@ -212,15 +212,6 @@
./backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/SparkWriteFilesCommitProtocol.scala
./cpp-ch/local-engine/Parser/aggregate_function_parser/BloomFilterAggParser.cpp
./gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala
-
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/FileSourceScanExecShim.scala
-
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala
-
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
-
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/WriteFiles.scala
-
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
-
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
-
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala.deprecated
-
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala
-
./shims/spark32/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
./shims/spark33/src/main/scala/org/apache/spark/sql/execution/FileSourceScanExecShim.scala
./shims/spark33/src/main/scala/org/apache/spark/sql/execution/datasources/WriteFiles.scala
./shims/spark33/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
diff --git a/dev/release/package-release.sh b/dev/release/package-release.sh
index 930a8f41da..e8b3da6c02 100755
--- a/dev/release/package-release.sh
+++ b/dev/release/package-release.sh
@@ -34,15 +34,28 @@ RELEASE_VERSION=${TAG_VERSION%-rc*}
CURRENT_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
GLUTEN_HOME=${CURRENT_DIR}/../../
if [ ! -d "$GLUTEN_HOME/release/" ]; then
- echo "Release directory does not exist."
+ echo "Release directory $GLUTEN_HOME/release/ does not exist."
+ exit 1
fi
+# The bundle JAR statically links third-party code, so per ASF policy the
binary
+# distribution must ship the LICENSE/NOTICE that cover it.
+LICENSE_BINARY="${GLUTEN_HOME}/LICENSE-binary"
+NOTICE_BINARY="${GLUTEN_HOME}/NOTICE-binary"
+for f in "$LICENSE_BINARY" "$NOTICE_BINARY"; do
+ if [[ ! -f "$f" ]]; then
+ echo "Missing $f, required for the binary distribution."
+ exit 1
+ fi
+done
+
pushd $GLUTEN_HOME/release/
-SPARK_VERSIONS="3.3 3.4 3.5 4.0"
+SPARK_VERSIONS="3.3 3.4 3.5 4.0 4.1"
for v in $SPARK_VERSIONS; do
- if [[ "$v" == "4.0" ]]; then
+ # Spark 4.x requires Scala 2.13; the spark-4.x Maven profiles enforce it.
+ if [[ "$v" == 4.* ]]; then
SCALA="2.13"
else
SCALA="2.12"
@@ -56,8 +69,16 @@ for v in $SPARK_VERSIONS; do
fi
echo "Packaging for Spark $v (Scala $SCALA)..."
- tar -czf apache-gluten-${RELEASE_VERSION}-bin-spark-${v}.tar.gz \
- $JAR
+ # Stage a versioned top-level directory so extracting does not scatter files
into the
+ # current directory, and so LICENSE/NOTICE travel with the JAR.
+ BIN_DIR="apache-gluten-${RELEASE_VERSION}-bin-spark-${v}"
+ rm -rf "${BIN_DIR}"
+ mkdir -p "${BIN_DIR}"
+ cp "$JAR" "${BIN_DIR}/"
+ cp "$LICENSE_BINARY" "${BIN_DIR}/LICENSE"
+ cp "$NOTICE_BINARY" "${BIN_DIR}/NOTICE"
+ tar -czf "${BIN_DIR}.tar.gz" "${BIN_DIR}"
+ rm -rf "${BIN_DIR}"
done
SRC_ZIP="${TAG}.zip"
diff --git a/docs/developers/HowToRelease.md b/docs/developers/HowToRelease.md
index 57dedb6a72..43dc4c6f9e 100644
--- a/docs/developers/HowToRelease.md
+++ b/docs/developers/HowToRelease.md
@@ -16,6 +16,52 @@ for a release of Apache Gluten project with the Velox
backend.
2. Linux
3. Docker
+## Pre-release Checks
+
+Complete these before tagging an RC. Each one corresponds to an issue
previously raised during a
+release vote, so skipping them tends to cost an extra release candidate.
+
+### Set the release version and remove leftover `-SNAPSHOT` strings
+
+```bash
+bash dev/release/bump-version.sh 1.7.0
+```
+
+`bump-version.sh` runs `versions:set` on the POMs under the repository root,
`tools/gluten-it`,
+`tools/qualification-tool` and `gluten-flink`. It does **not** touch versions
hardcoded elsewhere,
+for example `dev/info.sh` and the docs under `gluten-flink/docs/`. Check for
anything it missed:
+
+```bash
+git grep -nIE '[0-9]+\.[0-9]+\.[0-9]+-SNAPSHOT'
+```
+
+Gluten's own version must not appear as `-SNAPSHOT` in the result. References
to third-party
+snapshots, such as `velox4j` and `spark-sql-perf`, are expected to remain.
+
+A previous RC was voted `+0` because the source distribution still carried
`-SNAPSHOT` versions,
+including stale ones left over from an earlier release. Clean up all of them,
not just the current
+version.
+
+### Update `LICENSE-binary` and `NOTICE-binary`
+
+The bundle JAR statically links third-party code, and `package-release.sh`
ships `LICENSE-binary`
+and `NOTICE-binary` inside each binary tarball as `LICENSE` and `NOTICE`.
Whenever a bundled
+component is added or enabled, its copyright and license must be recorded
there.
+
+A previous RC was voted `-1` because Hudi and Paimon had been enabled in the
binaries without
+being added to `NOTICE-binary`.
+
+### Check license headers
+
+```bash
+# Requires the `regex` Python package: pip install regex
+python3 dev/check.py header main
+```
+
+Reviewers also run [Apache RAT](https://creadur.apache.org/rat/) against the
source archive. Note
+that RAT flags `.ipynb` notebooks, which cannot carry a comment header; the
convention is to place
+the license text in the notebook's first markdown cell.
+
## Steps to Create a Release
A standard release distribution can be created following the below steps.
@@ -26,8 +72,8 @@ Pull and download the build environment docker image. The
docker image is period
built and uploaded to DockerHub by scheduled GitHub Actions jobs.
```bash
-docker pull apache/gluten:vcpkg-centos-7
-docker run -it apache/gluten:vcpkg-centos-7 bash
+docker pull apache/gluten:vcpkg-centos-7-gcc13
+docker run -it apache/gluten:vcpkg-centos-7-gcc13 bash
```
### Clone the repository
@@ -76,13 +122,24 @@ Confirm that all the needed sources and binaries are
successfully created at the
`$GLUTEN_HOME/release/`.
```bash
-[root@8de83f716f0f workspace]# ls -l release/
-total 481628
--rw-r--r--. 1 root root 74396439 Oct 14 14:19
apache-gluten-1.6.0-example-src.tar.gz
--rw-r--r--. 1 root root 104790092 Oct 14 14:19
apache-gluten-1.6.0-example-bin-spark-3.2.tar.gz
--rw-r--r--. 1 root root 104767582 Oct 14 14:19
apache-gluten-1.6.0-example-bin-spark-3.3.tar.gz
--rw-r--r--. 1 root root 104625356 Oct 14 14:19
apache-gluten-1.6.0-example-bin-spark-3.4.tar.gz
--rw-r--r--. 1 root root 104595103 Oct 14 14:19
apache-gluten-1.6.0-example-bin-spark-3.5.tar.gz
+[root@8de83f716f0f workspace]# ls -1 release/*.tar.gz
+release/apache-gluten-1.6.0-example-src.tar.gz
+release/apache-gluten-1.6.0-example-bin-spark-3.3.tar.gz
+release/apache-gluten-1.6.0-example-bin-spark-3.4.tar.gz
+release/apache-gluten-1.6.0-example-bin-spark-3.5.tar.gz
+release/apache-gluten-1.6.0-example-bin-spark-4.0.tar.gz
+release/apache-gluten-1.6.0-example-bin-spark-4.1.tar.gz
+```
+
+Each binary tarball contains a versioned top-level directory holding the
bundle JAR along with
+the `LICENSE` and `NOTICE` that cover the third-party code statically linked
into it.
+
+```bash
+[root@8de83f716f0f workspace]# tar -tzf
release/apache-gluten-1.6.0-example-bin-spark-3.5.tar.gz
+apache-gluten-1.6.0-example-bin-spark-3.5/
+apache-gluten-1.6.0-example-bin-spark-3.5/LICENSE
+apache-gluten-1.6.0-example-bin-spark-3.5/NOTICE
+apache-gluten-1.6.0-example-bin-spark-3.5/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.6.0-example.jar
```
<!--- Moved from
https://github.com/apache/gluten-site/blob/main/_docs/v1.3.0/developers/HowToRelease.md
--->
@@ -144,6 +201,10 @@ $ svn ci -m "add gpg key"
$ for i in *.tar.gz; do echo $i; gpg --local-user xxxx --armor --output $i.asc
--detach-sig $i ; done
```
+Note the KEYS file is appended in the **release** SVN directory, not the `dev`
one, so that it is
+served from <https://downloads.apache.org/gluten/KEYS>. Do this **before**
starting the vote and
+reference that URL in the vote email: a previous vote had to be corrected
mid-thread because it
+pointed reviewers at a `dist.apache.org/repos/dist/dev/` KEYS link instead.
#### How to Generate checksums for the release artifacts.
@@ -160,7 +221,7 @@ $ for i in *.tar.gz; do echo $i; sha512sum $i > $i.sha512
; done
2. Create a directory for the release artifacts in the SVN repository.
`https://dist.apache.org/repos/dist/dev/gluten/{release-version}`
- release-version format: apache-gluten-#.#.#-rc#
+ release-version format: #.#.#-rc# (e.g. `1.7.0-rc0`)
3. Upload the release artifacts to the SVN repository.
```bash
@@ -171,11 +232,15 @@ $ svn commit -m "add Apache Gluten release artifacts for
{release-version}"
```
4. After the upload, please visit the link
`https://dist.apache.org/repos/dist/dev/gluten/{release-version}` to verify if
the file upload is successful or not.
- The upload release artifacts should be include
+ The uploaded artifacts should include the source archive and one binary
archive per supported
+ Spark version, each with its `.asc` signature and `.sha512` checksum:
```bash
-* apache-gluten-#.#.#-src.tar.gz
-* apache-gluten-#.#.#-src.tar.gz.asc
-* apache-gluten-#.#.#-src.tar.gz.sha512
+* apache-gluten-#.#.#-src.tar.gz{,.asc,.sha512}
+* apache-gluten-#.#.#-bin-spark-3.3.tar.gz{,.asc,.sha512}
+* apache-gluten-#.#.#-bin-spark-3.4.tar.gz{,.asc,.sha512}
+* apache-gluten-#.#.#-bin-spark-3.5.tar.gz{,.asc,.sha512}
+* apache-gluten-#.#.#-bin-spark-4.0.tar.gz{,.asc,.sha512}
+* apache-gluten-#.#.#-bin-spark-4.1.tar.gz{,.asc,.sha512}
```
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
