rdtr opened a new pull request, #13575:
URL: https://github.com/apache/gluten/pull/13575
## What changes are proposed in this pull request?
Part of #13573, its second checkbox. Spark 3.4.4, tagged on 2024-10-21, was
the last 3.4 release, and #13524 proposed dropping 3.4 the way #12902 dropped
3.3. This removes the Spark 3.4 build surface. #13574 moved the one
Gluten-authored case that only `gluten-ut/spark34` had, so it should land first.
Four directories go, 352 files and about 48k lines: `shims/spark34`,
`gluten-ut/spark34`, `gluten-delta/src-delta24` (Delta Lake 2.4, selected only
by the 3.4 profile's `delta.binary.version=24`) and
`backends-bolt/src-iceberg-spark34` (Bolt's Iceberg tests, selected by
`iceberg-test` through `spark.plain.version=34`). The `spark-3.4` profile is
gone from the root pom, `shims/pom.xml`, `gluten-ut/pom.xml` and
`tools/gluten-it/pom.xml`. The root pom's `requireActiveProfile` list now names
`spark-3.5,spark-4.0,spark-4.1,spark-4.2`, so `-Pspark-3.4` fails with `Missing
spark version profile: -Pspark-<version>`. `package/pom.xml` drops its
`*spark3.4*` clean exclude. `LICENSE` has no entries under the removed paths.
CI in `velox_backend_x86.yml`:
- `spark-test-spark34` and `spark-test-spark34-slow` are deleted.
`spark-test-spark35` runs the same profiles (`-Piceberg -Pdelta -Phudi
-Ppaimon`). The 3.4 job also passed `-Piceberg-test`, but for Velox that adds
no tests: `backends-velox` has no `src-iceberg-spark34`, and no test there uses
the `iceberg-open-api` test jars the profile adds.
- `tpc-test-ubuntu` and `tpc-test-centos8` drop 3.4 from their matrices and
excludes. 3.5 keeps the JDK 8 combination 3.4 had.
- The `shims34` change-detection flag and its path filter go.
- `build-cudf-centos-9` built its jar only with `-Pspark-3.4`, so it moves
to `-Pspark-3.5` instead of being deleted. The same goes for the cuDF build in
`velox_backend_cache.yml` and for
`dev/docker/cudf/Dockerfile.centos-9-jdk17-cuda12.9-cudf`
(`--spark_version=3.5`).
- `velox_nightly.yml` drops the 3.4 line from five bundle jobs.
`install-spark-resources.sh` no longer installs Spark 3.4.4, and
`install-spark-deps.sh` drops 3.4 from its hadoop-aws case.
Scripts: `builddeps-veloxbe.sh` (`SUPPORTED_SPARK_VERSIONS`, which
`buildbundle-veloxbe.sh` loops over for `ALL`), `bloop-test.sh`,
`run-scala-test.sh`, `format-scala-code.sh`, `release/build-release.sh`,
`release/package-release.sh`, `docker/install-ci-tools.sh` (the PySpark 3.4.4
virtualenv), the 3.4 step in `Dockerfile.ubuntu22-m2-cache`, and
`install-spark-resources.sh 3.4` in the CentOS 8 and 9 dynamic-build
Dockerfiles. Bolt's `jar_spark34` Makefile target goes too. The docs drop 3.4
from the supported versions, the build guide and the Delta Lake page. The Delta
Lake page loses the 3.4 / Delta Lake 2.4 row and the 3.4-only statuses. The
"since" column of the configuration docs and the 1.7.0 examples in
`HowToRelease.md` stay, since they describe past releases.
Things reviewers should know:
- The Docker images rebuild after this lands (`docker_image.yml` watches
`install-spark-resources.sh` and `install-spark-deps.sh`), and
`/opt/shims/spark34/spark_home` and `/opt/pyspark/3.4.4` disappear from them.
Nothing left in the repo refers to them.
- Bolt has its own 3.4 pieces (its `BoltTestSettings` for 3.4,
`src-iceberg-spark34`, `jar_spark34`), removed here with the rest. cc
@taiyang-li @Weixin-Xu in case Bolt still needs 3.4.
- ClickHouse has no 3.4 profile or packaging since #12902. Only its 3.4 test
settings go, with the module.
- The Spark 4.3 PRs #13519 and #13520 edit some of the same lines
(`requireActiveProfile`, `format-scala-code.sh`, `bloop-test.sh`,
`run-scala-test.sh`, the change-detection flags, `install-spark-resources.sh`).
Whichever lands second needs a small rebase.
Left for follow-ups under #13573: the shim API cleanup (like #12954) and the
leftover version checks (like #12981). Those include `OffloadDeltaScan`'s Spark
3.4 deletion vector fallback, `startsWith("3.4")` checks in
`VeloxHashJoinSuite`, `VeloxMetricsSuite` and their Bolt copies, two
`testWithMinSparkVersion(..., "3.4")` gates, and the `"3.5"` gates that become
constant (like #12992). `Makefile`'s `jar_spark33` target was already dead
after #12902 and is left alone here.
## How was this patch tested?
| check | result |
|-|-|
| `mvn validate -Pspark-3.4` | fails at `enforce-spark-profile` with
`Missing spark version profile: -Pspark-<version>`, as intended |
| `test-compile` with `-Pbackends-velox -Pspark-ut -Pjava-17` on 3.5
(`-Piceberg -Pdelta -Phudi -Ppaimon`), 4.0 (`-Pscala-2.13 -Piceberg -Pdelta
-Ppaimon`), 4.1 and 4.2 (`-Pscala-2.13 -Pdelta`), scalastyle and
`spotless:check` on | all pass |
| `dev/format-scala-code.sh check` (its profile list with `spark-3.4`
removed) | passes |
| `.github/workflows/util/check.py header branch` | passes |
| YAML parse of the four changed workflows, `bash -n` on every changed
script | passes |
I have no Docker on this machine, so the Dockerfile changes rest on reading.
Unit tests are what CI settles.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]