FelixYBW opened a new pull request, #13423:
URL: https://github.com/apache/gluten/pull/13423
## What changes are proposed in this pull request?
Follow-up to #13422, part of #12288. **Draft until #13422 is merged and
`docker_image.yml` has published the images.** Until then this PR's own CI will
fail, because the runtime images and PySpark venvs don't exist yet.
This removes the runtime package installs from the PR-triggered jobs:
- **Spark UTs** (`spark-test-spark34/35/35-scala213/40/41/42`, enhanced,
ANSI):
- A `Select PySpark <version>` step adds `/opt/pyspark/<version>/bin` to
`GITHUB_PATH`. It replaces installing python39/python3.11 plus `pip install
pyspark pandas pyarrow`.
- The per-step `yum install -y java-17-openjdk-devel` is dropped; JDK 17
is baked into every centos image.
- **TPC jobs:**
- `tpc-test-ubuntu` runs on `apache/gluten:ubuntu-22.04-runtime`.
- `tpc-test-centos8` and the Celeborn job run on
`apache/gluten:centos-8-runtime`.
- The ARM `tpc-test-centos8` runs on `apache/gluten:centos-9-runtime`.
- The tzdata, mirror-list and JDK install steps are removed. The jobs keep
selecting a JDK through `JAVA_HOME`.
- Hadoop and MinIO now come from the image via `install-spark-deps.sh`.
- **Runner-hosted jobs** (OOM, random-kill, Kafka Spark 3.5) use the
runner's preinstalled `JAVA_HOME_8_X64` / `JAVA_HOME_17_X64` instead of
`apt-get install openjdk-*`.
- **Native build, `cpp-test-udf-test`, Delta and `velox_native_build.yml`:**
the gh/jq/bison/python39, `clang-tools-extra` and `regex` installs are removed.
After this, the only remaining runtime installs in PR jobs are the cuDF CUDA
toolkit swap and `pip install requests` in the runner-hosted ANSI analysis jobs.
This conflicts with #13421 in `velox_kafka.yml` and
`velox_backend_enhanced.yml`. I'll rebase whichever lands second.
## How was this patch tested?
- `actionlint` on all changed workflows: no new findings. The 6 findings in
`velox_backend_ansi.yml` already exist on `main`.
- An install-step audit over the changed workflows finds no
`yum`/`dnf`/`apt-get`/`pip`/gh/jq installs left, apart from the two noted above.
- The image contents were smoke-tested in #13422. The end-to-end check is
this PR's CI once the images are published.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5-5
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
