gaogaotiantian opened a new pull request, #56913:
URL: https://github.com/apache/spark/pull/56913
### What changes were proposed in this pull request?
This backports the CI build-time optimization series to `branch-4.0`,
combining the work that landed on `branch-4.1` as #56798 (precompile artifact
sharing + unified Coursier cache) and #56911 (install `zstd` in the CI
container images) into a single PR.
- **`build_and_test.yml`**: a `precompile` job builds Spark once with SBT
and publishes the compile output (`target/` dirs, zstd-compressed) as an
artifact. The `build`, `pyspark`, `sparkr`, `tpcds-1g`,
`docker-integration-tests`, and `k8s-integration-tests` jobs download and
extract it, falling back to a local SBT build if the precompile job is absent
or fails. The `run-tests`-driven jobs export `SKIP_SCALA_BUILD` to skip the
local build. The per-job Coursier caches are unified under a single `${{
runner.os }}-coursier` key, with the `precompile` job as the sole writer and
all consumers restore-only.
- **`maven_test.yml`**: a `precompile-maven` job runs `clean install` once
and publishes `target/` plus `~/.m2/.../spark` as an artifact for the matrix
entries to consume.
- **`python_macos_test.yml`**: `branch-4.0`'s analog of `branch-4.1`'s
`python_hosted_runner_test.yml`. A `precompile` job builds Spark with SBT on
macOS and publishes a macOS-distinct artifact that the matrix consumes.
- **`dev/run-tests.py`**: guard `build_apache_spark` and
`build_spark_assembly_sbt` behind `SKIP_SCALA_BUILD` so the test run reuses the
precompiled output.
- **Dockerfiles**: install `zstd` in `dev/infra/Dockerfile` and the `docs`,
`lint`, `sparkr`, `python-311`, and `pypy-310` images under
`dev/spark-test-image/`.
Adaptations for `branch-4.0`:
- `maven_test.yml`: `branch-4.0` has no SPARK-51628 assembly-wipe, so the
full `target/` (including `assembly`) is shipped and the reuse path simply
skips the local `clean install` with no assembly rebuild needed for the connect
tests. No macOS cache guards are added, since SPARK-54466 is not on
`branch-4.0` and the existing build job already caches unconditionally.
- The `zstd` Dockerfile changes cover only the images the `branch-4.0`
scheduler (`branch40_scheduler.yml`) builds: `docs`, `lint`, `sparkr`,
`python-311`, and `pypy-310` (and `dev/infra`). `python-314` does not exist on
`branch-4.0` and is not scheduled, so it is omitted.
### Why are the changes needed?
To cut redundant Scala/Maven compilation and Coursier cache duplication on
`branch-4.0` CI, matching the optimization already present on the newer
branches. Without `zstd` in the container images, `build_and_test.yml`'s `zstd
-dc compile-artifact.tar.zst` extraction fails in the container jobs, and
`actions/cache` silently falls back to `gzip` and never restores caches saved
by host jobs.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
CI on this PR. The three workflow files validate with `python3 -c "import
yaml; yaml.safe_load(...)"`, and `dev/run-tests.py` compiles cleanly.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-8)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
