dongjoon-hyun opened a new pull request, #543:
URL: https://github.com/apache/spark-connect-swift/pull/543
### What changes were proposed in this pull request?
This PR adds a `setup-spark` composite action that caches the downloaded
Apache Spark distribution tarball, and switches all seven macOS integration
test jobs to use it.
- New `.github/actions/setup-spark/action.yml`:
- `actions/cache@v4` on `~/spark-dist/spark-<version>-bin-hadoop3.tgz`,
keyed by version and SHA-512, so a release candidate and its final release
never share a cache entry.
- Downloads from the ASF mirror redirector only on a cache miss. A `url`
input overrides it for release candidates.
- Verifies SHA-512 before extracting into `/tmp/spark`.
- `.github/workflows/build_and_test.yml`: the duplicated `curl`/`tar`/`mv`
preamble in seven jobs is replaced by a single `uses:
./.github/actions/setup-spark` step.
- SHA-512 verification is now applied to all versions. The values for 4.0.4
and 4.1.3 were taken from the official ASF `.sha512` files:
-
[spark-4.0.4-bin-hadoop3.tgz.sha512](https://downloads.apache.org/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz.sha512)
-
[spark-4.1.3-bin-hadoop3.tgz.sha512](https://downloads.apache.org/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz.sha512)
- `tar xvfz` is changed to `tar xfz` to remove thousands of lines of file
listings from the CI logs.
### Why are the changes needed?
Seven macOS jobs each download a Spark distribution from an ASF mirror on
every workflow run, roughly 2.8GB of mirror traffic per run:
| Version | Jobs |
| ------- | ---- |
| 4.0.4 | `integration-test-mac`, `integration-test-mac-spark4-iceberg` |
| 4.1.3 | `integration-test-mac-spark41`, `integration-test-token`,
`integration-test-mac-spark41-iceberg` |
| 4.2.0 | `integration-test-mac-spark42` |
| 4.3.0-rc1 | `integration-test-mac-spark43` |
1. macOS runners are billed at a 10x multiplier, so mirror download time is
the most expensive kind of CI time in this repository.
2. `closer.lua` can redirect to a slow mirror, which puts these jobs at risk
of hitting the 20-minute timeout for reasons unrelated to the change under test.
3. It reduces the load this repository puts on ASF mirrors.
4. Only 4.2.0 and 4.3.0 verified their download. Every version is verified
now.
5. The download and extraction logic was duplicated seven times, so adding a
version or fixing the logic meant editing seven places.
### Does this PR introduce _any_ user-facing change?
No. This is a CI-only change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
