abhinav-phi opened a new pull request, #19986:
URL: https://github.com/apache/hudi/pull/19986
### Describe the issue this Pull Request addresses
This implements the caching / pre-seeding follow-up that #19883 invites
after the dlcdn fast path from #19884 was merged: every `integration-tests` run
still moves the 382 MB Spark tarball over the network. While implementing it, I
found that a plain `actions/cache` is useless in this repository because the
`delete-all-caches` job in `scheduled_workflow.yml` deletes every Actions cache
every five minutes — so this PR also adds narrow exceptions to that purge for
the Spark archive and Trino m2 keys.
### Summary and Changelog
- Add a `Restore Spark archive` step (`actions/cache/restore@v4`) to the
`integration-tests` job in `bot.yml`. It restores the tarball to
`~/spark-archives/${{ matrix.sparkArchive }}` under the exact key
`spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}`. There are
deliberately no `restore-keys` prefixes, because a partially matching restore
(a different Spark or Hadoop flavor) would be worse than downloading the
correct archive. The purge exception for this key family lives in
`scheduled_workflow.yml` — a pointer comment sits next to the key so the two
stay in sync across key bumps.
- Split the old monolithic `IT` step in two: a `Fetch Spark archive` step
runs the existing dlcdn → archive fallback chain from #19884 unchanged on a
restore miss, and a `Verify and extract Spark archive` step checks Apache's
published `.sha512` on both the restored and the freshly downloaded paths, then
extracts. Cold-run download behavior is identical to master today.
- Persist the tarball with a separate `actions/cache/save@v4` step placed
immediately after a successful `tar -xf` and before `mvn verify`, gated on
push-to-`master` plus a restore miss. The extract itself — not Hudi test health
— is the integrity gate on the entry, and seeding no longer depends on a green
master IT. PR runs and failed extracts never write to the cache, so the 10 GB
repo quota only ever holds one ~382 MB entry per Spark archive generation, with
stale generations aging out through GitHub's normal LRU eviction.
- Exclude version-independent prefixes from the five-minute purge in
`scheduled_workflow.yml`: `spark-archive-` and `trino-m2-v2-`. Pinning a
literal version there would silently re-arm the purge on the next key bump. The
Trino exception fixes the same dead-cache problem for the `trino-m2-v2-` entry
in `hudi_trino_ci.yml`, which the purge was likewise deleting minutes after
creation.
### Impact
No public API or user-facing behavior change; this is CI-only. Warm runs
skip the 382 MB egress entirely — measured on six master runs since dlcdn
landed, the download itself is only 1–18s (median ~11s) against a ~36 minute
job with a ~4.5s extrapolated restore, so the honest saving is roughly 6–14s
per warm run plus removing the stale-pin tail risk: as Spark drops the pinned
version from the fastly CDN, CI falls back to the slower archive origin. Cold
runs keep the existing download path. No Spark version, profile, or `mvn`
invocation is changed. The only lasting cost is at most one ~382 MB cache entry
per archive generation against the 10 GB repo quota. No measured hosted-cache
speedup is claimed yet; that requires a real run (tracked separately).
### Risk Level
Low. The cache touches CI plumbing only: a cache miss, a failed save, or a
failed checksum all fall back to — or fail loudly before — the exact download
path merged in #19884. Seeding is gated on a successful `tar -xf` plus
`sha512sum -c`, so a corrupt entry can neither be written nor consumed
silently. The two judgment calls — keeping the Spark archive and Trino m2
caches out of the scheduled purge — are flagged here and on the issue for
maintainer sign-off, and can be dropped from this PR if preferred.
### Documentation Update
None.
### Validation
Local plus a hosted validation run on a temporary fork branch (see the
tracking issue linked below); nothing here has run on the apache/hudi cache
service yet.
- Behavior tests executed against the exact shell of the modified steps with
mocked `curl`/`sha512sum`/`mvn`: exact hit (zero download calls), miss (single
dlcdn call), dlcdn failure with archive success, both sources failing (non-zero
exit, nothing cached), checksum mismatch on either path (non-zero exit before
extraction), save skipped on non-master ref, and purge-filter cases (mixed keys
delete only unprotected keys; Spark-only and Trino-only lists delete nothing;
empty list deletes nothing).
- Both workflow files parse as valid YAML; `git diff --check` is clean.
- Hosted validation (temporary fork branch `validate-spark-cache`, tracked
in #20011): restore-miss → download → checksum → extract → save (key
`spark-archive-v1-Linux-spark-3.5.9`, 401321029 bytes) → full IT, run
35523562633 job 106111723466. Two caveats so this is not overclaimed: the
overall run reports cancelled on the unrelated test-utilities job, and only the
miss path was exercised, a warm cache hit has not been observed yet. The fork
cache entry has since been evicted, so the save gate evidence lives in the run
log rather than in a live cache listing. The temporary branch (widened save
gate, null-before-SHA bypass, one comment-only Java touch) was deleted after
the run; none of that scaffolding is in this PR.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable (CI change; validated with the
local behavior tests described above)
<sub>AI assistance was used for implementation, local validation, and this
description.</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]