voonhous opened a new pull request, #19905:
URL: https://github.com/apache/hudi/pull/19905
### Describe the issue this Pull Request addresses
Part of the CI improvement epic #19524 (#19529). Stacked on #19892; the
commits before "build: declare the Maven Build Cache Extension" are that PR's
and drop out on rebase.
Every Spark-lane job starts by compiling 10 to 23 modules from scratch, 6 to
8 minutes on a 4-vCPU runner, before it runs a test. Thirteen jobs do this per
run, and most PRs touch only a few of those modules. This PR keeps the compiled
modules from the last push to the base branch and restores the ones a PR did
not change.
### Summary and Changelog
- **`.mvn/extensions.xml`, `.mvn/maven-build-cache-config.xml`**: the Maven
Build Cache Extension 1.2.3, inert unless `-Dmaven.build.cache.enabled=true` is
passed. A module whose sources, resources, pom and upstream modules match a
cached build has its jars, class directories and the Scala plugin's compile
markers unpacked instead of compiled. Tests are never cached: surefire,
failsafe, scalatest and the jacoco agent are listed under `runAlways`, as is
`install`, so the local repository is updated on a hit. Local builds are
unchanged unless a developer opts in.
- **`bot.yml`**: each host job restores `~/.m2/build-cache` before its build
step and passes the enable flag to that step only. Cache keys carry the base
branch and the lane (JDK, Scala, Spark), so a PR against master reads what the
last master push saved for its lane. One job per lane saves after a push build:
`test-spark-client-and-hadoop-common` (JDK 11, the widest of that lane's
reactors) and `test-spark-java17-ut-other`. A restored file keeps the timestamp
it was compiled at, older than the checkout, so a step touches the restored
outputs after the build; without it the test steps' incremental compilers would
rebuild every module under test. `test-common-and-other-modules` builds inside
a docker image whose Maven 3.6.3 predates the extension and stays as it was.
- **`.github/workflows/README.md`**: one line on the cache.
### Impact
Runner-minutes: a job's build step drops from 6 to 8 minutes to about 1
minute when the PR touches nothing in its reactor, and to the downstream slice
when it does. Weighted over 60 recent PRs by the first module each one touches,
that is about 40 runner-minutes per run (7%), and the same ~3 minutes off every
job's wall clock, so the longest job moves by about that much. Measured numbers
go in the table below once this PR has run twice (cold, then warm).
Cache storage: one entry per lane per push, 60 to 100 MB each, in the
repository's 10 GB Actions cache alongside the `~/.m2/repository` entries
`setup-java` already keeps.
<details>
<summary>Local verification (hudi-spark reactor, 18 modules, JDK
17)</summary>
| step | result |
|---|---|
| cold build, empty cache | 1m51s, 57 MB saved |
| every `target/` deleted, sources touched, warm build | 5 s, 18 hits; jars,
class directories and the Scala plugin's compile markers back; install ran |
| one-line change in hudi-spark-common | exactly the four downstream modules
recompiled |
| `mvn test-compile -Punit-tests` on hudi-common, hudi-spark-common,
hudi-spark after a restore, cache off | same output as today's CI test steps:
nothing to compile |
| the same without the touch step | every module under test recompiled
("changed source code") |
| module checksum under `-Pthrift-gen-source`, no `clean`, no `-T`,
`-Djacoco.skip=false`, JDK 11 instead of 17 | unchanged; only the reactor shape
(`-am` or not) changes it, and every build step uses `-am` |
| Maven 3.6.3 (the docker image) with the extension declared | one warning,
cache disabled, build unaffected |
| Azure agents | Maven 3.9.16; the flag is never passed there, so nothing
changes |
</details>
### Risk Level
low
A stale hit is the risk: a module restored from the cache when its inputs
did change. The extension hashes every file under `src/`, the pom and the
upstream modules' hashes, and the JDK-17 and JDK-11 lanes keep separate keys.
Tests never come from the cache. Removing the enable flag from `bot.yml`
reverts to today's builds without touching the extension.
### Documentation Update
`.github/workflows/README.md`
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
