zozo123 opened a new pull request, #74173: URL: https://github.com/apache/airflow/pull/74173
Every job that prepares the CI image restores the stash and runs `docker image load` on it. Docker then unpacks and checksums every layer of the image again, which takes about 100–140 s in each of the dozens of jobs in a PR run. This adds a second way to hand the image over. The build job also saves a copy of the daemon's image store (`image/` and `overlay2/` under `/var/lib/docker`), holding only the CI image, with a small `.meta` file naming the daemon that wrote it. A job that prepares the CI image restores that copy into its stopped, empty daemon with one extraction, and falls back to the existing stash and `docker image load` whenever the snapshot does not fit: - there is no snapshot for the key; - the job's daemon has a different version, storage driver, architecture or data root; - the daemon already holds images that the restore would drop; - the restored image does not run (the store is emptied again first). The image stash itself is unchanged, so release branches and the other workflows that read it keep working. Snapshots use a new stash key, `ci-image-snapshot-v1-*`, and are only taken for the per-branch image, not for `image-stash-ref` builds. ### Measured on GitHub `ubuntu-22.04` runners These runs used the image as it was before #74076 removed ~800 MB from it. On the current image the saving per job should be somewhat smaller (estimated 65–95 s); this PR's own CI run shows the real numbers, and I will update them here. Restoring the published `ghcr.io/apache/airflow/main/ci/python3.10:latest`, with Docker on `/mnt` as in CI: | | Round 1 | Round 2 | |---|---|---| | `docker image load` of the stash (2.47 GB, zstd) | 113.8 s | 112.5 s | | Snapshot restore (2.65 GB): extract + daemon start | 32.5 s | 32.9 s | Creating the snapshot took 25.5 s. End to end: two rounds on a fork with the changed action and script ([run 1](https://github.com/zozo123/airflow/actions/runs/36990568450), [run 2](https://github.com/zozo123/airflow/actions/runs/36992717345)). One consumer took the snapshot. The other, asking for a stash with no snapshot behind it, took the fallback. | Sub-step of *Prepare breeze & CI image* | Run 1 | Run 2 | |---|---|---| | `breeze ci-image load` (fallback, today's path) | 142.6 s | 111.5 s | | Snapshot restore: extract, daemon start, `docker run` check | 36.6 s | 33.5 s | Both consumers end with the same image ID. Both then ran `airflow-core/tests/unit/utils/test_helpers.py` and `airflow-core/tests/unit/models/test_dagrun.py` through Breeze: 275 passed, 1 xfailed, on each path. Download times vary with the runner, and the snapshot is 7% larger than the stash. The cost is in the build job, which keeps writing the image stash and now also writes the snapshot. Taking the snapshot took 25.5 s in the benchmark and 99/84 s in the two end-to-end runs, and stashing it 23/27 s. So the build job gets about 1–2 minutes longer, and each job that loads the image gets 75–106 s shorter: a large saving in runner time, while the wall clock of a run changes by under a minute either way. The build job takes the snapshot right after the build, while the new layers may still be in the page cache, and after the mount-cache export, since the snapshot drops the build cache. This PR's own CI shows what that costs on a freshly built image. It also stores one extra artifact per CI image build: the snapshot (about 2.6 GB, kept 2 days) next to the existing stash. If that storage is a concern, the snapshot can sit behind a workflow input. Other notes: - The snapshot key follows the job's Python version, so the switch of the default to 3.11 (#74151) needs no change here. - The extra 1–2 minutes in the build job stay under the CI duration monitor's 3-minute absolute floor for per-job alerts (`scripts/ci/analyze_ci_job_durations.py`, `JOB_MIN_ABS_INCREASE_MINUTES`). - The draft #73095 edits the same restore step in `prepare_breeze_and_image`; whichever lands second needs a small rebase. ### Checks run - `scripts/tests/ci/test_docker_data_root_snapshot.py` (7 tests) drives the script with fake `docker`, `sudo` and `zstd`, covering every fallback decision and what `create` keeps. Removing the daemon check makes the "other daemon" case fail. - Rebased on main f070997ba6: the 7 tests pass, and `prek run --stage pre-commit` and `--stage manual` pass, including actionlint and shellcheck. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
