zozo123 opened a new pull request, #73997: URL: https://github.com/apache/airflow/pull/73997
A full CI run restores the same ~8 GB CI image in 184 jobs on AMD and 148 on ARM. Today the producer uploads a plain `docker image save` tar, `stash/save` zips it at zlib level 6 (135-199 s per image), and every consumer downloads that zip and inflates ~8 GB to `/mnt` before `docker image load` can start. That restore is about half of each consumer's "Prepare breeze & CI image" time (median 129 s of 261 s across 1200 recent consumer jobs); the other half is `docker image load`, which this PR does not change. This PR compresses the image once, on the producer, with `zstd -6 -T0`, stashes it with `compression-level: '0'` so it is not zipped again, and hands the `.tar.zst` straight to `docker image load`, which reads zstd natively. Consumers move ~12% fewer bytes and skip the inflate; the producer skips the level-6 upload. ## Results Controlled paired A/B on GitHub-hosted runners with the real `main` CI image (Python 3.10), 20 consumer jobs per arch, each job measuring every variant in a balanced order. Metric: `stash/restore` + `docker image load`, per job. | | v3 today: median / p90 | this PR: median / p90 | paired change [95% CI] | faster jobs | |---|---|---|---|---| | amd64 (`ubuntu-22.04`) | 212.4 s / 284.2 s | 165.9 s / 202.5 s | **-26.4%** [-28.3, -22.5] | 20/20 | | arm64 (`ubuntu-22.04-arm`) | 220.3 s / 241.2 s | 168.0 s / 182.7 s | **-24.0%** [-28.3, -21.3] | 20/20 | - Image IDs identical in 200/200 loads (and equal to upstream's stash); 0/200 failures, 0 download retries. - Artifact: 2.47 GB instead of 2.83 GB (amd64), 2.37 GB instead of 2.70 GB (arm64). - Producer save + upload per image: 333 s -> 162 s (amd64), 263 s -> 146 s (arm64). - Real CI downloads are slower than in the bench, and most of the saving is a roughly fixed ~50 s of avoided inflate, so expect about **20-22% at the median** in real CI rather than 24-26%. In absolute terms that is ~3 runner-hours per full AMD run and ~2.2 per ARM run. <details> <summary>Method, all variants, projection and caveats</summary> **Bench run:** [zozo123/airflow run 36771856900](https://github.com/zozo123/airflow/actions/runs/36771856900) (workflow and scripts on branch [`bench/zstd-image-stash`](https://github.com/zozo123/airflow/tree/bench/zstd-image-stash)). - Image: upstream's own `ci-image-save-v3-linux_<arch>-3.10-main` stash, i.e. `ghcr.io/apache/airflow/main/ci/python3.10:latest` (8.21 GB tar on amd64, 7.95 GB on arm64). - One producer per arch uploaded five variants with `apache/infrastructure-actions/stash/save@61dcea11`: `v3` (plain tar, default level 6, as today), `zst1` / `zst3` / `zst6` (`docker image save | zstd -N -T0`, level 0) and `raw` (plain tar, level 0). - 20 consumers per arch restored each variant with `stash/restore@61dcea11` (`only-current-branch: true`) into `/mnt` and ran `docker image load -i`, in a Williams-crossover order, clearing Docker state and the page cache before each variant. - Statistics are within-job paired differences, bootstrap 95% CI of the median (10k resamples). **Baseline:** 1200 consumer jobs from 7 upstream runs (restore + load median 260.9 s, p90 356.5 s): AMD schedule [36725664249](https://github.com/apache/airflow/actions/runs/36725664249), [36658374990](https://github.com/apache/airflow/actions/runs/36658374990); ARM schedule [36685452188](https://github.com/apache/airflow/actions/runs/36685452188), [36620372737](https://github.com/apache/airflow/actions/runs/36620372737); AMD PR [36734713052](https://github.com/apache/airflow/actions/runs/36734713052), [36423433196](https://github.com/apache/airflow/actions/runs/36423433196), [36547288909](https://github.com/apache/airflow/actions/runs/36547288909). **All variants, paired change in restore + load vs v3:** | | artifact (amd64) | amd64 | arm64 | |---|---|---|---| | zst1 | 2.87 GB | -21.8% [-25.2, -16.0] | -21.0% [-26.8, -18.2] | | zst3 | 2.61 GB | -25.4% | -18.0% [-22.9, -10.0] | | **zst6 (this PR)** | 2.47 GB | -26.4% [-28.3, -22.5] | -24.0% [-28.3, -21.3] | | raw (level 0, no zstd) | 8.21 GB | +23.3% | +35.6% | - `raw` shows that skipping GitHub's zlib alone is worse: bytes matter as much as the inflate. - zst1 moves as many bytes as today's zip, so it only saves the inflate. zst6 beats zst1 on amd64 (-8.1 s [-12.1, -5.5] per job) and on artifact size and p90; on arm64 the per-job difference is not significant (-11.2 s [-15.1, +0.5]). - `docker image load` of a `.tar.zst` is not slower than of the plain tar. **Projection to real CI.** Applying the bench's median paired saving per arch and region group to each of the 1200 baseline jobs: zst6 -21.9% amd64 (-20.6% without region weighting), -20.6% arm64 (-20.1%); zst1 would give about -15% to -17% amd64. At p90, real CI should gain roughly 40-60 s rather than the bench's 20-26%. **Caveats** - One bench run (2026-09-30), ~20 concurrent consumers; the CIs cover job-to-job, not run-to-run, variance. Bench downloads were faster than real CI's even within the same regions (west v3 restore median 109.6 s in the bench vs 181.8 s in the baseline, amd64). - One sample per variant per job; medians are robust, p90s less so. - Python 3.10 image only; producer measured once per variant on a cold page cache (CI saves from a warm one). - Consumers ran `docker image load -i` directly; `breeze ci-image load` adds ~10 s equally to every variant. </details> ## Changes **Breeze** - `ci-image save --image-file <name>.tar.zst` streams `docker image save` through `zstd -6 -T0` as one `bash -o pipefail` command via `run_command`, so `--dry-run`/`--verbose` behave as elsewhere; a failing stage exits with its own code and removes the partial file. Other names still produce a plain tar. - `ci-image load` accepts both `ci-image-save-v3-*.tar` and `ci-image-save-v4-*.tar.zst`; without `--image-file` it prefers v4 and falls back to v3. `--from-run`/`--from-pr` target v4. - `download_artifact_from_run_id` built the artifact prefix with `os.path.splitext`, which leaves `.tar` on a `.tar.zst` name, so no v4 artifact could match. **CI** - `ci-image-build.yml` picks the archive format from the checked-out sources: `v4` / `.tar.zst` / level 0 when both the checked-out `prepare_breeze_and_image` action and Breeze know v4, otherwise `v3` / `.tar` / level 6 exactly as today. All CI-image and commit-marker stash keys, globs, file names and the compression level come from that step. - `prepare_breeze_and_image` and `prepare_single_ci_image` restore `ci-image-save-v4-*` for CI images. - Unchanged on purpose: PROD image stashes, mount-cache stashes, and `build_ci_image_with_cache` (its own only consumer). **Docs:** `06_managing_docker_images.rst`, `ci/01_ci_environment.md`, `ci/02_images.md` (which used a non-existent `-i` flag), and the regenerated `ci-image load`/`save` command images. ## Compatibility - `release-constraints.yml` and `refresh-constraints.yml` (via `update-constraints-on-push.yml`) run main's `ci-image-build.yml` against a release branch or tag, whose Breeze and `prepare_breeze_and_image` still expect v3. Without the format-detection step those runs would restore a stale main image for up to two days and then fail. With it, such refs keep producing and consuming v3. These workflows do not run on PR CI, so this is covered by reasoning rather than a run. - Backporting to `v3-3-test` would move that branch to v4; already-cut tags keep working either way. - Local `breeze ci-image load` still reads v3 `.tar` files; loading a `.tar.zst` needs no host zstd. Saving one needs `zstd` on `PATH` (hosted runners ship 1.5.7). - The stash key changes, so the first run on each branch after merge misses anything seeded from an earlier stash. Rollback is a revert; the next producer run on each branch repopulates the v3 keys. ## Checks run - `uv run --directory dev/breeze pytest tests/test_ci_image_commands.py tests/test_github_utils.py -q`: 51 passed. New tests fail against the upstream sources, except the v3 regression guards. - `uv run --directory dev/breeze pytest tests -q -n 8`: 1181 passed, 6 failed; the 6 are unrelated (3 `test_publish_docs_to_s3.py::test_publish_stable_version_docs` cases also fail on `main`; 3 `test_shim_version_check.py` cases fail only under xdist). - `prek run --from-ref upstream/main --to-ref HEAD --stage pre-commit`: passed (yamllint, zizmor, ruff, mypy-dev, update-breeze-cmd-output, ...). Skipped for lack of Docker: `check-provider-yaml-valid`, `update-providers-build-files`, `lychee-docker`. - `prek run --from-ref upstream/main --to-ref HEAD --stage manual` (standard skips): passed. No newsfragment: CI and dev tooling only. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
