potiuk opened a new pull request, #76225: URL: https://github.com/apache/airflow/pull/76225
#76200 unblocked `main` after Docker renumbered its packages in the dhi.io apt repository, but it could not make the builds reproducible. The base images are pinned to digests, yet every CI and release build still installs its OS dependencies from dhi.io, a live repository that moves on and drops the versions it replaced whenever Docker rebuilds its images. The same commit could build a different image on a different day, or fail to build at all, and a dhi.io change broke `main` directly instead of an upgrade PR. This pins the OS-dependency layer itself, the same way the base images are pinned: - `install_os_dependencies.sh` now also installs the MySQL, MSSQL and Postgres clients, so the whole OS layer - system packages and database clients - comes from one place. It records a hash of everything that decides what it installs (the script, the client scripts and repository keys, the installation type, the apt dependency and client build args, and `DEPENDENCIES_EPOCH_NUMBER`). On an image that already carries a matching marker it installs nothing; on a mismatch it installs on top, so a PR that changes the script still tests its own change. - The Dockerfiles split each OS installation into a stage of its own (`ci-os-deps` in `Dockerfile.ci`, `dev-os-deps` and `runtime-os-deps` in `Dockerfile`), starting from `CI_OS_DEPS_IMAGE` / `DEV_OS_DEPS_IMAGE` / `RUNTIME_OS_DEPS_IMAGE`. They default to `BASE_IMAGE`, so a standalone `docker build` behaves as before. Images record where their OS dependencies came from in `AIRFLOW_OS_DEPS_IMAGE` and `/usr/share/airflow/os-dependencies.base-image`. - `breeze release-management build-os-deps-images` / `publish-os-deps-images` build those three images per Python version for amd64 and arm64 on the mirrored base images and publish them to `ghcr.io/apache/airflow/base/os-deps`. They also build ahead for a newer Python patch release the mirror already copied, so the upgrade moving to it has images to pin. - The "Mirror hardened base images" workflow builds and publishes them daily (the mirror itself keeps its 6-hourly schedule), in concurrency groups of their own, so the upgrade check's mirror-only runs never queue behind a build. - The upgrade check pins the latest published OS-dependencies image for each Python patch release in `hardened_os_deps_image_digests.json`, together with the base images, in the PR that has to pass the full CI. Only `main` carries these pins until there are images per branch. - CI and release builds pass the pinned images. Builds with a custom base image, a non-default Debian version, the legacy flavor, custom apt dependencies or custom database clients keep installing live. `--no-use-pinned-os-deps-images` (`USE_PINNED_OS_DEPS_IMAGES=false`) turns the pins off, and the "disable image cache" label does so in CI; the automatic cache-disabled retry of a failed image build keeps them. The pins file starts empty, so nothing changes in CI until the first daily build has published images and an upgrade PR has pinned them. Tested locally on linux/arm64 with Python 3.13 by real builds: the CI image and the PROD image starting from OS-dependencies images built from these stages (each OS installation reported "nothing to install"; the images run Airflow and carry the MariaDB, Postgres and MSSQL clients), the standalone PROD OS-dependencies stages, and the legacy-flavor PROD image. Breeze and upgrade-script unit tests pass. The workflow changes can only be exercised on `main` of apache/airflow, and amd64 was not built locally. After the first push, `ghcr.io/apache/airflow/base/os-deps` has to be made public, like `base/python`. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Opus 5 Generated-by: Claude Opus 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
