This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch pin-os-deps-image-layer in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d4336aa350763cf79932154571cc1543315c0130 Author: Jarek Potiuk <[email protected]> AuthorDate: Sun Nov 1 12:14:06 2026 +0100 Build each OS-dependencies stage fresh once and record the base image it used Forcing the whole build fresh made the runtime OS-dependencies image repeat the dev installation it copies the Python distribution from, although the dev image had just been built in the same job. Forcing only the target stage fresh reuses it. An image built on a pinned OS-dependencies image recorded the BASE_IMAGE it was given, while its layers come from the OS-dependencies image, which may be built on an older digest - the images now record which image their OS dependencies came fro [...] Generated-by: Claude Opus 5 --- Dockerfile | 16 ++++++++++++++++ Dockerfile.ci | 10 ++++++++++ .../commands/release_management_commands.py | 8 ++++++-- dev/breeze/tests/test_os_deps_images.py | 17 ++++++++++++++++- docker-stack-docs/build.rst | 3 +++ scripts/docker/install_os_dependencies.sh | 7 +++++++ 6 files changed, 58 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index e83df12c028..fd50894b59b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -899,6 +899,10 @@ fi mkdir -p "$(dirname "${OS_DEPENDENCIES_MARKER}")" echo "${OS_DEPENDENCIES_INPUTS_HASH}" > "${OS_DEPENDENCIES_MARKER}" +OS_DEPENDENCIES_BASE_IMAGE_FILE="$(dirname "${OS_DEPENDENCIES_MARKER}")/os-dependencies.base-image" +if [[ ! -f "${OS_DEPENDENCIES_BASE_IMAGE_FILE}" ]]; then + echo "${BASE_IMAGE:-unknown}" > "${OS_DEPENDENCIES_BASE_IMAGE_FILE}" +fi EOF # The content below is automatically copied from scripts/docker/install_mysql.sh @@ -2246,6 +2250,12 @@ SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "nounset", "-o", "n ARG BASE_IMAGE +# BASE_IMAGE is the base image the build was given. When DEV_OS_DEPS_IMAGE names a pinned OS-dependencies +# image, the OS dependencies and Python come from that image instead - it may be built on an older digest +# of the base image, which /usr/share/airflow/os-dependencies.base-image records. +ARG DEV_OS_DEPS_IMAGE +ENV AIRFLOW_OS_DEPS_IMAGE=${DEV_OS_DEPS_IMAGE} + # Make sure noninteractive debian install is used and language variables set ENV BASE_IMAGE=${BASE_IMAGE} \ DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \ @@ -2512,6 +2522,12 @@ SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "nounset", "-o", "n ARG BASE_IMAGE +# BASE_IMAGE is the base image the build was given. When RUNTIME_OS_DEPS_IMAGE names a pinned OS-dependencies +# image, the OS dependencies and Python come from that image instead - it may be built on an older digest +# of the base image, which /usr/share/airflow/os-dependencies.base-image records. +ARG RUNTIME_OS_DEPS_IMAGE +ENV AIRFLOW_OS_DEPS_IMAGE=${RUNTIME_OS_DEPS_IMAGE} + # Make sure noninteractive debian install is used and language variables set ENV BASE_IMAGE=${BASE_IMAGE} \ DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \ diff --git a/Dockerfile.ci b/Dockerfile.ci index 4d460f96800..c1d7fa61d0e 100644 --- a/Dockerfile.ci +++ b/Dockerfile.ci @@ -829,6 +829,10 @@ fi mkdir -p "$(dirname "${OS_DEPENDENCIES_MARKER}")" echo "${OS_DEPENDENCIES_INPUTS_HASH}" > "${OS_DEPENDENCIES_MARKER}" +OS_DEPENDENCIES_BASE_IMAGE_FILE="$(dirname "${OS_DEPENDENCIES_MARKER}")/os-dependencies.base-image" +if [[ ! -f "${OS_DEPENDENCIES_BASE_IMAGE_FILE}" ]]; then + echo "${BASE_IMAGE:-unknown}" > "${OS_DEPENDENCIES_BASE_IMAGE_FILE}" +fi EOF # The content below is automatically copied from scripts/docker/install_mysql.sh @@ -2102,6 +2106,12 @@ SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "nounset", "-o", "n ARG BASE_IMAGE +# BASE_IMAGE is the base image the build was given. When CI_OS_DEPS_IMAGE names a pinned OS-dependencies +# image, the OS dependencies and Python come from that image instead - it may be built on an older digest +# of the base image, which /usr/share/airflow/os-dependencies.base-image records. +ARG CI_OS_DEPS_IMAGE +ENV AIRFLOW_OS_DEPS_IMAGE=${CI_OS_DEPS_IMAGE} + # By increasing this number we can do force build of all dependencies. # NOTE! When you want to make sure dependencies are installed from scratch in your PR after removing # some dependencies, you also need to set "disable image cache" in your PR to make sure the image is diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py index 1266c27f5af..12b0171a9d3 100644 --- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py @@ -5581,8 +5581,12 @@ def build_os_deps_images( build.image_type.target, "--platform", platform, - # The point of the image is a fresh installation from the apt repositories. - "--no-cache", + # The point of the image is a fresh installation from the apt repositories - but only of + # the target stage: the runtime stage copies from one built on the dev OS dependencies, + # which the dev image built just before in the same job leaves in the cache, so this + # does not install them a second time. The installation types are built in this order. + "--no-cache-filter", + build.image_type.target, "--pull", # A plain image manifest per platform, for publish-os-deps-images to merge. "--provenance=false", diff --git a/dev/breeze/tests/test_os_deps_images.py b/dev/breeze/tests/test_os_deps_images.py index 9bc5be09918..dd7ed3b8ef9 100644 --- a/dev/breeze/tests/test_os_deps_images.py +++ b/dev/breeze/tests/test_os_deps_images.py @@ -267,7 +267,8 @@ def test_build_os_deps_images_builds_and_pushes_the_platform_image(registry): assert command[command.index("--platform") + 1] == "linux/arm64" assert command[command.index("--tag") + 1] == f"{REPOSITORY}:{build.content_tag}-arm64" assert "--push" in command - assert "--no-cache" in command + assert command[command.index("--no-cache-filter") + 1] == "runtime-os-deps" + assert "--no-cache" not in command @pytest.mark.parametrize("existing", ["merged", "platform"]) @@ -513,3 +514,17 @@ def test_publish_os_deps_images_prints_the_plan_in_dry_run(mock_dry_run): [f"{REPOSITORY}:{build.content_tag}"], [f"{REPOSITORY}:ci-{PYTHON_TAG}"], ] + + +def test_dev_os_deps_image_is_built_before_the_runtime_one_that_reuses_it(): + names = [image_type.name for image_type in OS_DEPS_IMAGE_TYPES] + + assert names.index("dev") < names.index("runtime") + + +def test_os_deps_stages_expose_the_image_their_os_dependencies_come_from(): + for image_type in OS_DEPS_IMAGE_TYPES: + stage = extract_dockerfile_stage( + (AIRFLOW_ROOT_PATH / image_type.dockerfile).read_text(), image_type.target + ) + assert f"ENV AIRFLOW_OS_DEPS_IMAGE=${{{image_type.build_arg}}}" in stage diff --git a/docker-stack-docs/build.rst b/docker-stack-docs/build.rst index 7275c73b737..ef6a401bc93 100644 --- a/docker-stack-docs/build.rst +++ b/docker-stack-docs/build.rst @@ -978,6 +978,9 @@ previous images that can matter when you extend or run the image are: by the same upgrade, passed as ``DEV_OS_DEPS_IMAGE`` and ``RUNTIME_OS_DEPS_IMAGE``. The script records what it installed with and does nothing when nothing differs, so a build with other apt dependencies still installs them. A plain ``docker build`` leaves both at ``BASE_IMAGE`` and installs everything as before. + The ``AIRFLOW_OS_DEPS_IMAGE`` environment variable of an image names the image its OS dependencies came + from, and ``/usr/share/airflow/os-dependencies.base-image`` the base image they were installed on - which + can be an older digest than ``BASE_IMAGE``. * **Packages come from Docker's apt repository first.** The image keeps stock Debian's apt sources beside ``dhi.io``, and ``/etc/apt/preferences.d/airflow-prefer-dhi`` ranks ``dhi.io`` higher, so apt installs Docker's build of a package whenever ``dhi.io`` publishes it and falls back to Debian's otherwise - also diff --git a/scripts/docker/install_os_dependencies.sh b/scripts/docker/install_os_dependencies.sh index 73ad398efb4..8128142eee6 100644 --- a/scripts/docker/install_os_dependencies.sh +++ b/scripts/docker/install_os_dependencies.sh @@ -804,3 +804,10 @@ fi mkdir -p "$(dirname "${OS_DEPENDENCIES_MARKER}")" echo "${OS_DEPENDENCIES_INPUTS_HASH}" > "${OS_DEPENDENCIES_MARKER}" +# The base image the OS dependencies were first installed on. An image built on a pinned OS-dependencies +# image records its own BASE_IMAGE, but its layers come from this one, which may be an older digest. An +# installation on top keeps the record, as the layers underneath still come from that image. +OS_DEPENDENCIES_BASE_IMAGE_FILE="$(dirname "${OS_DEPENDENCIES_MARKER}")/os-dependencies.base-image" +if [[ ! -f "${OS_DEPENDENCIES_BASE_IMAGE_FILE}" ]]; then + echo "${BASE_IMAGE:-unknown}" > "${OS_DEPENDENCIES_BASE_IMAGE_FILE}" +fi
