This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch hardened-python-base-images in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 02d924e954d2597983f6785dac589336e8038e58 Author: Jarek Potiuk <[email protected]> AuthorDate: Mon Oct 12 15:30:51 2026 +0100 Keep apt from replacing the hardened images' Python Debian 13's python3 packages conflict with the hardened images' python-3.X packages, so installing NodeSource's nodejs (which depends on python3) into the lang-SDK runtime image made apt remove the image's Python 3.11, and the migration pods crash-looped. The image now holds its Python packages, so such an install fails instead, and the runtime image copies the node binary from the official node image rather than installing the deb. Generated-by: Claude Opus 5 --- Dockerfile | 16 ++++++++++++++++ Dockerfile.ci | 16 ++++++++++++++++ docker-stack-docs/build.rst | 6 ++++++ kubernetes-tests/lang_sdk/Dockerfile.runtimes | 23 ++++++++--------------- scripts/docker/install_os_dependencies.sh | 16 ++++++++++++++++ 5 files changed, 62 insertions(+), 15 deletions(-) diff --git a/Dockerfile b/Dockerfile index 2066ae8ccd4..3c50b473d1d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -456,6 +456,20 @@ function compile_python_stdlib() { /usr/python/bin/python -m compileall -q -j "$(nproc)" -o 0 -o 1 -o 2 "${stdlib}" || true } +function hold_hardened_python() { + # The Debian 13 hardened images install their Python as packages that conflict with Debian's own + # python3. A package depending on python3 - for example NodeSource's nodejs - therefore makes apt + # remove the image's Python and install Debian's, without asking. Holding the packages turns that + # into an apt error instead, in this build and in images extending it. + local packages + packages=$(dpkg-query -W -f='${db:Status-Abbrev} ${Package}\n' 'python-3.*' 'libpython-3.*' 2>/dev/null \ + | awk '$1 == "ii" {print $2}' || true) + if [[ -n "${packages}" ]]; then + # shellcheck disable=SC2086 + apt-mark hold ${packages} + fi +} + function check_no_system_python() { # Python - from the hardened base image or compiled from sources - must stay the only Python in the # image. A system Python pulled in as a dependency of an apt package shares its shared libraries @@ -623,6 +637,7 @@ function install_debian_runtime_dependencies() { check_no_system_python link_python if [[ "${AIRFLOW_IMAGE_FLAVOR}" == "hardened" ]]; then + hold_hardened_python restore_thread_stack_size compile_python_stdlib fi @@ -728,6 +743,7 @@ else check_no_system_python if [[ "${AIRFLOW_IMAGE_FLAVOR}" == "hardened" ]]; then link_python + hold_hardened_python restore_thread_stack_size compile_python_stdlib else diff --git a/Dockerfile.ci b/Dockerfile.ci index 93084a3c3de..286dcb95275 100644 --- a/Dockerfile.ci +++ b/Dockerfile.ci @@ -392,6 +392,20 @@ function compile_python_stdlib() { /usr/python/bin/python -m compileall -q -j "$(nproc)" -o 0 -o 1 -o 2 "${stdlib}" || true } +function hold_hardened_python() { + # The Debian 13 hardened images install their Python as packages that conflict with Debian's own + # python3. A package depending on python3 - for example NodeSource's nodejs - therefore makes apt + # remove the image's Python and install Debian's, without asking. Holding the packages turns that + # into an apt error instead, in this build and in images extending it. + local packages + packages=$(dpkg-query -W -f='${db:Status-Abbrev} ${Package}\n' 'python-3.*' 'libpython-3.*' 2>/dev/null \ + | awk '$1 == "ii" {print $2}' || true) + if [[ -n "${packages}" ]]; then + # shellcheck disable=SC2086 + apt-mark hold ${packages} + fi +} + function check_no_system_python() { # Python - from the hardened base image or compiled from sources - must stay the only Python in the # image. A system Python pulled in as a dependency of an apt package shares its shared libraries @@ -559,6 +573,7 @@ function install_debian_runtime_dependencies() { check_no_system_python link_python if [[ "${AIRFLOW_IMAGE_FLAVOR}" == "hardened" ]]; then + hold_hardened_python restore_thread_stack_size compile_python_stdlib fi @@ -664,6 +679,7 @@ else check_no_system_python if [[ "${AIRFLOW_IMAGE_FLAVOR}" == "hardened" ]]; then link_python + hold_hardened_python restore_thread_stack_size compile_python_stdlib else diff --git a/docker-stack-docs/build.rst b/docker-stack-docs/build.rst index 69fc721b02a..24496484d6d 100644 --- a/docker-stack-docs/build.rst +++ b/docker-stack-docs/build.rst @@ -938,6 +938,12 @@ previous images that can matter when you extend or run the image are: wrong Python. The image build fails when this happens, but a custom image that installs extra apt packages is not checked. Check what ``apt-get install --simulate`` would install before adding packages that might depend on ``python3``, and prefer the runtime library packages over their ``-dev`` variants. +* **The image's Python packages are held.** On the Debian 13 images, Debian's own ``python3`` packages + conflict with the image's ``python-3.X`` packages, so a package that apt can only satisfy with them - + for example NodeSource's ``nodejs`` - would make apt remove the image's Python without asking. The + image marks its Python packages with ``apt-mark hold``, so apt reports the ``python3`` conflict and the + installation fails instead. Install that software some other way - for Node.js, copy the ``node`` binary + from the official ``node`` image - rather than releasing the hold. * **C extensions are compiled more strictly.** The Debian 13 images come with GCC 14, which turns incompatible pointer types, integer conversions and implicit function declarations into errors, and the hardened Python passes ``-Werror=implicit-function-declaration`` on to every C extension it builds. Old diff --git a/kubernetes-tests/lang_sdk/Dockerfile.runtimes b/kubernetes-tests/lang_sdk/Dockerfile.runtimes index 00bd97fc6f5..7b1883bdce4 100644 --- a/kubernetes-tests/lang_sdk/Dockerfile.runtimes +++ b/kubernetes-tests/lang_sdk/Dockerfile.runtimes @@ -23,7 +23,14 @@ # both runtimes to parse native Java and TypeScript Dags, and using one image # everywhere keeps the pod templates and the Helm image override identical # across components. +# +# Node comes as the official binary rather than a deb: the NodeSource nodejs +# package depends on python3, which apt would satisfy on the hardened images by +# replacing the image's own Python with Debian's. ARG BASE_IMAGE +ARG NODE_IMAGE="node:22-bookworm-slim" +FROM ${NODE_IMAGE} AS node + FROM ${BASE_IMAGE} USER root @@ -31,19 +38,5 @@ RUN apt-get update \ && apt-get install --no-install-recommends -y default-jre-headless \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* -# The keyring step pipes curl into gpg, so a failure mid-pipe has to fail the build. -SHELL ["/bin/bash", "-o", "pipefail", "-c"] -RUN apt-get update \ - && apt-get install --no-install-recommends -y ca-certificates curl gnupg \ - && mkdir -p /etc/apt/keyrings \ - && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key \ - | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \ - && echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_22.x nodistro main" \ - > /etc/apt/sources.list.d/nodesource.list \ - && apt-get update \ - && apt-get install --no-install-recommends -y nodejs \ - && apt-get purge -y --auto-remove gnupg \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* -SHELL ["/bin/sh", "-c"] +COPY --from=node /usr/local/bin/node /usr/local/bin/node USER airflow diff --git a/scripts/docker/install_os_dependencies.sh b/scripts/docker/install_os_dependencies.sh index 649e490339d..66bb1349078 100644 --- a/scripts/docker/install_os_dependencies.sh +++ b/scripts/docker/install_os_dependencies.sh @@ -367,6 +367,20 @@ function compile_python_stdlib() { /usr/python/bin/python -m compileall -q -j "$(nproc)" -o 0 -o 1 -o 2 "${stdlib}" || true } +function hold_hardened_python() { + # The Debian 13 hardened images install their Python as packages that conflict with Debian's own + # python3. A package depending on python3 - for example NodeSource's nodejs - therefore makes apt + # remove the image's Python and install Debian's, without asking. Holding the packages turns that + # into an apt error instead, in this build and in images extending it. + local packages + packages=$(dpkg-query -W -f='${db:Status-Abbrev} ${Package}\n' 'python-3.*' 'libpython-3.*' 2>/dev/null \ + | awk '$1 == "ii" {print $2}' || true) + if [[ -n "${packages}" ]]; then + # shellcheck disable=SC2086 + apt-mark hold ${packages} + fi +} + function check_no_system_python() { # Python - from the hardened base image or compiled from sources - must stay the only Python in the # image. A system Python pulled in as a dependency of an apt package shares its shared libraries @@ -534,6 +548,7 @@ function install_debian_runtime_dependencies() { check_no_system_python link_python if [[ "${AIRFLOW_IMAGE_FLAVOR}" == "hardened" ]]; then + hold_hardened_python restore_thread_stack_size compile_python_stdlib fi @@ -639,6 +654,7 @@ else check_no_system_python if [[ "${AIRFLOW_IMAGE_FLAVOR}" == "hardened" ]]; then link_python + hold_hardened_python restore_thread_stack_size compile_python_stdlib else
