This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new e50c2265b65 Document Python 3.10 client RC reproducibility + verify
helper (#67898)
e50c2265b65 is described below
commit e50c2265b659b0637ed87f1a72f19cda13b5ea12
Author: Jarek Potiuk <[email protected]>
AuthorDate: Wed Jun 3 13:23:30 2026 +0200
Document Python 3.10 client RC reproducibility + verify helper (#67898)
* Document Python 3.10 client RC reproducibility and add verify helper
Documents that the Python client reproducible build must run under Python
3.10
(the trigger_dag_run_post_body.py AST re-emit is interpreter-version
sensitive),
and adds dev/verify_python_client_rc.sh — a headless contributor smoke test
that
boots airflow standalone inside breeze and runs test_python_client.py
against
the live API.
* Fail fast when building Python client under non-default Python
prepare-python-client must run under DEFAULT_PYTHON_MAJOR_MINOR_VERSION
(3.10): the client generator re-emits trigger_dag_run_post_body.py with
ast.unparse, which uses the running interpreter's grammar, so any other
Python yields a non-reproducible client. The existing guard only ran
after full code generation (which itself can fail or emit nothing under
the wrong Python), so the command never failed cleanly.
Hoist the check to the start of the command so it exits immediately with
actionable guidance, and document that the command now refuses to run
under any other Python.
---
dev/README_RELEASE_PYTHON_CLIENT.md | 35 +++++++++
.../commands/release_management_commands.py | 47 ++++++++----
.../tests/test_release_management_commands.py | 30 ++++++++
dev/verify_python_client_rc.sh | 86 ++++++++++++++++++++++
4 files changed, 182 insertions(+), 16 deletions(-)
diff --git a/dev/README_RELEASE_PYTHON_CLIENT.md
b/dev/README_RELEASE_PYTHON_CLIENT.md
index f6b267a855d..8d29d7e0432 100644
--- a/dev/README_RELEASE_PYTHON_CLIENT.md
+++ b/dev/README_RELEASE_PYTHON_CLIENT.md
@@ -525,6 +525,20 @@ This is generally faster and requires less
resources/network bandwidth.
Both commands should produce reproducible `.whl`, `.tar.gz` packages in dist
folder and "-source.tar.gz"
file containing airflow sources in dist folder.
+> [!IMPORTANT]
+> Run the build with Python 3.10 — the project's
`DEFAULT_PYTHON_MAJOR_MINOR_VERSION`. The
+> client generator applies the `trigger_dag_run_post_body.py` AST patch with
`ast.unparse`, which
+> re-emits that file using the running interpreter's grammar, so building
under a different Python
+> (e.g. the host's 3.11+/3.13) produces a non-reproducible client and the
`prepare-python-client`
+> step may not even emit the wheel/sdist. `prepare-python-client` refuses to
run under any other
+> Python and exits early with this guidance, so pin the interpreter explicitly:
+>
+> ```shell
+> UV_PYTHON=3.10 breeze release-management prepare-python-client
--distribution-format both --version-suffix ""
+> # or equivalently
+> breeze --python 3.10 release-management prepare-python-client
--distribution-format both --version-suffix ""
+> ```
+
4) Change to the directory where you have the packages from svn and check if
they are identical to the ones
you just built:
@@ -758,6 +772,27 @@ Give the server 20-30 seconds to serialize the example
Dags to DB
python /opt/airflow/clients/python/test_python_client.py
```
+### Non-interactive smoke test
+
+If you would rather not drive the interactive `start-airflow` session, the
+[`dev/verify_python_client_rc.sh`](verify_python_client_rc.sh) helper performs
the same end-to-end
+check in a single non-interactive `breeze shell` invocation: it boots `airflow
standalone`, waits for
+the API server, installs the client, and runs `test_python_client.py` against
the live API. `breeze
+shell` already configures `SimpleAuthManager` with `admin`/`admin` and a
per-shell JWT secret, so the
+server and the test share the same credentials.
+
+```shell script
+# install the RC from PyPI (published with the rcN suffix)
+CLIENT_VERSION=${VERSION_RC} breeze shell --load-example-dags --backend sqlite
\
+ "bash dev/verify_python_client_rc.sh"
+
+# or test the exact SVN wheel (copy it under ./dist first so it is visible at
/opt/airflow/dist)
+CLIENT_WHEEL=/opt/airflow/dist/apache_airflow_client-${VERSION}-py3-none-any.whl
\
+ breeze shell --load-example-dags --backend sqlite "bash
dev/verify_python_client_rc.sh"
+```
+
+The script prints `PYTHON_CLIENT_RC_VERIFY: OK` on success.
+
# Publish the final Apache Airflow client release
diff --git
a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index 773e35f41f4..c039aa3df97 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -3895,6 +3895,32 @@ def
_build_client_packages_with_docker(source_date_epoch: int, distribution_form
run_command(["docker", "rm", "--force", container_id], check=False,
stdout=DEVNULL, stderr=DEVNULL)
+def _ensure_default_python_for_reproducible_client() -> None:
+ """Fail fast unless running under the default Python.
+
+ The client generator post-processes ``trigger_dag_run_post_body.py`` with
``ast.unparse``,
+ which re-emits that file using the running interpreter's grammar. Building
under any Python
+ other than ``DEFAULT_PYTHON_MAJOR_MINOR_VERSION`` therefore produces a
non-reproducible client
+ (and the generation step may not even emit the wheel/sdist). Refuse to
continue rather than
+ silently producing a bad package.
+ """
+ current_python_version =
f"{sys.version_info.major}.{sys.version_info.minor}"
+ if current_python_version == DEFAULT_PYTHON_MAJOR_MINOR_VERSION:
+ return
+ console_print(
+ f"[error]Python version mismatch: current version is
{current_python_version}, "
+ f"but the reproducible client must be built with Python
{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}.[/]"
+ )
+ console_print(f"[info]Please rerun breeze with Python
{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}.[/]")
+ console_print(
+ "\n - For the recommended uvx-based setup, set UV_PYTHON before
invoking breeze:\n"
+ f" UV_PYTHON={DEFAULT_PYTHON_MAJOR_MINOR_VERSION} breeze ...\n"
+ " - For a legacy global install, reinstall with the right Python:\n"
+ f" uv tool install --python
{DEFAULT_PYTHON_MAJOR_MINOR_VERSION} -e ./dev/breeze --force\n"
+ )
+ sys.exit(1)
+
+
@release_management_group.command(name="prepare-python-client", help="Prepares
python client packages.")
@option_distribution_format
@option_version_suffix
@@ -3928,6 +3954,7 @@ def prepare_python_client(
only_publish_build_scripts: bool,
security_schemes: str,
):
+ _ensure_default_python_for_reproducible_client()
shutil.rmtree(PYTHON_CLIENT_TMP_DIR, ignore_errors=True)
PYTHON_CLIENT_TMP_DIR.mkdir(parents=True, exist_ok=True)
shutil.copy(src=SOURCE_API_YAML_PATH, dst=TARGET_API_YAML_PATH)
@@ -4001,23 +4028,11 @@ def prepare_python_client(
This patch:
- Locates the `_dict = self.model_dump(...)` line in `to_dict()`
- Inserts a conditional to add `"logical_date": None` if it's missing
- """
- current_python_version =
f"{sys.version_info.major}.{sys.version_info.minor}"
- if current_python_version != DEFAULT_PYTHON_MAJOR_MINOR_VERSION:
- console_print(
- f"[error]Python version mismatch: current version is
{current_python_version}, "
- f"but default version is {DEFAULT_PYTHON_MAJOR_MINOR_VERSION}
- this might cause "
- f"reproducibility problems with prepared package.[/]"
- )
- console_print(f"[info]Please rerun breeze with Python
{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}.[/]")
- console_print(
- "\n - For the recommended uvx-based setup, set UV_PYTHON
before invoking breeze:\n"
- f" UV_PYTHON={DEFAULT_PYTHON_MAJOR_MINOR_VERSION}
breeze ...\n"
- " - For a legacy global install, reinstall with the right
Python:\n"
- f" uv tool install --python
{DEFAULT_PYTHON_MAJOR_MINOR_VERSION} -e ./dev/breeze --force\n"
- )
- sys.exit(1)
+ The interpreter is already pinned to
``DEFAULT_PYTHON_MAJOR_MINOR_VERSION`` by
+ ``_ensure_default_python_for_reproducible_client`` at the start of the
command, so the
+ ``ast.unparse`` re-emit below is reproducible.
+ """
TRIGGER_MODEL_PATH = PYTHON_CLIENT_TMP_DIR / Path(
"airflow_client/client/models/trigger_dag_run_post_body.py"
)
diff --git a/dev/breeze/tests/test_release_management_commands.py
b/dev/breeze/tests/test_release_management_commands.py
index 6077f55d859..6a2f272cd39 100644
--- a/dev/breeze/tests/test_release_management_commands.py
+++ b/dev/breeze/tests/test_release_management_commands.py
@@ -16,12 +16,16 @@
# under the License.
from __future__ import annotations
+from types import SimpleNamespace
+
import pytest
from airflow_breeze.commands.release_management_commands import (
+ _ensure_default_python_for_reproducible_client,
_is_initial_provider_release,
_should_include_provider_in_issue,
)
+from airflow_breeze.global_constants import DEFAULT_PYTHON_MAJOR_MINOR_VERSION
@pytest.mark.parametrize(
@@ -60,3 +64,29 @@ def test_should_include_provider_in_issue(
)
is expected
)
+
+
+def _fake_version_info(version: str) -> SimpleNamespace:
+ major, minor = (int(part) for part in version.split("."))
+ return SimpleNamespace(major=major, minor=minor, micro=0,
releaselevel="final", serial=0)
+
+
+def
test_ensure_default_python_for_reproducible_client_passes_on_default(monkeypatch):
+ monkeypatch.setattr(
+ "airflow_breeze.commands.release_management_commands.sys.version_info",
+ _fake_version_info(DEFAULT_PYTHON_MAJOR_MINOR_VERSION),
+ )
+ # Must not raise / exit when running under the default Python.
+ _ensure_default_python_for_reproducible_client()
+
+
[email protected]("wrong_version", ["3.11", "3.13", "3.9"])
+def
test_ensure_default_python_for_reproducible_client_exits_on_mismatch(monkeypatch,
wrong_version):
+ assert wrong_version != DEFAULT_PYTHON_MAJOR_MINOR_VERSION
+ monkeypatch.setattr(
+ "airflow_breeze.commands.release_management_commands.sys.version_info",
+ _fake_version_info(wrong_version),
+ )
+ with pytest.raises(SystemExit) as exc_info:
+ _ensure_default_python_for_reproducible_client()
+ assert exc_info.value.code == 1
diff --git a/dev/verify_python_client_rc.sh b/dev/verify_python_client_rc.sh
new file mode 100755
index 00000000000..60cf6fae299
--- /dev/null
+++ b/dev/verify_python_client_rc.sh
@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Non-interactive contributor smoke test for an Apache Airflow Python Client
+# release candidate.
+#
+# The documented contributor flow (README_RELEASE_PYTHON_CLIENT.md) uses the
+# interactive ``breeze start-airflow`` + ``test_python_client.py``. This script
+# is the headless equivalent: it is meant to be run *inside* the Breeze
+# container so a single command brings up Airflow and exercises the client
+# end-to-end. ``breeze shell`` configures SimpleAuthManager with admin/admin
and
+# a per-shell JWT secret, so booting ``airflow standalone`` and running the
test
+# in the *same* shell is all that is needed.
+#
+# Usage (from the repo root, with the chosen RC version):
+#
+# CLIENT_VERSION=3.2.2rc1 breeze shell --load-example-dags --backend sqlite \
+# "bash dev/verify_python_client_rc.sh"
+#
+# By default the client is installed from PyPI (the RC is published there with
+# the rcN suffix). To test the exact SVN artifact instead, mount/copy the wheel
+# into the repo (it is available under /opt/airflow inside the container) and
+# point CLIENT_WHEEL at it:
+#
+#
CLIENT_WHEEL=/opt/airflow/dist/apache_airflow_client-3.2.2-py3-none-any.whl \
+# breeze shell --load-example-dags "bash dev/verify_python_client_rc.sh"
+set -uo pipefail
+
+cd /opt/airflow || exit 1
+export AIRFLOW__CORE__LOAD_EXAMPLES=True
+
+CLIENT_VERSION="${CLIENT_VERSION:-}"
+CLIENT_WHEEL="${CLIENT_WHEEL:-}"
+
+echo "### Starting airflow standalone (background) ###"
+airflow standalone > /tmp/standalone.log 2>&1 &
+SA=$!
+
+echo "### Waiting for API server on :8080 ###"
+up=0
+for i in $(seq 1 90); do
+ if curl -fsS -o /dev/null --max-time 3
http://localhost:8080/api/v2/monitor/health 2>/dev/null; then
+ echo "API healthy after ~$((i * 4))s"; up=1; break
+ fi
+ sleep 4
+done
+if [[ "${up}" != "1" ]]; then
+ echo "ERROR: API server did not become healthy"; tail -40
/tmp/standalone.log; kill "${SA}" 2>/dev/null; exit 1
+fi
+
+echo "### Installing Apache Airflow Python Client ###"
+if [[ -n "${CLIENT_WHEEL}" ]]; then
+ pip install --force-reinstall "${CLIENT_WHEEL}"
+elif [[ -n "${CLIENT_VERSION}" ]]; then
+ pip install --force-reinstall "apache-airflow-client==${CLIENT_VERSION}"
+else
+ echo "ERROR: set CLIENT_VERSION (e.g. 3.2.2rc1) or CLIENT_WHEEL"; kill
"${SA}" 2>/dev/null; exit 2
+fi
+python -c "import importlib.metadata as m; print('installed client version:',
m.version('apache-airflow-client'))"
+
+echo "### Running test_python_client.py against the live API ###"
+python /opt/airflow/clients/python/test_python_client.py
+rc=$?
+
+kill "${SA}" 2>/dev/null || true
+if [[ "${rc}" == "0" ]]; then
+ echo "PYTHON_CLIENT_RC_VERIFY: OK"
+else
+ echo "PYTHON_CLIENT_RC_VERIFY: FAILED (rc=${rc})"
+fi
+exit "${rc}"
