This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch drop-py310/9-docs in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f97c34e689b481ca2d7e287ae98283327eec4ba8 Author: Shahar Epstein <[email protected]> AuthorDate: Sat Oct 3 16:04:43 2026 +0300 Update docs for the Python 3.11 baseline Contributing docs, Breeze docs, Docker stack docs and the user-facing installation pages used Python 3.10 as the example and default version. Examples and defaults now use 3.11; historical changelog entries and ADRs keep their original wording. --- .../pr-management-triage-comment-templates.md | 4 +- .apache-magpie-overrides/release-verify-rc.md | 2 +- .apache-magpie-overrides/runtime-invocation.md | 2 +- airflow-core/docs/extra-packages-ref.rst | 2 +- .../docs/installation/installing-from-pypi.rst | 24 +++--- .../vulnerabilities-in-3rd-party-dependencies.rst | 2 +- airflow-core/docs/start.rst | 6 +- airflow-ctl/README.md | 2 +- airflow-ctl/docs/installation/prerequisites.rst | 4 +- constraints/README.md | 6 +- contributing-docs/03_contributors_quick_start.rst | 24 +++--- contributing-docs/05_pull_requests.rst | 2 +- contributing-docs/07_local_virtualenv.rst | 2 +- contributing-docs/08_static_code_checks.rst | 2 +- .../13_airflow_dependencies_and_extras.rst | 8 +- contributing-docs/testing/docker_compose_tests.rst | 8 +- contributing-docs/testing/k8s_tests.rst | 86 +++++++++++----------- .../testing/task_sdk_integration_tests.rst | 4 +- contributing-docs/testing/unit_tests.rst | 10 +-- dev/README_RELEASE_AIRFLOW.md | 8 +- dev/README_RELEASE_PROVIDERS.md | 2 +- dev/README_RELEASE_PYTHON_CLIENT.md | 6 +- dev/breeze/doc/01_installation.rst | 2 +- dev/breeze/doc/03_developer_tasks.rst | 16 ++-- dev/breeze/doc/04_troubleshooting.rst | 8 +- dev/breeze/doc/05_test_commands.rst | 10 +-- dev/breeze/doc/06_managing_docker_images.rst | 8 +- dev/breeze/doc/09_release_management_tasks.rst | 2 +- dev/breeze/doc/14_advanced_breeze_topics.rst | 2 +- dev/breeze/doc/ci/02_images.md | 58 +++++++-------- dev/breeze/doc/ci/03_github_variables.md | 4 +- dev/breeze/doc/ci/04_selective_checks.md | 12 +-- dev/breeze/doc/ci/07_running_ci_locally.md | 4 +- docker-stack-docs/build.rst | 14 ++-- .../restricted/restricted_environments.sh | 8 +- docker-stack-docs/entrypoint.rst | 14 ++-- providers-summary-docs/installing-from-pypi.rst | 2 +- registry/README.md | 2 +- 38 files changed, 190 insertions(+), 192 deletions(-) diff --git a/.apache-magpie-overrides/pr-management-triage-comment-templates.md b/.apache-magpie-overrides/pr-management-triage-comment-templates.md index 2b05f27b204..d77a79e5157 100644 --- a/.apache-magpie-overrides/pr-management-triage-comment-templates.md +++ b/.apache-magpie-overrides/pr-management-triage-comment-templates.md @@ -120,8 +120,8 @@ check or other violation. For this project, the bullet uses the This overrides the framework default. The triage comment must **not** enumerate the failing check names underneath the category (e.g. avoid `:x: **Kubernetes tests** — Failing: -Kubernetes tests / K8S System:LocalExecutor-3.10-v1.30.13-false, -Kubernetes tests / K8S System:KubernetesExecutor-3.10-..., +Kubernetes tests / K8S System:LocalExecutor-3.11-v1.30.13-false, +Kubernetes tests / K8S System:KubernetesExecutor-3.11-..., (+1 more). See docs.`). The same applies to the per-category remediation snippets the diff --git a/.apache-magpie-overrides/release-verify-rc.md b/.apache-magpie-overrides/release-verify-rc.md index 72d4819e705..318c8d1269f 100644 --- a/.apache-magpie-overrides/release-verify-rc.md +++ b/.apache-magpie-overrides/release-verify-rc.md @@ -80,7 +80,7 @@ script calling `git` via `subprocess` works. ```bash breeze start-airflow --mount-sources remove --use-distributions-from-dist \ - --use-airflow-version wheel --python 3.10 --backend postgres --load-default-connections + --use-airflow-version wheel --python 3.11 --backend postgres --load-default-connections ``` 4. Show the user a table of the tasks and what each log should contain, plus any diff --git a/.apache-magpie-overrides/runtime-invocation.md b/.apache-magpie-overrides/runtime-invocation.md index 48b033c3f93..c02cb253b52 100644 --- a/.apache-magpie-overrides/runtime-invocation.md +++ b/.apache-magpie-overrides/runtime-invocation.md @@ -67,7 +67,7 @@ you run with: - Build or refresh the CI image before a campaign so the first reproducer does not pay the build cost inside its timeout: `breeze ci-image build --python <python>`. -- Python versions: `3.10` (default), `3.11`, `3.12`, `3.13`, `3.14` — +- Python versions: `3.11` (default), `3.12`, `3.13`, `3.14` — `ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS` in `dev/breeze/src/airflow_breeze/global_constants.py`; `requires-python` in `airflow-core/pyproject.toml`. Use the default unless the issue names diff --git a/airflow-core/docs/extra-packages-ref.rst b/airflow-core/docs/extra-packages-ref.rst index be7d787a2dc..6bf6c7b4f5e 100644 --- a/airflow-core/docs/extra-packages-ref.rst +++ b/airflow-core/docs/extra-packages-ref.rst @@ -170,7 +170,7 @@ with a consistent set of dependencies based on constraint files provided by Airf :substitutions: pip install apache-airflow[google,amazon,apache-spark]==|version| \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.10.txt" + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.11.txt" Note, that this will install providers in the versions that were released at the time of Airflow |version| release. You can later upgrade those providers manually if you want to use latest versions of the providers. diff --git a/airflow-core/docs/installation/installing-from-pypi.rst b/airflow-core/docs/installation/installing-from-pypi.rst index 87727f19710..92f2fa3b761 100644 --- a/airflow-core/docs/installation/installing-from-pypi.rst +++ b/airflow-core/docs/installation/installing-from-pypi.rst @@ -65,7 +65,7 @@ Typical command to install Airflow from scratch in a reproducible way from PyPI .. code-block:: bash - pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.10.txt" + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.11.txt" Typically, you can add other dependencies and providers as separate command after the reproducible @@ -135,14 +135,14 @@ You can create the URL to the file substituting the variables in the template be where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version -- ``PYTHON_VERSION`` Python version e.g. ``3.10``, ``3.11`` +- ``PYTHON_VERSION`` Python version e.g. ``3.11``, ``3.12`` The examples below assume that you want to use install Airflow in a reproducible way with the ``celery`` extra, but you can pick your own set of extras and providers to install. .. code-block:: bash - pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.10.txt" + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.11.txt" .. note:: @@ -173,7 +173,7 @@ performing dependency resolution. .. code-block:: bash - pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.10.txt" + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.11.txt" pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1 You can also downgrade or upgrade other dependencies this way - even if they are not compatible with @@ -181,7 +181,7 @@ those dependencies that are stored in the original constraints file: .. code-block:: bash - pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.10.txt" + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.11.txt" pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0 .. warning:: @@ -224,7 +224,7 @@ one provided by the community. .. code-block:: bash - pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.10.txt" + pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.11.txt" pip install "apache-airflow==|version|" dbt-core==0.20.0 pip freeze > my-constraints.txt @@ -353,9 +353,9 @@ dependencies compatible with just Airflow core at the moment Airflow was release AIRFLOW_VERSION=|version| PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')" - # For example: 3.10 + # For example: 3.11 CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt"; - # For example: https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-no-providers-3.10.txt + # For example: https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-no-providers-3.11.txt pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" Troubleshooting @@ -379,7 +379,7 @@ Symbol not found: ``_Py_GetArgcArgv`` ===================================== If you see ``Symbol not found: _Py_GetArgcArgv`` while starting or importing ``airflow``, this may mean that you are using an incompatible version of Python. -For a homebrew installed version of Python, this is generally caused by using Python in ``/usr/local/opt/bin`` rather than the Frameworks installation (e.g. for ``python 3.10``: ``/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.10``). +For a homebrew installed version of Python, this is generally caused by using Python in ``/usr/local/opt/bin`` rather than the Frameworks installation (e.g. for ``python 3.11``: ``/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.11``). The crux of the issue is that a library Airflow depends on, ``setproctitle``, uses a non-public Python API which is not available from the standard installation ``/usr/local/opt/`` (which symlinks to a path under ``/usr/local/Cellar``). @@ -388,9 +388,9 @@ An easy fix is just to ensure you use a version of Python that has a dylib of th .. code-block:: bash - # Note: these instructions are for python3.10 but can be loosely modified for other versions - brew install [email protected] - virtualenv -p /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/bin/python3 .toy-venv + # Note: these instructions are for python3.11 but can be loosely modified for other versions + brew install [email protected] + virtualenv -p /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.11/bin/python3 .toy-venv source .toy-venv/bin/activate pip install apache-airflow python diff --git a/airflow-core/docs/security/vulnerabilities-in-3rd-party-dependencies.rst b/airflow-core/docs/security/vulnerabilities-in-3rd-party-dependencies.rst index 810a0b3621f..837f4b45f79 100644 --- a/airflow-core/docs/security/vulnerabilities-in-3rd-party-dependencies.rst +++ b/airflow-core/docs/security/vulnerabilities-in-3rd-party-dependencies.rst @@ -141,7 +141,7 @@ used by Airflow and you would like to get rid of those. There are a few things y git clone [email protected]:apache/airflow.git cd airflow ./scripts/tools/setup_breeze - breeze release-management constraints-version-check --python 3.10 --package PACKAGE_NAME --explain-why + breeze release-management constraints-version-check --python 3.11 --package PACKAGE_NAME --explain-why Fragment of example output of such tool is shown below - indicating that ``apache-beam`` blocks upgrade of ``grpcio`` package to version 1.56.0: diff --git a/airflow-core/docs/start.rst b/airflow-core/docs/start.rst index 7768fa153fc..5c12875923a 100644 --- a/airflow-core/docs/start.rst +++ b/airflow-core/docs/start.rst @@ -24,11 +24,11 @@ This quick start guide will help you bootstrap an Airflow standalone instance on .. note:: - Successful installation requires a Python 3 environment. Starting with Airflow 3.2.0, Airflow supports Python 3.10, 3.11, 3.12, 3.13, 3.14. + Successful installation requires a Python 3 environment. Starting with Airflow 3.2.0, Airflow supports Python 3.11, 3.12, 3.13, 3.14. Officially supported installation methods are ``pip`` or ``uv``. - Run ``pip install apache-airflow[EXTRAS]==AIRFLOW_VERSION --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-AIRFLOW_VERSION/constraints-PYTHON_VERSION.txt"``, for example ``pip install "apache-airflow[celery]==3.0.0" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.10.txt"`` to install Airflow in a reproducible way. You can also use - much faster - ``uv`` - by adding ``uv`` before the command. + Run ``pip install apache-airflow[EXTRAS]==AIRFLOW_VERSION --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-AIRFLOW_VERSION/constraints-PYTHON_VERSION.txt"``, for example ``pip install "apache-airflow[celery]==3.0.0" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt"`` to install Airflow in a reproducible way. You can also use - much faster - ``uv`` - by adding ``uv`` before the command. @@ -113,7 +113,7 @@ before running ``pip install`` commands: PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')" CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"; - # For example this would install 3.1.1 with python 3.10: https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.10.txt + # For example this would install 3.1.1 with python 3.11: https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.11.txt uv pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" diff --git a/airflow-ctl/README.md b/airflow-ctl/README.md index 809929b94a2..5cd15ca5513 100644 --- a/airflow-ctl/README.md +++ b/airflow-ctl/README.md @@ -31,7 +31,7 @@ A command-line tool for interacting with Apache Airflow instances through the Ai ## Requirements -- Python 3.10 or later (compatible with 3.10 <= Python <= 3.14) +- Python 3.11 or later (compatible with 3.11 <= Python <= 3.14) - Network access to an Apache Airflow instance with REST API enabled - \[Recommended\] Keyring backend installed in operating system for secure token storage. - In case there's no keyring available (common in headless environments) you can provide the token to each command. See the [Security page](https://airflow.apache.org/docs/apache-airflow-ctl/stable/security.html) for more information. diff --git a/airflow-ctl/docs/installation/prerequisites.rst b/airflow-ctl/docs/installation/prerequisites.rst index 4777b662476..9169ff9e483 100644 --- a/airflow-ctl/docs/installation/prerequisites.rst +++ b/airflow-ctl/docs/installation/prerequisites.rst @@ -65,7 +65,7 @@ use cases. Simply install them to make them available: Python Version Compatibility ---------------------------- -``airflowctl`` is compatible with versions of Python 3.10 through Python 3.14. +``airflowctl`` is compatible with versions of Python 3.11 through Python 3.14. .. list-table:: :widths: 15 85 @@ -73,8 +73,6 @@ Python Version Compatibility * - Python Version - Supported - * - 3.10 - - Yes * - 3.11 - Yes * - 3.12 diff --git a/constraints/README.md b/constraints/README.md index 758048a9006..7fa572d6b35 100644 --- a/constraints/README.md +++ b/constraints/README.md @@ -29,12 +29,12 @@ This allows you to iterate on dependencies without having to run `--upgrade-to-n Typical workflow in this case is: * download and copy the constraint file to the folder (for example via -[The GitHub Raw Link](https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.10.txt)) +[The GitHub Raw Link](https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt)) * modify the constraint file in "constraints" folder * build the image using this command ```bash -breeze ci-image build --python 3.10 --airflow-constraints-location constraints/constraints-3.10.txt +breeze ci-image build --python 3.11 --airflow-constraints-location constraints/constraints-3.11.txt ``` You can continue iterating and updating the constraint file (and rebuilding the image) @@ -46,7 +46,7 @@ pip freeze | sort | \ grep -v "apache_airflow" | \ grep -v "apache-airflow==" | \ grep -v "@" | \ - grep -v "/opt/airflow" > /opt/airflow/constraints/constraints-3.10.txt + grep -v "/opt/airflow" > /opt/airflow/constraints/constraints-3.11.txt ``` If you are working with others on updating the dependencies, you can also commit the constraint diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index ae53fd1e4f9..c51ae89d3fc 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -270,8 +270,8 @@ To avoid burden on our CI infrastructure and to save time, prek hooks can be run We have recently started to recommend ``uv`` for our local development. .. note:: - Remember to have global python set to Python >= 3.10 - Python 3.10 is end-of-life already and we've - started to use Python 3.10+ features in Airflow and accompanying scripts. + Remember to have global python set to Python >= 3.11 - Python 3.10 is end-of-life already and we've + started to use Python 3.11+ features in Airflow and accompanying scripts. Installing prek is best done with ``uv`` (recommended) or ``pipx``. @@ -447,7 +447,7 @@ see in CI in your local environment. .. code-block:: bash - breeze --python 3.10 --backend postgres + breeze --python 3.11 --backend postgres .. note:: If you encounter an error like "docker.credentials.errors.InitializationError: @@ -477,7 +477,7 @@ see in CI in your local environment. .. code-block:: bash - breeze --python 3.10 --backend postgres + breeze --python 3.11 --backend postgres 5. When you enter the Breeze environment you should see a prompt similar to ``[Breeze:3.10.19] root@e4756f6ac886:/opt/airflow#``. This @@ -524,7 +524,7 @@ Using Breeze ------------ 1. Starting the Breeze environment using ``breeze start-airflow`` starts the Breeze environment with last configuration run( - In this case Python version and backend are picked up from last execution ``breeze --python 3.10 --backend postgres``) + In this case Python version and backend are picked up from last execution ``breeze --python 3.11 --backend postgres``) It also automatically starts the API server (FastAPI api and UI), triggerer, dag processor and scheduler. It drops you in tmux with triggerer to the right, and Scheduler, API server (FastAPI api and UI), Dag processor from left to right at the bottom. Use ``[Ctrl + B] and Arrow keys`` to navigate. @@ -535,9 +535,9 @@ Using Breeze Use CI image. Branch name: main - Docker image: ghcr.io/apache/airflow/main/ci/python3.10:latest + Docker image: ghcr.io/apache/airflow/main/ci/python3.11:latest Airflow source version: 2.4.0.dev0 - Python version: 3.10 + Python version: 3.11 Backend: mysql 5.7 * Port forwarding: @@ -574,7 +574,7 @@ Using Breeze .. code-block:: bash - breeze --python 3.10 --backend postgres + breeze --python 3.11 --backend postgres 2. Open tmux @@ -657,7 +657,7 @@ After returning to the host shell, stop the remaining Docker Compose services: breeze down -If ``breeze`` was started with ``breeze --python 3.10 --backend postgres`` (or similar): +If ``breeze`` was started with ``breeze --python 3.11 --backend postgres`` (or similar): .. code-block:: bash @@ -779,20 +779,20 @@ All Tests are inside ./tests directory. .. code-block:: bash - breeze --backend postgres --postgres-version 15 --python 3.10 --db-reset testing tests --test-type All + breeze --backend postgres --postgres-version 15 --python 3.11 --db-reset testing tests --test-type All - Running specific type of test .. code-block:: bash - breeze --backend postgres --postgres-version 15 --python 3.10 --db-reset testing tests --test-type Core + breeze --backend postgres --postgres-version 15 --python 3.11 --db-reset testing tests --test-type Core - Running Integration test for specific test type .. code-block:: bash - breeze --backend postgres --postgres-version 15 --python 3.10 --db-reset testing tests --test-type All --integration mongo + breeze --backend postgres --postgres-version 15 --python 3.11 --db-reset testing tests --test-type All --integration mongo - For more information on Testing visit |09_testing.rst| diff --git a/contributing-docs/05_pull_requests.rst b/contributing-docs/05_pull_requests.rst index e7c3eab2ea0..4052986d8ba 100644 --- a/contributing-docs/05_pull_requests.rst +++ b/contributing-docs/05_pull_requests.rst @@ -154,7 +154,7 @@ these guidelines: you can push your code to PR and see results of the tests in the CI. - You can use any supported python version to run the tests, but the best is to check - if it works for the oldest supported version (Python 3.10 currently). In rare cases + if it works for the oldest supported version (Python 3.11 currently). In rare cases tests might fail with the oldest version when you use features that are available in newer Python versions. For that purpose we have ``airflow.compat`` package where we keep back-ported useful features from newer versions. diff --git a/contributing-docs/07_local_virtualenv.rst b/contributing-docs/07_local_virtualenv.rst index b75db8d4dbd..4729ab5e4ea 100644 --- a/contributing-docs/07_local_virtualenv.rst +++ b/contributing-docs/07_local_virtualenv.rst @@ -36,7 +36,7 @@ Required Software Packages Use system-level package managers like yum, apt-get for Linux, or Homebrew for macOS to install required software packages: -* Python (One of: 3.10, 3.11, 3.12, 3.13, 3.14) +* Python (One of: 3.11, 3.12, 3.13, 3.14) * MySQL 5.7+ * libxml * helm (only for helm chart tests) diff --git a/contributing-docs/08_static_code_checks.rst b/contributing-docs/08_static_code_checks.rst index 9c34dfd16b1..11eee3ebb53 100644 --- a/contributing-docs/08_static_code_checks.rst +++ b/contributing-docs/08_static_code_checks.rst @@ -357,7 +357,7 @@ To show unused mypy ignores, run: prek mypy-airflow-core --all-files For **providers**, mypy still runs via breeze (``breeze run mypy``) as a separate CI job and requires -``breeze ci-image build --python 3.10`` to be built locally. Providers use a separate docker-volume +``breeze ci-image build --python 3.11`` to be built locally. Providers use a separate docker-volume (called ``mypy-cache-volume``) that keeps the cache of last MyPy execution. To clear all mypy caches (the Docker volume used by providers, any legacy repo-root ``.mypy_cache``, diff --git a/contributing-docs/13_airflow_dependencies_and_extras.rst b/contributing-docs/13_airflow_dependencies_and_extras.rst index 1a64d5e2ca9..794b835c086 100644 --- a/contributing-docs/13_airflow_dependencies_and_extras.rst +++ b/contributing-docs/13_airflow_dependencies_and_extras.rst @@ -355,7 +355,7 @@ from the PyPI package: .. code-block:: bash pip install "apache-airflow[google,amazon,async]==3.0.0" \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.10.txt"; + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt"; The last one can be used to install Airflow in "minimal" mode - i.e when bare Airflow is installed without extras. @@ -367,7 +367,7 @@ requirements). .. code-block:: bash pip install -e ".[devel]" \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.10.txt"; + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.11.txt"; This also works with extras - for example: @@ -375,7 +375,7 @@ This also works with extras - for example: .. code-block:: bash pip install ".[ssh]" \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.10.txt"; + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.11.txt"; There are different set of fixed constraint files for different python major/minor versions and you should @@ -387,7 +387,7 @@ using ``constraints-no-providers`` constraint files as well. .. code-block:: bash pip install . --upgrade \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.10.txt"; + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.11.txt"; These constraint files are generated from the ``uv.lock`` file committed in the repository, using diff --git a/contributing-docs/testing/docker_compose_tests.rst b/contributing-docs/testing/docker_compose_tests.rst index 8c8dcb8306c..a225264323d 100644 --- a/contributing-docs/testing/docker_compose_tests.rst +++ b/contributing-docs/testing/docker_compose_tests.rst @@ -50,7 +50,7 @@ Running complete test with breeze: .. code-block:: bash - breeze prod-image build --python 3.10 + breeze prod-image build --python 3.11 breeze testing docker-compose-tests In case the test fails, it will dump the logs from the running containers to the console and it @@ -67,8 +67,8 @@ to see the output of the test as it happens (it can be also set via The test can be also run manually with ``pytest docker_tests/test_docker_compose_quick_start.py`` command, provided that you have a local Airflow venv with ``dev`` extra set and the ``DOCKER_IMAGE`` environment variable is set to the image you want to test. The variable defaults -to ``ghcr.io/apache/airflow/main/prod/python3.10:latest`` which is built by default -when you run ``breeze prod-image build --python 3.10``. also the switches ``--skip-docker-compose-deletion`` +to ``ghcr.io/apache/airflow/main/prod/python3.11:latest`` which is built by default +when you run ``breeze prod-image build --python 3.11``. also the switches ``--skip-docker-compose-deletion`` and ``--wait-for-containers-timeout`` can only be passed via environment variables. If you want to debug the deployment using ``docker compose`` commands after ``SKIP_DOCKER_COMPOSE_DELETION`` @@ -89,7 +89,7 @@ the prod image build command above. .. code-block:: bash - export AIRFLOW_IMAGE_NAME=ghcr.io/apache/airflow/main/prod/python3.10:latest + export AIRFLOW_IMAGE_NAME=ghcr.io/apache/airflow/main/prod/python3.11:latest and follow the instructions in the `Running Airflow in Docker <https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html>`_ diff --git a/contributing-docs/testing/k8s_tests.rst b/contributing-docs/testing/k8s_tests.rst index f21774e999a..f0b121bcee7 100644 --- a/contributing-docs/testing/k8s_tests.rst +++ b/contributing-docs/testing/k8s_tests.rst @@ -49,7 +49,7 @@ per each combination of Python and Kubernetes version. This is used during CI wh tests against those different clusters - even in parallel. The cluster name follows the pattern ``airflow-python-X.Y-vA.B.C`` where X.Y is a major/minor Python version -and A.B.C is Kubernetes version. Example cluster name: ``airflow-python-3.10-v1.24.0`` +and A.B.C is Kubernetes version. Example cluster name: ``airflow-python-3.11-v1.24.0`` Most of the commands can be executed in parallel for multiple images/clusters by adding ``--run-in-parallel`` to create clusters or deploy airflow. Similarly checking for status, dumping logs and deleting clusters @@ -271,7 +271,7 @@ The typical session for tests with Kubernetes looks like follows: .. code-block:: text - Config created in /Users/jarek/IdeaProjects/airflow/.build/.k8s-clusters/airflow-python-3.10-v1.24.2/.kindconfig.yaml: + Config created in /Users/jarek/IdeaProjects/airflow/.build/.k8s-clusters/airflow-python-3.11-v1.24.2/.kindconfig.yaml: kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 @@ -288,7 +288,7 @@ The typical session for tests with Kubernetes looks like follows: listenAddress: "127.0.0.1" protocol: TCP - Creating cluster "airflow-python-3.10-v1.24.2" ... + Creating cluster "airflow-python-3.11-v1.24.2" ... ✓ Ensuring node image (kindest/node:v1.24.2) 🖼 ✓ Preparing nodes 📦 📦 ✓ Writing configuration 📜 @@ -296,10 +296,10 @@ The typical session for tests with Kubernetes looks like follows: ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜 - Set kubectl context to "kind-airflow-python-3.10-v1.24.2" + Set kubectl context to "kind-airflow-python-3.11-v1.24.2" You can now use your cluster with: - kubectl cluster-info --context kind-airflow-python-3.10-v1.24.2 + kubectl cluster-info --context kind-airflow-python-3.11-v1.24.2 Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/ @@ -307,9 +307,9 @@ The typical session for tests with Kubernetes looks like follows: Connecting to localhost:18150. Num try: 1 Error when connecting to localhost:18150 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) - Airflow webserver is not available at port 18150. Run `breeze k8s deploy-airflow --python 3.10 --kubernetes-version v1.24.2` to (re)deploy airflow + Airflow webserver is not available at port 18150. Run `breeze k8s deploy-airflow --python 3.11 --kubernetes-version v1.24.2` to (re)deploy airflow - KinD cluster airflow-python-3.10-v1.24.2 created! + KinD cluster airflow-python-3.11-v1.24.2 created! NEXT STEP: You might now configure your cluster by: @@ -323,20 +323,20 @@ The typical session for tests with Kubernetes looks like follows: .. code-block:: text - Configuring airflow-python-3.10-v1.24.2 to be ready for Airflow deployment - Deleting K8S namespaces for kind-airflow-python-3.10-v1.24.2 + Configuring airflow-python-3.11-v1.24.2 to be ready for Airflow deployment + Deleting K8S namespaces for kind-airflow-python-3.11-v1.24.2 Error from server (NotFound): namespaces "airflow" not found Error from server (NotFound): namespaces "test-namespace" not found Creating namespaces namespace/airflow created namespace/test-namespace created - Created K8S namespaces for cluster kind-airflow-python-3.10-v1.24.2 + Created K8S namespaces for cluster kind-airflow-python-3.11-v1.24.2 - Deploying test resources for cluster kind-airflow-python-3.10-v1.24.2 + Deploying test resources for cluster kind-airflow-python-3.11-v1.24.2 persistentvolume/test-volume created persistentvolumeclaim/test-volume created service/airflow-webserver-node-port created - Deployed test resources for cluster kind-airflow-python-3.10-v1.24.2 + Deployed test resources for cluster kind-airflow-python-3.11-v1.24.2 NEXT STEP: You might now build your k8s image by: @@ -354,45 +354,45 @@ The typical session for tests with Kubernetes looks like follows: .. code-block:: text ======================================================================================================================== - Cluster: airflow-python-3.10-v1.24.2 + Cluster: airflow-python-3.11-v1.24.2 - * KUBECONFIG=/Users/jarek/IdeaProjects/airflow/.build/.k8s-clusters/airflow-python-3.10-v1.24.2/.kubeconfig - * KINDCONFIG=/Users/jarek/IdeaProjects/airflow/.build/.k8s-clusters/airflow-python-3.10-v1.24.2/.kindconfig.yaml + * KUBECONFIG=/Users/jarek/IdeaProjects/airflow/.build/.k8s-clusters/airflow-python-3.11-v1.24.2/.kubeconfig + * KINDCONFIG=/Users/jarek/IdeaProjects/airflow/.build/.k8s-clusters/airflow-python-3.11-v1.24.2/.kindconfig.yaml - Cluster info: airflow-python-3.10-v1.24.2 + Cluster info: airflow-python-3.11-v1.24.2 Kubernetes control plane is running at https://127.0.0.1:48366 CoreDNS is running at https://127.0.0.1:48366/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. - Storage class for airflow-python-3.10-v1.24.2 + Storage class for airflow-python-3.11-v1.24.2 NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 83s - Running pods for airflow-python-3.10-v1.24.2 + Running pods for airflow-python-3.11-v1.24.2 NAME READY STATUS RESTARTS AGE coredns-6d4b75cb6d-rwp9d 1/1 Running 0 71s coredns-6d4b75cb6d-vqnrc 1/1 Running 0 71s - etcd-airflow-python-3.10-v1.24.2-control-plane 1/1 Running 0 84s + etcd-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s kindnet-ckc8l 1/1 Running 0 69s kindnet-qqt8k 1/1 Running 0 71s - kube-apiserver-airflow-python-3.10-v1.24.2-control-plane 1/1 Running 0 84s - kube-controller-manager-airflow-python-3.10-v1.24.2-control-plane 1/1 Running 0 84s + kube-apiserver-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s + kube-controller-manager-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s kube-proxy-6g7hn 1/1 Running 0 69s kube-proxy-dwfvp 1/1 Running 0 71s - kube-scheduler-airflow-python-3.10-v1.24.2-control-plane 1/1 Running 0 84s + kube-scheduler-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s KinD Cluster API server URL: http://localhost:48366 Connecting to localhost:18150. Num try: 1 Error when connecting to localhost:18150 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) - Airflow webserver is not available at port 18150. Run `breeze k8s deploy-airflow --python 3.10 --kubernetes-version v1.24.2` to (re)deploy airflow + Airflow webserver is not available at port 18150. Run `breeze k8s deploy-airflow --python 3.11 --kubernetes-version v1.24.2` to (re)deploy airflow - Cluster healthy: airflow-python-3.10-v1.24.2 + Cluster healthy: airflow-python-3.11-v1.24.2 #. Build the image base on PROD Airflow image. You need to build the PROD image first (the command will guide you if you did not) either by running the build separately or passing ``--rebuild-base-image`` @@ -410,15 +410,15 @@ The typical session for tests with Kubernetes looks like follows: .. code-block:: text - Building the K8S image for Python 3.10 using Airflow base image: ghcr.io/apache/airflow/main/prod/python3.10:latest + Building the K8S image for Python 3.11 using Airflow base image: ghcr.io/apache/airflow/main/prod/python3.11:latest [+] Building 0.1s (8/8) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 301B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 35B 0.0s - => [internal] load metadata for ghcr.io/apache/airflow/main/prod/python3.10:latest 0.0s - => [1/3] FROM ghcr.io/apache/airflow/main/prod/python3.10:latest 0.0s + => [internal] load metadata for ghcr.io/apache/airflow/main/prod/python3.11:latest 0.0s + => [1/3] FROM ghcr.io/apache/airflow/main/prod/python3.11:latest 0.0s => [internal] load build context 0.0s => => transferring context: 3.00kB 0.0s => CACHED [2/3] COPY airflow/example_dags/ /opt/airflow/dags/ 0.0s @@ -426,7 +426,7 @@ The typical session for tests with Kubernetes looks like follows: => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:c0bdd363c549c3b0731b8e8ce34153d081f239ee2b582355b7b3ffd5394c40bb 0.0s - => => naming to ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:latest + => => naming to ghcr.io/apache/airflow/main/prod/python3.11-kubernetes:latest NEXT STEP: You might now upload your k8s image by: @@ -445,9 +445,9 @@ The typical session for tests with Kubernetes looks like follows: Good version of kubectl installed: 1.25.0 in /Users/jarek/IdeaProjects/airflow/kubernetes-tests/.venv/bin Good version of helm installed: 3.9.2 in /Users/jarek/IdeaProjects/airflow/kubernetes-tests/.venv/bin Stable repo is already added - Uploading Airflow image ghcr.io/apache/airflow/main/prod/python3.10-kubernetes to cluster airflow-python-3.10-v1.24.2 - Image: "ghcr.io/apache/airflow/main/prod/python3.10-kubernetes" with ID "sha256:fb6195f7c2c2ad97788a563a3fe9420bf3576c85575378d642cd7985aff97412" not yet present on node "airflow-python-3.10-v1.24.2-worker", loading... - Image: "ghcr.io/apache/airflow/main/prod/python3.10-kubernetes" with ID "sha256:fb6195f7c2c2ad97788a563a3fe9420bf3576c85575378d642cd7985aff97412" not yet present on node "airflow-python-3.10-v1.24.2-control-plane", loading... + Uploading Airflow image ghcr.io/apache/airflow/main/prod/python3.11-kubernetes to cluster airflow-python-3.11-v1.24.2 + Image: "ghcr.io/apache/airflow/main/prod/python3.11-kubernetes" with ID "sha256:fb6195f7c2c2ad97788a563a3fe9420bf3576c85575378d642cd7985aff97412" not yet present on node "airflow-python-3.11-v1.24.2-worker", loading... + Image: "ghcr.io/apache/airflow/main/prod/python3.11-kubernetes" with ID "sha256:fb6195f7c2c2ad97788a563a3fe9420bf3576c85575378d642cd7985aff97412" not yet present on node "airflow-python-3.11-v1.24.2-control-plane", loading... NEXT STEP: You might now deploy Airflow by: @@ -461,8 +461,8 @@ The typical session for tests with Kubernetes looks like follows: .. code-block:: text - Deploying Airflow for cluster airflow-python-3.10-v1.24.2 - Deploying kind-airflow-python-3.10-v1.24.2 with Airflow Helm Chart. + Deploying Airflow for cluster airflow-python-3.11-v1.24.2 + Deploying kind-airflow-python-3.11-v1.24.2 with Airflow Helm Chart. Copied chart sources to /private/var/folders/v3/gvj4_mw152q556w2rrh7m46w0000gn/T/chart_edu__kir/chart Deploying Airflow from /private/var/folders/v3/gvj4_mw152q556w2rrh7m46w0000gn/T/chart_edu__kir/chart NAME: airflow @@ -504,12 +504,12 @@ The typical session for tests with Kubernetes looks like follows: Information on how to set a static webserver secret key can be found here: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#api-secret-key - Deployed kind-airflow-python-3.10-v1.24.2 with Airflow Helm Chart. + Deployed kind-airflow-python-3.11-v1.24.2 with Airflow Helm Chart. - Airflow for Python 3.10 and K8S version v1.24.2 has been successfully deployed. + Airflow for Python 3.11 and K8S version v1.24.2 has been successfully deployed. - The KinD cluster name: airflow-python-3.10-v1.24.2 - The kubectl cluster name: kind-airflow-python-3.10-v1.24.2. + The KinD cluster name: airflow-python-3.11-v1.24.2 + The kubectl cluster name: kind-airflow-python-3.11-v1.24.2. KinD Cluster API server URL: http://localhost:48366 @@ -552,7 +552,7 @@ The typical session for tests with Kubernetes looks like follows: Good version of helm installed: 3.16.4 in /Users/jarek/IdeaProjects/airflow/kubernetes-tests/.venv/bin Stable repo is already added - Running tests with kind-airflow-python-3.10-v1.29.12 cluster. + Running tests with kind-airflow-python-3.11-v1.29.12 cluster. Command to run: uv run pytest kubernetes-tests/tests/ Installed 74 packages in 179ms /Users/jarek/IdeaProjects/airflow/.venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset. @@ -586,7 +586,7 @@ The typical session for tests with Kubernetes looks like follows: Entering interactive k8s shell. - (kind-airflow-python-3.10-v1.24.2:KubernetesExecutor)> + (kind-airflow-python-3.11-v1.24.2:KubernetesExecutor)> In a separate terminal you can open the k9s CLI: @@ -636,7 +636,7 @@ The typical session for tests with Kubernetes looks like follows: echo ${KUBECONFIG} - /home/jarek/code/airflow/.build/.k8s-clusters/airflow-python-3.10-v1.28.13/.kube/config + /home/jarek/code/airflow/.build/.k8s-clusters/airflow-python-3.11-v1.28.13/.kube/config .. image:: images/kubeconfig-env.png :align: center @@ -704,9 +704,9 @@ The typical session for tests with Kubernetes looks like follows: .. code-block:: text - Deleting KinD cluster airflow-python-3.10-v1.24.2! - Deleting cluster "airflow-python-3.10-v1.24.2" ... - KinD cluster airflow-python-3.10-v1.24.2 deleted! + Deleting KinD cluster airflow-python-3.11-v1.24.2! + Deleting cluster "airflow-python-3.11-v1.24.2" ... + KinD cluster airflow-python-3.11-v1.24.2 deleted! Running complete k8s tests diff --git a/contributing-docs/testing/task_sdk_integration_tests.rst b/contributing-docs/testing/task_sdk_integration_tests.rst index fc629b0f609..df9ad268d23 100644 --- a/contributing-docs/testing/task_sdk_integration_tests.rst +++ b/contributing-docs/testing/task_sdk_integration_tests.rst @@ -88,7 +88,7 @@ Then, you should build the base image once before running the tests. You can do .. code-block:: bash # From the Airflow repository root - breeze prod-image build --python 3.10 + breeze prod-image build --python 3.11 The first build may take a while as it needs to download base image, build Python, install dependencies and set up the environment. Subsequent builds will be much faster as they will use cached layers. @@ -99,7 +99,7 @@ If you use ``breeze`` to run the integration tests and you do not have the image ``breeze`` will prompt you to build it, and the building will proceed automatically after 20 seconds if you do not answer ``no``. -This will build the right image ``ghcr.io/apache/airflow/main/prod/python3.10.latest`` (with the right +This will build the right image ``ghcr.io/apache/airflow/main/prod/python3.11.latest`` (with the right Python version) that will be used to run the tests. The ``breeze prod-image build`` command by default - when run from sources of airflow - will use the local sources and build the image using ``uv`` to speed up the build process. Also, when building from sources it will check if the assets are built diff --git a/contributing-docs/testing/unit_tests.rst b/contributing-docs/testing/unit_tests.rst index faa145bb7a4..77e036d1941 100644 --- a/contributing-docs/testing/unit_tests.rst +++ b/contributing-docs/testing/unit_tests.rst @@ -215,7 +215,7 @@ rerun in Breeze as needed (``-n auto`` will parallelize tests using the ``pytest .. code-block:: bash - breeze shell --backend none --python 3.10 + breeze shell --backend none --python 3.11 > pytest airflow-core/tests --skip-db-tests -n auto .. AGENT-SKILL-START @@ -261,7 +261,7 @@ out non-DB tests. (You can specify the whole ``tests`` directory or any specific You can also run DB tests within the ``breeze`` dockerized environment. You can choose the backend with the ``--backend`` flag. The default is ``sqlite``, but you can also use ``postgres`` or ``mysql``. You can also select the backend version and Python version. Breeze will list the available test types via ``--help`` and provide auto-complete. -The example below runs ``Core`` tests with the ``postgres`` backend and Python ``3.10``: +The example below runs ``Core`` tests with the ``postgres`` backend and Python ``3.11``: You can also run the commands via ``breeze testing core-tests`` or ``breeze testing providers-tests`` by adding the parallel flags manually: @@ -280,14 +280,14 @@ If you want to iterate on tests, you can enter the interactive shell and run tes .. code-block:: bash - breeze shell --backend postgres --python 3.10 + breeze shell --backend postgres --python 3.11 > pytest airflow-core/tests --run-db-tests-only As explained before, you cannot run DB tests in parallel using the ``pytest-xdist`` plugin. However, ``breeze`` supports splitting all tests into test-types to run in separate containers with separate databases using the ``--run-in-parallel`` flag. .. code-block:: bash - breeze testing core-tests --run-db-tests-only --backend postgres --python 3.10 --run-in-parallel + breeze testing core-tests --run-db-tests-only --backend postgres --python 3.11 --run-in-parallel .. AGENT-SKILL-START type: agents-md-commands @@ -1254,7 +1254,7 @@ Here is how to reproduce it: .. code-block:: bash - breeze ci-image build --python 3.10 + breeze ci-image build --python 3.11 2. Build providers from latest sources: diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md index 8516657ab9d..425366f2968 100644 --- a/dev/README_RELEASE_AIRFLOW.md +++ b/dev/README_RELEASE_AIRFLOW.md @@ -1289,7 +1289,7 @@ Optionally it can be followed with constraints ```shell script pip install apache-airflow==<VERSION>rc<X> \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>rc<X>/constraints-3.10.txt" + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>rc<X>/constraints-3.11.txt" ``` Note that the constraints contain python version that you are installing it with. @@ -1301,7 +1301,7 @@ There is also an easy way of installation with Breeze if you have the latest sou Running the following command will use tmux inside breeze, create `admin` user and run Webserver & Scheduler: ```shell script -breeze start-airflow --use-airflow-version 3.1.3rc1 --python 3.10 --backend postgres +breeze start-airflow --use-airflow-version 3.1.3rc1 --python 3.11 --backend postgres ``` You can also choose different executors and extras to install when you are installing airflow this way. For @@ -1309,7 +1309,7 @@ example in order to run Airflow with CeleryExecutor and install celery, google a Airflow 2.7.0, you need to have celery provider installed to run Airflow with CeleryExecutor) you can run: ```shell script -breeze start-airflow --use-airflow-version 3.1.3rc1 --python 3.10 --backend postgres \ +breeze start-airflow --use-airflow-version 3.1.3rc1 --python 3.11 --backend postgres \ --executor CeleryExecutor --airflow-extras "celery,google,amazon" ``` @@ -1476,7 +1476,7 @@ the older branches, you should set the "skip" field to true. ## Verify production images ```shell script -for PYTHON in 3.10 3.11 3.12 3.13 3.14 +for PYTHON in 3.11 3.12 3.13 3.14 do docker pull apache/airflow:${VERSION}-python${PYTHON} breeze prod-image verify --image-name apache/airflow:${VERSION}-python${PYTHON} diff --git a/dev/README_RELEASE_PROVIDERS.md b/dev/README_RELEASE_PROVIDERS.md index 11fb728286f..be9b0e7cb62 100644 --- a/dev/README_RELEASE_PROVIDERS.md +++ b/dev/README_RELEASE_PROVIDERS.md @@ -1649,7 +1649,7 @@ pip install apache-airflow-providers-<provider>==<VERSION>rc<X> You can use any Airflow 3.X.Y version, like 3.2.0, to install specific version for testing, using breeze. ```shell -breeze start-airflow --use-airflow-version 3.1.3 --python 3.10 --backend postgres \ +breeze start-airflow --use-airflow-version 3.1.3 --python 3.11 --backend postgres \ --load-example-dags --load-default-connections ``` diff --git a/dev/README_RELEASE_PYTHON_CLIENT.md b/dev/README_RELEASE_PYTHON_CLIENT.md index 88e2a8b4d14..c2e5036750a 100644 --- a/dev/README_RELEASE_PYTHON_CLIENT.md +++ b/dev/README_RELEASE_PYTHON_CLIENT.md @@ -526,7 +526,7 @@ Both commands should produce reproducible `.whl`, `.tar.gz` packages in dist fol file containing airflow sources in dist folder. > [!IMPORTANT] -> Run the build with Python 3.10 — the project's `DEFAULT_PYTHON_MAJOR_MINOR_VERSION`. The +> Run the build with Python 3.11 — the project's `DEFAULT_PYTHON_MAJOR_MINOR_VERSION`. The > client generator applies the `trigger_dag_run_post_body.py` AST patch with > `ast.unparse`, which > re-emits that file using the running interpreter's grammar, so building > under a different Python > (e.g. the host's 3.11+/3.13) produces a non-reproducible client and the > `prepare-python-client` @@ -534,9 +534,9 @@ file containing airflow sources in dist folder. > Python and exits early with this guidance, so pin the interpreter explicitly: > > ```shell -> UV_PYTHON=3.10 breeze release-management prepare-python-client --distribution-format both --version-suffix "" +> UV_PYTHON=3.11 breeze release-management prepare-python-client --distribution-format both --version-suffix "" > # or equivalently -> breeze --python 3.10 release-management prepare-python-client --distribution-format both --version-suffix "" +> breeze --python 3.11 release-management prepare-python-client --distribution-format both --version-suffix "" > ``` 4) Change to the directory where you have the packages from svn and check if they are identical to the ones diff --git a/dev/breeze/doc/01_installation.rst b/dev/breeze/doc/01_installation.rst index 72f3e7bf99e..a7f84982b3e 100644 --- a/dev/breeze/doc/01_installation.rst +++ b/dev/breeze/doc/01_installation.rst @@ -399,7 +399,7 @@ that Breeze works on .. code-block:: bash - UV_PYTHON=3.10 breeze ... + UV_PYTHON=3.11 breeze ... or set it permanently in your shell rc. ``uvx`` will rebuild its cached environment with that interpreter on the next call. diff --git a/dev/breeze/doc/03_developer_tasks.rst b/dev/breeze/doc/03_developer_tasks.rst index b94b4b01727..fe0f9f0c48c 100644 --- a/dev/breeze/doc/03_developer_tasks.rst +++ b/dev/breeze/doc/03_developer_tasks.rst @@ -34,12 +34,12 @@ You can use additional ``breeze`` flags to choose your environment. You can spec version to use, and backend (the meta-data database). Thanks to that, with Breeze, you can recreate the same environments as we have in matrix builds in the CI. See next chapter for backend selection. -For example, you can choose to run Python 3.10 tests with MySQL as backend and with mysql version 8 +For example, you can choose to run Python 3.11 tests with MySQL as backend and with mysql version 8 as follows: .. code-block:: bash - breeze --python 3.10 --backend mysql --mysql-version 8.0 + breeze --python 3.11 --backend mysql --mysql-version 8.0 .. note:: Note for Windows WSL2 users @@ -57,9 +57,9 @@ with ``--builder``. For example: .. code-block:: bash - breeze --docker-host unix://$HOME/.colima/default/docker.sock --python 3.10 --backend mysql --mysql-version 8.0 + breeze --docker-host unix://$HOME/.colima/default/docker.sock --python 3.11 --backend mysql --mysql-version 8.0 # or: - breeze --builder=default --python 3.10 --backend mysql --mysql-version 8.0 + breeze --builder=default --python 3.11 --backend mysql --mysql-version 8.0 The choices you make are persisted in the ``./.build/`` cache directory so that next time when you use the ``breeze`` script, it could use the values that were used previously. This way you do not have to specify @@ -486,7 +486,7 @@ When you are starting Airflow from local sources, www asset compilation is autom .. code-block:: bash - breeze --python 3.10 --backend mysql start-airflow + breeze --python 3.11 --backend mysql start-airflow You can also use it to start different executor. @@ -499,7 +499,7 @@ You can also use it to start any released version of Airflow from ``PyPI`` with .. code-block:: bash - breeze start-airflow --python 3.10 --backend mysql --use-airflow-version 2.7.0 + breeze start-airflow --python 3.11 --backend mysql --use-airflow-version 2.7.0 When you are installing version from PyPI, it's also possible to specify extras that should be used when installing Airflow - you can provide several extras separated by coma - for example to install @@ -527,13 +527,13 @@ To start Airflow with an integration, use the following command: .. code-block:: bash - breeze --python 3.10 --backend postgres --integration <integration_name> + breeze --python 3.11 --backend postgres --integration <integration_name> For example, to run Airflow with Kafka: .. code-block:: bash - breeze --python 3.10 --backend postgres --integration kafka + breeze --python 3.11 --backend postgres --integration kafka Check the available integrations by running: diff --git a/dev/breeze/doc/04_troubleshooting.rst b/dev/breeze/doc/04_troubleshooting.rst index eaf357e624f..cf4ba231100 100644 --- a/dev/breeze/doc/04_troubleshooting.rst +++ b/dev/breeze/doc/04_troubleshooting.rst @@ -94,7 +94,7 @@ describe your problem. stated in `This comment <https://github.com/moby/moby/issues/43361#issuecomment-1227617516>`_ and allows to run Breeze with no problems. -Cannot import name 'cache' or Python >=3.10 required +Cannot import name 'cache' or Python >=3.11 required --------------------------------------------------- When you see this error: @@ -107,7 +107,7 @@ or .. code-block:: - ERROR: Package 'blacken-docs' requires a different Python: 3.8.18 not in '>=3.10' + ERROR: Package 'blacken-docs' requires a different Python: 3.8.18 not in '>=3.11' It means that your prek hook is installed with (already End-Of-Life) Python 3.8 and you should reinstall @@ -118,7 +118,7 @@ This can be done with ``uv tool`` to install ``prek``) .. code-block:: bash uv tool uninstall prek - uv tool install prek --python 3.10 --force + uv tool install prek --python 3.11 --force prek clean prek install @@ -127,7 +127,7 @@ You can also use ``pipx`` .. code-block:: bash pipx uninstall prek - pipx install prek --python $(which python3.10) --force + pipx install prek --python $(which python3.11) --force prek clean prek install diff --git a/dev/breeze/doc/05_test_commands.rst b/dev/breeze/doc/05_test_commands.rst index 68d749d215f..db45622671c 100644 --- a/dev/breeze/doc/05_test_commands.rst +++ b/dev/breeze/doc/05_test_commands.rst @@ -793,7 +793,7 @@ as executor you use, similar to: .. code-block:: bash - (kind-airflow-python-3.10-v1.24.0:KubernetesExecutor)> + (kind-airflow-python-3.11-v1.24.0:KubernetesExecutor)> The shell automatically activates the virtual environment that has all appropriate dependencies @@ -802,7 +802,7 @@ be created and Airflow deployed to it before running the tests): .. code-block:: bash - (kind-airflow-python-3.10-v1.24.0:KubernetesExecutor)> pytest test_kubernetes_executor.py + (kind-airflow-python-3.11-v1.24.0:KubernetesExecutor)> pytest test_kubernetes_executor.py ================================================= test session starts ================================================= platform linux -- Python 3.10.6, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /home/jarek/code/airflow/kubernetes-tests/.venv/bin/python cachedir: .pytest_cache @@ -814,14 +814,14 @@ be created and Airflow deployed to it before running the tests): test_kubernetes_executor.py::TestKubernetesExecutor::test_integration_run_dag_with_scheduler_failure PASSED [100%] ================================================== warnings summary =================================================== - kubernetes-tests/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py:1233 - /home/jarek/code/airflow/kubernetes-tests/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: asyncio_mode + kubernetes-tests/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py:1233 + /home/jarek/code/airflow/kubernetes-tests/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: asyncio_mode self._warn_or_fail_if_strict(f"Unknown config option: {key}\n") -- Docs: https://docs.pytest.org/en/stable/warnings.html ============================================ 2 passed, 1 warning in 38.62s ============================================ - (kind-airflow-python-3.10-v1.24.0:KubernetesExecutor)> + (kind-airflow-python-3.11-v1.24.0:KubernetesExecutor)> All parameters of the command are here: diff --git a/dev/breeze/doc/06_managing_docker_images.rst b/dev/breeze/doc/06_managing_docker_images.rst index b64fc2f0f8a..4531f8c229d 100644 --- a/dev/breeze/doc/06_managing_docker_images.rst +++ b/dev/breeze/doc/06_managing_docker_images.rst @@ -120,13 +120,13 @@ To load the image from specific PR, you can use the following command: .. code-block:: bash - breeze ci-image load --from-pr 12345 --python 3.10 --github-token <your_github_token> + breeze ci-image load --from-pr 12345 --python 3.11 --github-token <your_github_token> To load the image from specific job run (for example 12538475388), you can use the following command, find the run id from github action runs. .. code-block:: bash - breeze ci-image load --from-run 12538475388 --python 3.10 --github-token <your_github_token> + breeze ci-image load --from-run 12538475388 --python 3.11 --github-token <your_github_token> After you load the image, you can reproduce the very exact environment that was used in the CI run by entering breeze container without mounting your local sources: @@ -220,10 +220,10 @@ suffix and they need to also be paired with corresponding runtime dependency add .. code-block:: bash - breeze prod-image build --python 3.10 --additional-dev-deps "libasound2-dev" \ + breeze prod-image build --python 3.11 --additional-dev-deps "libasound2-dev" \ --additional-runtime-apt-deps "libasound2" -Same as above but uses python 3.10. +Same as above but uses python 3.11. Building PROD image ................... diff --git a/dev/breeze/doc/09_release_management_tasks.rst b/dev/breeze/doc/09_release_management_tasks.rst index 52ec6559771..9174d252bae 100644 --- a/dev/breeze/doc/09_release_management_tasks.rst +++ b/dev/breeze/doc/09_release_management_tasks.rst @@ -1236,7 +1236,7 @@ Example usage: .. code-block:: bash - breeze release-management constraints-version-check --python 3.10 --airflow-constraints-mode constraints-source-providers --explain-why + breeze release-management constraints-version-check --python 3.11 --airflow-constraints-mode constraints-source-providers --explain-why ----- diff --git a/dev/breeze/doc/14_advanced_breeze_topics.rst b/dev/breeze/doc/14_advanced_breeze_topics.rst index 4009c5fe1ba..4168fa58620 100644 --- a/dev/breeze/doc/14_advanced_breeze_topics.rst +++ b/dev/breeze/doc/14_advanced_breeze_topics.rst @@ -49,7 +49,7 @@ make sure to follow these steps: this will bypass the check we run in Breeze to see if there are new requirements to install for it See example configuration for PyCharm which has run/debug configuration for -``breeze sbom generate-providers-requirements --provider-id sqlite --python 3.10`` +``breeze sbom generate-providers-requirements --provider-id sqlite --python 3.11`` .. raw:: html diff --git a/dev/breeze/doc/ci/02_images.md b/dev/breeze/doc/ci/02_images.md index ff50ee6954a..265c26cc49f 100644 --- a/dev/breeze/doc/ci/02_images.md +++ b/dev/breeze/doc/ci/02_images.md @@ -129,17 +129,17 @@ The images are built with default extras - different extras for CI and production image and you can change the extras via the `--airflow-extras` parameters and add new ones with `--additional-airflow-extras`. -For example if you want to build Python 3.10 version of production image +For example if you want to build Python 3.11 version of production image with "all" extras installed you should run this command: ``` bash -breeze prod-image build --python 3.10 --airflow-extras "all" +breeze prod-image build --python 3.11 --airflow-extras "all" ``` If you just want to add new extras you can add them like that: ``` bash -breeze prod-image build --python 3.10 --additional-airflow-extras "all" +breeze prod-image build --python 3.11 --additional-airflow-extras "all" ``` The command that builds the CI image is optimized to minimize the time @@ -168,7 +168,7 @@ You can also build production images from PIP packages via providing `--install-airflow-version` parameter to Breeze: ``` bash -breeze prod-image build --python 3.10 --additional-airflow-extras=trino --install-airflow-version=2.0.0 +breeze prod-image build --python 3.11 --additional-airflow-extras=trino --install-airflow-version=2.0.0 ``` This will build the image using command similar to: @@ -176,7 +176,7 @@ This will build the image using command similar to: ``` bash pip install \ apache-airflow[async,amazon,celery,cncf.kubernetes,docker,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv]==2.0.0 \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.10.txt"; + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.11.txt"; ``` > [!NOTE] @@ -207,7 +207,7 @@ HEAD of development for constraints): ``` bash pip install "https://github.com/apache/airflow/archive/<tag>.tar.gz#egg=apache-airflow" \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.10.txt"; + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt"; ``` You can also skip installing airflow and install it from locally @@ -215,7 +215,7 @@ provided files by using `--install-distributions-from-context` parameter to Breeze: ``` bash -breeze prod-image build --python 3.10 --additional-airflow-extras=trino --install-distributions-from-context +breeze prod-image build --python 3.11 --additional-airflow-extras=trino --install-distributions-from-context ``` In this case you airflow and all packages (.whl files) should be placed @@ -251,20 +251,20 @@ flags: `registry` (default), `local`, or `disabled` flags when you run Breeze commands. For example: ``` bash -breeze ci-image build --python 3.10 --docker-cache local +breeze ci-image build --python 3.11 --docker-cache local ``` Will build the CI image using local build cache (note that it will take quite a long time the first time you run it). ``` bash -breeze prod-image build --python 3.10 --docker-cache registry +breeze prod-image build --python 3.11 --docker-cache registry ``` Will build the production image with cache used from registry. ``` bash -breeze prod-image build --python 3.10 --docker-cache disabled +breeze prod-image build --python 3.11 --docker-cache disabled ``` Will build the production image from the scratch. @@ -367,7 +367,7 @@ you can build the image in the Here just a few examples are presented which should give you general understanding of what you can customize. -This builds the production image in version 3.10 with additional airflow +This builds the production image in version 3.11 with additional airflow extras from 2.0.0 PyPI package and additional apt dev and runtime dependencies. @@ -379,7 +379,7 @@ plugin installed. ``` bash DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \ --pull \ - --build-arg PYTHON_BASE_IMAGE="python:3.10-slim-bookworm" \ + --build-arg PYTHON_BASE_IMAGE="python:3.11-slim-bookworm" \ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" \ --build-arg ADDITIONAL_PYTHON_DEPS="pandas" \ --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" \ @@ -390,7 +390,7 @@ the same image can be built using `breeze` (it supports auto-completion of the options): ``` bash -breeze ci-image build --python 3.10 --additional-airflow-extras=jdbc --additional-python-deps="pandas" \ +breeze ci-image build --python 3.11 --additional-airflow-extras=jdbc --additional-python-deps="pandas" \ --additional-dev-apt-deps="gcc g++" ``` @@ -404,7 +404,7 @@ comment](https://github.com/apache/airflow/issues/8605#issuecomment-690065621): ``` bash DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \ --pull \ - --build-arg PYTHON_BASE_IMAGE="python:3.10-slim-bookworm" \ + --build-arg PYTHON_BASE_IMAGE="python:3.11-slim-bookworm" \ --build-arg AIRFLOW_INSTALLATION_METHOD="apache-airflow" \ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="slack" \ --build-arg ADDITIONAL_PYTHON_DEPS="apache-airflow-providers-odbc \ @@ -429,8 +429,8 @@ can be used for CI images: | Build argument | Default value | Description | |-----------------------------------|-----------------------------|-------------------------------------------------------------------------------------------------------------------| -| `PYTHON_BASE_IMAGE` | `python:3.10-slim-bookworm` | Base Python image | -| `PYTHON_MAJOR_MINOR_VERSION` | `3.10` | major/minor version of Python (should match base image) | +| `PYTHON_BASE_IMAGE` | `python:3.11-slim-bookworm` | Base Python image | +| `PYTHON_MAJOR_MINOR_VERSION` | `3.11` | major/minor version of Python (should match base image) | | `DEPENDENCIES_EPOCH_NUMBER` | `2` | increasing this number will reinstall all apt dependencies | | `ADDITIONAL_PIP_INSTALL_FLAGS` | | additional `pip` flags passed to the installation commands (except when reinstalling `pip` itself) | | `HOME` | `/root` | Home directory of the root user (CI image has root user as default) | @@ -460,59 +460,59 @@ can be used for CI images: Here are some examples of how CI images can built manually. CI is always built from local sources. -This builds the CI image in version 3.10 with default extras ("all"). +This builds the CI image in version 3.11 with default extras ("all"). ``` bash DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \ --pull \ - --build-arg PYTHON_BASE_IMAGE="python:3.10-slim-bookworm" --tag my-image:0.0.1 + --build-arg PYTHON_BASE_IMAGE="python:3.11-slim-bookworm" --tag my-image:0.0.1 ``` -This builds the CI image in version 3.10 with "gcp" extra only. +This builds the CI image in version 3.11 with "gcp" extra only. ``` bash DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \ --pull \ - --build-arg PYTHON_BASE_IMAGE="python:3.10-slim-bookworm" \ + --build-arg PYTHON_BASE_IMAGE="python:3.11-slim-bookworm" \ --build-arg AIRFLOW_EXTRAS=gcp --tag my-image:0.0.1 ``` -This builds the CI image in version 3.10 with "apache-beam" extra added. +This builds the CI image in version 3.11 with "apache-beam" extra added. ``` bash DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \ --pull \ - --build-arg PYTHON_BASE_IMAGE="python:3.10-slim-bookworm" \ + --build-arg PYTHON_BASE_IMAGE="python:3.11-slim-bookworm" \ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam" --tag my-image:0.0.1 ``` -This builds the CI image in version 3.10 with "mssql" additional package +This builds the CI image in version 3.11 with "mssql" additional package added. ``` bash DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \ --pull \ - --build-arg PYTHON_BASE_IMAGE="python:3.10-slim-bookworm" \ + --build-arg PYTHON_BASE_IMAGE="python:3.11-slim-bookworm" \ --build-arg ADDITIONAL_PYTHON_DEPS="mssql" --tag my-image:0.0.1 ``` -This builds the CI image in version 3.10 with "gcc" and "g++" additional +This builds the CI image in version 3.11 with "gcc" and "g++" additional apt dev dependencies added. ``` DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \ --pull - --build-arg PYTHON_BASE_IMAGE="python:3.10-slim-bookworm" \ + --build-arg PYTHON_BASE_IMAGE="python:3.11-slim-bookworm" \ --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" --tag my-image:0.0.1 ``` -This builds the CI image in version 3.10 with "jdbc" extra and +This builds the CI image in version 3.11 with "jdbc" extra and "default-jre-headless" additional apt runtime dependencies added. ``` DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \ --pull \ - --build-arg PYTHON_BASE_IMAGE="python:3.10-slim-bookworm" \ + --build-arg PYTHON_BASE_IMAGE="python:3.11-slim-bookworm" \ --build-arg AIRFLOW_EXTRAS=jdbc \ --tag my-image:0.0.1 ``` @@ -574,7 +574,7 @@ percent-encoded when you access them via UI (/ = %2F) | PROD image | airflow/\<BRANCH\>/prod/python\<X.Y\> | faster to build or pull. Production image optimized for size. | - \<BRANCH\> might be either "main" or "v2-\*-test" -- \<X.Y\> - Python version (Major + Minor).Should be one of \["3.10", "3.11", "3.12", "3.13", "3.14" \]. +- \<X.Y\> - Python version (Major + Minor).Should be one of \["3.11", "3.12", "3.13", "3.14" \]. ---- diff --git a/dev/breeze/doc/ci/03_github_variables.md b/dev/breeze/doc/ci/03_github_variables.md index 545779fad96..3aef457e6bd 100644 --- a/dev/breeze/doc/ci/03_github_variables.md +++ b/dev/breeze/doc/ci/03_github_variables.md @@ -65,8 +65,8 @@ workflow: ``` bash # GITHUB_REPOSITORY is set automatically in GitHub Actions so we need to override it with flag # -breeze ci-image build --github-repository apache/airflow --python 3.10 -docker tag ghcr.io/apache/airflow/main/ci/python3.10 your-image-name:tag +breeze ci-image build --github-repository apache/airflow --python 3.11 +docker tag ghcr.io/apache/airflow/main/ci/python3.11 your-image-name:tag ``` ----- diff --git a/dev/breeze/doc/ci/04_selective_checks.md b/dev/breeze/doc/ci/04_selective_checks.md index 5d96ed6d910..ec177bd87a8 100644 --- a/dev/breeze/doc/ci/04_selective_checks.md +++ b/dev/breeze/doc/ci/04_selective_checks.md @@ -575,8 +575,8 @@ GitHub Actions to pass the list of parameters to a command to execute | Output | Meaning of the output | Example value | List | |---------------------------------------------------------|---------------------------------------------------------------------------------------------------------|------------------------------------------|------| -| all-python-versions | List of all python versions there are available in the form of JSON array | \['3.10', '3.11'\] | | -| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string | 3.10 3.11 | * | +| all-python-versions | List of all python versions there are available in the form of JSON array | \['3.11', '3.12'\] | | +| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string | 3.11 3.12 | * | | all-versions | If set to true, then all python, k8s, DB versions are used for tests. | false | | | basic-checks-only | Whether to run all static checks ("false") or only basic set of static checks ("true") | false | | | ci-image-build | Whether CI image build is needed | true | | @@ -589,7 +589,7 @@ GitHub Actions to pass the list of parameters to a command to execute | default-kubernetes-version | Which Kubernetes version to use as default | v1.25.2 | | | default-mysql-version | Which MySQL version to use as default | 5.7 | | | default-postgres-version | Which Postgres version to use as default | 10 | | -| default-python-version | Which Python version to use as default | 3.10 | | +| default-python-version | Which Python version to use as default | 3.11 | | | disable-airflow-repo-cache | Disables cache of the repo main cache in CI - airflow will be installed without main installation cache | true | | | docker-cache | Which cache should be used for images ("registry", "local" , "disabled") | registry | | | docs-build | Whether to build documentation ("true"/"false") | true | | @@ -608,7 +608,7 @@ GitHub Actions to pass the list of parameters to a command to execute | is-legacy-ui-api-labeled | Whether the PR is labeled as legacy UI/API | false | | | java-sdk-version | JDK version used to build the lang-SDK Java artifacts natively in CI | 17 | | | kind-version | Which Kind version to use for tests | v0.24.0 | | -| kubernetes-combos-list-as-string | All combinations of Python version and Kubernetes version to use for tests as space-separated string | 3.10-v1.25.2 3.11-v1.28.13 | * | +| kubernetes-combos-list-as-string | All combinations of Python version and Kubernetes version to use for tests as space-separated string | 3.11-v1.25.2 3.12-v1.28.13 | * | | kubernetes-versions | All Kubernetes versions to use for tests as JSON array | \['v1.25.2'\] | | | kubernetes-versions-list-as-string | All Kubernetes versions to use for tests as space-separated string | v1.25.2 | * | | latest-versions-only | If set, the number of Python, Kubernetes, DB versions will be limited to the latest ones. | false | | @@ -621,8 +621,8 @@ GitHub Actions to pass the list of parameters to a command to execute | providers-compatibility-tests-matrix | Matrix of providers compatibility tests: (python_version, airflow_version, removed_providers) | \[{}\] | | | providers-test-types-list-as-strings-in-json | Which test types should be run for unit tests for providers | Providers Providers\[-google\] | * | | pyproject-toml-changed | When pyproject.toml changed in the PR. | false | | -| python-versions | List of python versions to use for that build | \['3.10'\] | | -| python-versions-list-as-string | Which versions of MySQL to use for tests as space-separated string | 3.10 | * | +| python-versions | List of python versions to use for that build | \['3.11'\] | | +| python-versions-list-as-string | Which versions of MySQL to use for tests as space-separated string | 3.11 | * | | run-agent-framework-tests | Whether the common.ai agent framework adapter tests should be run ("true"/"false") | true | | | run-amazon-tests | Whether Amazon tests should be run ("true"/"false") | true | | | run-api-codegen | Whether "api-codegen" are needed to run ("true"/"false") | true | | diff --git a/dev/breeze/doc/ci/07_running_ci_locally.md b/dev/breeze/doc/ci/07_running_ci_locally.md index 2bc268bfb64..c2dc02565b0 100644 --- a/dev/breeze/doc/ci/07_running_ci_locally.md +++ b/dev/breeze/doc/ci/07_running_ci_locally.md @@ -76,14 +76,14 @@ this will change soon. To load the image from specific PR, you can use the following command: ```bash -breeze ci-image load --from-pr 12345 --python 3.10 --github-token <your_github_token> +breeze ci-image load --from-pr 12345 --python 3.11 --github-token <your_github_token> ``` To load the image from specific run (for example 12538475388), you can use the following command, find the run id from GitHub action runs. ```bash -breeze ci-image load --from-run 12538475388 --python 3.10 --github-token <your_github_token> +breeze ci-image load --from-run 12538475388 --python 3.11 --github-token <your_github_token> ``` After you load the image, you can reproduce the very exact environment that was used in the CI run by diff --git a/docker-stack-docs/build.rst b/docker-stack-docs/build.rst index 79ffe2bd7b2..daf02a00644 100644 --- a/docker-stack-docs/build.rst +++ b/docker-stack-docs/build.rst @@ -215,7 +215,7 @@ In the simplest case building your image consists of those steps: 1) Create your own ``Dockerfile`` (name it ``Dockerfile``) where you add: -* information what your image should be based on (for example ``FROM: apache/airflow:|airflow-version|-python3.10`` +* information what your image should be based on (for example ``FROM: apache/airflow:|airflow-version|-python3.11`` * additional steps that should be executed in your image (typically in the form of ``RUN <command>``) @@ -320,9 +320,9 @@ Naming conventions for the images: +----------------+--------------------------+---------------------------------+--------------------------------------+ | Default | 3.13 | apache/airflow:X.Y.Z | apache/airflow:slim-X.Y.Z | +----------------+--------------------------+---------------------------------+--------------------------------------+ -| Latest | 3.10,3.11,3.12,3.13,3.14 | apache/airflow:latest-pythonN.M | apache/airflow:slim-latest-pythonN.M | +| Latest | 3.11,3.12,3.13,3.14 | apache/airflow:latest-pythonN.M | apache/airflow:slim-latest-pythonN.M | +----------------+--------------------------+---------------------------------+--------------------------------------+ -| Specific | 3.10,3.11,3.12,3.13,3.14 | apache/airflow:X.Y.Z-pythonN.M | apache/airflow:slim-X.Y.Z-pythonN.M | +| Specific | 3.11,3.12,3.13,3.14 | apache/airflow:X.Y.Z-pythonN.M | apache/airflow:slim-X.Y.Z-pythonN.M | +----------------+--------------------------+---------------------------------+--------------------------------------+ * The "latest" image is always the latest released stable version available. @@ -736,7 +736,7 @@ Building from PyPI packages This is the basic way of building the custom images from sources. -The following example builds the production image in version ``3.10`` with latest PyPI-released Airflow, +The following example builds the production image in version ``3.11`` with latest PyPI-released Airflow, with default set of Airflow extras and dependencies. The latest PyPI-released Airflow constraints are used automatically. .. exampleinclude:: docker-examples/customizing/stable-airflow.sh @@ -744,7 +744,7 @@ with default set of Airflow extras and dependencies. The latest PyPI-released Ai :start-after: [START build] :end-before: [END build] -The following example builds the production image in version ``3.10`` with default extras from ``2.3.0`` Airflow +The following example builds the production image in version ``3.11`` with default extras from ``2.3.0`` Airflow package. The ``2.3.0`` constraints are used automatically. .. exampleinclude:: docker-examples/customizing/pypi-selected-version.sh @@ -752,7 +752,7 @@ package. The ``2.3.0`` constraints are used automatically. :start-after: [START build] :end-before: [END build] -The following example builds the production image in version ``3.10`` with additional Airflow extras +The following example builds the production image in version ``3.11`` with additional Airflow extras (``mssql,hdfs``) from ``2.3.0`` PyPI package, and additional dependency (``oauth2client``). .. exampleinclude:: docker-examples/customizing/pypi-extras-and-deps.sh @@ -779,7 +779,7 @@ have more complex dependencies to build. Building optimized images ......................... -The following example builds the production image in version ``3.10`` with additional Airflow extras from +The following example builds the production image in version ``3.11`` with additional Airflow extras from PyPI package but it includes additional apt dev and runtime dependencies. The dev dependencies are those that require ``build-essential`` and usually need to involve recompiling diff --git a/docker-stack-docs/docker-examples/restricted/restricted_environments.sh b/docker-stack-docs/docker-examples/restricted/restricted_environments.sh index 05b1a9b2da2..ac15ebf8e3b 100755 --- a/docker-stack-docs/docker-examples/restricted/restricted_environments.sh +++ b/docker-stack-docs/docker-examples/restricted/restricted_environments.sh @@ -29,8 +29,8 @@ mkdir -p docker-context-files export AIRFLOW_VERSION="2.5.3" rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true -curl -Lo "docker-context-files/constraints-3.10.txt" \ - "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.10.txt"; +curl -Lo "docker-context-files/constraints-3.11.txt" \ + "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.11.txt"; echo echo "Make sure you use the right python version here (should be same as in constraints)!" @@ -38,7 +38,7 @@ echo python --version pip download --dest docker-context-files \ - --constraint docker-context-files/constraints-3.10.txt \ + --constraint docker-context-files/constraints-3.11.txt \ "apache-airflow[async,celery,elasticsearch,kubernetes,postgres,redis,ssh,statsd,virtualenv]==${AIRFLOW_VERSION}" # [END download] @@ -56,7 +56,7 @@ docker build . \ --build-arg INSTALL_POSTGRES_CLIENT="true" \ --build-arg DOCKER_CONTEXT_FILES="docker-context-files" \ --build-arg INSTALL_DISTRIBUTIONS_FROM_CONTEXT="true" \ - --build-arg AIRFLOW_CONSTRAINTS_LOCATION="/docker-context-files/constraints-3.10.txt" \ + --build-arg AIRFLOW_CONSTRAINTS_LOCATION="/docker-context-files/constraints-3.11.txt" \ --tag airflow-my-restricted-environment:0.0.1 # [END build] diff --git a/docker-stack-docs/entrypoint.rst b/docker-stack-docs/entrypoint.rst index 04a9a9861d7..23185e137ca 100644 --- a/docker-stack-docs/entrypoint.rst +++ b/docker-stack-docs/entrypoint.rst @@ -132,7 +132,7 @@ if you specify extra arguments. For example: .. code-block:: bash - docker run -it apache/airflow:3.4.0-python3.10 bash -c "ls -la" + docker run -it apache/airflow:3.4.0-python3.11 bash -c "ls -la" total 16 drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 . drwxr-xr-x 1 root root 4096 Jun 5 18:12 .. @@ -144,7 +144,7 @@ you pass extra parameters. For example: .. code-block:: bash - > docker run -it apache/airflow:3.4.0-python3.10 python -c "print('test')" + > docker run -it apache/airflow:3.4.0-python3.11 python -c "print('test')" test If first argument equals to ``airflow`` - the rest of the arguments is treated as an Airflow command @@ -152,13 +152,13 @@ to execute. Example: .. code-block:: bash - docker run -it apache/airflow:3.4.0-python3.10 airflow webserver + docker run -it apache/airflow:3.4.0-python3.11 airflow webserver If there are any other arguments - they are simply passed to the "airflow" command .. code-block:: bash - > docker run -it apache/airflow:3.4.0-python3.10 help + > docker run -it apache/airflow:3.4.0-python3.11 help usage: airflow [-h] GROUP_OR_COMMAND ... Positional Arguments: @@ -375,7 +375,7 @@ database and creating an ``admin/admin`` Admin user with the following command: --env "_AIRFLOW_DB_MIGRATE=true" \ --env "_AIRFLOW_WWW_USER_CREATE=true" \ --env "_AIRFLOW_WWW_USER_PASSWORD=admin" \ - apache/airflow:3.4.0-python3.10 webserver + apache/airflow:3.4.0-python3.11 webserver .. code-block:: bash @@ -384,7 +384,7 @@ database and creating an ``admin/admin`` Admin user with the following command: --env "_AIRFLOW_DB_MIGRATE=true" \ --env "_AIRFLOW_WWW_USER_CREATE=true" \ --env "_AIRFLOW_WWW_USER_PASSWORD_CMD=echo admin" \ - apache/airflow:3.4.0-python3.10 webserver + apache/airflow:3.4.0-python3.11 webserver The commands above perform initialization of the SQLite database, create admin user with admin password and Admin role. They also forward local port ``8080`` to the webserver port and finally start the webserver. @@ -424,6 +424,6 @@ Example: --env "_AIRFLOW_DB_MIGRATE=true" \ --env "_AIRFLOW_WWW_USER_CREATE=true" \ --env "_AIRFLOW_WWW_USER_PASSWORD_CMD=echo admin" \ - apache/airflow:3.4.0-python3.10 webserver + apache/airflow:3.4.0-python3.11 webserver This method is only available starting from Docker image of Airflow 2.1.1 and above. diff --git a/providers-summary-docs/installing-from-pypi.rst b/providers-summary-docs/installing-from-pypi.rst index 6dbe6c32338..ce7a0548fba 100644 --- a/providers-summary-docs/installing-from-pypi.rst +++ b/providers-summary-docs/installing-from-pypi.rst @@ -45,6 +45,6 @@ Typical command to install Airflow from PyPI looks like below (you need to use t .. code-block:: - pip install "apache-airflow-providers-celery" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.10.2/constraints-3.10.txt"; + pip install "apache-airflow-providers-celery" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.10.2/constraints-3.11.txt"; This is an example, see :doc:`apache-airflow:installation/installing-from-pypi` for more examples, including how to upgrade the providers. diff --git a/registry/README.md b/registry/README.md index 9502927a86e..154bf19dcb1 100644 --- a/registry/README.md +++ b/registry/README.md @@ -27,7 +27,7 @@ and deployed to `airflow.apache.org/registry/`. ### Prerequisites -- Python 3.10+ (for metadata extraction) +- Python 3.11+ (for metadata extraction) - Node.js 20+ and pnpm 9+ - `pyyaml` Python package (`uv pip install pyyaml`)
