This is an automated email from the ASF dual-hosted git repository. vatsrahul1001 pushed a commit to branch cherry-pick-71171-v3-3-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9a2a7433f800f5c73cdea0bd0b8158304d39a19e Author: Jarek Potiuk <[email protected]> AuthorDate: Wed Aug 5 18:49:10 2026 +0800 Refresh the image registry cache in a workflow that cannot be cancelled (#71171) * Refresh the image registry cache in a workflow that cannot be cancelled A branch built on push refreshed its registry cache from inside the CI run, which cancels in progress on the next push. With no cache the build runs cold, which widens the window the next push cancels in, which leaves the cache missing -- v3-3-test held only the default Python version's amd64 cache because that was the one matrix entry that finished in time. The refresh also has to cover every Python version: a version with no cache entry builds from scratch on every run, and selective checks narrows the list on a text-only push. * Refresh constraints from a workflow of its own, covering every flavour The manual constraints refresh shared a workflow with the automatic uv.lock push, so it was filtered by that workflow's paths and buried under its name in the Actions list. The documented procedures pointed at local builds -- refresh_images.sh for the image cache, breeze commands for the constraints -- which need a buildx/qemu setup, a committer login to ghcr.io, and enough bandwidth to push the layers. The workflows need none of that, and unlike a local run they always cover every Python version and every constraint flavour. * Exempt a manually started cache refresh from the push cancellation A push-triggered refresh should give way to the next one -- it is building the cache of a commit that is no longer the tip. A manual run should not: it is started because the branch has no cache and the push runs are the thing that keeps getting cancelled. (cherry picked from commit e69c1881b32c36abb827bae3717eaf46424427bd) --- .github/workflows/additional-ci-image-checks.yml | 78 +----------------- .github/workflows/ci-amd.yml | 7 -- .github/workflows/ci-arm.yml | 7 -- .github/workflows/refresh-constraints.yml | 72 +++++++++++++++++ ...-cache.yml => refresh-image-registry-cache.yml} | 59 ++++++++------ .github/workflows/update-constraints-on-push.yml | 56 +++++++++---- dev/MANUALLY_BUILDING_IMAGES.md | 6 +- ...UALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md | 89 ++++++++++++++------- dev/breeze/tests/test_selective_checks.py | 23 +----- dev/images/update_constraints_run_workflow.png | Bin 0 -> 505265 bytes 10 files changed, 215 insertions(+), 182 deletions(-) diff --git a/.github/workflows/additional-ci-image-checks.yml b/.github/workflows/additional-ci-image-checks.yml index 051fbd02029..42b71e1f32b 100644 --- a/.github/workflows/additional-ci-image-checks.yml +++ b/.github/workflows/additional-ci-image-checks.yml @@ -28,18 +28,10 @@ on: # yamllint disable-line rule:truthy description: "Platform for the build - 'linux/amd64' or 'linux/arm64'" required: true type: string - python-versions: - description: "The list of python versions (stringified JSON array) to run the tests on." - required: true - type: string branch: description: "Branch used to run the CI jobs in (main/v*_*_test)." required: true type: string - constraints-branch: - description: "Branch used to get constraints from" - required: true - type: string default-python-version: description: "Which version of python should be used by default" required: true @@ -52,85 +44,19 @@ on: # yamllint disable-line rule:truthy description: "Whether to skip prek hooks (true/false)" required: true type: string - docker-cache: - description: "Docker cache specification to build the image (registry, local, disabled)." - required: true - type: string - disable-airflow-repo-cache: - description: "Disable airflow repo cache read from main." - required: true - type: string - canary-run: - description: "Whether this is a canary run (true/false)" - required: true - type: string latest-versions-only: description: "Whether to run only latest versions (true/false)" required: true type: string - include-success-outputs: - description: "Whether to include success outputs (true/false)" - required: true - type: string debug-resources: description: "Whether to debug resources (true/false)" required: true type: string - use-uv: - description: "Whether to use uv to build the image (true/false)" - required: true - type: string permissions: contents: read jobs: - # Push early BuildX cache to GitHub Registry in Apache repository. This cache does not wait for all the - # tests to complete - it is run very early in the build process in order to refresh cache using the - # current constraints. This will speed up cache refresh in cases when pyproject.toml changes or in case - # of Dockerfile changes. Failure in this step is not a problem (at most it will delay cache refresh). - # It does not attempt to upgrade to newer dependencies. - # We only push CI cache as PROD cache usually does not gain as much from fresh cache because - # it uses prepared airflow and provider distributions that invalidate the cache anyway most of the time - # - # Only for the scheduled canary and manual dispatch -- both of which run to completion. Push events - # are refreshed by `refresh-image-cache.yml` instead: this workflow is part of the CI run, which - # cancels in progress on the next push, and a cache refresh that keeps being cancelled leaves the - # branch building cold forever (see that workflow's header for the loop it broke). - # - # Pull requests stay excluded -- `canary-run` is also true for a PR carrying the `canary` label, and - # pushing cache from an unmerged branch is not something a label should authorise. This is the same - # condition the Regular cache push uses. - # - # Both paths cover every Python version: `refresh-image-cache.yml` forces it, and on `schedule` / - # `workflow_dispatch` selective checks always resolves `python-versions` to all of them - # (`_should_run_all_tests_and_versions`), which `test_all_python_versions_on_scheduled_canary` pins. - push-early-buildx-cache-to-github-registry: - name: Push Early Image Cache - uses: ./.github/workflows/push-image-cache.yml - permissions: - contents: read - # This write is only given here for `push` events from "apache/airflow" repo. It is not given for PRs - # from forks. This is to prevent malicious PRs from creating images in the "apache/airflow" repo. - packages: write - with: - runners: ${{ inputs.runners }} - cache-type: "Early" - include-prod-images: "false" - push-latest-images: "false" - platform: ${{ inputs.platform }} - python-versions: ${{ inputs.python-versions }} - branch: ${{ inputs.branch }} - constraints-branch: ${{ inputs.constraints-branch }} - use-uv: ${{ inputs.use-uv }} - include-success-outputs: ${{ inputs.include-success-outputs }} - docker-cache: ${{ inputs.docker-cache }} - disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }} - if: > - inputs.canary-run == 'true' && github.event_name != 'pull_request' - && github.event_name != 'push' - - # Check that after earlier cache push, breeze command will build quickly - # This build is a bit slow from in-the scratch builds, so we should run it only in - # regular PRs + # Check that the image builds quickly from the registry cache. This build is slow from + # scratch, so we only run it on regular PRs. check-that-image-builds-quickly: timeout-minutes: 25 name: Check that image builds quickly diff --git a/.github/workflows/ci-amd.yml b/.github/workflows/ci-amd.yml index fcff7f4924c..f8dca261ed4 100644 --- a/.github/workflows/ci-amd.yml +++ b/.github/workflows/ci-amd.yml @@ -285,19 +285,12 @@ jobs: with: runners: ${{ needs.build-info.outputs.runner-type }} platform: ${{ needs.build-info.outputs.platform }} - python-versions: ${{ needs.build-info.outputs.python-versions }} branch: ${{ needs.build-info.outputs.default-branch }} - constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }} default-python-version: "${{ needs.build-info.outputs.default-python-version }}" upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }} skip-prek-hooks: ${{ needs.build-info.outputs.skip-prek-hooks }} - docker-cache: ${{ needs.build-info.outputs.docker-cache }} - disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }} - canary-run: ${{ needs.build-info.outputs.canary-run }} latest-versions-only: ${{ needs.build-info.outputs.latest-versions-only }} - include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }} debug-resources: ${{ needs.build-info.outputs.debug-resources }} - use-uv: ${{ needs.build-info.outputs.use-uv }} generate-constraints: name: "Generate constraints" diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml index 0d4f34f67d9..acb7cd8f456 100644 --- a/.github/workflows/ci-arm.yml +++ b/.github/workflows/ci-arm.yml @@ -278,19 +278,12 @@ jobs: with: runners: ${{ needs.build-info.outputs.runner-type }} platform: ${{ needs.build-info.outputs.platform }} - python-versions: ${{ needs.build-info.outputs.python-versions }} branch: ${{ needs.build-info.outputs.default-branch }} - constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }} default-python-version: "${{ needs.build-info.outputs.default-python-version }}" upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }} skip-prek-hooks: ${{ needs.build-info.outputs.skip-prek-hooks }} - docker-cache: ${{ needs.build-info.outputs.docker-cache }} - disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }} - canary-run: ${{ needs.build-info.outputs.canary-run }} latest-versions-only: ${{ needs.build-info.outputs.latest-versions-only }} - include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }} debug-resources: ${{ needs.build-info.outputs.debug-resources }} - use-uv: ${{ needs.build-info.outputs.use-uv }} generate-constraints: name: "Generate constraints" diff --git a/.github/workflows/refresh-constraints.yml b/.github/workflows/refresh-constraints.yml new file mode 100644 index 00000000000..58295b6c092 --- /dev/null +++ b/.github/workflows/refresh-constraints.yml @@ -0,0 +1,72 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: Refresh constraints +# Refreshes the pinned constraint files on demand, when waiting for the automatic path is not an +# option -- a `main` / `vX-Y-test` build that cannot be made green, or newly released providers +# that a release candidate has to pick up before it is promoted. +# +# All three constraint flavours are refreshed, not just the PyPI one: +# +# * `constraints-source-providers-X.Y.txt` -- providers resolved from this checkout's sources, +# which is what a contributor's Breeze environment and CI install from; +# * `constraints-no-providers-X.Y.txt` -- core dependencies alone, used to install Airflow +# without any provider pinned; +# * `constraints-X.Y.txt` -- providers as published on PyPI, which is what users install with. +# +# Refreshing only the PyPI flavour would leave CI and Breeze on the older pins, so the three +# would disagree about the same dependency. +# +# This does the same work as the automatic `uv.lock` push (whose workflow it calls), against the +# ref you name rather than the branch the push landed on. Run it from `main`: the workflow +# definition and `breeze` come from `main`, while the sources come from `ref`, so refreshing a +# release line needs no cherry-pick to `vX-Y-test` / `vX-Y-stable` first. The `constraints-X-Y` +# branch written to is derived from that ref's `branch_defaults.py`. +# +# See dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md. +on: # yamllint disable-line rule:truthy + workflow_dispatch: + inputs: + ref: + description: "Repo reference to refresh constraints from (e.g. v3-3-stable). Empty = this branch." + required: false + default: "" + type: string + upgrade-to-newer-dependencies: + description: "Upgrade deps to newest from PyPI" + required: false + type: boolean + default: true +permissions: + contents: read +concurrency: + # Two refreshes of the same ref would race each other onto the same `constraints-X-Y` branch. + group: refresh-constraints-${{ inputs.ref || github.ref }} + cancel-in-progress: false +jobs: + refresh-constraints: + name: "Refresh constraints" + uses: ./.github/workflows/update-constraints-on-push.yml + permissions: + contents: write + packages: write + with: + ref: ${{ inputs.ref }} + upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies }} + secrets: + SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} diff --git a/.github/workflows/refresh-image-cache.yml b/.github/workflows/refresh-image-registry-cache.yml similarity index 61% rename from .github/workflows/refresh-image-cache.yml rename to .github/workflows/refresh-image-registry-cache.yml index 43e9e5facd4..23b3e6221dc 100644 --- a/.github/workflows/refresh-image-cache.yml +++ b/.github/workflows/refresh-image-registry-cache.yml @@ -16,18 +16,19 @@ # under the License. # --- -name: Refresh image cache -# Refreshes the CI image cache in the GitHub registry, in its own workflow run. +name: Refresh image registry cache +# Refreshes the CI image cache in the GitHub registry, in a run of its own. # -# It lives outside `ci-amd.yml` / `ci-arm.yml` because those cancel in progress runs on the next -# push (`cancel-in-progress: true`), and a cache refresh is precisely the job that must not be -# cancelled: a branch with no cache builds cold (~20 minutes per Python version), which widens the -# window in which the next push cancels the run, which leaves the cache missing. `v3-3-test` sat in -# exactly that loop -- its `linux/amd64` cache existed for the default Python version only, because -# that was the single matrix entry that finished before the next push killed the rest. +# It has to be its own workflow rather than a job inside `ci-amd.yml` / `ci-arm.yml`, because a +# job cannot opt out of its run being cancelled, and those runs are cancelled by the next push. +# With no cache a branch builds cold (~20 minutes per Python version), which widens the window in +# which the next push cancels the run, which leaves the cache missing. `v3-3-test` sat in that +# loop -- its `linux/amd64` cache covered the default Python version alone, the single matrix +# entry that finished before the next push killed the other four seconds later. Being a separate +# workflow is what lets a manual run escape that (see the concurrency group below). # -# Cancelling here would recreate the loop, so this workflow does not cancel. A newer run queues -# behind the one in flight instead of replacing it. +# Every Python version is refreshed in both trigger paths. That is what makes the cache useful: +# a version with no cache entry builds from scratch on every run, whichever way it got skipped. on: # yamllint disable-line rule:truthy workflow_dispatch: inputs: @@ -40,16 +41,25 @@ on: # yamllint disable-line rule:truthy - "linux/arm64" default: "both" push: - # Release-prep and providers branches only. `main` refreshes its cache through the scheduled - # canary in `ci-amd.yml` / `ci-arm.yml`, which nothing cancels -- it is not built on push. + # Release-prep and providers branches only -- they are built on push, so the CI run that + # would have refreshed their cache is also the run that keeps being cancelled. `main` is not + # built on push; its cache is refreshed by the Push Image Cache job at the end of the + # scheduled canary, which runs to completion. branches: - v[0-9]+-[0-9]+-test - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+ permissions: contents: read concurrency: - group: refresh-image-cache-${{ github.ref }} - cancel-in-progress: false + # Push runs are never cancelled -- being cancelled by the next merge is the whole failure this + # workflow exists to escape, so a newer one queues behind the one in flight rather than + # replacing it. GitHub keeps at most one run pending per group, so the queue cannot grow. + # + # Manual runs are grouped separately, which is what keeps a merge from cancelling one, and they + # do supersede each other: firing the workflow again for the same branch says the run in flight + # is not the one you want. + group: refresh-image-registry-cache-${{ github.ref }}-${{ github.event_name }} + cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' }} jobs: build-info: name: "Build info" @@ -76,11 +86,10 @@ jobs: - name: Selective checks id: selective-checks env: - # A cache is only useful for the Python versions that are actually built, so the refresh - # always covers all of them. Selective checks would otherwise narrow `python-versions` - # down to the default one -- on a push where only text files changed it does exactly - # that (`_should_run_all_tests_and_versions`), which is the other way a branch ends up - # with cache for one Python version and nothing for the rest. + # Forces every Python version. Selective checks narrows `python-versions` down to the + # default one on a push where only text files changed + # (`_should_run_all_tests_and_versions`), which would leave the rest of the versions + # with no cache -- the failure mode this workflow exists to prevent. PR_LABELS: '["all versions"]' COMMIT_REF: "${{ github.sha }}" VERBOSE: "false" @@ -98,11 +107,11 @@ jobs: runners: '["ubuntu-22.04"]' platform: "linux/amd64" cache-type: "Refresh" - # PROD cache is built from distributions prepared earlier in a CI run, which a standalone - # refresh has no artifact for. CI cache is also where the build time actually goes. + # The PROD cache is built from distributions prepared earlier in a CI run, and a standalone + # refresh has no such artifact. The CI cache is also where the build time goes. include-prod-images: "false" - # `latest` is what CI pulls when it has no cache at all; moving it belongs to a full canary - # run that ran the tests, not to a cache refresh. + # `latest` is what a build pulls when it has no cache at all. Moving it belongs to a canary + # run that has passed its tests, not to a cache refresh. push-latest-images: "false" python-versions: ${{ needs.build-info.outputs.python-versions }} branch: ${{ needs.build-info.outputs.default-branch }} @@ -111,7 +120,7 @@ jobs: include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }} docker-cache: ${{ needs.build-info.outputs.docker-cache }} disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }} - if: inputs.platform == 'both' || inputs.platform == 'linux/amd64' || github.event_name == 'push' + if: github.event_name == 'push' || inputs.platform == 'both' || inputs.platform == 'linux/amd64' refresh-ci-cache-arm: name: "Refresh cache linux/arm64" @@ -133,4 +142,4 @@ jobs: include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }} docker-cache: ${{ needs.build-info.outputs.docker-cache }} disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }} - if: inputs.platform == 'both' || inputs.platform == 'linux/arm64' || github.event_name == 'push' + if: github.event_name == 'push' || inputs.platform == 'both' || inputs.platform == 'linux/arm64' diff --git a/.github/workflows/update-constraints-on-push.yml b/.github/workflows/update-constraints-on-push.yml index 75e185dc1a1..c0fc95b3e16 100644 --- a/.github/workflows/update-constraints-on-push.yml +++ b/.github/workflows/update-constraints-on-push.yml @@ -16,14 +16,15 @@ # under the License. # --- -name: Update constraints (on uv.lock push or manual dispatch) +name: Update constraints (on uv.lock push) # This workflow refreshes the pinned constraint files stored in the # `constraints-*` branches. It runs automatically whenever `uv.lock` changes on -# `main` or a `vX-Y-test` branch, and can also be triggered manually via the -# "Run workflow" button (workflow_dispatch) - for example to pick up newly -# released providers/dependencies from PyPI just before promoting an RC. +# `main` or a `vX-Y-test` branch. The manual path lives in +# `refresh-constraints.yml`, which calls this workflow -- keeping the two triggers +# in separate workflows means a manual refresh is not filtered by `paths: uv.lock` +# and shows up in the Actions list under its own name. # -# The manual run is always launched from `main` (so the workflow definition and +# A manual run is always launched from `main` (so the workflow definition and # `breeze` come from `main`), and the `ref` input selects the commit-ish # (branch, tag or commit hash) whose sources the constraints are refreshed from. # The `constraints-X-Y` branch to push to is derived from that ref's @@ -37,22 +38,27 @@ on: # yamllint disable-line rule:truthy - v[0-9]+-[0-9]+-test paths: - 'uv.lock' - workflow_dispatch: + workflow_call: inputs: + # `ref` is optional. Leave it empty to refresh constraints for the branch + # you run the workflow from (normally `main` -> `constraints-main`). Set a + # branch, tag or commit to refresh a specific line, e.g. + # `v3-3-stable` -> `constraints-3-3`. The target `constraints-X-Y` branch + # is derived from that ref's branch_defaults.py. ref: - description: >- - Commit-ish to refresh constraints from (branch, tag or commit hash), - e.g. `v3-3-test`, `v3-3-stable`, `constraints-3-3` or a tag/hash. - The matching `constraints-X-Y` branch is derived from that ref. - required: true + description: "Optional repo reference to build constraints (e.g. v3-3-stable)" + required: false + default: "" type: string upgrade-to-newer-dependencies: - description: >- - Re-resolve to the newest matching dependencies (picks up newly - released providers/dependencies from PyPI). Leave enabled when - refreshing constraints before promoting an RC. + description: "Upgrade deps to newest from PyPI" + required: false type: boolean default: true + secrets: + SLACK_BOT_TOKEN: + description: "Slack bot token used to post provider-downgrade alerts (optional)." + required: false permissions: contents: read @@ -123,6 +129,26 @@ jobs: VERBOSE: "false" GITHUB_CONTEXT_INPUT: "${{ runner.temp }}/github_context.json" run: breeze ci selective-check 2>> ${GITHUB_OUTPUT} + - name: "Parameters summary" + shell: bash + env: + EVENT_NAME: ${{ github.event_name }} + REF: ${{ inputs.ref }} + UPGRADE: ${{ inputs.upgrade-to-newer-dependencies }} + CONSTRAINTS_BRANCH: ${{ steps.selective-checks.outputs.default-constraints-branch }} + run: | + { + echo "## Update constraints" + echo "" + echo "| Parameter | Value |" + echo "|---|---|" + echo "| Triggered by | \`${EVENT_NAME}\` |" + if [[ "${EVENT_NAME}" == "workflow_dispatch" ]]; then + echo "| Refresh from ref | \`${REF}\` |" + echo "| Upgrade to newer dependencies | \`${UPGRADE}\` |" + fi + echo "| Target constraints branch | \`${CONSTRAINTS_BRANCH}\` |" + } | tee -a "${GITHUB_STEP_SUMMARY}" build-ci-images: name: "Build CI images" diff --git a/dev/MANUALLY_BUILDING_IMAGES.md b/dev/MANUALLY_BUILDING_IMAGES.md index 18ba10dd210..aa0c8a4c14f 100644 --- a/dev/MANUALLY_BUILDING_IMAGES.md +++ b/dev/MANUALLY_BUILDING_IMAGES.md @@ -94,7 +94,7 @@ The benefit of this method is that the images can be built and pushed in a singl to build and push them separately and perform the additional manifest merge step. ```bash -docker buildx create --name airflow_cache --driver docker-container unix:///var/run/docker.sock # your local builder (you might want to use +docker buildx create --name airflow_cache --driver docker-container unix:///var/run/docker.sock # your local builder docker buildx create --name airflow_cache --append HOST:PORT # your remote builder ``` @@ -167,7 +167,7 @@ with appropriate versions). You can then merge the images into a single multi-platform image with: ```bash -docker release-management merge-prod-images --airflow-version "${VERSION}" --metadata-folder dist +breeze release-management merge-prod-images --airflow-version "${VERSION}" --metadata-folder dist ``` The same can be repeated for slim images by adding `--slim-images` option to the command, the manifests are @@ -189,5 +189,5 @@ breeze release-management release-prod-images --slim-images --airflow-version "$ Merging the images: ```bash -docker release-management merge-prod-images --slim-images --airflow-version "${VERSION}" --metadata-folder dist +breeze release-management merge-prod-images --slim-images --airflow-version "${VERSION}" --metadata-folder dist ``` diff --git a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md index 4a794333ab6..cd07deec80b 100644 --- a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md +++ b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md @@ -25,8 +25,9 @@ - [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci) - [Manually refreshing the image cache](#manually-refreshing-the-image-cache) - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually) - - [Prerequisites](#prerequisites) - - [How to refresh the image cache](#how-to-refresh-the-image-cache) + - [How to refresh the image cache via the CI workflow (recommended)](#how-to-refresh-the-image-cache-via-the-ci-workflow-recommended) + - [Prerequisites for refreshing locally](#prerequisites-for-refreshing-locally) + - [How to refresh the image cache locally](#how-to-refresh-the-image-cache-locally) - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache) - [What the command does](#what-the-command-does) - [Manually generating constraint files](#manually-generating-constraint-files) @@ -69,17 +70,17 @@ rebuilding of [Breeze](./breeze/doc/README.rst) images for development purpose. * The latest [constraints](/contributing-docs/13_airflow_dependencies_and_extras.rst#pinned-constraint-files) are pushed to appropriate branch after all tests succeed in the `canary` build. -* The [images](breeze/doc/ci/02_images.md) in `ghcr.io` registry are refreshed early at the beginning of the - `canary` build. This is done twice during the canary build: - * By the `Push Early Image Cache` job that is run at the beginning of the `canary` build. This cover the - case when there are new dependencies added or Dockerfile/scripts change. Thanks to that step, subsequent - PRs will be faster when they use the new Dockerfile/script. Those jobs **might fail** occasionally, - if the latest PR added some conflicting dependencies with current constraints. This is not a problem - and when it happens, it will be fixed by the next step. - * By the `Push Image Cache` job that is run at the end of the `canary` build. This covers the case when - cache is also refreshed after than `main` build succeeds after the new constraints are pushed. This - step makes sure that constraints are committed and pushed just before the cache is refreshed, so - there is no problem with conflicting dependencies. +* The [images](breeze/doc/ci/02_images.md) in `ghcr.io` registry are refreshed by the `Push Image Cache` + job at the end of the `canary` build, once the tests have passed. Running it last is what makes the + constraints and the cache agree: the constraints are committed and pushed just before the cache is + refreshed from them. + +* Branches that are built on `push` rather than by the scheduled `canary` -- the release-prep + (`vX-Y-test`) and providers branches -- are refreshed by the + [`Refresh image registry cache`](../.github/workflows/refresh-image-registry-cache.yml) workflow + instead. It runs on its own rather than inside the CI run, because the CI run is cancelled by the + next push and a cache refresh that keeps being cancelled never lands. It covers every Python version + on both platforms, and it is the same workflow you run by hand (see below). # Manually refreshing the image cache @@ -87,14 +88,35 @@ rebuilding of [Breeze](./breeze/doc/README.rst) images for development purpose. ## Why we need to update image cache manually Sometimes, when we have a problem with our CI running and flakiness of GitHub Actions runners or our -tests, the refresh might not be triggered. This has been mitigated by "Push Early Image Cache" job added in -our CI, but there are other reasons you might want to refresh the cache. Sometimes we want to refresh the +tests, the refresh might not be triggered. Sometimes we want to refresh the image cache in `vX_Y_test` branch (following our convention of branch names `vX_Y_test` branch is the branch used to release all `X.Y.*` versions of airflow) before we attempt to push a change there. There are no PRs happening in this branch, so manual refresh before we make a PR might speed up the PR build. Or sometimes we just refreshed the constraints (see below) and we want the cache to include those. -## Prerequisites +## How to refresh the image cache via the CI workflow (recommended) + +Run the [`Refresh image registry cache`](../.github/workflows/refresh-image-registry-cache.yml) workflow +from the Actions tab, selecting the branch whose cache you want to refresh in GitHub's branch dropdown -- +the cache is namespaced per branch, so the branch you pick is the namespace written to. The `platform` +input refreshes `linux/amd64`, `linux/arm64`, or both (the default). + +Prefer this over the local route below. The workflow needs no buildx / qemu setup on your machine, it +does not depend on your upload bandwidth, it always covers every Python version, and it runs with the +registry credentials the CI already has, so you do not need to be logged in to `ghcr.io` as a committer. + +A run you start by hand is not cancelled by merges landing while it works -- it uses a concurrency +group of its own for that reason. Starting it again for the same branch does replace the run in +flight. Push-triggered refreshes are never cancelled either; a newer one waits for the one in +flight to finish. + +The workflow definition has to exist on the branch you select, so a release branch needs the workflow +backported before it can be refreshed there. + +## Prerequisites for refreshing locally + +The rest of this chapter describes refreshing the cache from your own machine. It is a fallback for when +the workflow cannot be used at all -- it is slower, and easy to get subtly wrong. Note that in order to refresh images you have to not only have `buildx` command installed for docker, but you should also make sure that you have the buildkit builder configured and set. @@ -146,7 +168,7 @@ docker buildx ls airflow_cache1 tcp://127.0.0.1:2375 ``` -## How to refresh the image cache +## How to refresh the image cache locally The images can be rebuilt and refreshed after the constraints are pushed. Refreshing image for all python version is as simple as running the [refresh_images.sh](refresh_images.sh) script which will @@ -217,11 +239,17 @@ and you need to be sure what you are doing, but you can always do it manually if ## How to refresh constraints via the CI workflow (recommended) The easiest way to refresh the constraints - for example to pick up newly released -providers/dependencies from PyPI just before promoting an RC - is to trigger the -[`Update constraints`](../.github/workflows/update-constraints-on-push.yml) workflow manually -instead of running the `breeze` commands locally. The workflow runs exactly the same steps -that run automatically when `uv.lock` changes, builds the CI images, generates all constraint -flavours and commits/pushes them to the matching `constraints-*` branch. +providers/dependencies from PyPI just before promoting an RC - is to run the +[`Refresh constraints`](../.github/workflows/refresh-constraints.yml) workflow instead of +running the `breeze` commands locally. It runs exactly the same steps that run automatically +when `uv.lock` changes (it calls that workflow), builds the CI images, and commits/pushes to the +matching `constraints-*` branch. + +It refreshes **all three constraint flavours**, not only the PyPI one: +`constraints-source-providers-X.Y.txt` (providers from the sources, which is what CI and Breeze +install), `constraints-no-providers-X.Y.txt` (core alone) and `constraints-X.Y.txt` (providers as +published on PyPI, which is what users install). Refreshing only the PyPI flavour would leave CI +and Breeze pinned to the older versions, so the three would disagree about the same dependency. The manual run is always launched from `main` and takes a `ref` input that selects the commit-ish (branch, tag or commit hash) whose sources the constraints are refreshed from. You @@ -236,18 +264,21 @@ workflow); the automatic `uv.lock`-push runs are not restricted. To run it: -1. Go to the [`Update constraints`](https://github.com/apache/airflow/actions/workflows/update-constraints-on-push.yml) + + +1. Go to the [`Refresh constraints`](https://github.com/apache/airflow/actions/workflows/refresh-constraints.yml) workflow in the Actions tab. 2. Click **Run workflow** and keep the branch set to `main` (this is where the workflow runs from - it is not the branch whose constraints get refreshed). -3. In the **ref** field, enter the commit-ish to refresh constraints from - for example - `v3-3-test`, `v3-3-stable`, `constraints-3-3`, an RC tag, or a commit hash. The matching - `constraints-X-Y` branch to push to is derived automatically from that ref's +3. In the **Repo reference to refresh constraints from** field, enter the ref to refresh + constraints from - for example `v3-3-test`, `v3-3-stable`, `constraints-3-3`, an RC tag, or a + commit hash. Leave it empty to refresh the branch you run from (`main` -> `constraints-main`). + The matching `constraints-X-Y` branch to push to is derived automatically from that ref's `dev/breeze/src/airflow_breeze/branch_defaults.py`, so pointing at anything on the 3.3 line refreshes `constraints-3-3`. -4. Keep **Re-resolve to the newest matching dependencies** enabled (the default) so the run - picks up the latest released providers/dependencies from PyPI. Disable it only if you want - to regenerate constraints strictly from that ref's `uv.lock` without upgrading. +4. Keep **Upgrade deps to newest from PyPI** enabled (the default) so the run picks up the + latest released providers/dependencies from PyPI. Disable it only if you want to regenerate + constraints strictly from that ref's `uv.lock` without upgrading. 5. Once the run finishes, verify the new commit on the matching constraints branch (for example `constraints-3-3` for a 3.3 refresh): diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py index 535d4786934..beab00b8ba2 100644 --- a/dev/breeze/tests/test_selective_checks.py +++ b/dev/breeze/tests/test_selective_checks.py @@ -2697,29 +2697,12 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(mock_get, gith # The image cache is pushed for `python-versions`, so a narrowed list leaves the remaining -# versions with no cache at all and they build cold on every run. The two tests below pin the -# two paths that refresh it: the canary in `additional-ci-image-checks.yml` and the standalone -# `refresh-image-cache.yml`. - - [email protected]( - "github_event", - [GithubEvents.SCHEDULE, GithubEvents.WORKFLOW_DISPATCH], -) -def test_all_python_versions_on_scheduled_canary(github_event): - """Only a `push` may narrow to the default version on a text-only change; the canary may not.""" - stderr = SelectiveChecks( - files=("INTHEWILD.md",), - commit_ref=NEUTRAL_COMMIT, - github_event=github_event, - pr_labels=(), - default_branch="main", - ) - assert_outputs_are_printed({"python-versions": CURRENT_PYTHON_VERSIONS_AS_LIST}, str(stderr)) +# versions with no cache and they build from scratch on every run. `refresh-image-registry-cache.yml` +# forces the label below for exactly that reason; this pins the behaviour it relies on. def test_all_python_versions_with_all_versions_label(): - """`refresh-image-cache.yml` forces this label, on the event that would otherwise narrow.""" + """Set on the event that would otherwise narrow: a text-only push.""" stderr = SelectiveChecks( files=("INTHEWILD.md",), commit_ref=NEUTRAL_COMMIT, diff --git a/dev/images/update_constraints_run_workflow.png b/dev/images/update_constraints_run_workflow.png new file mode 100644 index 00000000000..16eead4701b Binary files /dev/null and b/dev/images/update_constraints_run_workflow.png differ
