This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new e69c1881b32 Refresh the image registry cache in a workflow that cannot
be cancelled (#71171)
e69c1881b32 is described below
commit e69c1881b32c36abb827bae3717eaf46424427bd
Author: Jarek Potiuk <[email protected]>
AuthorDate: Wed Aug 5 18:49:10 2026 +0800
Refresh the image registry cache in a workflow that cannot be cancelled
(#71171)
* Refresh the image registry cache in a workflow that cannot be cancelled
A branch built on push refreshed its registry cache from inside the CI run,
which
cancels in progress on the next push. With no cache the build runs cold,
which widens
the window the next push cancels in, which leaves the cache missing --
v3-3-test held
only the default Python version's amd64 cache because that was the one
matrix entry
that finished in time.
The refresh also has to cover every Python version: a version with no cache
entry
builds from scratch on every run, and selective checks narrows the list on
a text-only
push.
* Refresh constraints from a workflow of its own, covering every flavour
The manual constraints refresh shared a workflow with the automatic uv.lock
push, so it
was filtered by that workflow's paths and buried under its name in the
Actions list.
The documented procedures pointed at local builds -- refresh_images.sh for
the image
cache, breeze commands for the constraints -- which need a buildx/qemu
setup, a
committer login to ghcr.io, and enough bandwidth to push the layers. The
workflows need
none of that, and unlike a local run they always cover every Python version
and every
constraint flavour.
* Exempt a manually started cache refresh from the push cancellation
A push-triggered refresh should give way to the next one -- it is building
the cache of a
commit that is no longer the tip. A manual run should not: it is started
because the
branch has no cache and the push runs are the thing that keeps getting
cancelled.
---
.github/workflows/additional-ci-image-checks.yml | 73 +----------
.github/workflows/ci-amd.yml | 7 -
.github/workflows/ci-arm.yml | 7 -
.github/workflows/refresh-constraints.yml | 72 ++++++++++
.github/workflows/refresh-image-registry-cache.yml | 145 +++++++++++++++++++++
.github/workflows/update-constraints-on-push.yml | 18 ++-
dev/MANUALLY_BUILDING_IMAGES.md | 6 +-
...UALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md | 78 +++++++----
dev/breeze/tests/test_selective_checks.py | 21 +++
9 files changed, 308 insertions(+), 119 deletions(-)
diff --git a/.github/workflows/additional-ci-image-checks.yml
b/.github/workflows/additional-ci-image-checks.yml
index d0a4cb55fa6..42b71e1f32b 100644
--- a/.github/workflows/additional-ci-image-checks.yml
+++ b/.github/workflows/additional-ci-image-checks.yml
@@ -28,18 +28,10 @@ on: # yamllint disable-line rule:truthy
description: "Platform for the build - 'linux/amd64' or 'linux/arm64'"
required: true
type: string
- python-versions:
- description: "The list of python versions (stringified JSON array) to
run the tests on."
- required: true
- type: string
branch:
description: "Branch used to run the CI jobs in (main/v*_*_test)."
required: true
type: string
- constraints-branch:
- description: "Branch used to get constraints from"
- required: true
- type: string
default-python-version:
description: "Which version of python should be used by default"
required: true
@@ -52,80 +44,19 @@ on: # yamllint disable-line rule:truthy
description: "Whether to skip prek hooks (true/false)"
required: true
type: string
- docker-cache:
- description: "Docker cache specification to build the image (registry,
local, disabled)."
- required: true
- type: string
- disable-airflow-repo-cache:
- description: "Disable airflow repo cache read from main."
- required: true
- type: string
- canary-run:
- description: "Whether this is a canary run (true/false)"
- required: true
- type: string
latest-versions-only:
description: "Whether to run only latest versions (true/false)"
required: true
type: string
- include-success-outputs:
- description: "Whether to include success outputs (true/false)"
- required: true
- type: string
debug-resources:
description: "Whether to debug resources (true/false)"
required: true
type: string
- use-uv:
- description: "Whether to use uv to build the image (true/false)"
- required: true
- type: string
permissions:
contents: read
jobs:
- # Push early BuildX cache to GitHub Registry in Apache repository. This
cache does not wait for all the
- # tests to complete - it is run very early in the build process in order to
refresh cache using the
- # current constraints. This will speed up cache refresh in cases when
pyproject.toml changes or in case
- # of Dockerfile changes. Failure in this step is not a problem (at most it
will delay cache refresh).
- # It does not attempt to upgrade to newer dependencies.
- # We only push CI cache as PROD cache usually does not gain as much from
fresh cache because
- # it uses prepared airflow and provider distributions that invalidate the
cache anyway most of the time
- #
- # Runs on `push` as well, not only on the scheduled canary. Release branches
(`v*-*-test`) are built
- # by `push` only, and their other refresh path -- the Regular cache push in
`finalize-tests.yml` -- sits
- # behind the whole test matrix. A branch that is red therefore never
refreshes its cache at all, and the
- # staler the cache gets the slower it is to turn green again. `main` is
unaffected either way: it is not
- # built on push, so it keeps refreshing through the scheduled canary.
- #
- # Pull requests stay excluded -- `canary-run` is also true for a PR carrying
the `canary` label, and
- # pushing cache from an unmerged branch is not something a label should
authorise. This is the same
- # condition the Regular cache push uses.
- push-early-buildx-cache-to-github-registry:
- name: Push Early Image Cache
- uses: ./.github/workflows/push-image-cache.yml
- permissions:
- contents: read
- # This write is only given here for `push` events from "apache/airflow"
repo. It is not given for PRs
- # from forks. This is to prevent malicious PRs from creating images in
the "apache/airflow" repo.
- packages: write
- with:
- runners: ${{ inputs.runners }}
- cache-type: "Early"
- include-prod-images: "false"
- push-latest-images: "false"
- platform: ${{ inputs.platform }}
- python-versions: ${{ inputs.python-versions }}
- branch: ${{ inputs.branch }}
- constraints-branch: ${{ inputs.constraints-branch }}
- use-uv: ${{ inputs.use-uv }}
- include-success-outputs: ${{ inputs.include-success-outputs }}
- docker-cache: ${{ inputs.docker-cache }}
- disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }}
- if: inputs.canary-run == 'true' && github.event_name != 'pull_request'
-
- # Check that after earlier cache push, breeze command will build quickly
- # This build is a bit slow from in-the scratch builds, so we should run it
only in
- # regular PRs
+ # Check that the image builds quickly from the registry cache. This build is
slow from
+ # scratch, so we only run it on regular PRs.
check-that-image-builds-quickly:
timeout-minutes: 25
name: Check that image builds quickly
diff --git a/.github/workflows/ci-amd.yml b/.github/workflows/ci-amd.yml
index 72ef23494d7..63b07e4ce6d 100644
--- a/.github/workflows/ci-amd.yml
+++ b/.github/workflows/ci-amd.yml
@@ -297,19 +297,12 @@ jobs:
with:
runners: ${{ needs.build-info.outputs.runner-type }}
platform: ${{ needs.build-info.outputs.platform }}
- python-versions: ${{ needs.build-info.outputs.python-versions }}
branch: ${{ needs.build-info.outputs.default-branch }}
- constraints-branch: ${{
needs.build-info.outputs.default-constraints-branch }}
default-python-version: "${{
needs.build-info.outputs.default-python-version }}"
upgrade-to-newer-dependencies: ${{
needs.build-info.outputs.upgrade-to-newer-dependencies }}
skip-prek-hooks: ${{ needs.build-info.outputs.skip-prek-hooks }}
- docker-cache: ${{ needs.build-info.outputs.docker-cache }}
- disable-airflow-repo-cache: ${{
needs.build-info.outputs.disable-airflow-repo-cache }}
- canary-run: ${{ needs.build-info.outputs.canary-run }}
latest-versions-only: ${{ needs.build-info.outputs.latest-versions-only
}}
- include-success-outputs: ${{
needs.build-info.outputs.include-success-outputs }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
- use-uv: ${{ needs.build-info.outputs.use-uv }}
generate-constraints:
name: "Generate constraints"
diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml
index 8c62297ea7e..0c179002e87 100644
--- a/.github/workflows/ci-arm.yml
+++ b/.github/workflows/ci-arm.yml
@@ -286,19 +286,12 @@ jobs:
with:
runners: ${{ needs.build-info.outputs.runner-type }}
platform: ${{ needs.build-info.outputs.platform }}
- python-versions: ${{ needs.build-info.outputs.python-versions }}
branch: ${{ needs.build-info.outputs.default-branch }}
- constraints-branch: ${{
needs.build-info.outputs.default-constraints-branch }}
default-python-version: "${{
needs.build-info.outputs.default-python-version }}"
upgrade-to-newer-dependencies: ${{
needs.build-info.outputs.upgrade-to-newer-dependencies }}
skip-prek-hooks: ${{ needs.build-info.outputs.skip-prek-hooks }}
- docker-cache: ${{ needs.build-info.outputs.docker-cache }}
- disable-airflow-repo-cache: ${{
needs.build-info.outputs.disable-airflow-repo-cache }}
- canary-run: ${{ needs.build-info.outputs.canary-run }}
latest-versions-only: ${{ needs.build-info.outputs.latest-versions-only
}}
- include-success-outputs: ${{
needs.build-info.outputs.include-success-outputs }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
- use-uv: ${{ needs.build-info.outputs.use-uv }}
generate-constraints:
name: "Generate constraints"
diff --git a/.github/workflows/refresh-constraints.yml
b/.github/workflows/refresh-constraints.yml
new file mode 100644
index 00000000000..58295b6c092
--- /dev/null
+++ b/.github/workflows/refresh-constraints.yml
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Refresh constraints
+# Refreshes the pinned constraint files on demand, when waiting for the
automatic path is not an
+# option -- a `main` / `vX-Y-test` build that cannot be made green, or newly
released providers
+# that a release candidate has to pick up before it is promoted.
+#
+# All three constraint flavours are refreshed, not just the PyPI one:
+#
+# * `constraints-source-providers-X.Y.txt` -- providers resolved from this
checkout's sources,
+# which is what a contributor's Breeze environment and CI install from;
+# * `constraints-no-providers-X.Y.txt` -- core dependencies alone, used to
install Airflow
+# without any provider pinned;
+# * `constraints-X.Y.txt` -- providers as published on PyPI, which is what
users install with.
+#
+# Refreshing only the PyPI flavour would leave CI and Breeze on the older
pins, so the three
+# would disagree about the same dependency.
+#
+# This does the same work as the automatic `uv.lock` push (whose workflow it
calls), against the
+# ref you name rather than the branch the push landed on. Run it from `main`:
the workflow
+# definition and `breeze` come from `main`, while the sources come from `ref`,
so refreshing a
+# release line needs no cherry-pick to `vX-Y-test` / `vX-Y-stable` first. The
`constraints-X-Y`
+# branch written to is derived from that ref's `branch_defaults.py`.
+#
+# See dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md.
+on: # yamllint disable-line rule:truthy
+ workflow_dispatch:
+ inputs:
+ ref:
+ description: "Repo reference to refresh constraints from (e.g.
v3-3-stable). Empty = this branch."
+ required: false
+ default: ""
+ type: string
+ upgrade-to-newer-dependencies:
+ description: "Upgrade deps to newest from PyPI"
+ required: false
+ type: boolean
+ default: true
+permissions:
+ contents: read
+concurrency:
+ # Two refreshes of the same ref would race each other onto the same
`constraints-X-Y` branch.
+ group: refresh-constraints-${{ inputs.ref || github.ref }}
+ cancel-in-progress: false
+jobs:
+ refresh-constraints:
+ name: "Refresh constraints"
+ uses: ./.github/workflows/update-constraints-on-push.yml
+ permissions:
+ contents: write
+ packages: write
+ with:
+ ref: ${{ inputs.ref }}
+ upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies
}}
+ secrets:
+ SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
diff --git a/.github/workflows/refresh-image-registry-cache.yml
b/.github/workflows/refresh-image-registry-cache.yml
new file mode 100644
index 00000000000..23b3e6221dc
--- /dev/null
+++ b/.github/workflows/refresh-image-registry-cache.yml
@@ -0,0 +1,145 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Refresh image registry cache
+# Refreshes the CI image cache in the GitHub registry, in a run of its own.
+#
+# It has to be its own workflow rather than a job inside `ci-amd.yml` /
`ci-arm.yml`, because a
+# job cannot opt out of its run being cancelled, and those runs are cancelled
by the next push.
+# With no cache a branch builds cold (~20 minutes per Python version), which
widens the window in
+# which the next push cancels the run, which leaves the cache missing.
`v3-3-test` sat in that
+# loop -- its `linux/amd64` cache covered the default Python version alone,
the single matrix
+# entry that finished before the next push killed the other four seconds
later. Being a separate
+# workflow is what lets a manual run escape that (see the concurrency group
below).
+#
+# Every Python version is refreshed in both trigger paths. That is what makes
the cache useful:
+# a version with no cache entry builds from scratch on every run, whichever
way it got skipped.
+on: # yamllint disable-line rule:truthy
+ workflow_dispatch:
+ inputs:
+ platform:
+ description: "Platform to refresh the cache for"
+ type: choice
+ options:
+ - "both"
+ - "linux/amd64"
+ - "linux/arm64"
+ default: "both"
+ push:
+ # Release-prep and providers branches only -- they are built on push, so
the CI run that
+ # would have refreshed their cache is also the run that keeps being
cancelled. `main` is not
+ # built on push; its cache is refreshed by the Push Image Cache job at the
end of the
+ # scheduled canary, which runs to completion.
+ branches:
+ - v[0-9]+-[0-9]+-test
+ - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+
+permissions:
+ contents: read
+concurrency:
+ # Push runs are never cancelled -- being cancelled by the next merge is the
whole failure this
+ # workflow exists to escape, so a newer one queues behind the one in flight
rather than
+ # replacing it. GitHub keeps at most one run pending per group, so the queue
cannot grow.
+ #
+ # Manual runs are grouped separately, which is what keeps a merge from
cancelling one, and they
+ # do supersede each other: firing the workflow again for the same branch
says the run in flight
+ # is not the one you want.
+ group: refresh-image-registry-cache-${{ github.ref }}-${{ github.event_name
}}
+ cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' }}
+jobs:
+ build-info:
+ name: "Build info"
+ # No outputs yet at this stage, so the runner has to be hard-coded to a
public one.
+ runs-on: ["ubuntu-22.04"]
+ outputs:
+ default-branch: ${{ steps.selective-checks.outputs.default-branch }}
+ default-constraints-branch: ${{
steps.selective-checks.outputs.default-constraints-branch }}
+ disable-airflow-repo-cache: ${{
steps.selective-checks.outputs.disable-airflow-repo-cache }}
+ docker-cache: ${{ steps.selective-checks.outputs.docker-cache }}
+ include-success-outputs: ${{
steps.selective-checks.outputs.include-success-outputs }}
+ python-versions: ${{ steps.selective-checks.outputs.python-versions }}
+ use-uv: ${{ steps.selective-checks.outputs.force-pip == 'true' &&
'false' || 'true' }}
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: sudo rm -rf ${GITHUB_WORKSPACE}/*
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ persist-credentials: false
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ - name: Selective checks
+ id: selective-checks
+ env:
+ # Forces every Python version. Selective checks narrows
`python-versions` down to the
+ # default one on a push where only text files changed
+ # (`_should_run_all_tests_and_versions`), which would leave the rest
of the versions
+ # with no cache -- the failure mode this workflow exists to prevent.
+ PR_LABELS: '["all versions"]'
+ COMMIT_REF: "${{ github.sha }}"
+ VERBOSE: "false"
+ run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}
+
+ refresh-ci-cache-amd:
+ name: "Refresh cache linux/amd64"
+ needs: [build-info]
+ uses: ./.github/workflows/push-image-cache.yml
+ permissions:
+ contents: read
+ # Only granted for `push` / `workflow_dispatch` on apache/airflow
itself, never for forks.
+ packages: write
+ with:
+ runners: '["ubuntu-22.04"]'
+ platform: "linux/amd64"
+ cache-type: "Refresh"
+ # The PROD cache is built from distributions prepared earlier in a CI
run, and a standalone
+ # refresh has no such artifact. The CI cache is also where the build
time goes.
+ include-prod-images: "false"
+ # `latest` is what a build pulls when it has no cache at all. Moving it
belongs to a canary
+ # run that has passed its tests, not to a cache refresh.
+ push-latest-images: "false"
+ python-versions: ${{ needs.build-info.outputs.python-versions }}
+ branch: ${{ needs.build-info.outputs.default-branch }}
+ constraints-branch: ${{
needs.build-info.outputs.default-constraints-branch }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
+ include-success-outputs: ${{
needs.build-info.outputs.include-success-outputs }}
+ docker-cache: ${{ needs.build-info.outputs.docker-cache }}
+ disable-airflow-repo-cache: ${{
needs.build-info.outputs.disable-airflow-repo-cache }}
+ if: github.event_name == 'push' || inputs.platform == 'both' ||
inputs.platform == 'linux/amd64'
+
+ refresh-ci-cache-arm:
+ name: "Refresh cache linux/arm64"
+ needs: [build-info]
+ uses: ./.github/workflows/push-image-cache.yml
+ permissions:
+ contents: read
+ packages: write
+ with:
+ runners: '["ubuntu-22.04-arm"]'
+ platform: "linux/arm64"
+ cache-type: "Refresh"
+ include-prod-images: "false"
+ push-latest-images: "false"
+ python-versions: ${{ needs.build-info.outputs.python-versions }}
+ branch: ${{ needs.build-info.outputs.default-branch }}
+ constraints-branch: ${{
needs.build-info.outputs.default-constraints-branch }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
+ include-success-outputs: ${{
needs.build-info.outputs.include-success-outputs }}
+ docker-cache: ${{ needs.build-info.outputs.docker-cache }}
+ disable-airflow-repo-cache: ${{
needs.build-info.outputs.disable-airflow-repo-cache }}
+ if: github.event_name == 'push' || inputs.platform == 'both' ||
inputs.platform == 'linux/arm64'
diff --git a/.github/workflows/update-constraints-on-push.yml
b/.github/workflows/update-constraints-on-push.yml
index e7d273f319b..c0fc95b3e16 100644
--- a/.github/workflows/update-constraints-on-push.yml
+++ b/.github/workflows/update-constraints-on-push.yml
@@ -16,14 +16,15 @@
# under the License.
#
---
-name: Update constraints (on uv.lock push or manual dispatch)
+name: Update constraints (on uv.lock push)
# This workflow refreshes the pinned constraint files stored in the
# `constraints-*` branches. It runs automatically whenever `uv.lock` changes on
-# `main` or a `vX-Y-test` branch, and can also be triggered manually via the
-# "Run workflow" button (workflow_dispatch) - for example to pick up newly
-# released providers/dependencies from PyPI just before promoting an RC.
+# `main` or a `vX-Y-test` branch. The manual path lives in
+# `refresh-constraints.yml`, which calls this workflow -- keeping the two
triggers
+# in separate workflows means a manual refresh is not filtered by `paths:
uv.lock`
+# and shows up in the Actions list under its own name.
#
-# The manual run is always launched from `main` (so the workflow definition and
+# A manual run is always launched from `main` (so the workflow definition and
# `breeze` come from `main`), and the `ref` input selects the commit-ish
# (branch, tag or commit hash) whose sources the constraints are refreshed
from.
# The `constraints-X-Y` branch to push to is derived from that ref's
@@ -37,7 +38,7 @@ on: # yamllint disable-line rule:truthy
- v[0-9]+-[0-9]+-test
paths:
- 'uv.lock'
- workflow_dispatch:
+ workflow_call:
inputs:
# `ref` is optional. Leave it empty to refresh constraints for the branch
# you run the workflow from (normally `main` -> `constraints-main`). Set
a
@@ -51,8 +52,13 @@ on: # yamllint disable-line rule:truthy
type: string
upgrade-to-newer-dependencies:
description: "Upgrade deps to newest from PyPI"
+ required: false
type: boolean
default: true
+ secrets:
+ SLACK_BOT_TOKEN:
+ description: "Slack bot token used to post provider-downgrade alerts
(optional)."
+ required: false
permissions:
contents: read
diff --git a/dev/MANUALLY_BUILDING_IMAGES.md b/dev/MANUALLY_BUILDING_IMAGES.md
index 18ba10dd210..aa0c8a4c14f 100644
--- a/dev/MANUALLY_BUILDING_IMAGES.md
+++ b/dev/MANUALLY_BUILDING_IMAGES.md
@@ -94,7 +94,7 @@ The benefit of this method is that the images can be built
and pushed in a singl
to build and push them separately and perform the additional manifest merge
step.
```bash
-docker buildx create --name airflow_cache --driver docker-container
unix:///var/run/docker.sock # your local builder (you might want to use
+docker buildx create --name airflow_cache --driver docker-container
unix:///var/run/docker.sock # your local builder
docker buildx create --name airflow_cache --append HOST:PORT # your remote
builder
```
@@ -167,7 +167,7 @@ with appropriate versions).
You can then merge the images into a single multi-platform image with:
```bash
-docker release-management merge-prod-images --airflow-version "${VERSION}"
--metadata-folder dist
+breeze release-management merge-prod-images --airflow-version "${VERSION}"
--metadata-folder dist
```
The same can be repeated for slim images by adding `--slim-images` option to
the command, the manifests are
@@ -189,5 +189,5 @@ breeze release-management release-prod-images --slim-images
--airflow-version "$
Merging the images:
```bash
-docker release-management merge-prod-images --slim-images --airflow-version
"${VERSION}" --metadata-folder dist
+breeze release-management merge-prod-images --slim-images --airflow-version
"${VERSION}" --metadata-folder dist
```
diff --git a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md
b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md
index 01bdd7f7504..cd07deec80b 100644
--- a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md
+++ b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md
@@ -25,8 +25,9 @@
- [Automated image cache and constraints refreshing in
CI](#automated-image-cache-and-constraints-refreshing-in-ci)
- [Manually refreshing the image cache](#manually-refreshing-the-image-cache)
- [Why we need to update image cache
manually](#why-we-need-to-update-image-cache-manually)
- - [Prerequisites](#prerequisites)
- - [How to refresh the image cache](#how-to-refresh-the-image-cache)
+ - [How to refresh the image cache via the CI workflow
(recommended)](#how-to-refresh-the-image-cache-via-the-ci-workflow-recommended)
+ - [Prerequisites for refreshing
locally](#prerequisites-for-refreshing-locally)
+ - [How to refresh the image cache
locally](#how-to-refresh-the-image-cache-locally)
- [Is it safe to refresh the image
cache?](#is-it-safe-to-refresh-the-image-cache)
- [What the command does](#what-the-command-does)
- [Manually generating constraint files](#manually-generating-constraint-files)
@@ -69,17 +70,17 @@ rebuilding of [Breeze](./breeze/doc/README.rst) images for
development purpose.
* The latest
[constraints](/contributing-docs/13_airflow_dependencies_and_extras.rst#pinned-constraint-files)
are pushed to appropriate branch after all tests succeed in the
`canary` build.
-* The [images](breeze/doc/ci/02_images.md) in `ghcr.io` registry are refreshed
early at the beginning of the
- `canary` build. This is done twice during the canary build:
- * By the `Push Early Image Cache` job that is run at the beginning of the
`canary` build. This cover the
- case when there are new dependencies added or Dockerfile/scripts change.
Thanks to that step, subsequent
- PRs will be faster when they use the new Dockerfile/script. Those jobs
**might fail** occasionally,
- if the latest PR added some conflicting dependencies with current
constraints. This is not a problem
- and when it happens, it will be fixed by the next step.
- * By the `Push Image Cache` job that is run at the end of the `canary`
build. This covers the case when
- cache is also refreshed after than `main` build succeeds after the new
constraints are pushed. This
- step makes sure that constraints are committed and pushed just before the
cache is refreshed, so
- there is no problem with conflicting dependencies.
+* The [images](breeze/doc/ci/02_images.md) in `ghcr.io` registry are refreshed
by the `Push Image Cache`
+ job at the end of the `canary` build, once the tests have passed. Running it
last is what makes the
+ constraints and the cache agree: the constraints are committed and pushed
just before the cache is
+ refreshed from them.
+
+* Branches that are built on `push` rather than by the scheduled `canary` --
the release-prep
+ (`vX-Y-test`) and providers branches -- are refreshed by the
+ [`Refresh image registry
cache`](../.github/workflows/refresh-image-registry-cache.yml) workflow
+ instead. It runs on its own rather than inside the CI run, because the CI
run is cancelled by the
+ next push and a cache refresh that keeps being cancelled never lands. It
covers every Python version
+ on both platforms, and it is the same workflow you run by hand (see below).
# Manually refreshing the image cache
@@ -87,14 +88,35 @@ rebuilding of [Breeze](./breeze/doc/README.rst) images for
development purpose.
## Why we need to update image cache manually
Sometimes, when we have a problem with our CI running and flakiness of GitHub
Actions runners or our
-tests, the refresh might not be triggered. This has been mitigated by "Push
Early Image Cache" job added in
-our CI, but there are other reasons you might want to refresh the cache.
Sometimes we want to refresh the
+tests, the refresh might not be triggered. Sometimes we want to refresh the
image cache in `vX_Y_test` branch (following our convention of branch names
`vX_Y_test` branch is the branch
used to release all `X.Y.*` versions of airflow) before we attempt to push a
change there.
There are no PRs happening in this branch, so manual refresh before we make a
PR might speed up the PR build.
Or sometimes we just refreshed the constraints (see below) and we want the
cache to include those.
-## Prerequisites
+## How to refresh the image cache via the CI workflow (recommended)
+
+Run the [`Refresh image registry
cache`](../.github/workflows/refresh-image-registry-cache.yml) workflow
+from the Actions tab, selecting the branch whose cache you want to refresh in
GitHub's branch dropdown --
+the cache is namespaced per branch, so the branch you pick is the namespace
written to. The `platform`
+input refreshes `linux/amd64`, `linux/arm64`, or both (the default).
+
+Prefer this over the local route below. The workflow needs no buildx / qemu
setup on your machine, it
+does not depend on your upload bandwidth, it always covers every Python
version, and it runs with the
+registry credentials the CI already has, so you do not need to be logged in to
`ghcr.io` as a committer.
+
+A run you start by hand is not cancelled by merges landing while it works --
it uses a concurrency
+group of its own for that reason. Starting it again for the same branch does
replace the run in
+flight. Push-triggered refreshes are never cancelled either; a newer one waits
for the one in
+flight to finish.
+
+The workflow definition has to exist on the branch you select, so a release
branch needs the workflow
+backported before it can be refreshed there.
+
+## Prerequisites for refreshing locally
+
+The rest of this chapter describes refreshing the cache from your own machine.
It is a fallback for when
+the workflow cannot be used at all -- it is slower, and easy to get subtly
wrong.
Note that in order to refresh images you have to not only have `buildx`
command installed for docker,
but you should also make sure that you have the buildkit builder configured
and set.
@@ -146,7 +168,7 @@ docker buildx ls
airflow_cache1 tcp://127.0.0.1:2375
```
-## How to refresh the image cache
+## How to refresh the image cache locally
The images can be rebuilt and refreshed after the constraints are pushed.
Refreshing image for all
python version is as simple as running the
[refresh_images.sh](refresh_images.sh) script which will
@@ -217,11 +239,17 @@ and you need to be sure what you are doing, but you can
always do it manually if
## How to refresh constraints via the CI workflow (recommended)
The easiest way to refresh the constraints - for example to pick up newly
released
-providers/dependencies from PyPI just before promoting an RC - is to trigger
the
-[`Update constraints`](../.github/workflows/update-constraints-on-push.yml)
workflow manually
-instead of running the `breeze` commands locally. The workflow runs exactly
the same steps
-that run automatically when `uv.lock` changes, builds the CI images, generates
all constraint
-flavours and commits/pushes them to the matching `constraints-*` branch.
+providers/dependencies from PyPI just before promoting an RC - is to run the
+[`Refresh constraints`](../.github/workflows/refresh-constraints.yml) workflow
instead of
+running the `breeze` commands locally. It runs exactly the same steps that run
automatically
+when `uv.lock` changes (it calls that workflow), builds the CI images, and
commits/pushes to the
+matching `constraints-*` branch.
+
+It refreshes **all three constraint flavours**, not only the PyPI one:
+`constraints-source-providers-X.Y.txt` (providers from the sources, which is
what CI and Breeze
+install), `constraints-no-providers-X.Y.txt` (core alone) and
`constraints-X.Y.txt` (providers as
+published on PyPI, which is what users install). Refreshing only the PyPI
flavour would leave CI
+and Breeze pinned to the older versions, so the three would disagree about the
same dependency.
The manual run is always launched from `main` and takes a `ref` input that
selects the
commit-ish (branch, tag or commit hash) whose sources the constraints are
refreshed from. You
@@ -236,13 +264,13 @@ workflow); the automatic `uv.lock`-push runs are not
restricted.
To run it:
-
+
-1. Go to the [`Update
constraints`](https://github.com/apache/airflow/actions/workflows/update-constraints-on-push.yml)
+1. Go to the [`Refresh
constraints`](https://github.com/apache/airflow/actions/workflows/refresh-constraints.yml)
workflow in the Actions tab.
2. Click **Run workflow** and keep the branch set to `main` (this is where the
workflow runs
from - it is not the branch whose constraints get refreshed).
-3. In the **Optional repo reference to build constraints** field, enter the
ref to refresh
+3. In the **Repo reference to refresh constraints from** field, enter the ref
to refresh
constraints from - for example `v3-3-test`, `v3-3-stable`,
`constraints-3-3`, an RC tag, or a
commit hash. Leave it empty to refresh the branch you run from (`main` ->
`constraints-main`).
The matching `constraints-X-Y` branch to push to is derived automatically
from that ref's
diff --git a/dev/breeze/tests/test_selective_checks.py
b/dev/breeze/tests/test_selective_checks.py
index a99d5c7bf86..d516aaeee4f 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -29,6 +29,7 @@ from airflow_breeze.global_constants import (
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS,
CI_AMD_PLATFORM,
CI_ARM_PLATFORM,
+ CURRENT_PYTHON_MAJOR_MINOR_VERSIONS,
DEFAULT_KUBERNETES_VERSION,
DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
JAVA_SDK_VERSION,
@@ -56,6 +57,9 @@ ALL_KUBERNETES_VERSIONS_AS_STRING = "
".join(ALLOWED_KUBERNETES_VERSIONS)
ALL_KUBERNETES_VERSIONS_AS_LIST = "[" + ", ".join([f"'{v}'" for v in
ALLOWED_KUBERNETES_VERSIONS]) + "]"
ALL_PYTHON_VERSIONS_AS_STRING = " ".join(ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS)
ALL_PYTHON_VERSIONS_AS_LIST = "[" + ", ".join([f"'{v}'" for v in
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS]) + "]"
+CURRENT_PYTHON_VERSIONS_AS_LIST = (
+ "[" + ", ".join([f"'{v}'" for v in CURRENT_PYTHON_MAJOR_MINOR_VERSIONS]) +
"]"
+)
DEFAULT_HELM_K8S_VERSION = ALLOWED_KUBERNETES_VERSIONS[0].lstrip("v")
LAST_HELM_K8S_VERSION = ALLOWED_KUBERNETES_VERSIONS[-1].lstrip("v")
@@ -2884,6 +2888,23 @@ def
test_no_commit_provided_trigger_full_build_for_any_event_type(github_event):
)
+# The image cache is pushed for `python-versions`, so a narrowed list leaves
the remaining
+# versions with no cache and they build from scratch on every run.
`refresh-image-registry-cache.yml`
+# forces the label below for exactly that reason; this pins the behaviour it
relies on.
+
+
+def test_all_python_versions_with_all_versions_label():
+ """Set on the event that would otherwise narrow: a text-only push."""
+ stderr = SelectiveChecks(
+ files=("INTHEWILD.md",),
+ commit_ref=NEUTRAL_COMMIT,
+ github_event=GithubEvents.PUSH,
+ pr_labels=("all versions",),
+ default_branch="main",
+ )
+ assert_outputs_are_printed({"python-versions":
CURRENT_PYTHON_VERSIONS_AS_LIST}, str(stderr))
+
+
@pytest.mark.parametrize(
"github_event",
[
