This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch refresh-image-registry-cache-workflow in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 95de7e7b07667da1bca4bcd8bfd81e0c7844c2ca Author: Jarek Potiuk <[email protected]> AuthorDate: Wed Aug 5 18:19:50 2026 +0800 Refresh constraints from a workflow of its own, covering every flavour The manual constraints refresh shared a workflow with the automatic uv.lock push, so it was filtered by that workflow's paths and buried under its name in the Actions list. The documented procedures pointed at local builds -- refresh_images.sh for the image cache, breeze commands for the constraints -- which need a buildx/qemu setup, a committer login to ghcr.io, and enough bandwidth to push the layers. The workflows need none of that, and unlike a local run they always cover every Python version and every constraint flavour. --- .github/workflows/refresh-constraints.yml | 72 +++++++++++++++++++++ .github/workflows/update-constraints-on-push.yml | 18 ++++-- dev/MANUALLY_BUILDING_IMAGES.md | 6 +- ...UALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md | 73 ++++++++++++++-------- 4 files changed, 135 insertions(+), 34 deletions(-) diff --git a/.github/workflows/refresh-constraints.yml b/.github/workflows/refresh-constraints.yml new file mode 100644 index 00000000000..58295b6c092 --- /dev/null +++ b/.github/workflows/refresh-constraints.yml @@ -0,0 +1,72 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: Refresh constraints +# Refreshes the pinned constraint files on demand, when waiting for the automatic path is not an +# option -- a `main` / `vX-Y-test` build that cannot be made green, or newly released providers +# that a release candidate has to pick up before it is promoted. +# +# All three constraint flavours are refreshed, not just the PyPI one: +# +# * `constraints-source-providers-X.Y.txt` -- providers resolved from this checkout's sources, +# which is what a contributor's Breeze environment and CI install from; +# * `constraints-no-providers-X.Y.txt` -- core dependencies alone, used to install Airflow +# without any provider pinned; +# * `constraints-X.Y.txt` -- providers as published on PyPI, which is what users install with. +# +# Refreshing only the PyPI flavour would leave CI and Breeze on the older pins, so the three +# would disagree about the same dependency. +# +# This does the same work as the automatic `uv.lock` push (whose workflow it calls), against the +# ref you name rather than the branch the push landed on. Run it from `main`: the workflow +# definition and `breeze` come from `main`, while the sources come from `ref`, so refreshing a +# release line needs no cherry-pick to `vX-Y-test` / `vX-Y-stable` first. The `constraints-X-Y` +# branch written to is derived from that ref's `branch_defaults.py`. +# +# See dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md. +on: # yamllint disable-line rule:truthy + workflow_dispatch: + inputs: + ref: + description: "Repo reference to refresh constraints from (e.g. v3-3-stable). Empty = this branch." + required: false + default: "" + type: string + upgrade-to-newer-dependencies: + description: "Upgrade deps to newest from PyPI" + required: false + type: boolean + default: true +permissions: + contents: read +concurrency: + # Two refreshes of the same ref would race each other onto the same `constraints-X-Y` branch. + group: refresh-constraints-${{ inputs.ref || github.ref }} + cancel-in-progress: false +jobs: + refresh-constraints: + name: "Refresh constraints" + uses: ./.github/workflows/update-constraints-on-push.yml + permissions: + contents: write + packages: write + with: + ref: ${{ inputs.ref }} + upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies }} + secrets: + SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} diff --git a/.github/workflows/update-constraints-on-push.yml b/.github/workflows/update-constraints-on-push.yml index e7d273f319b..c0fc95b3e16 100644 --- a/.github/workflows/update-constraints-on-push.yml +++ b/.github/workflows/update-constraints-on-push.yml @@ -16,14 +16,15 @@ # under the License. # --- -name: Update constraints (on uv.lock push or manual dispatch) +name: Update constraints (on uv.lock push) # This workflow refreshes the pinned constraint files stored in the # `constraints-*` branches. It runs automatically whenever `uv.lock` changes on -# `main` or a `vX-Y-test` branch, and can also be triggered manually via the -# "Run workflow" button (workflow_dispatch) - for example to pick up newly -# released providers/dependencies from PyPI just before promoting an RC. +# `main` or a `vX-Y-test` branch. The manual path lives in +# `refresh-constraints.yml`, which calls this workflow -- keeping the two triggers +# in separate workflows means a manual refresh is not filtered by `paths: uv.lock` +# and shows up in the Actions list under its own name. # -# The manual run is always launched from `main` (so the workflow definition and +# A manual run is always launched from `main` (so the workflow definition and # `breeze` come from `main`), and the `ref` input selects the commit-ish # (branch, tag or commit hash) whose sources the constraints are refreshed from. # The `constraints-X-Y` branch to push to is derived from that ref's @@ -37,7 +38,7 @@ on: # yamllint disable-line rule:truthy - v[0-9]+-[0-9]+-test paths: - 'uv.lock' - workflow_dispatch: + workflow_call: inputs: # `ref` is optional. Leave it empty to refresh constraints for the branch # you run the workflow from (normally `main` -> `constraints-main`). Set a @@ -51,8 +52,13 @@ on: # yamllint disable-line rule:truthy type: string upgrade-to-newer-dependencies: description: "Upgrade deps to newest from PyPI" + required: false type: boolean default: true + secrets: + SLACK_BOT_TOKEN: + description: "Slack bot token used to post provider-downgrade alerts (optional)." + required: false permissions: contents: read diff --git a/dev/MANUALLY_BUILDING_IMAGES.md b/dev/MANUALLY_BUILDING_IMAGES.md index 18ba10dd210..aa0c8a4c14f 100644 --- a/dev/MANUALLY_BUILDING_IMAGES.md +++ b/dev/MANUALLY_BUILDING_IMAGES.md @@ -94,7 +94,7 @@ The benefit of this method is that the images can be built and pushed in a singl to build and push them separately and perform the additional manifest merge step. ```bash -docker buildx create --name airflow_cache --driver docker-container unix:///var/run/docker.sock # your local builder (you might want to use +docker buildx create --name airflow_cache --driver docker-container unix:///var/run/docker.sock # your local builder docker buildx create --name airflow_cache --append HOST:PORT # your remote builder ``` @@ -167,7 +167,7 @@ with appropriate versions). You can then merge the images into a single multi-platform image with: ```bash -docker release-management merge-prod-images --airflow-version "${VERSION}" --metadata-folder dist +breeze release-management merge-prod-images --airflow-version "${VERSION}" --metadata-folder dist ``` The same can be repeated for slim images by adding `--slim-images` option to the command, the manifests are @@ -189,5 +189,5 @@ breeze release-management release-prod-images --slim-images --airflow-version "$ Merging the images: ```bash -docker release-management merge-prod-images --slim-images --airflow-version "${VERSION}" --metadata-folder dist +breeze release-management merge-prod-images --slim-images --airflow-version "${VERSION}" --metadata-folder dist ``` diff --git a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md index 01bdd7f7504..6ebbf9e964f 100644 --- a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md +++ b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md @@ -25,8 +25,9 @@ - [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci) - [Manually refreshing the image cache](#manually-refreshing-the-image-cache) - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually) - - [Prerequisites](#prerequisites) - - [How to refresh the image cache](#how-to-refresh-the-image-cache) + - [How to refresh the image cache via the CI workflow (recommended)](#how-to-refresh-the-image-cache-via-the-ci-workflow-recommended) + - [Prerequisites for refreshing locally](#prerequisites-for-refreshing-locally) + - [How to refresh the image cache locally](#how-to-refresh-the-image-cache-locally) - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache) - [What the command does](#what-the-command-does) - [Manually generating constraint files](#manually-generating-constraint-files) @@ -69,17 +70,17 @@ rebuilding of [Breeze](./breeze/doc/README.rst) images for development purpose. * The latest [constraints](/contributing-docs/13_airflow_dependencies_and_extras.rst#pinned-constraint-files) are pushed to appropriate branch after all tests succeed in the `canary` build. -* The [images](breeze/doc/ci/02_images.md) in `ghcr.io` registry are refreshed early at the beginning of the - `canary` build. This is done twice during the canary build: - * By the `Push Early Image Cache` job that is run at the beginning of the `canary` build. This cover the - case when there are new dependencies added or Dockerfile/scripts change. Thanks to that step, subsequent - PRs will be faster when they use the new Dockerfile/script. Those jobs **might fail** occasionally, - if the latest PR added some conflicting dependencies with current constraints. This is not a problem - and when it happens, it will be fixed by the next step. - * By the `Push Image Cache` job that is run at the end of the `canary` build. This covers the case when - cache is also refreshed after than `main` build succeeds after the new constraints are pushed. This - step makes sure that constraints are committed and pushed just before the cache is refreshed, so - there is no problem with conflicting dependencies. +* The [images](breeze/doc/ci/02_images.md) in `ghcr.io` registry are refreshed by the `Push Image Cache` + job at the end of the `canary` build, once the tests have passed. Running it last is what makes the + constraints and the cache agree: the constraints are committed and pushed just before the cache is + refreshed from them. + +* Branches that are built on `push` rather than by the scheduled `canary` -- the release-prep + (`vX-Y-test`) and providers branches -- are refreshed by the + [`Refresh image registry cache`](../.github/workflows/refresh-image-registry-cache.yml) workflow + instead. It runs on its own rather than inside the CI run, because the CI run is cancelled by the + next push and a cache refresh that keeps being cancelled never lands. It covers every Python version + on both platforms, and it is the same workflow you run by hand (see below). # Manually refreshing the image cache @@ -87,14 +88,30 @@ rebuilding of [Breeze](./breeze/doc/README.rst) images for development purpose. ## Why we need to update image cache manually Sometimes, when we have a problem with our CI running and flakiness of GitHub Actions runners or our -tests, the refresh might not be triggered. This has been mitigated by "Push Early Image Cache" job added in -our CI, but there are other reasons you might want to refresh the cache. Sometimes we want to refresh the +tests, the refresh might not be triggered. Sometimes we want to refresh the image cache in `vX_Y_test` branch (following our convention of branch names `vX_Y_test` branch is the branch used to release all `X.Y.*` versions of airflow) before we attempt to push a change there. There are no PRs happening in this branch, so manual refresh before we make a PR might speed up the PR build. Or sometimes we just refreshed the constraints (see below) and we want the cache to include those. -## Prerequisites +## How to refresh the image cache via the CI workflow (recommended) + +Run the [`Refresh image registry cache`](../.github/workflows/refresh-image-registry-cache.yml) workflow +from the Actions tab, selecting the branch whose cache you want to refresh in GitHub's branch dropdown -- +the cache is namespaced per branch, so the branch you pick is the namespace written to. The `platform` +input refreshes `linux/amd64`, `linux/arm64`, or both (the default). + +Prefer this over the local route below. The workflow needs no buildx / qemu setup on your machine, it +does not depend on your upload bandwidth, it always covers every Python version, and it runs with the +registry credentials the CI already has, so you do not need to be logged in to `ghcr.io` as a committer. + +The workflow definition has to exist on the branch you select, so a release branch needs the workflow +backported before it can be refreshed there. + +## Prerequisites for refreshing locally + +The rest of this chapter describes refreshing the cache from your own machine. It is a fallback for when +the workflow cannot be used at all -- it is slower, and easy to get subtly wrong. Note that in order to refresh images you have to not only have `buildx` command installed for docker, but you should also make sure that you have the buildkit builder configured and set. @@ -146,7 +163,7 @@ docker buildx ls airflow_cache1 tcp://127.0.0.1:2375 ``` -## How to refresh the image cache +## How to refresh the image cache locally The images can be rebuilt and refreshed after the constraints are pushed. Refreshing image for all python version is as simple as running the [refresh_images.sh](refresh_images.sh) script which will @@ -217,11 +234,17 @@ and you need to be sure what you are doing, but you can always do it manually if ## How to refresh constraints via the CI workflow (recommended) The easiest way to refresh the constraints - for example to pick up newly released -providers/dependencies from PyPI just before promoting an RC - is to trigger the -[`Update constraints`](../.github/workflows/update-constraints-on-push.yml) workflow manually -instead of running the `breeze` commands locally. The workflow runs exactly the same steps -that run automatically when `uv.lock` changes, builds the CI images, generates all constraint -flavours and commits/pushes them to the matching `constraints-*` branch. +providers/dependencies from PyPI just before promoting an RC - is to run the +[`Refresh constraints`](../.github/workflows/refresh-constraints.yml) workflow instead of +running the `breeze` commands locally. It runs exactly the same steps that run automatically +when `uv.lock` changes (it calls that workflow), builds the CI images, and commits/pushes to the +matching `constraints-*` branch. + +It refreshes **all three constraint flavours**, not only the PyPI one: +`constraints-source-providers-X.Y.txt` (providers from the sources, which is what CI and Breeze +install), `constraints-no-providers-X.Y.txt` (core alone) and `constraints-X.Y.txt` (providers as +published on PyPI, which is what users install). Refreshing only the PyPI flavour would leave CI +and Breeze pinned to the older versions, so the three would disagree about the same dependency. The manual run is always launched from `main` and takes a `ref` input that selects the commit-ish (branch, tag or commit hash) whose sources the constraints are refreshed from. You @@ -236,13 +259,13 @@ workflow); the automatic `uv.lock`-push runs are not restricted. To run it: - + -1. Go to the [`Update constraints`](https://github.com/apache/airflow/actions/workflows/update-constraints-on-push.yml) +1. Go to the [`Refresh constraints`](https://github.com/apache/airflow/actions/workflows/refresh-constraints.yml) workflow in the Actions tab. 2. Click **Run workflow** and keep the branch set to `main` (this is where the workflow runs from - it is not the branch whose constraints get refreshed). -3. In the **Optional repo reference to build constraints** field, enter the ref to refresh +3. In the **Repo reference to refresh constraints from** field, enter the ref to refresh constraints from - for example `v3-3-test`, `v3-3-stable`, `constraints-3-3`, an RC tag, or a commit hash. Leave it empty to refresh the branch you run from (`main` -> `constraints-main`). The matching `constraints-X-Y` branch to push to is derived automatically from that ref's
