This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v3-3-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-3-test by this push:
new 052346847bb [v3-3-test] Run short CI bookkeeping jobs on ubuntu-slim
runners (#72852) (#72874)
052346847bb is described below
commit 052346847bb17e98550c689a866f35d5af830d5b
Author: Jarek Potiuk <[email protected]>
AuthorDate: Thu Sep 10 10:24:40 2026 +0200
[v3-3-test] Run short CI bookkeeping jobs on ubuntu-slim runners (#72852)
(#72874)
* Run short CI bookkeeping jobs on ubuntu-slim runners
A large share of our workflow jobs never build an image, never install
Airflow and never run a test suite - they compute a matrix, post a Slack
message, close a stale issue or check a newsfragment name. Those jobs held
a 2-core runner for their whole (short) lifetime, competing with the builds
and test shards that genuinely need one.
ubuntu-slim is a single-core, 5 GB, unprivileged container image, which is
the right size for that class of work and keeps the larger runners free for
jobs that cannot fit on it. Its constraints - no Docker daemon, an
externally-managed system Python, a minimal toolset - decided which jobs
moved: anything touching Breeze, Docker, Gradle or a real test suite stayed
where it was.
Three of the moved jobs installed uv by extracting the version from uv.lock
and pip-installing it into the system interpreter, which PEP 668 refuses on
this image. They now use astral-sh/setup-uv with version-file: uv.lock,
which resolves the same locked version and lets uv provision its own Python.
Claude-Session: https://claude.ai/code/session_01UL4HBX63zq4um1L4XBcJYy
* Update dev/breeze/doc/ci/05_workflows.md
---------
(cherry picked from commit 1685bd6f0ca02acfe1d2a0f88b5e29a977323432)
Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Wei Lee <[email protected]>
---
.github/workflows/automatic-backport.yml | 2 +-
.github/workflows/check-newsfragment-pr-number.yml | 2 +-
.github/workflows/ci-amd.yml | 4 +--
.github/workflows/ci-arm.yml | 4 +--
.github/workflows/ci-duration-monitor.yml | 2 +-
.github/workflows/ci-notification.yml | 11 +++++---
.github/workflows/codeql-analysis.yml | 2 +-
.github/workflows/e2e-flaky-tests-report.yml | 2 +-
.github/workflows/java-sdk-dependency-security.yml | 2 +-
.github/workflows/milestone-tag-assistant.yml | 2 +-
.github/workflows/notify-uv-lock-conflicts.yml | 15 +++--------
.github/workflows/recheck-old-bug-report.yml | 2 +-
.github/workflows/registry-backfill.yml | 2 +-
.github/workflows/registry-tests.yml | 6 +++--
.../scheduled-verify-release-calendar.yml | 15 +++--------
.github/workflows/stale.yml | 2 +-
.../update-constraints-on-push-stable.yml | 2 +-
.github/workflows/update-constraints-on-push.yml | 2 +-
dev/breeze/doc/ci/05_workflows.md | 31 ++++++++++++++++++++++
19 files changed, 66 insertions(+), 44 deletions(-)
diff --git a/.github/workflows/automatic-backport.yml
b/.github/workflows/automatic-backport.yml
index afe20ce780d..daa142e9dfc 100644
--- a/.github/workflows/automatic-backport.yml
+++ b/.github/workflows/automatic-backport.yml
@@ -26,7 +26,7 @@ permissions:
jobs:
get-pr-info:
name: "Get PR information"
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
outputs:
branches: ${{ steps.pr-info.outputs.branches }}
commit-sha: ${{ github.sha }}
diff --git a/.github/workflows/check-newsfragment-pr-number.yml
b/.github/workflows/check-newsfragment-pr-number.yml
index 2c9fce37b82..3f81dbb8d0e 100644
--- a/.github/workflows/check-newsfragment-pr-number.yml
+++ b/.github/workflows/check-newsfragment-pr-number.yml
@@ -34,7 +34,7 @@ concurrency:
jobs:
check-newsfragment-pr-number:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip
newsfragment check') }}
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
timeout-minutes: 5
steps:
- name: Check newsfragment PR number
diff --git a/.github/workflows/ci-amd.yml b/.github/workflows/ci-amd.yml
index ef4a445a0a8..567df5b02c1 100644
--- a/.github/workflows/ci-amd.yml
+++ b/.github/workflows/ci-amd.yml
@@ -225,7 +225,7 @@ jobs:
print-platform:
name: "Platform: AMD"
needs: [build-info]
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
steps:
- name: "Print architecture"
run: "echo '## Architecture: AMD' >> $GITHUB_STEP_SUMMARY"
@@ -1109,7 +1109,7 @@ jobs:
!cancelled() &&
github.event_name == 'schedule' &&
github.run_attempt == 1
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml
index 55a984725c3..7ea253c0987 100644
--- a/.github/workflows/ci-arm.yml
+++ b/.github/workflows/ci-arm.yml
@@ -218,7 +218,7 @@ jobs:
print-platform:
name: "Platform: ARM"
needs: [build-info]
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
steps:
- name: "Print architecture"
run: "echo '## Architecture: ARM' >> $GITHUB_STEP_SUMMARY"
@@ -1102,7 +1102,7 @@ jobs:
!cancelled() &&
github.event_name == 'schedule' &&
github.run_attempt == 1
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
diff --git a/.github/workflows/ci-duration-monitor.yml
b/.github/workflows/ci-duration-monitor.yml
index 4e878002dd4..52f2d06c2fb 100644
--- a/.github/workflows/ci-duration-monitor.yml
+++ b/.github/workflows/ci-duration-monitor.yml
@@ -34,7 +34,7 @@ jobs:
monitor-ci-durations:
name: "Monitor CI durations on main"
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
diff --git a/.github/workflows/ci-notification.yml
b/.github/workflows/ci-notification.yml
index 169b8fb3cff..4cf6f087bda 100644
--- a/.github/workflows/ci-notification.yml
+++ b/.github/workflows/ci-notification.yml
@@ -40,18 +40,21 @@ jobs:
# Track AMD; ARM is the canary slot and is reported separately by
# `ci-arm.yml`'s notify-slack job on schedule events.
workflow-id: ["ci-amd.yml"]
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
with:
persist-credentials: false
+ - name: "Install uv"
+ uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d #
v10.0.1
+ with:
+ version-file: uv.lock
+
- name: "Find workflow run status"
id: find-workflow-run-status
- run: |
- python3 -m pip install uv
- uv run ./dev/breeze/src/airflow_breeze/utils/workflow_status.py
+ run: uv run ./dev/breeze/src/airflow_breeze/utils/workflow_status.py
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
workflow_branch: ${{ matrix.branch }}
diff --git a/.github/workflows/codeql-analysis.yml
b/.github/workflows/codeql-analysis.yml
index 201a1b18843..7379d4de4d7 100644
--- a/.github/workflows/codeql-analysis.yml
+++ b/.github/workflows/codeql-analysis.yml
@@ -35,7 +35,7 @@ concurrency:
jobs:
detect-languages:
name: Detect languages to scan
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
permissions:
contents: read
pull-requests: read
diff --git a/.github/workflows/e2e-flaky-tests-report.yml
b/.github/workflows/e2e-flaky-tests-report.yml
index bc922671d46..712dd81da39 100644
--- a/.github/workflows/e2e-flaky-tests-report.yml
+++ b/.github/workflows/e2e-flaky-tests-report.yml
@@ -33,7 +33,7 @@ jobs:
analyze-flaky-tests:
name: "Analyze E2E flaky tests"
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
diff --git a/.github/workflows/java-sdk-dependency-security.yml
b/.github/workflows/java-sdk-dependency-security.yml
index eab793b05ec..6810de173d8 100644
--- a/.github/workflows/java-sdk-dependency-security.yml
+++ b/.github/workflows/java-sdk-dependency-security.yml
@@ -42,7 +42,7 @@ jobs:
dependency-review:
name: Reject vulnerable dependency changes
if: github.event_name == 'pull_request'
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
steps:
- name: Review dependency changes
uses:
actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 #
v5.0.0
diff --git a/.github/workflows/milestone-tag-assistant.yml
b/.github/workflows/milestone-tag-assistant.yml
index 3592bc5235c..79a9c11f965 100644
--- a/.github/workflows/milestone-tag-assistant.yml
+++ b/.github/workflows/milestone-tag-assistant.yml
@@ -35,7 +35,7 @@ permissions:
jobs:
get-pr-info:
name: "Get PR information"
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
outputs:
should-run: ${{ steps.pr-info.outputs.should-run }}
pr-number: ${{ steps.pr-info.outputs.pr-number }}
diff --git a/.github/workflows/notify-uv-lock-conflicts.yml
b/.github/workflows/notify-uv-lock-conflicts.yml
index 95dace06996..ec0ea2c28f6 100644
--- a/.github/workflows/notify-uv-lock-conflicts.yml
+++ b/.github/workflows/notify-uv-lock-conflicts.yml
@@ -29,7 +29,7 @@ permissions:
jobs:
notify:
name: "Notify open PRs that conflict on uv.lock"
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
timeout-minutes: 10
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -37,16 +37,9 @@ jobs:
with:
persist-credentials: false
- name: "Install uv"
- # Extract uv version from uv.lock. The format is stable: the line
- # immediately after `name = "uv"` is `version = "<X.Y.Z>"`.
- run: |
- UV_VERSION=$(sed -n '/^name = "uv"$/{n;s/^version =
"\(.*\)"$/\1/p;}' uv.lock)
- if [[ -z "${UV_VERSION}" ]]; then
- echo "Failed to extract uv version from uv.lock" >&2
- exit 1
- fi
- echo "Installing uv==${UV_VERSION}"
- pip install "uv==${UV_VERSION}"
+ uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d #
v10.0.1
+ with:
+ version-file: uv.lock
- name: "Notify open PRs"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/recheck-old-bug-report.yml
b/.github/workflows/recheck-old-bug-report.yml
index dcda4355553..049071dfad0 100644
--- a/.github/workflows/recheck-old-bug-report.yml
+++ b/.github/workflows/recheck-old-bug-report.yml
@@ -26,7 +26,7 @@ permissions:
issues: write
jobs:
recheck-old-bug-report:
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
steps:
- uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0
with:
diff --git a/.github/workflows/registry-backfill.yml
b/.github/workflows/registry-backfill.yml
index 05224c3f417..48768822c96 100644
--- a/.github/workflows/registry-backfill.yml
+++ b/.github/workflows/registry-backfill.yml
@@ -78,7 +78,7 @@ jobs:
disable-airflow-repo-cache: "false"
prepare:
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
outputs:
matrix: ${{ steps.matrix.outputs.matrix }}
bucket: ${{ steps.destination.outputs.bucket }}
diff --git a/.github/workflows/registry-tests.yml
b/.github/workflows/registry-tests.yml
index b14daf9ea0e..9f9a8343356 100644
--- a/.github/workflows/registry-tests.yml
+++ b/.github/workflows/registry-tests.yml
@@ -41,8 +41,10 @@ concurrency:
jobs:
registry-tests:
name: "Registry extraction tests"
- runs-on: ubuntu-latest
- timeout-minutes: 5
+ runs-on: ubuntu-slim
+ # ubuntu-slim is single-core, so the uv sync plus the suite need more
headroom than
+ # the 5 minutes that sufficed on a 2-core runner.
+ timeout-minutes: 10
steps:
- name: "Checkout repository"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
diff --git a/.github/workflows/scheduled-verify-release-calendar.yml
b/.github/workflows/scheduled-verify-release-calendar.yml
index ec08a2863a0..a4b18d10136 100644
--- a/.github/workflows/scheduled-verify-release-calendar.yml
+++ b/.github/workflows/scheduled-verify-release-calendar.yml
@@ -27,7 +27,7 @@ permissions:
jobs:
verify-release-calendar:
name: "Verify release calendar"
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
timeout-minutes: 10
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -35,16 +35,9 @@ jobs:
with:
persist-credentials: false
- name: "Install uv"
- # Extract uv version from uv.lock. The format is stable: the line
- # immediately after `name = "uv"` is `version = "<X.Y.Z>"`.
- run: |
- UV_VERSION=$(sed -n '/^name = "uv"$/{n;s/^version =
"\(.*\)"$/\1/p;}' uv.lock)
- if [[ -z "${UV_VERSION}" ]]; then
- echo "Failed to extract uv version from uv.lock" >&2
- exit 1
- fi
- echo "Installing uv==${UV_VERSION}"
- pip install "uv==${UV_VERSION}"
+ uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d #
v10.0.1
+ with:
+ version-file: uv.lock
- name: "Verify release calendar"
run: uv run dev/verify_release_calendar.py
# yamllint disable rule:line-length
diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml
index e2e7d2bd599..f12dbf36061 100644
--- a/.github/workflows/stale.yml
+++ b/.github/workflows/stale.yml
@@ -27,7 +27,7 @@ permissions:
issues: write
jobs:
stale:
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
steps:
# Handle all PRs (45-day stale) and pending-response issues (14-day
stale)
- uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0
diff --git a/.github/workflows/update-constraints-on-push-stable.yml
b/.github/workflows/update-constraints-on-push-stable.yml
index aebe4bc9760..c194c120a9a 100644
--- a/.github/workflows/update-constraints-on-push-stable.yml
+++ b/.github/workflows/update-constraints-on-push-stable.yml
@@ -159,7 +159,7 @@ jobs:
notify-on-failure:
name: "Notify on failure"
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
needs: [build-info, build-ci-images, generate-constraints,
update-constraints]
if: failure()
env:
diff --git a/.github/workflows/update-constraints-on-push.yml
b/.github/workflows/update-constraints-on-push.yml
index c0fc95b3e16..9e753a6527a 100644
--- a/.github/workflows/update-constraints-on-push.yml
+++ b/.github/workflows/update-constraints-on-push.yml
@@ -234,7 +234,7 @@ jobs:
notify-on-failure:
name: "Notify on failure"
- runs-on: ["ubuntu-22.04"]
+ runs-on: ["ubuntu-slim"]
needs: [build-info, build-ci-images, generate-constraints,
update-constraints]
if: failure()
env:
diff --git a/dev/breeze/doc/ci/05_workflows.md
b/dev/breeze/doc/ci/05_workflows.md
index b06daa28894..5d1f51c87eb 100644
--- a/dev/breeze/doc/ci/05_workflows.md
+++ b/dev/breeze/doc/ci/05_workflows.md
@@ -28,6 +28,7 @@
- [Workflow Architecture Overview](#workflow-architecture-overview)
- [Branch-Specific Behavior](#branch-specific-behavior)
- [Tests Workflow Structure](#tests-workflow-structure)
+ - [Runners](#runners)
- [Implementation Details](#implementation-details)
- [CodeQL scan](#codeql-scan)
- [Publishing documentation](#publishing-documentation)
@@ -311,6 +312,36 @@ Special tests (integration and system tests) run
selectively:
- In canary runs for scheduled quality checks
- When dependency upgrades require thorough testing
+## Runners
+
+Two kinds of GitHub-hosted runner are used, and which one a job gets depends
on what
+the job actually needs.
+
+**`ubuntu-22.04` / `ubuntu-22.04-arm`** (2 cores, 7-16 GB RAM, privileged) is
the
+default for anything that builds or runs a container, installs Airflow, or
runs a real
+test suite. Jobs that get their runner from selective checks (`amd-runners` /
+`arm-runners`, see [04_selective_checks.md](04_selective_checks.md)) always
land here.
+
+**`ubuntu-slim`** (1 core, 5 GB RAM, 14 GB disk) is used for the short
bookkeeping jobs
+around the edges of CI — computing a matrix, posting a Slack notification,
closing stale
+issues, checking a newsfragment name. It is a container rather than a VM and
runs
+unprivileged, which constrains what can go on it:
+
+- **No Docker daemon.** The Docker *client* is on the image, but nothing can
build an
+ image, start a container, or use a Docker-container action. Anything
touching Breeze
+ is out.
+- **`python3` is Ubuntu's system interpreter**, so it is [PEP
668](https://peps.python.org/pep-0668/) externally managed and
+ a bare `pip install` fails. Stdlib-only scripts are fine; a job that needs
+ dependencies must bring its own interpreter. For anything driven by `uv
run`, use
+ `astral-sh/setup-uv` with `version-file: uv.lock` — that installs the uv
version the
+ workspace is locked to, and `uv run` then provisions its own Python.
+- **Minimal toolset** — `git`, `gh`, `jq`, `node`, `curl` are present; Java
and Go are
+ not. `actions/setup-*` still works, since those actions download into the
tool cache.
+- **One core**, so give a job that does real work a timeout with some slack in
it.
+
+When adding a job, reach for `ubuntu-slim` if it only shuffles metadata
around, and
+`ubuntu-22.04` otherwise.
+
## Implementation Details
Here's how the composite workflow system is organized in practice.
