This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch use-ubuntu-slim-for-short-jobs in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 567ea20809e08850431a7ce84010d7dc911cc2a9 Author: Jarek Potiuk <[email protected]> AuthorDate: Thu Sep 10 01:47:10 2026 +0200 Run short CI bookkeeping jobs on ubuntu-slim runners A large share of our workflow jobs never build an image, never install Airflow and never run a test suite - they compute a matrix, post a Slack message, close a stale issue or check a newsfragment name. Those jobs held a 2-core runner for their whole (short) lifetime, competing with the builds and test shards that genuinely need one. ubuntu-slim is a single-core, 5 GB, unprivileged container image, which is the right size for that class of work and keeps the larger runners free for jobs that cannot fit on it. Its constraints - no Docker daemon, an externally-managed system Python, a minimal toolset - decided which jobs moved: anything touching Breeze, Docker, Gradle or a real test suite stayed where it was. Three of the moved jobs installed uv by extracting the version from uv.lock and pip-installing it into the system interpreter, which PEP 668 refuses on this image. They now use astral-sh/setup-uv with version-file: uv.lock, which resolves the same locked version and lets uv provision its own Python. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UL4HBX63zq4um1L4XBcJYy --- .github/workflows/automatic-backport.yml | 2 +- .github/workflows/check-newsfragment-pr-number.yml | 2 +- .github/workflows/ci-amd.yml | 4 +-- .github/workflows/ci-arm.yml | 4 +-- .github/workflows/ci-duration-monitor.yml | 2 +- .github/workflows/ci-notification.yml | 11 +++++--- .github/workflows/codeql-analysis.yml | 2 +- .github/workflows/e2e-flaky-tests-report.yml | 2 +- .github/workflows/java-sdk-dependency-security.yml | 2 +- .github/workflows/milestone-tag-assistant.yml | 2 +- .github/workflows/notify-uv-lock-conflicts.yml | 15 +++-------- .github/workflows/recheck-old-bug-report.yml | 2 +- .github/workflows/registry-backfill.yml | 2 +- .github/workflows/registry-tests.yml | 6 +++-- .../scheduled-verify-release-calendar.yml | 15 +++-------- .github/workflows/stale.yml | 2 +- .github/workflows/ts-sdk-dependency-review.yml | 2 +- .../update-constraints-on-push-stable.yml | 2 +- .github/workflows/update-constraints-on-push.yml | 2 +- dev/breeze/doc/ci/05_workflows.md | 31 ++++++++++++++++++++++ 20 files changed, 67 insertions(+), 45 deletions(-) diff --git a/.github/workflows/automatic-backport.yml b/.github/workflows/automatic-backport.yml index afe20ce780d..daa142e9dfc 100644 --- a/.github/workflows/automatic-backport.yml +++ b/.github/workflows/automatic-backport.yml @@ -26,7 +26,7 @@ permissions: jobs: get-pr-info: name: "Get PR information" - runs-on: ubuntu-latest + runs-on: ubuntu-slim outputs: branches: ${{ steps.pr-info.outputs.branches }} commit-sha: ${{ github.sha }} diff --git a/.github/workflows/check-newsfragment-pr-number.yml b/.github/workflows/check-newsfragment-pr-number.yml index 2c9fce37b82..3f81dbb8d0e 100644 --- a/.github/workflows/check-newsfragment-pr-number.yml +++ b/.github/workflows/check-newsfragment-pr-number.yml @@ -34,7 +34,7 @@ concurrency: jobs: check-newsfragment-pr-number: if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip newsfragment check') }} - runs-on: ubuntu-latest + runs-on: ubuntu-slim timeout-minutes: 5 steps: - name: Check newsfragment PR number diff --git a/.github/workflows/ci-amd.yml b/.github/workflows/ci-amd.yml index 7c9aa502e23..b043a911156 100644 --- a/.github/workflows/ci-amd.yml +++ b/.github/workflows/ci-amd.yml @@ -238,7 +238,7 @@ jobs: print-platform: name: "Platform: AMD" needs: [build-info] - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] steps: - name: "Print architecture" run: "echo '## Architecture: AMD' >> $GITHUB_STEP_SUMMARY" @@ -1201,7 +1201,7 @@ jobs: !cancelled() && github.event_name == 'schedule' && github.run_attempt == 1 - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] steps: - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml index 0d73eda9b37..592d41f0f48 100644 --- a/.github/workflows/ci-arm.yml +++ b/.github/workflows/ci-arm.yml @@ -227,7 +227,7 @@ jobs: print-platform: name: "Platform: ARM" needs: [build-info] - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] steps: - name: "Print architecture" run: "echo '## Architecture: ARM' >> $GITHUB_STEP_SUMMARY" @@ -1190,7 +1190,7 @@ jobs: !cancelled() && github.event_name == 'schedule' && github.run_attempt == 1 - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] steps: - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/.github/workflows/ci-duration-monitor.yml b/.github/workflows/ci-duration-monitor.yml index 4e878002dd4..52f2d06c2fb 100644 --- a/.github/workflows/ci-duration-monitor.yml +++ b/.github/workflows/ci-duration-monitor.yml @@ -34,7 +34,7 @@ jobs: monitor-ci-durations: name: "Monitor CI durations on main" - runs-on: ubuntu-latest + runs-on: ubuntu-slim steps: - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/.github/workflows/ci-notification.yml b/.github/workflows/ci-notification.yml index 169b8fb3cff..4cf6f087bda 100644 --- a/.github/workflows/ci-notification.yml +++ b/.github/workflows/ci-notification.yml @@ -40,18 +40,21 @@ jobs: # Track AMD; ARM is the canary slot and is reported separately by # `ci-arm.yml`'s notify-slack job on schedule events. workflow-id: ["ci-amd.yml"] - runs-on: ubuntu-latest + runs-on: ubuntu-slim steps: - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false + - name: "Install uv" + uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + with: + version-file: uv.lock + - name: "Find workflow run status" id: find-workflow-run-status - run: | - python3 -m pip install uv - uv run ./dev/breeze/src/airflow_breeze/utils/workflow_status.py + run: uv run ./dev/breeze/src/airflow_breeze/utils/workflow_status.py env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} workflow_branch: ${{ matrix.branch }} diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index 9f6afae5c8d..53604cecd5b 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -49,7 +49,7 @@ concurrency: jobs: detect-languages: name: Detect languages to scan - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] permissions: contents: read pull-requests: read diff --git a/.github/workflows/e2e-flaky-tests-report.yml b/.github/workflows/e2e-flaky-tests-report.yml index bc922671d46..712dd81da39 100644 --- a/.github/workflows/e2e-flaky-tests-report.yml +++ b/.github/workflows/e2e-flaky-tests-report.yml @@ -33,7 +33,7 @@ jobs: analyze-flaky-tests: name: "Analyze E2E flaky tests" - runs-on: ubuntu-latest + runs-on: ubuntu-slim steps: - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/.github/workflows/java-sdk-dependency-security.yml b/.github/workflows/java-sdk-dependency-security.yml index dceacdcd627..805ca491628 100644 --- a/.github/workflows/java-sdk-dependency-security.yml +++ b/.github/workflows/java-sdk-dependency-security.yml @@ -42,7 +42,7 @@ jobs: dependency-review: name: Reject vulnerable dependency changes if: github.event_name == 'pull_request' - runs-on: ubuntu-latest + runs-on: ubuntu-slim steps: - name: Review dependency changes uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0 diff --git a/.github/workflows/milestone-tag-assistant.yml b/.github/workflows/milestone-tag-assistant.yml index 3592bc5235c..79a9c11f965 100644 --- a/.github/workflows/milestone-tag-assistant.yml +++ b/.github/workflows/milestone-tag-assistant.yml @@ -35,7 +35,7 @@ permissions: jobs: get-pr-info: name: "Get PR information" - runs-on: ubuntu-latest + runs-on: ubuntu-slim outputs: should-run: ${{ steps.pr-info.outputs.should-run }} pr-number: ${{ steps.pr-info.outputs.pr-number }} diff --git a/.github/workflows/notify-uv-lock-conflicts.yml b/.github/workflows/notify-uv-lock-conflicts.yml index 95dace06996..ec0ea2c28f6 100644 --- a/.github/workflows/notify-uv-lock-conflicts.yml +++ b/.github/workflows/notify-uv-lock-conflicts.yml @@ -29,7 +29,7 @@ permissions: jobs: notify: name: "Notify open PRs that conflict on uv.lock" - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] timeout-minutes: 10 steps: - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" @@ -37,16 +37,9 @@ jobs: with: persist-credentials: false - name: "Install uv" - # Extract uv version from uv.lock. The format is stable: the line - # immediately after `name = "uv"` is `version = "<X.Y.Z>"`. - run: | - UV_VERSION=$(sed -n '/^name = "uv"$/{n;s/^version = "\(.*\)"$/\1/p;}' uv.lock) - if [[ -z "${UV_VERSION}" ]]; then - echo "Failed to extract uv version from uv.lock" >&2 - exit 1 - fi - echo "Installing uv==${UV_VERSION}" - pip install "uv==${UV_VERSION}" + uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + with: + version-file: uv.lock - name: "Notify open PRs" env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/recheck-old-bug-report.yml b/.github/workflows/recheck-old-bug-report.yml index dcda4355553..049071dfad0 100644 --- a/.github/workflows/recheck-old-bug-report.yml +++ b/.github/workflows/recheck-old-bug-report.yml @@ -26,7 +26,7 @@ permissions: issues: write jobs: recheck-old-bug-report: - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] steps: - uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0 with: diff --git a/.github/workflows/registry-backfill.yml b/.github/workflows/registry-backfill.yml index 27838ae73d4..bbd14630981 100644 --- a/.github/workflows/registry-backfill.yml +++ b/.github/workflows/registry-backfill.yml @@ -79,7 +79,7 @@ jobs: disable-airflow-repo-cache: "false" prepare: - runs-on: ubuntu-latest + runs-on: ubuntu-slim outputs: matrix: ${{ steps.matrix.outputs.matrix }} bucket: ${{ steps.destination.outputs.bucket }} diff --git a/.github/workflows/registry-tests.yml b/.github/workflows/registry-tests.yml index a31cf55f352..a3e789eb89f 100644 --- a/.github/workflows/registry-tests.yml +++ b/.github/workflows/registry-tests.yml @@ -45,8 +45,10 @@ concurrency: jobs: registry-tests: name: "Registry extraction tests" - runs-on: ubuntu-latest - timeout-minutes: 5 + runs-on: ubuntu-slim + # ubuntu-slim is single-core, so the uv sync plus the suite need more headroom than + # the 5 minutes that sufficed on a 2-core runner. + timeout-minutes: 10 steps: - name: "Checkout repository" uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/.github/workflows/scheduled-verify-release-calendar.yml b/.github/workflows/scheduled-verify-release-calendar.yml index ec08a2863a0..a4b18d10136 100644 --- a/.github/workflows/scheduled-verify-release-calendar.yml +++ b/.github/workflows/scheduled-verify-release-calendar.yml @@ -27,7 +27,7 @@ permissions: jobs: verify-release-calendar: name: "Verify release calendar" - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] timeout-minutes: 10 steps: - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" @@ -35,16 +35,9 @@ jobs: with: persist-credentials: false - name: "Install uv" - # Extract uv version from uv.lock. The format is stable: the line - # immediately after `name = "uv"` is `version = "<X.Y.Z>"`. - run: | - UV_VERSION=$(sed -n '/^name = "uv"$/{n;s/^version = "\(.*\)"$/\1/p;}' uv.lock) - if [[ -z "${UV_VERSION}" ]]; then - echo "Failed to extract uv version from uv.lock" >&2 - exit 1 - fi - echo "Installing uv==${UV_VERSION}" - pip install "uv==${UV_VERSION}" + uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + with: + version-file: uv.lock - name: "Verify release calendar" run: uv run dev/verify_release_calendar.py # yamllint disable rule:line-length diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index e2e7d2bd599..f12dbf36061 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -27,7 +27,7 @@ permissions: issues: write jobs: stale: - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] steps: # Handle all PRs (45-day stale) and pending-response issues (14-day stale) - uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0 diff --git a/.github/workflows/ts-sdk-dependency-review.yml b/.github/workflows/ts-sdk-dependency-review.yml index 2d74108d66f..37bcdb9c814 100644 --- a/.github/workflows/ts-sdk-dependency-review.yml +++ b/.github/workflows/ts-sdk-dependency-review.yml @@ -36,7 +36,7 @@ concurrency: jobs: dependency-review: name: Reject vulnerable dependency changes - runs-on: ubuntu-latest + runs-on: ubuntu-slim steps: - name: Review dependency changes uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0 diff --git a/.github/workflows/update-constraints-on-push-stable.yml b/.github/workflows/update-constraints-on-push-stable.yml index aebe4bc9760..c194c120a9a 100644 --- a/.github/workflows/update-constraints-on-push-stable.yml +++ b/.github/workflows/update-constraints-on-push-stable.yml @@ -159,7 +159,7 @@ jobs: notify-on-failure: name: "Notify on failure" - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] needs: [build-info, build-ci-images, generate-constraints, update-constraints] if: failure() env: diff --git a/.github/workflows/update-constraints-on-push.yml b/.github/workflows/update-constraints-on-push.yml index c0fc95b3e16..9e753a6527a 100644 --- a/.github/workflows/update-constraints-on-push.yml +++ b/.github/workflows/update-constraints-on-push.yml @@ -234,7 +234,7 @@ jobs: notify-on-failure: name: "Notify on failure" - runs-on: ["ubuntu-22.04"] + runs-on: ["ubuntu-slim"] needs: [build-info, build-ci-images, generate-constraints, update-constraints] if: failure() env: diff --git a/dev/breeze/doc/ci/05_workflows.md b/dev/breeze/doc/ci/05_workflows.md index b06daa28894..711e2ddf934 100644 --- a/dev/breeze/doc/ci/05_workflows.md +++ b/dev/breeze/doc/ci/05_workflows.md @@ -28,6 +28,7 @@ - [Workflow Architecture Overview](#workflow-architecture-overview) - [Branch-Specific Behavior](#branch-specific-behavior) - [Tests Workflow Structure](#tests-workflow-structure) + - [Runners](#runners) - [Implementation Details](#implementation-details) - [CodeQL scan](#codeql-scan) - [Publishing documentation](#publishing-documentation) @@ -311,6 +312,36 @@ Special tests (integration and system tests) run selectively: - In canary runs for scheduled quality checks - When dependency upgrades require thorough testing +## Runners + +Two kinds of GitHub-hosted runner are used, and which one a job gets depends on what +the job actually needs. + +**`ubuntu-22.04` / `ubuntu-22.04-arm`** (2 cores, 7-16 GB RAM, privileged) is the +default for anything that builds or runs a container, installs Airflow, or runs a real +test suite. Jobs that get their runner from selective checks (`amd-runners` / +`arm-runners`, see [04_selective_checks.md](04_selective_checks.md)) always land here. + +**`ubuntu-slim`** (1 core, 5 GB RAM, 14 GB disk) is used for the short bookkeeping jobs +around the edges of CI — computing a matrix, posting a Slack notification, closing stale +issues, checking a newsfragment name. It is a container rather than a VM and runs +unprivileged, which constrains what can go on it: + +- **No Docker daemon.** The Docker *client* is on the image, but nothing can build an + image, start a container, or use a Docker-container action. Anything touching Breeze + is out. +- **`python3` is Ubuntu's system interpreter**, so it is PEP 668 externally managed and + a bare `pip install` fails. Stdlib-only scripts are fine; a job that needs + dependencies must bring its own interpreter. For anything driven by `uv run`, use + `astral-sh/setup-uv` with `version-file: uv.lock` — that installs the uv version the + workspace is locked to, and `uv run` then provisions its own Python. +- **Minimal toolset** — `git`, `gh`, `jq`, `node`, `curl` are present; Java and Go are + not. `actions/setup-*` still works, since those actions download into the tool cache. +- **One core**, so give a job that does real work a timeout with some slack in it. + +When adding a job, reach for `ubuntu-slim` if it only shuffles metadata around, and +`ubuntu-22.04` otherwise. + ## Implementation Details Here's how the composite workflow system is organized in practice.
