This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch ci/debounce-release-branch-ci in repository https://gitbox.apache.org/repos/asf/airflow.git
commit dd9f72d38158a61fae692f5c2eb5d1b5d20336eb Author: Jarek Potiuk <[email protected]> AuthorDate: Thu Oct 22 18:12:15 2026 +0200 Run full CI on release branches on a schedule instead of every push While a release is prepared, backports land on vX-Y-test in bursts and every push started a full AMD and ARM run. In the week of 15-21 October v3-4-test triggered 177 such runs; 133 of them were cancelled by the next push after already using hundreds of runner-minutes each, which made release-branch pushes one of the largest single consumers of the ASF's shared runners. Each backport PR has already run CI against the branch, so the branch needs a full run of its current HEAD a few times a day rather than one per merged commit. Release managers still need quick feedback on a broken backport, so a push still runs the tests for what changed since the last successful run of the branch. Testing from the last success rather than from the pushed commit alone means changes whose runs failed or were cancelled by a newer push are not left untested. Scheduled canaries that have nothing new to test - no commit since the last successful full run - only repeat that run, so they now finish without building anything. Generated-by: Claude Opus 5 --- .github/workflows/ci-amd.yml | 108 +++++++++++++-- .github/workflows/ci-arm.yml | 112 ++++++++++++--- .github/workflows/refresh-image-registry-cache.yml | 11 +- .github/workflows/release-branch-ci.yml | 151 +++++++++++++++++++++ dev/README_AIRFLOW3_DEV.md | 9 +- dev/README_RELEASE_AIRFLOW.md | 59 +++++++- dev/README_RELEASE_AIRFLOWCTL.md | 12 +- dev/breeze/doc/ci/04_selective_checks.md | 49 ++++--- dev/breeze/doc/ci/05_workflows.md | 66 ++++++++- .../doc/images/output_ci_selective-check.svg | 54 +++++--- .../doc/images/output_ci_selective-check.txt | 2 +- .../src/airflow_breeze/commands/ci_commands.py | 55 +++++++- .../airflow_breeze/commands/ci_commands_config.py | 2 + .../src/airflow_breeze/utils/selective_checks.py | 13 +- dev/breeze/tests/test_ci_commands.py | 107 +++++++++++++++ dev/breeze/tests/test_pr_info.py | 2 +- dev/breeze/tests/test_selective_checks.py | 33 +++++ scripts/ci/prek/check_ci_workflows_in_sync.py | 27 +++- 18 files changed, 771 insertions(+), 101 deletions(-) diff --git a/.github/workflows/ci-amd.yml b/.github/workflows/ci-amd.yml index abb0aca0d84..628d7cc4fde 100644 --- a/.github/workflows/ci-amd.yml +++ b/.github/workflows/ci-amd.yml @@ -16,8 +16,8 @@ # under the License. # --- -# AMD CI workflow. Triggered on every PR and on push to release branches. -# Companion to `ci-arm.yml`, which runs the same pipeline on ARM as the +# AMD CI workflow. Triggered on every PR, on the canary cron, and on release branches by +# `release-branch-ci.yml`. Companion to `ci-arm.yml`, which runs the same pipeline on ARM as the # scheduled canary. The two workflows are physical copies of each other # (GitHub Actions has no cross-file YAML include), differing only in their # trigger surface, the hardcoded `runner-type` / `platform` values in @@ -43,11 +43,10 @@ on: # yamllint disable-line rule:truthy - airflow-ctl/v[0-9]+-[0-9]+-stable types: [opened, reopened, synchronize, ready_for_review] push: - # Post-merge pushes to release-prep / providers branches run on both - # AMD and ARM (the matching block lives in the other wrapper too — - # `check-ci-workflows-in-sync` enforces they stay identical). `main` is - # intentionally NOT in this list: main pushes stay AMD-only via - # `ci-amd.yml`; ARM main coverage is the cron-driven canary above. + # Pushes to release-prep / providers branches run only the tests needed by the changes since the + # last successful run of the branch (selected as for a PR). Full runs of these branches, on AMD + # and ARM, are dispatched by `release-branch-ci.yml`. `main` does not build on push; its full + # runs are the canary above. branches: - v[0-9]+-[0-9]+-test - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+ @@ -66,12 +65,88 @@ env: VERBOSE: "true" concurrency: - group: ci-amd-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true + # Push runs have a group of their own per branch: a newer push cancels the older push run (the + # newer run then also tests the cancelled run's changes, see `check-run-needed`), and a push never + # cancels a full run of the branch. + group: >- + ci-amd-${{ github.event.pull_request.number || github.ref }}${{ + github.event_name == 'push' && '-push' || '' }} + # A scheduled canary queues behind a run in progress instead of cancelling it: it may well be a + # manual run of the same commit, which the canary would otherwise cancel and then skip itself. + cancel-in-progress: ${{ github.event_name != 'schedule' }} jobs: + check-run-needed: + name: "Check if the run is needed" + # A scheduled canary is not needed when the last completed full run of this workflow on the + # branch tested the same commit and succeeded: it would only repeat that run. A push run tests + # the changes since the last successful run of this workflow on the branch, found here. + # This job runs for every event, with no steps for the others: every job of the run depends on + # it, and a skipped job here would skip all of them, whatever their own `if:` says. + runs-on: ["ubuntu-slim"] + timeout-minutes: 5 + permissions: + actions: read + contents: read + outputs: + run-needed: ${{ steps.schedule.outputs.run-needed || 'true' }} + push-base-commit: ${{ steps.push.outputs.push-base-commit }} + steps: + - name: "Find the last successful run of the branch" + if: github.event_name == 'push' + id: push + shell: bash + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + WORKFLOW_REF: ${{ github.workflow_ref }} + BRANCH: ${{ github.ref_name }} + run: | + set -euo pipefail + workflow_path="${WORKFLOW_REF%@*}" + # A successful push or full run tested everything up to its commit. Changes after it -- from + # pushes whose runs failed, or were cancelled by a newer push -- are all tested by this run. + # A successful run of this same commit is passed over, so that a re-run tests again. + base="$(gh api "repos/${REPOSITORY}/actions/workflows/${workflow_path##*/}/runs" -X GET \ + -f branch="${BRANCH}" -f status=success -F per_page=30 \ + --jq "[.workflow_runs[] | select((.event == \"push\" or .event == \"workflow_dispatch\") + and .head_sha != \"${GITHUB_SHA}\")][0].head_sha // \"\"")" + # With no successful run to build on, selective checks run all tests. + echo "push-base-commit=${base}" >> "${GITHUB_OUTPUT}" + echo "Testing the changes since ${base:-the start: no successful run yet, so all tests run}." \ + >> "${GITHUB_STEP_SUMMARY}" + - name: "Check the last full run of ${{ github.sha }}" + if: github.event_name == 'schedule' + id: schedule + shell: bash + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + WORKFLOW_REF: ${{ github.workflow_ref }} + BRANCH: ${{ github.ref_name }} + run: | + set -euo pipefail + workflow_path="${WORKFLOW_REF%@*}" + workflow="${workflow_path##*/}" + # Full runs are the scheduled canaries and manual `workflow_dispatch` runs; this run is + # still in progress, so `status=completed` leaves it out. + last="$(gh api \ + "repos/${REPOSITORY}/actions/workflows/${workflow}/runs?branch=${BRANCH}&status=completed" \ + -F per_page=20 -X GET \ + --jq '[.workflow_runs[] | select(.event == "schedule" or .event == "workflow_dispatch")][0] + | "\(.head_sha // "") \(.conclusion // "") \(.html_url // "")"')" + read -r last_sha last_conclusion last_url <<< "${last}" + if [[ "${last_sha}" == "${GITHUB_SHA}" && "${last_conclusion}" == "success" ]]; then + echo "run-needed=false" >> "${GITHUB_OUTPUT}" + echo "No new commits since the last successful full run: ${last_url}" >> "${GITHUB_STEP_SUMMARY}" + else + echo "run-needed=true" >> "${GITHUB_OUTPUT}" + fi + build-info: name: "Build info" + needs: [check-run-needed] + if: needs.check-run-needed.outputs.run-needed == 'true' # At build-info stage we do not yet have outputs so we need to hard-code the runs-on to public runners runs-on: ["ubuntu-22.04"] outputs: @@ -190,6 +265,16 @@ jobs: ref: ${{ github.sha }} fetch-depth: 2 persist-credentials: false + - name: "Fetch the last tested push commit" + # Selective checks of a push diff against this commit. A two-commit diff only needs the + # commit itself, not the history in between; if it cannot be fetched, all tests run. + if: needs.check-run-needed.outputs.push-base-commit != '' + shell: bash + env: + PUSH_BASE_COMMIT: ${{ needs.check-run-needed.outputs.push-base-commit }} + run: | + git fetch --no-tags --depth=1 origin "${PUSH_BASE_COMMIT}" \ + || echo "::warning::Could not fetch ${PUSH_BASE_COMMIT}, all tests will run" - name: "Install Breeze" uses: ./.github/actions/breeze id: breeze @@ -231,6 +316,8 @@ jobs: VERBOSE: "false" GITHUB_CONTEXT_INPUT: "${{ runner.temp }}/github_context.json" PLATFORM: "linux/amd64" + SELECTIVE_TESTS_ON_PUSH: "true" + PUSH_BASE_COMMIT: ${{ needs.check-run-needed.outputs.push-base-commit }} run: breeze ci selective-check 2>> ${GITHUB_OUTPUT} - name: env run: printenv @@ -1202,7 +1289,7 @@ jobs: contents: write packages: write # This will fire when all the jobs from "needs" are either successful or skipped - if: always() && !failure() && !cancelled() + if: always() && !failure() && !cancelled() && needs.build-info.result == 'success' needs: - additional-ci-image-checks - additional-prod-image-tests @@ -1261,6 +1348,7 @@ jobs: always() && !cancelled() && github.event_name == 'schedule' && + needs.build-info.result != 'skipped' && github.run_attempt == 1 runs-on: ["ubuntu-slim"] steps: diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml index 4b3c716437c..549db2cab03 100644 --- a/.github/workflows/ci-arm.yml +++ b/.github/workflows/ci-arm.yml @@ -16,9 +16,9 @@ # under the License. # --- -# ARM canary CI workflow. Triggered on the canary cron only — ARM runners are -# reserved for the scheduled slot. Companion to `ci-amd.yml`, which runs the -# same pipeline on AMD for every PR and push. The two workflows are physical +# ARM canary CI workflow. Triggered on the canary cron and on release branches by +# `release-branch-ci.yml` — never on PRs. Companion to `ci-amd.yml`, which runs the +# same pipeline on AMD for every PR. The two workflows are physical # copies of each other (GitHub Actions has no cross-file YAML include), # differing only in their trigger surface, the hardcoded `runner-type` / # `platform` values in `build-info`'s outputs, and the architecture-specific @@ -31,17 +31,6 @@ on: # yamllint disable-line rule:truthy # architecture). The `:28` vs AMD's `:58` offset keeps the two from competing for # runners at exactly the same minute. - cron: '28 7,19 * * *' - push: - # Post-merge pushes to release-prep / providers branches run on both - # AMD and ARM (the matching block lives in the other wrapper too — - # `check-ci-workflows-in-sync` enforces they stay identical). `main` is - # intentionally NOT in this list: main pushes stay AMD-only via - # `ci-amd.yml`; ARM main coverage is the cron-driven canary above. - branches: - - v[0-9]+-[0-9]+-test - - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+ - - chart/v[0-9]+-[0-9]+x-test - - airflow-ctl/v[0-9]+-[0-9]+-test workflow_dispatch: permissions: # All other permissions are set to none by default @@ -55,12 +44,88 @@ env: VERBOSE: "true" concurrency: - group: ci-arm-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true + # Push runs have a group of their own per branch: a newer push cancels the older push run (the + # newer run then also tests the cancelled run's changes, see `check-run-needed`), and a push never + # cancels a full run of the branch. + group: >- + ci-arm-${{ github.event.pull_request.number || github.ref }}${{ + github.event_name == 'push' && '-push' || '' }} + # A scheduled canary queues behind a run in progress instead of cancelling it: it may well be a + # manual run of the same commit, which the canary would otherwise cancel and then skip itself. + cancel-in-progress: ${{ github.event_name != 'schedule' }} jobs: + check-run-needed: + name: "Check if the run is needed" + # A scheduled canary is not needed when the last completed full run of this workflow on the + # branch tested the same commit and succeeded: it would only repeat that run. A push run tests + # the changes since the last successful run of this workflow on the branch, found here. + # This job runs for every event, with no steps for the others: every job of the run depends on + # it, and a skipped job here would skip all of them, whatever their own `if:` says. + runs-on: ["ubuntu-slim"] + timeout-minutes: 5 + permissions: + actions: read + contents: read + outputs: + run-needed: ${{ steps.schedule.outputs.run-needed || 'true' }} + push-base-commit: ${{ steps.push.outputs.push-base-commit }} + steps: + - name: "Find the last successful run of the branch" + if: github.event_name == 'push' + id: push + shell: bash + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + WORKFLOW_REF: ${{ github.workflow_ref }} + BRANCH: ${{ github.ref_name }} + run: | + set -euo pipefail + workflow_path="${WORKFLOW_REF%@*}" + # A successful push or full run tested everything up to its commit. Changes after it -- from + # pushes whose runs failed, or were cancelled by a newer push -- are all tested by this run. + # A successful run of this same commit is passed over, so that a re-run tests again. + base="$(gh api "repos/${REPOSITORY}/actions/workflows/${workflow_path##*/}/runs" -X GET \ + -f branch="${BRANCH}" -f status=success -F per_page=30 \ + --jq "[.workflow_runs[] | select((.event == \"push\" or .event == \"workflow_dispatch\") + and .head_sha != \"${GITHUB_SHA}\")][0].head_sha // \"\"")" + # With no successful run to build on, selective checks run all tests. + echo "push-base-commit=${base}" >> "${GITHUB_OUTPUT}" + echo "Testing the changes since ${base:-the start: no successful run yet, so all tests run}." \ + >> "${GITHUB_STEP_SUMMARY}" + - name: "Check the last full run of ${{ github.sha }}" + if: github.event_name == 'schedule' + id: schedule + shell: bash + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + WORKFLOW_REF: ${{ github.workflow_ref }} + BRANCH: ${{ github.ref_name }} + run: | + set -euo pipefail + workflow_path="${WORKFLOW_REF%@*}" + workflow="${workflow_path##*/}" + # Full runs are the scheduled canaries and manual `workflow_dispatch` runs; this run is + # still in progress, so `status=completed` leaves it out. + last="$(gh api \ + "repos/${REPOSITORY}/actions/workflows/${workflow}/runs?branch=${BRANCH}&status=completed" \ + -F per_page=20 -X GET \ + --jq '[.workflow_runs[] | select(.event == "schedule" or .event == "workflow_dispatch")][0] + | "\(.head_sha // "") \(.conclusion // "") \(.html_url // "")"')" + read -r last_sha last_conclusion last_url <<< "${last}" + if [[ "${last_sha}" == "${GITHUB_SHA}" && "${last_conclusion}" == "success" ]]; then + echo "run-needed=false" >> "${GITHUB_OUTPUT}" + echo "No new commits since the last successful full run: ${last_url}" >> "${GITHUB_STEP_SUMMARY}" + else + echo "run-needed=true" >> "${GITHUB_OUTPUT}" + fi + build-info: name: "Build info" + needs: [check-run-needed] + if: needs.check-run-needed.outputs.run-needed == 'true' # At build-info stage we do not yet have outputs so we need to hard-code the runs-on to public runners runs-on: ["ubuntu-22.04"] outputs: @@ -179,6 +244,16 @@ jobs: ref: ${{ github.sha }} fetch-depth: 2 persist-credentials: false + - name: "Fetch the last tested push commit" + # Selective checks of a push diff against this commit. A two-commit diff only needs the + # commit itself, not the history in between; if it cannot be fetched, all tests run. + if: needs.check-run-needed.outputs.push-base-commit != '' + shell: bash + env: + PUSH_BASE_COMMIT: ${{ needs.check-run-needed.outputs.push-base-commit }} + run: | + git fetch --no-tags --depth=1 origin "${PUSH_BASE_COMMIT}" \ + || echo "::warning::Could not fetch ${PUSH_BASE_COMMIT}, all tests will run" - name: "Install Breeze" uses: ./.github/actions/breeze id: breeze @@ -220,6 +295,8 @@ jobs: VERBOSE: "false" GITHUB_CONTEXT_INPUT: "${{ runner.temp }}/github_context.json" PLATFORM: "linux/arm64" + SELECTIVE_TESTS_ON_PUSH: "true" + PUSH_BASE_COMMIT: ${{ needs.check-run-needed.outputs.push-base-commit }} run: breeze ci selective-check 2>> ${GITHUB_OUTPUT} - name: env run: printenv @@ -1191,7 +1268,7 @@ jobs: contents: write packages: write # This will fire when all the jobs from "needs" are either successful or skipped - if: always() && !failure() && !cancelled() + if: always() && !failure() && !cancelled() && needs.build-info.result == 'success' needs: - additional-ci-image-checks - additional-prod-image-tests @@ -1250,6 +1327,7 @@ jobs: always() && !cancelled() && github.event_name == 'schedule' && + needs.build-info.result != 'skipped' && github.run_attempt == 1 runs-on: ["ubuntu-slim"] steps: diff --git a/.github/workflows/refresh-image-registry-cache.yml b/.github/workflows/refresh-image-registry-cache.yml index 23b3e6221dc..e133222cf2d 100644 --- a/.github/workflows/refresh-image-registry-cache.yml +++ b/.github/workflows/refresh-image-registry-cache.yml @@ -20,7 +20,7 @@ name: Refresh image registry cache # Refreshes the CI image cache in the GitHub registry, in a run of its own. # # It has to be its own workflow rather than a job inside `ci-amd.yml` / `ci-arm.yml`, because a -# job cannot opt out of its run being cancelled, and those runs are cancelled by the next push. +# job cannot opt out of its run being cancelled, and those runs are cancelled by the next run. # With no cache a branch builds cold (~20 minutes per Python version), which widens the window in # which the next push cancels the run, which leaves the cache missing. `v3-3-test` sat in that # loop -- its `linux/amd64` cache covered the default Python version alone, the single matrix @@ -41,10 +41,11 @@ on: # yamllint disable-line rule:truthy - "linux/arm64" default: "both" push: - # Release-prep and providers branches only -- they are built on push, so the CI run that - # would have refreshed their cache is also the run that keeps being cancelled. `main` is not - # built on push; its cache is refreshed by the Push Image Cache job at the end of the - # scheduled canary, which runs to completion. + # Release-prep and providers branches only -- their CI runs are no longer started on push but + # by `release-branch-ci.yml` a few times a day, and a newer run can still cancel the one in + # flight, so the cache is refreshed here on every merge instead. `main` is not built on push; + # its cache is refreshed by the Push Image Cache job at the end of the scheduled canary, + # which runs to completion. branches: - v[0-9]+-[0-9]+-test - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+ diff --git a/.github/workflows/release-branch-ci.yml b/.github/workflows/release-branch-ci.yml new file mode 100644 index 00000000000..a258687a67d --- /dev/null +++ b/.github/workflows/release-branch-ci.yml @@ -0,0 +1,151 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +# Starts the full CI (`ci-amd.yml` / `ci-arm.yml`) on release-prep and providers branches on a +# schedule, instead of on every push to them. During release preparation backports land on +# `vX-Y-test` in bursts, and every push used to start a full AMD + ARM canary that the next push +# cancelled after it had already used hundreds of runner-minutes. Each backport PR has already run +# CI against the branch, so the branch needs a full run of its current HEAD a few times a day, not +# one per merged commit. +# +# Each slot dispatches one architecture, for every release branch unless the last full run of that +# workflow on the branch tested its current HEAD and succeeded (or is still in progress): a run is +# repeated only when there are new commits, or when the previous run of the same commit failed or +# was cancelled. Branches without a commit in the last `MAX_IDLE_DAYS` days are skipped, so the +# long tail of old release branches is never built. Release managers can start a run at any time +# with `workflow_dispatch` here, or directly on `ci-amd.yml` / `ci-arm.yml` with +# `gh workflow run ci-amd.yml --repo apache/airflow --ref vX-Y-test`. +# +# `schedule` runs only from the default branch, which is why this lives on `main` and dispatches +# the workflows on the release branch: the dispatched run uses the workflow files of that branch. +name: Release branch CI +on: # yamllint disable-line rule:truthy + schedule: + # AMD at 04:43 and 16:43, ARM at 10:43 and 22:43 UTC: between the `main` canaries + # (AMD 01:58/13:58, ARM 07:28/19:28), so the two never compete for runners at the same time. + # Keep these strings in sync with AMD_CRON below. + - cron: '43 4,16 * * *' + - cron: '43 10,22 * * *' + workflow_dispatch: + inputs: + platform: + description: "Which CI workflow to start" + type: choice + options: + - "both" + - "amd" + - "arm" + default: "both" + branch: + description: "Release branch to build (empty: every release branch with new commits)" + type: string + default: "" + force: + description: "Start the run even if the last run of the branch HEAD succeeded" + type: boolean + default: false +permissions: + contents: read +concurrency: + group: release-branch-ci + cancel-in-progress: false +jobs: + dispatch: + name: "Dispatch release branch CI" + if: github.repository == 'apache/airflow' + runs-on: ["ubuntu-slim"] + timeout-minutes: 10 + permissions: + actions: write + contents: read + steps: + - name: "Start CI for release branches with new commits" + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + EVENT_NAME: ${{ github.event_name }} + SCHEDULE: ${{ github.event.schedule }} + AMD_CRON: '43 4,16 * * *' + PLATFORM: ${{ inputs.platform }} + BRANCH: ${{ inputs.branch }} + FORCE: ${{ inputs.force }} + MAX_IDLE_DAYS: "7" + # Same branches as the `pull_request` filter of `ci-amd.yml`, minus the `-stable` ones. + RELEASE_BRANCH_REGEX: >- + ^(v[0-9]+-[0-9]+-test|providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+|chart/v[0-9]+-[0-9]+x-test|airflow-ctl/v[0-9]+-[0-9]+-test)$ + run: | + set -euo pipefail + if [[ "${EVENT_NAME}" == "schedule" ]]; then + if [[ "${SCHEDULE}" == "${AMD_CRON}" ]]; then + workflows=("ci-amd.yml") + else + workflows=("ci-arm.yml") + fi + else + case "${PLATFORM}" in + amd) workflows=("ci-amd.yml") ;; + arm) workflows=("ci-arm.yml") ;; + *) workflows=("ci-amd.yml" "ci-arm.yml") ;; + esac + fi + if [[ -n "${BRANCH}" && ! "${BRANCH}" =~ ${RELEASE_BRANCH_REGEX} ]]; then + echo "::error::'${BRANCH}' is not a release branch (expected to match ${RELEASE_BRANCH_REGEX})" + exit 1 + fi + cutoff="$(date -u -d "-${MAX_IDLE_DAYS} days" +%s)" + { + echo "| Branch | Workflow | HEAD | Action |" + echo "| --- | --- | --- | --- |" + } >> "${GITHUB_STEP_SUMMARY}" + gh api --paginate "repos/${REPOSITORY}/branches?per_page=100" \ + --jq '.[] | [.name, .commit.sha] | @tsv' > "${RUNNER_TEMP}/branches.tsv" + while IFS=$'\t' read -r branch head_sha; do + [[ "${branch}" =~ ${RELEASE_BRANCH_REGEX} ]] || continue + [[ -z "${BRANCH}" || "${branch}" == "${BRANCH}" ]] || continue + if [[ -z "${BRANCH}" ]]; then + committed="$(gh api "repos/${REPOSITORY}/commits/${head_sha}" --jq '.commit.committer.date')" + if (( $(date -u -d "${committed}" +%s) < cutoff )); then + continue + fi + fi + for workflow in "${workflows[@]}"; do + # The newest full run of this workflow on the branch. Push runs only run the selective + # tests of their commit, so they do not count. + last="$(gh api \ + "repos/${REPOSITORY}/actions/workflows/${workflow}/runs?branch=${branch//\//%2F}" \ + -X GET -f event=workflow_dispatch -F per_page=1 \ + --jq '.workflow_runs[0] | "\(.head_sha // "") \(.status // "") \(.conclusion // "")"')" + read -r last_sha last_status last_conclusion <<< "${last}" + if [[ "${FORCE}" != "true" && "${last_sha}" == "${head_sha}" \ + && ( "${last_status}" != "completed" || "${last_conclusion}" == "success" ) ]]; then + action="skipped: last run of HEAD ${last_conclusion:-${last_status}}" + else + # A branch cut before `ci-amd.yml` / `ci-arm.yml` existed cannot run them; report it and + # carry on with the other branches. + if gh workflow run "${workflow}" --repo "${REPOSITORY}" --ref "${branch}"; then + action="started" + else + action="could not start" + echo "::warning::Could not start ${workflow} on ${branch}" + fi + fi + echo "${branch} ${workflow} ${head_sha:0:10}: ${action}" + echo "| \`${branch}\` | \`${workflow}\` | \`${head_sha:0:10}\` | ${action} |" \ + >> "${GITHUB_STEP_SUMMARY}" + done + done < "${RUNNER_TEMP}/branches.tsv" diff --git a/dev/README_AIRFLOW3_DEV.md b/dev/README_AIRFLOW3_DEV.md index d99a81a60f7..9249f063551 100644 --- a/dev/README_AIRFLOW3_DEV.md +++ b/dev/README_AIRFLOW3_DEV.md @@ -78,10 +78,11 @@ PRs should **never** target `v3-2-stable` directly unless explicitly instructed > RC testing. In this case, the release manager can prepare the RC directly > from the `v3-X-test` branch > without opening a PR to `v3-X-stable`. This saves the overhead of creating > and managing a PR that will > likely need additional changes before GA. However, when using this shortcut, > the release manager **must** -> verify that the `v3-X-test` push CI action ("Tests" workflow) has succeeded before cutting the RC. You can -> check this at: -> https://github.com/apache/airflow/actions/workflows/ci-amd.yml?query=event%3Apush+branch%3Av3-2-test -> (adjust the branch filter for the relevant `v3-X-test` branch). +> verify that the full CI runs ("Tests (AMD)" and "Tests (ARM)" workflows) of the `v3-X-test` HEAD have +> succeeded before cutting the RC. They are started by `release-branch-ci.yml`, not on push; see +> [CI of the `vX-Y-test` branch](README_RELEASE_AIRFLOW.md#ci-of-the-vx-y-test-branch). You can see them at: +> https://github.com/apache/airflow/actions/workflows/ci-amd.yml?query=event%3Aworkflow_dispatch+branch%3Av3-2-test +> (adjust the branch filter for the relevant `v3-X-test` branch, and check `ci-arm.yml` the same way). ## Developing for Airflow 3 diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md index c7dd2f54855..5c81a5f8461 100644 --- a/dev/README_RELEASE_AIRFLOW.md +++ b/dev/README_RELEASE_AIRFLOW.md @@ -23,6 +23,7 @@ - [Collect ambiguities during the release (for a follow-up doc PR)](#collect-ambiguities-during-the-release-for-a-follow-up-doc-pr) - [Perform review of security issues that are marked for the release](#perform-review-of-security-issues-that-are-marked-for-the-release) - [Selecting what to put into the release](#selecting-what-to-put-into-the-release) + - [CI of the `vX-Y-test` branch](#ci-of-the-vx-y-test-branch) - [Beta releases: fast-forwarding `vX-Y-test` to `main`](#beta-releases-fast-forwarding-vx-y-test-to-main) - [i18n workflow](#i18n-workflow) - [Selecting what to cherry-pick](#selecting-what-to-cherry-pick) @@ -105,6 +106,54 @@ The first step of a release is to work out what is being included. This differs - For a *patch* release, you will be selecting specific commits to cherry-pick and backport into the existing release branch. +## CI of the `vX-Y-test` branch + +Release branches do not run the full CI on every push. Backports land in bursts while a release is +prepared, and a full AMD + ARM run per merged commit was mostly cancelled by the next merge. Instead: + +- **Full runs** are started by + [`release-branch-ci.yml`](../.github/workflows/release-branch-ci.yml) on every release branch whose + HEAD has not had a successful full run yet: `Tests (AMD)` at 04:43 and 16:43 UTC, `Tests (ARM)` at + 10:43 and 22:43 UTC. A failed or cancelled run of HEAD is repeated at the next slot. They are + `workflow_dispatch` runs of `ci-amd.yml` / `ci-arm.yml` on the branch, run the full matrix, and + refresh the branch's constraints and image cache, as the push runs used to. +- **Incremental tests on push**: every push starts a `Tests (AMD)` run that tests only what changed + since the last successful run of `Tests (AMD)` on the branch (push or full run), selected the same + way as for a PR with those changes. A newer push cancels the push run still in progress, and its + run then also covers the cancelled run's changes; changes from a failed run are tested again by + the next one. A broken backport therefore shows up within the hour, not at the next full run. + +Nothing needs to be switched on or off for a release. Branches cut from `main` after this was +introduced have it already. An older release branch only behaves this way once the change to +`ci-amd.yml` / `ci-arm.yml` that introduced it has been backported to it; until then it keeps +running the full CI on every push. + +### Before cutting an RC: check the full run of the branch HEAD + +The incremental push runs do not replace the full run. Before you cut an RC (or open the PR syncing +`vX-Y-test` into `vX-Y-stable`), make sure both the AMD and the ARM full run of the current HEAD of +`vX-Y-test` have succeeded: + +```shell script +git ls-remote https://github.com/apache/airflow.git refs/heads/vX-Y-test +gh run list --repo apache/airflow --workflow ci-amd.yml --branch vX-Y-test --event workflow_dispatch \ + --limit 1 --json headSha,status,conclusion,url +gh run list --repo apache/airflow --workflow ci-arm.yml --branch vX-Y-test --event workflow_dispatch \ + --limit 1 --json headSha,status,conclusion,url +``` + +The `headSha` of each run must be the SHA `git ls-remote` printed. If the last run is for an older +commit, start the full runs now instead of waiting for the next slot, and check again when they finish: + +```shell script +gh workflow run release-branch-ci.yml --repo apache/airflow -f branch=vX-Y-test +``` + +This starts both architectures for the branch HEAD (add `-f platform=amd` or `-f platform=arm` for +one of them, and `-f force=true` to repeat a run that already succeeded for HEAD). You can also start +one directly with `gh workflow run ci-amd.yml --repo apache/airflow --ref vX-Y-test`. Starting a new +full run cancels the full run of the same workflow already in progress on the branch. + ## Beta releases: fast-forwarding `vX-Y-test` to `main` For a *major* or *minor* release, the `vX-Y-test` branch is created early (see @@ -608,10 +657,12 @@ still works but is no longer recommended. > RC testing. In this case, the release manager can prepare the RC directly > from the `v3-X-test` branch > without opening a PR to `v3-X-stable`. This saves the overhead of creating > and managing a PR that will > likely need additional changes before GA. However, when using this shortcut, > the release manager **must** -> verify that the `v3-X-test` push CI action ("Tests" workflow) has succeeded before cutting the RC. You can -> check this at: -> https://github.com/apache/airflow/actions/workflows/ci-amd.yml?query=event%3Apush+branch%3Av3-2-test -> (adjust the branch filter for the relevant `v3-X-test` branch). +> verify that the full CI runs ("Tests (AMD)" and "Tests (ARM)" workflows) of the `v3-X-test` HEAD have +> succeeded before cutting the RC, as described in +> [Before cutting an RC: check the full run of the branch HEAD](#before-cutting-an-rc-check-the-full-run-of-the-branch-head). +> You can see them at: +> https://github.com/apache/airflow/actions/workflows/ci-amd.yml?query=event%3Aworkflow_dispatch+branch%3Av3-2-test +> (adjust the branch filter for the relevant `v3-X-test` branch, and check `ci-arm.yml` the same way). - When the PR is approved (or when using the shortcut above), install `dev/breeze` in a virtualenv: diff --git a/dev/README_RELEASE_AIRFLOWCTL.md b/dev/README_RELEASE_AIRFLOWCTL.md index 7e8b8fa2673..e87f018cb89 100644 --- a/dev/README_RELEASE_AIRFLOWCTL.md +++ b/dev/README_RELEASE_AIRFLOWCTL.md @@ -233,8 +233,16 @@ create the X.Y branches" above before proceeding. > for a new minor (e.g. `0.2.0rc1`), the release manager may cut the RC > directly from `airflow-ctl/vX-Y-test` without opening a sync PR to > `airflow-ctl/vX-Y-stable`. However, they **must** verify that the -> `airflow-ctl/vX-Y-test` push CI action ("Tests" workflow) has succeeded -> before cutting the RC. +> full CI runs ("Tests (AMD)" and "Tests (ARM)" workflows) of the +> `airflow-ctl/vX-Y-test` HEAD have succeeded before cutting the RC. + +Release branches do not run the full CI on push: `release-branch-ci.yml` +starts it a few times a day until the branch HEAD has a successful full +run, and each push runs only the tests for the changes since the last +successful run. Check the full run of the HEAD before cutting an RC as +described in +[CI of the `vX-Y-test` branch](README_RELEASE_AIRFLOW.md#ci-of-the-vx-y-test-branch) +(use `airflow-ctl/vX-Y-test` as the branch name). Before cutting an RC, confirm the set of commits on the maintenance branch is what you expect: diff --git a/dev/breeze/doc/ci/04_selective_checks.md b/dev/breeze/doc/ci/04_selective_checks.md index 2f788e669f4..7b383f95566 100644 --- a/dev/breeze/doc/ci/04_selective_checks.md +++ b/dev/breeze/doc/ci/04_selective_checks.md @@ -144,8 +144,9 @@ flowchart TD A([_should_run_all_tests_and_versions]) --> B{event is<br/>PUSH / SCHEDULE / DISPATCH?} B -->|yes| C{PUSH and only<br/>.txt/.md files changed?} C -->|yes| NO1[False<br/>skip full tests] - C -->|no| D{PUSH and<br/>branch != main?} - D -->|yes - release branch push| NO2[False<br/>selective only] + C -->|no| D{PUSH with<br/>--selective-tests-on-push?} + D -->|yes - release branch push in ci-amd.yml| NO2[treated like PULL_REQUEST<br/>selective] + NO2 --> E D -->|no| YES1[True<br/>run everything] B -->|no - PULL_REQUEST| E{commit_ref missing?} E -->|yes| YES2[True] @@ -184,9 +185,12 @@ Two related booleans share this machinery: versions instead of just the defaults. Labels (`default versions only`, `latest versions only`, `all versions`) override; otherwise it follows `_should_run_all_tests_and_versions()`. * **`is_canary_run`** — `True` for SCHEDULE/PUSH/DISPATCH events **on `apache/airflow`** (not forks) - when more than just `.txt`/`.md` changed, or whenever the `canary` label is set. Canary runs always - run unit tests even when `full_tests_needed` is `False` (this is what makes a *release-branch push* - still run its relevant unit tests — see the examples). + when more than just `.txt`/`.md` changed, or whenever the `canary` label is set. A PUSH with + `--selective-tests-on-push` (env `SELECTIVE_TESTS_ON_PUSH`) is **not** a canary: `ci-amd.yml` passes + that flag, so a push to a release branch runs the selective tests of the changes since the last + successful run of the branch (`--push-base-commit`), exactly as a pull request would. Workflows that run on push for other reasons (constraints, image cache) do not + pass it, and keep getting the full set of Python versions they need. Canary runs always run unit + tests even when `full_tests_needed` is `False`. ### Stage 2 — decide which jobs run @@ -289,13 +293,13 @@ flowchart TD PR1[event=PULL_REQUEST<br/>files=providers/amazon/.../s3.py] --> PR2[full_tests_needed=False<br/>all_versions=False<br/>is_canary_run=False] PR2 --> PR3[unit tests: Providers amazon + Always<br/>default versions only<br/>amazon prek/mypy only] end - subgraph PUSHMAIN[Push / merge to main] - PM1[event=PUSH, branch=main] --> PM2[full_tests_needed=True<br/>all_versions=True<br/>is_canary_run=True] - PM2 --> PM3[full matrix, all versions,<br/>all canary-only jobs] + subgraph RELFULL[Release branch full run, dispatched] + RF1[event=WORKFLOW_DISPATCH<br/>branch=v3-x-test] --> RF2[full_tests_needed=True<br/>all_versions=True<br/>is_canary_run=True] + RF2 --> RF3[full matrix, all versions,<br/>all canary-only jobs] end subgraph RELPUSH[Push to release branch v3-x-test] - RP1[event=PUSH, branch!=main] --> RP2[full_tests_needed=False<br/>is_canary_run=True] - RP2 --> RP3[unit tests run via canary<br/>but selective test types<br/>default versions] + RP1[event=PUSH<br/>--selective-tests-on-push] --> RP2[full_tests_needed=False<br/>is_canary_run=False] + RP2 --> RP3[same selection as a PR<br/>with the files changed since<br/>the last successful run<br/>default versions] end subgraph CANARY[Scheduled nightly canary] CN1[event=SCHEDULE on apache/airflow] --> CN2[full_tests_needed=True<br/>all_versions=True<br/>is_canary_run=True] @@ -315,13 +319,16 @@ flowchart TD PR runs the **full matrix** — same as a canary — because the change can affect everything. (A *non-test* workflow such as `codeql-analysis.yml`, or a `scripts/ci/prek/...` static-check hook, does **not** force the full matrix — see the `Environment files` carve-outs above.) -4. **Push / merge to `main`.** `_should_run_all_tests_and_versions()` is true for PUSH on `main` → - `full_tests_needed=True`, `all_versions=True`, `is_canary_run=True`. The full matrix plus all - canary-only jobs run. This is the safety net that backstops aggressive PR-time optimisation. -5. **Push to a release branch** (`v3-1-test`). PUSH but `branch != main` → `full_tests_needed=False` - (selective), yet `is_canary_run=True`, so `run_unit_tests=True`. The branch runs its *relevant* - unit tests on default versions rather than the entire matrix — release branches don't need the full - cross-version sweep on every push. +4. **Full run of a release branch** (WORKFLOW_DISPATCH on `v3-1-test`, started by + `release-branch-ci.yml` a few times a day or by a release manager). `full_tests_needed=True`, + `all_versions=True`, `is_canary_run=True`. The full matrix plus all canary-only jobs run, and the + branch's constraints and image cache are refreshed. `main` gets the same from the scheduled canary; + neither builds the full matrix on push. +5. **Push to a release branch** (`v3-1-test`). `ci-amd.yml` passes `--selective-tests-on-push` and + `--push-base-commit` (the commit of the last successful run of the workflow on the branch), so the + push is treated like a pull request whose files are those changed since that commit: + `full_tests_needed` and `is_canary_run` follow the PR rules, and the run selects only the tests + those changes need, on default versions. If the diff cannot be computed, all tests run. 6. **Scheduled nightly canary** (SCHEDULE on `apache/airflow`). Full matrix on all versions, every canary-only job. The most thorough run; it is what gives us confidence that the PR-time shortcuts never let a regression through. @@ -433,7 +440,7 @@ together using `pytest-xdist` (pytest-xdist distributes the tests among parallel ## Selective check decision rules -* `Full tests` case is enabled when the event is PUSH **to `main`**, SCHEDULE or WORKFLOW_DISPATCH, or we +* `Full tests` case is enabled when the event is PUSH, SCHEDULE or WORKFLOW_DISPATCH, or we miss commit info, or any of the important environment files (`pyproject.toml`, `Dockerfile`, `scripts`, etc.) changed, or the API *contract* changed (the generated OpenAPI spec or the client generator — plain API source/test edits that leave the committed spec @@ -443,9 +450,9 @@ together using `pytest-xdist` (pytest-xdist distributes the tests among parallel set. Any other changed `tests_common` helper only selects the tests that import it. That enables all matrix combinations of variables (representative) and all possible test type. No further checks are performed. See also [1] note below. Two exceptions narrow this: a PUSH that changed **only** - `.txt`/`.md` files skips full tests, and a PUSH to a **release branch** (`v3-X-test`, i.e. not `main`) - runs selective tests only (it is still a `canary` run, so its *relevant* unit tests run on default - versions). The high-level flow and worked examples for these cases are in + `.txt`/`.md` files skips full tests, and a PUSH with `--selective-tests-on-push` (which `ci-amd.yml` + passes for pushes to release branches) is treated like a pull request — selective tests on default + versions, not a `canary` run. The high-level flow and worked examples for these cases are in [The decision pipeline](#the-decision-pipeline) and [Worked examples](#worked-examples-pull-requests-pushes-and-canary-runs) above. * Python, Kubernetes, Backend, Kind, Helm versions are limited to "defaults" only unless `Full tests` mode diff --git a/dev/breeze/doc/ci/05_workflows.md b/dev/breeze/doc/ci/05_workflows.md index e62dc41a283..c59950dc89f 100644 --- a/dev/breeze/doc/ci/05_workflows.md +++ b/dev/breeze/doc/ci/05_workflows.md @@ -94,8 +94,9 @@ label in the PR if you are a committer and want to force dependency upgrade. ## Canary run -This workflow is triggered when a pull request is merged into the `main` -branch or pushed to any of the `v*-*-test` branches. The `canary` run +This workflow runs on a schedule on the `main` branch, and a few times a day +on the release branches that have new commits (see +[Release branch runs](#release-branch-runs)). The `canary` run aims to upgrade dependencies to their latest versions and promptly pushes a preview of the CI/PROD image cache to the GitHub Registry. This allows pull requests to quickly utilize the new cache, which is @@ -116,9 +117,18 @@ constraints will not be updated, meaning that regular PRs will continue using the older version of dependencies that passed one of the previous `canary` runs. -The `canary` runs are executed 6 times a day on schedule, you can also +The `canary` runs are scheduled 4 times a day on `main`, alternating between +`ci-amd.yml` (01:58, 13:58 UTC) and `ci-arm.yml` (07:28, 19:28 UTC). You can also trigger the `canary` run manually via `workflow-dispatch` mechanism. +A scheduled canary that has nothing new to test finishes at once: its first job, +`Check if the run is needed`, looks up the last completed full run (`schedule` or +`workflow_dispatch`) of the same workflow on the branch, and when that run tested the +same commit and succeeded, every other job is skipped and the run succeeds. When the +last run failed or was cancelled, or new commits landed since, the canary runs in full. +A skipped canary does not post to Slack. A canary on an unchanged commit therefore does +not pick up dependencies released since the last run; the next commit's canary does. + # Workflows Apache Airflow's CI system has evolved into what we call a composite workflow architecture. This approach breaks down our testing process into manageable, reusable pieces that work together to validate code changes. @@ -175,13 +185,61 @@ Release branches (named `v*-*-test`) get a more focused set of tests. These bran This approach saves time and resources while still ensuring release quality. +### Release branch runs + +Release-prep and providers branches (`vX-Y-test`, `providers-<name>/vX-Y`, `chart/vX-Yx-test`, +`airflow-ctl/vX-Y-test`) do **not** run the full CI on push. Backports land in bursts while a release +is prepared, and a full AMD + ARM run per merged commit was mostly cancelled by the next merge after +it had already used hundreds of runner-minutes. Every backport PR has run CI against the branch +already, so the branch needs a full run of its current HEAD a few times a day, not one per commit. + +**Full runs.** [`release-branch-ci.yml`](../../../../.github/workflows/release-branch-ci.yml) runs from +`main` on a schedule and starts `ci-amd.yml` or `ci-arm.yml` with `workflow_dispatch` on each release +branch, unless the last `workflow_dispatch` run of that workflow on the branch tested its current HEAD +and succeeded, or is still running. A run of an unchanged HEAD is repeated only when the previous one +failed or was cancelled: + +| Slot (UTC) | Workflow started | +|--------------|------------------| +| 04:43, 16:43 | `ci-amd.yml` | +| 10:43, 22:43 | `ci-arm.yml` | + +A `workflow_dispatch` run on a release branch is a canary run: it runs the full matrix and refreshes +the branch's constraints and image cache. Branches without a commit in the last 7 days are skipped. +`release-branch-ci.yml` can also be started by hand, with optional `branch`, `platform` and `force` +inputs. + +**Incremental tests on push.** `ci-amd.yml` still has a `push` trigger for these branches (AMD only), +but a push run only tests what changed since the last successful run of `ci-amd.yml` on the branch, +whether that was a push run or a full run. The first job, `Check if the run is needed`, finds that +run's commit through the Actions API on `ubuntu-slim`. `build-info` fetches that one commit (a +two-commit diff needs no history in between) and passes it to selective checks as +`--push-base-commit`, together with `--selective-tests-on-push`, so the run selects tests from the +files changed since that commit as for a pull request, and is not a canary run. If there is no +successful run to build on, or the base commit cannot be fetched or diffed, all tests run. A +successful run of the same commit is passed over, so re-running a push run tests it again. + +Push runs have a concurrency group of their own per branch: a newer push cancels the push run still +in progress, and the newer run then covers the cancelled run's changes too, because a cancelled run +is not a successful one. A failed run's changes are tested again the same way. A push never cancels a +full run of the branch. + +Nothing has to be switched on or off for a release. + +The workflow files of the release branch itself decide what a run does, so an older release branch +only gets this behaviour once the change that introduced it has been backported to it. Until then it +keeps running the full CI on every push. + +The CI image cache of `vX-Y-test` and providers branches is still refreshed on every push, by +[`refresh-image-registry-cache.yml`](../../../../.github/workflows/refresh-image-registry-cache.yml). + ### Branch Detection Logic The system determines which tests to run through the `build-info` job. This job checks the `AIRFLOW_BRANCH` setting in `airflow_breeze/branch_defaults.py` to understand what branch type it's working with, then enables or disables test groups accordingly. ## Tests Workflow Structure -The Tests workflow handles all code validation for Apache Airflow. When changes get pushed to `main` or release branches, these runs become "canary" builds that also update shared infrastructure like dependency constraints and Docker images. +The Tests workflow handles all code validation for Apache Airflow. Its scheduled runs on `main` and its full runs on release branches (see [Release branch runs](#release-branch-runs)) are "canary" builds that also update shared infrastructure like dependency constraints and Docker images. ### Workflow Execution Flow diff --git a/dev/breeze/doc/images/output_ci_selective-check.svg b/dev/breeze/doc/images/output_ci_selective-check.svg index 6c5e3064794..86043541c57 100644 --- a/dev/breeze/doc/images/output_ci_selective-check.svg +++ b/dev/breeze/doc/images/output_ci_selective-check.svg @@ -1,4 +1,4 @@ -<svg class="rich-terminal" viewBox="0 0 1482 757.5999999999999" xmlns="http://www.w3.org/2000/svg";> +<svg class="rich-terminal" viewBox="0 0 1482 855.1999999999999" xmlns="http://www.w3.org/2000/svg";> <!-- Generated with Rich https://www.textualize.io --> <style> @@ -43,7 +43,7 @@ <defs> <clipPath id="breeze-ci-selective-check-clip-terminal"> - <rect x="0" y="0" width="1463.0" height="706.5999999999999" /> + <rect x="0" y="0" width="1463.0" height="804.1999999999999" /> </clipPath> <clipPath id="breeze-ci-selective-check-line-0"> <rect x="0" y="1.5" width="1464" height="24.65"/> @@ -129,9 +129,21 @@ <clipPath id="breeze-ci-selective-check-line-27"> <rect x="0" y="660.3" width="1464" height="24.65"/> </clipPath> +<clipPath id="breeze-ci-selective-check-line-28"> + <rect x="0" y="684.7" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-selective-check-line-29"> + <rect x="0" y="709.1" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-selective-check-line-30"> + <rect x="0" y="733.5" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-selective-check-line-31"> + <rect x="0" y="757.9" width="1464" height="24.65"/> + </clipPath> </defs> - <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="755.6" rx="8"/><text class="breeze-ci-selective-check-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci selective-check</text> + <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="853.2" rx="8"/><text class="breeze-ci-selective-check-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci selective-check</text> <g transform="translate(26,22)"> <circle cx="0" cy="0" r="7" fill="#ff5f57"/> <circle cx="22" cy="0" r="7" fill="#febc2e"/> @@ -154,22 +166,26 @@ </text><text class="breeze-ci-selective-check-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-10)">│</text><text class="breeze-ci-selective-check-r6" x="390.4" y="264" textLength="73.2" clip-path="url(#breeze-ci-selective-check-line-10)">(TEXT)</text><text class="breeze-ci-selective-check-r5" x="1451.8" y="264" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-10)">│</text><text class="breeze-ci-selective-check-r1" x="1464" y="264" textLe [...] </text><text class="breeze-ci-selective-check-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-11)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="288.4" textLength="341.6" clip-path="url(#breeze-ci-selective-check-line-11)">--platform                  </text><text class="breeze-ci-selective-check-r1" x="390.4" y="288.4" textLength="1049.2" clip-pa [...] </text><text class="breeze-ci-selective-check-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-12)">│</text><text class="breeze-ci-selective-check-r1" x="390.4" y="312.8" textLength="170.8" clip-path="url(#breeze-ci-selective-check-line-12)">are testable. </text><text class="breeze-ci-selective-check-r5" x="561.2" y="312.8" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-12)">[default: linux/amd64]</text><text class="br [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="337.2" textLength="1464" clip-path="url(#breeze-ci-selective-check-line-13)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-selective-check-r1" x="1464" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-13)"> -</text><text class="breeze-ci-selective-check-r5" x="0" y="361.6" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-14)">╭─</text><text class="breeze-ci-selective-check-r5" x="24.4" y="361.6" textLength="231.8" clip-path="url(#breeze-ci-selective-check-line-14)"> GitHub parameters </text><text class="breeze-ci-selective-check-r5" x="256.2" y="361.6" textLength="1183.4" clip-path="url(#breeze-ci-selective-check-line-14)">────────────────────────────────────── [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-15)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="386" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-15)">--github-event-name   </text><text class="breeze-ci-selective-check-r1" x="366" y="386" textLength="610" clip-path="url(#breeze-ci-selective-check-line-15)">Name of the GitHub event [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-16)">│</text><text class="breeze-ci-selective-check-r6" x="366" y="410.4" textLength="1073.6" clip-path="url(#breeze-ci-selective-check-line-16)">| pull_request_review | pull_request_target | pull_request_workflow | push | schedule | </text><text class="breeze-ci-selective-check-r5" x="1451.8" y="410.4" t [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-17)">│</text><text class="breeze-ci-selective-check-r6" x="366" y="434.8" textLength="402.6" clip-path="url(#breeze-ci-selective-check-line-17)">workflow_dispatch | workflow_run)</text><text class="breeze-ci-selective-check-r5" x="1451.8" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-17)">│</text><text class="breeze-ci-s [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-18)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="459.2" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-18)">--github-repository   </text><text class="breeze-ci-selective-check-r7" x="317.2" y="459.2" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-18)">-g</text><text class="breeze-ci-select [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-19)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="483.6" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-19)">--github-actor        </text><text class="breeze-ci-selective-check-r1" x="366" y="483.6" textLength="549" clip-path="url(#breeze-ci-selective-check-line-19)">Actor that& [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-20)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="508" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-20)">--github-context      </text><text class="breeze-ci-selective-check-r1" x="366" y="508" textLength="695.4" clip-path="url(#breeze-ci-selective-check-line-20)">GitHub context (JSON [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-21)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="532.4" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-21)">--github-context-input</text><text class="breeze-ci-selective-check-r1" x="366" y="532.4" textLength="1073.6" clip-path="url(#breeze-ci-selective-check-line-21)">File input (might be `-`) wi [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-22)">│</text><text class="breeze-ci-selective-check-r4" x="366" y="556.8" textLength="195.2" clip-path="url(#breeze-ci-selective-check-line-22)">--github-context</text><text class="breeze-ci-selective-check-r1" x="561.2" y="556.8" textLength="744.2" clip-path="url(#breeze-ci-selective-check-line-22)"> for large contexts that would [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="581.2" textLength="1464" clip-path="url(#breeze-ci-selective-check-line-23)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-selective-check-r1" x="1464" y="581.2" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-23)"> -</text><text class="breeze-ci-selective-check-r5" x="0" y="605.6" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-24)">╭─</text><text class="breeze-ci-selective-check-r5" x="24.4" y="605.6" textLength="195.2" clip-path="url(#breeze-ci-selective-check-line-24)"> Common options </text><text class="breeze-ci-selective-check-r5" x="219.6" y="605.6" textLength="1220" clip-path="url(#breeze-ci-selective-check-line-24)">─────────────────────────────────────────── [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-25)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="630" textLength="109.8" clip-path="url(#breeze-ci-selective-check-line-25)">--verbose</text><text class="breeze-ci-selective-check-r7" x="158.6" y="630" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-25)">-v</text><text class="breeze-ci-selective-check-r1" x="207.4" y="630" te [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-26)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="654.4" textLength="109.8" clip-path="url(#breeze-ci-selective-check-line-26)">--dry-run</text><text class="breeze-ci-selective-check-r7" x="158.6" y="654.4" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-26)">-D</text><text class="breeze-ci-selective-check-r1" x="207.4" y="6 [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-27)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="678.8" textLength="109.8" clip-path="url(#breeze-ci-selective-check-line-27)">--help   </text><text class="breeze-ci-selective-check-r7" x="158.6" y="678.8" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-27)">-h</text><text class="breeze-ci-selective-check-r1" [...] -</text><text class="breeze-ci-selective-check-r5" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-ci-selective-check-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-selective-check-r1" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-28)"> +</text><text class="breeze-ci-selective-check-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-13)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="337.2" textLength="341.6" clip-path="url(#breeze-ci-selective-check-line-13)">--selective-tests-on-push   </text><text class="breeze-ci-selective-check-r1" x="390.4" y="337.2" textLength="1049.2" clip-path="url(#breeze-ci-selective-check-line-13)">Select tests for [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-14)">│</text><text class="breeze-ci-selective-check-r1" x="390.4" y="361.6" textLength="1049.2" clip-path="url(#breeze-ci-selective-check-line-14)">of running all tests.                              [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-15)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="386" textLength="341.6" clip-path="url(#breeze-ci-selective-check-line-15)">--push-base-commit          </text><text class="breeze-ci-selective-check-r1" x="390.4" y="386" textLength="61" clip-path="url(#breeze-ci-selective-check-line-15)">With& [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-16)">│</text><text class="breeze-ci-selective-check-r1" x="390.4" y="410.4" textLength="427" clip-path="url(#breeze-ci-selective-check-line-16)">Without it, a push runs all tests. </text><text class="breeze-ci-selective-check-r6" x="817.4" y="410.4" textLength="73.2" clip-path="url(#breeze-ci-selective-check-line-16)">(TEXT) [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="434.8" textLength="1464" clip-path="url(#breeze-ci-selective-check-line-17)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-selective-check-r1" x="1464" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-17)"> +</text><text class="breeze-ci-selective-check-r5" x="0" y="459.2" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-18)">╭─</text><text class="breeze-ci-selective-check-r5" x="24.4" y="459.2" textLength="231.8" clip-path="url(#breeze-ci-selective-check-line-18)"> GitHub parameters </text><text class="breeze-ci-selective-check-r5" x="256.2" y="459.2" textLength="1183.4" clip-path="url(#breeze-ci-selective-check-line-18)">────────────────────────────────────── [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-19)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="483.6" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-19)">--github-event-name   </text><text class="breeze-ci-selective-check-r1" x="366" y="483.6" textLength="610" clip-path="url(#breeze-ci-selective-check-line-19)">Name of the GitHub ev [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-20)">│</text><text class="breeze-ci-selective-check-r6" x="366" y="508" textLength="1073.6" clip-path="url(#breeze-ci-selective-check-line-20)">| pull_request_review | pull_request_target | pull_request_workflow | push | schedule | </text><text class="breeze-ci-selective-check-r5" x="1451.8" y="508" textLen [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-21)">│</text><text class="breeze-ci-selective-check-r6" x="366" y="532.4" textLength="402.6" clip-path="url(#breeze-ci-selective-check-line-21)">workflow_dispatch | workflow_run)</text><text class="breeze-ci-selective-check-r5" x="1451.8" y="532.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-21)">│</text><text class="breeze-ci-s [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-22)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="556.8" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-22)">--github-repository   </text><text class="breeze-ci-selective-check-r7" x="317.2" y="556.8" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-22)">-g</text><text class="breeze-ci-select [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-23)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="581.2" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-23)">--github-actor        </text><text class="breeze-ci-selective-check-r1" x="366" y="581.2" textLength="549" clip-path="url(#breeze-ci-selective-check-line-23)">Actor that& [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-24)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="605.6" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-24)">--github-context      </text><text class="breeze-ci-selective-check-r1" x="366" y="605.6" textLength="695.4" clip-path="url(#breeze-ci-selective-check-line-24)">GitHub context  [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-25)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="630" textLength="268.4" clip-path="url(#breeze-ci-selective-check-line-25)">--github-context-input</text><text class="breeze-ci-selective-check-r1" x="366" y="630" textLength="1073.6" clip-path="url(#breeze-ci-selective-check-line-25)">File input (might be `-`) with [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-26)">│</text><text class="breeze-ci-selective-check-r4" x="366" y="654.4" textLength="195.2" clip-path="url(#breeze-ci-selective-check-line-26)">--github-context</text><text class="breeze-ci-selective-check-r1" x="561.2" y="654.4" textLength="744.2" clip-path="url(#breeze-ci-selective-check-line-26)"> for large contexts that would [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="678.8" textLength="1464" clip-path="url(#breeze-ci-selective-check-line-27)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-selective-check-r1" x="1464" y="678.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-27)"> +</text><text class="breeze-ci-selective-check-r5" x="0" y="703.2" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-28)">╭─</text><text class="breeze-ci-selective-check-r5" x="24.4" y="703.2" textLength="195.2" clip-path="url(#breeze-ci-selective-check-line-28)"> Common options </text><text class="breeze-ci-selective-check-r5" x="219.6" y="703.2" textLength="1220" clip-path="url(#breeze-ci-selective-check-line-28)">─────────────────────────────────────────── [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-29)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="727.6" textLength="109.8" clip-path="url(#breeze-ci-selective-check-line-29)">--verbose</text><text class="breeze-ci-selective-check-r7" x="158.6" y="727.6" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-29)">-v</text><text class="breeze-ci-selective-check-r1" x="207.4" y="7 [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-30)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="752" textLength="109.8" clip-path="url(#breeze-ci-selective-check-line-30)">--dry-run</text><text class="breeze-ci-selective-check-r7" x="158.6" y="752" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-30)">-D</text><text class="breeze-ci-selective-check-r1" x="207.4" y="752" te [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-31)">│</text><text class="breeze-ci-selective-check-r4" x="24.4" y="776.4" textLength="109.8" clip-path="url(#breeze-ci-selective-check-line-31)">--help   </text><text class="breeze-ci-selective-check-r7" x="158.6" y="776.4" textLength="24.4" clip-path="url(#breeze-ci-selective-check-line-31)">-h</text><text class="breeze-ci-selective-check-r1" [...] +</text><text class="breeze-ci-selective-check-r5" x="0" y="800.8" textLength="1464" clip-path="url(#breeze-ci-selective-check-line-32)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-selective-check-r1" x="1464" y="800.8" textLength="12.2" clip-path="url(#breeze-ci-selective-check-line-32)"> </text> </g> </g> diff --git a/dev/breeze/doc/images/output_ci_selective-check.txt b/dev/breeze/doc/images/output_ci_selective-check.txt index 625c3a3b750..f5dd36b26d9 100644 --- a/dev/breeze/doc/images/output_ci_selective-check.txt +++ b/dev/breeze/doc/images/output_ci_selective-check.txt @@ -1 +1 @@ -bc06116a5cbad927acd981e56b6a5411 +1b069f3d7eeb1446639dba6563dff528 diff --git a/dev/breeze/src/airflow_breeze/commands/ci_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_commands.py index 887498f6355..d3bb71fbad8 100644 --- a/dev/breeze/src/airflow_breeze/commands/ci_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/ci_commands.py @@ -194,6 +194,21 @@ def get_changed_files(commit_ref: str | None) -> tuple[str, ...]: return changed_files +def get_changed_files_since(base_ref: str, commit_ref: str) -> tuple[str, ...] | None: + """Return the files changed between two commits, or None when they cannot be computed.""" + result = run_command( + ["git", "diff", "--name-only", base_ref, commit_ref], check=False, capture_output=True, text=True + ) + if result.returncode != 0: + console_print(f"[warning] Error when running diff command [/]\n{result.stdout}\n{result.stderr}") + return None + changed_files = tuple(result.stdout.splitlines()) if result.stdout else () + console_print(f"\n[info]Changed files since {base_ref}:[/]\n") + console_print(changed_files) + console_print() + return changed_files + + @ci_group.command( name="selective-check", help="Checks what kind of tests should be run for an incoming commit." ) @@ -248,6 +263,20 @@ def get_changed_files(commit_ref: str | None) -> tuple[str, ...]: envvar="PLATFORM", show_default=True, ) [email protected]( + "--selective-tests-on-push", + help="Select tests for a push event from the changed files, as for a pull request, " + "instead of running all tests.", + is_flag=True, + envvar="SELECTIVE_TESTS_ON_PUSH", +) [email protected]( + "--push-base-commit", + help="With --selective-tests-on-push: select tests from the files changed since this commit. " + "Without it, a push runs all tests.", + envvar="PUSH_BASE_COMMIT", + default="", +) @click.option( "--github-context", help="GitHub context (JSON formatted) passed by GitHub Actions", @@ -275,6 +304,8 @@ def selective_check( github_context: str, github_context_input: StringIO | None, ci_platform: str, + selective_tests_on_push: bool, + push_base_commit: str, ): try: from airflow_breeze.utils.selective_checks import SelectiveChecks @@ -286,10 +317,23 @@ def selective_check( github_context = github_context_input.read() github_context_dict = json.loads(github_context) if github_context else {} github_event = GithubEvents(github_event_name) - if commit_ref is not None: + changed_files: tuple[str, ...] = () + if commit_ref is not None and selective_tests_on_push and github_event == GithubEvents.PUSH: + # The base is the last successful run of the branch, so this run also covers the commits + # whose push runs failed or were cancelled by a newer push. Without one, nothing after the + # branch point is known to pass, so everything runs. + pushed_files = ( + get_changed_files_since(base_ref=push_base_commit, commit_ref=commit_ref) + if push_base_commit + else None + ) + if pushed_files is None: + console_print("[warning]No tested base commit for the push, running all tests[/]") + commit_ref = None + else: + changed_files = pushed_files + elif commit_ref is not None: changed_files = get_changed_files(commit_ref=commit_ref) - else: - changed_files = () sc = SelectiveChecks( commit_ref=commit_ref, files=changed_files, @@ -301,6 +345,7 @@ def selective_check( github_actor=github_actor, github_context_dict=github_context_dict, platform=ci_platform, + selective_tests_on_push=selective_tests_on_push, ) print(str(sc), file=sys.stderr) except Exception: @@ -346,10 +391,12 @@ class WorkflowInfo(NamedTuple): return PUBLIC_AMD_RUNNERS def is_canary_run(self) -> str: + # A push is never a canary: the CI workflows build on push only to run the selective tests + # of the pushed commit, and full runs of release branches are dispatched by + # `release-branch-ci.yml`. if ( self.event_name in [ - GithubEvents.PUSH.value, GithubEvents.WORKFLOW_DISPATCH.value, GithubEvents.SCHEDULE.value, ] diff --git a/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py b/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py index e7aa2259b80..f96b516018a 100644 --- a/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py +++ b/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py @@ -47,6 +47,8 @@ CI_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = { "--default-branch", "--default-constraints-branch", "--platform", + "--selective-tests-on-push", + "--push-base-commit", ], }, { diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py index 82f18b2df9d..c68d0f59307 100644 --- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py +++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py @@ -907,6 +907,7 @@ class SelectiveChecks: github_actor: str = "", github_context_dict: dict[str, Any] | None = None, platform: str = CI_AMD_PLATFORM, + selective_tests_on_push: bool = False, ): self._files = files # A changed test helper selects the tests that import it, as if those test files had changed; @@ -933,6 +934,7 @@ class SelectiveChecks: self._github_actor = github_actor self._github_context_dict = github_context_dict or {} self._platform = platform + self._selective_tests_on_push = selective_tests_on_push self._new_toml: dict[str, Any] = {} self._old_toml: dict[str, Any] = {} @@ -986,8 +988,15 @@ class SelectiveChecks: def default_constraints_branch(self) -> str: return self._default_constraints_branch + def _get_full_run_events(self) -> list[GithubEvents]: + # Only `ci-amd.yml` opts in: other push-triggered workflows (constraints, image cache) rely on + # a push selecting all Python versions. + if self._selective_tests_on_push: + return [GithubEvents.SCHEDULE, GithubEvents.WORKFLOW_DISPATCH] + return [GithubEvents.PUSH, GithubEvents.SCHEDULE, GithubEvents.WORKFLOW_DISPATCH] + def _should_run_all_tests_and_versions(self) -> bool: - if self._github_event in [GithubEvents.PUSH, GithubEvents.SCHEDULE, GithubEvents.WORKFLOW_DISPATCH]: + if self._github_event in self._get_full_run_events(): if self.only_text_non_doc_files_changed and self._github_event == GithubEvents.PUSH: console_print( f"[warning]Only text non doc files changed in {self._github_event}, skip full tests[/]" @@ -2279,7 +2288,7 @@ class SelectiveChecks: def _is_canary_run(self): return ( - self._github_event in [GithubEvents.SCHEDULE, GithubEvents.PUSH, GithubEvents.WORKFLOW_DISPATCH] + self._github_event in self._get_full_run_events() and self._github_repository == APACHE_AIRFLOW_GITHUB_REPOSITORY and not self.only_text_non_doc_files_changed ) or CANARY_LABEL in self._pr_labels diff --git a/dev/breeze/tests/test_ci_commands.py b/dev/breeze/tests/test_ci_commands.py new file mode 100644 index 00000000000..044ee4e21bb --- /dev/null +++ b/dev/breeze/tests/test_ci_commands.py @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from subprocess import CompletedProcess +from unittest.mock import patch + +import pytest +from click.testing import CliRunner + +from airflow_breeze.commands.ci_commands import get_changed_files_since, selective_check + +PUSHED_COMMIT = "b" * 40 +BASE_COMMIT = "a" * 40 + + [email protected]( + ("returncode", "stdout", "expected"), + [ + pytest.param(0, "a.py\nb/c.py\n", ("a.py", "b/c.py"), id="changed files"), + pytest.param(0, "", (), id="no changes"), + pytest.param(128, "", None, id="git diff failed"), + ], +) +@patch("airflow_breeze.commands.ci_commands.run_command", autospec=True) +def test_get_changed_files_since(mock_run_command, returncode, stdout, expected): + mock_run_command.return_value = CompletedProcess(args=[], returncode=returncode, stdout=stdout, stderr="") + + assert get_changed_files_since(base_ref=BASE_COMMIT, commit_ref=PUSHED_COMMIT) == expected + assert mock_run_command.call_args.args[0] == ["git", "diff", "--name-only", BASE_COMMIT, PUSHED_COMMIT] + + +def _invoke_selective_check(*extra_args: str): + return CliRunner().invoke( + selective_check, + ["--commit-ref", PUSHED_COMMIT, "--github-event-name", "push", *extra_args], + catch_exceptions=False, + ) + + [email protected]( + ("changed_since_base", "expected_commit_ref", "expected_files"), + [ + pytest.param(("a.py", "b.py"), PUSHED_COMMIT, ("a.py", "b.py"), id="files since the base commit"), + pytest.param(None, None, (), id="all tests when the diff cannot be computed"), + ], +) +@patch("airflow_breeze.commands.ci_commands.get_changed_files", autospec=True) +@patch("airflow_breeze.commands.ci_commands.get_changed_files_since", autospec=True) +@patch("airflow_breeze.utils.selective_checks.SelectiveChecks", autospec=True) +def test_selective_tests_on_push_select_from_files_since_base_commit( + mock_selective_checks, + mock_get_changed_files_since, + mock_get_changed_files, + changed_since_base, + expected_commit_ref, + expected_files, +): + mock_get_changed_files_since.return_value = changed_since_base + + _invoke_selective_check("--selective-tests-on-push", "--push-base-commit", BASE_COMMIT) + + mock_get_changed_files_since.assert_called_once_with(base_ref=BASE_COMMIT, commit_ref=PUSHED_COMMIT) + mock_get_changed_files.assert_not_called() + assert mock_selective_checks.call_args.kwargs["commit_ref"] == expected_commit_ref + assert mock_selective_checks.call_args.kwargs["files"] == expected_files + + +@patch("airflow_breeze.commands.ci_commands.get_changed_files", autospec=True) +@patch("airflow_breeze.commands.ci_commands.get_changed_files_since", autospec=True) +@patch("airflow_breeze.utils.selective_checks.SelectiveChecks", autospec=True) +def test_selective_tests_on_push_without_a_base_commit_run_all_tests( + mock_selective_checks, mock_get_changed_files_since, mock_get_changed_files +): + _invoke_selective_check("--selective-tests-on-push") + + mock_get_changed_files_since.assert_not_called() + mock_get_changed_files.assert_not_called() + assert mock_selective_checks.call_args.kwargs["commit_ref"] is None + assert mock_selective_checks.call_args.kwargs["files"] == () + + +@patch("airflow_breeze.commands.ci_commands.get_changed_files", autospec=True, return_value=("a.py",)) +@patch("airflow_breeze.commands.ci_commands.get_changed_files_since", autospec=True) +@patch("airflow_breeze.utils.selective_checks.SelectiveChecks", autospec=True) +def test_push_without_selective_tests_on_push_uses_the_pushed_commit( + mock_selective_checks, mock_get_changed_files_since, mock_get_changed_files +): + _invoke_selective_check("--push-base-commit", BASE_COMMIT) + + mock_get_changed_files_since.assert_not_called() + mock_get_changed_files.assert_called_once_with(commit_ref=PUSHED_COMMIT) + assert mock_selective_checks.call_args.kwargs["files"] == ("a.py",) diff --git a/dev/breeze/tests/test_pr_info.py b/dev/breeze/tests/test_pr_info.py index 2a2f814c7dd..878ef9acbe3 100644 --- a/dev/breeze/tests/test_pr_info.py +++ b/dev/breeze/tests/test_pr_info.py @@ -50,7 +50,7 @@ def test_push_info(): assert wi.event_name == "push" assert wi.pr_number is None assert wi.get_runs_on() == '["ubuntu-22.04"]' - assert wi.is_canary_run() == "true" + assert wi.is_canary_run() == "false" assert wi.run_coverage() == "true" diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py index e420bef52f4..26bc452fd24 100644 --- a/dev/breeze/tests/test_selective_checks.py +++ b/dev/breeze/tests/test_selective_checks.py @@ -2914,6 +2914,39 @@ def test_expected_output_push( assert_outputs_are_printed(expected_outputs, str(stderr)) [email protected]( + ("selective_tests_on_push", "expected_outputs"), + [ + pytest.param( + False, + { + "all-python-versions": ALL_PYTHON_VERSIONS_AS_LIST, + "full-tests-needed": "true", + }, + id="Push runs all tests by default", + ), + pytest.param( + True, + { + "all-python-versions": f"['{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}']", + "full-tests-needed": "false", + "run-unit-tests": "true", + }, + id="Push selects tests from the changed files when selective tests on push are enabled", + ), + ], +) +def test_selective_tests_on_push(selective_tests_on_push: bool, expected_outputs: dict[str, str]): + stderr = SelectiveChecks( + files=("airflow-core/src/airflow/api.py",), + commit_ref=NEUTRAL_COMMIT, + github_event=GithubEvents.PUSH, + default_branch="v3-4-test", + selective_tests_on_push=selective_tests_on_push, + ) + assert_outputs_are_printed(expected_outputs, str(stderr)) + + @pytest.mark.parametrize( ("files", "expected_outputs"), [ diff --git a/scripts/ci/prek/check_ci_workflows_in_sync.py b/scripts/ci/prek/check_ci_workflows_in_sync.py index 14d195f067b..346aaa61926 100755 --- a/scripts/ci/prek/check_ci_workflows_in_sync.py +++ b/scripts/ci/prek/check_ci_workflows_in_sync.py @@ -27,12 +27,12 @@ Documented divergences: 1. Header intro comment (different prose explaining what each file is for) 2. Workflow ``name:`` — ``Tests (ARM)`` vs ``Tests (AMD)`` -3. Triggers — ARM = its own schedule + push (to release-prep / - providers branches) + workflow_dispatch; AMD = its own schedule - (offset from ARM's cron) + the same push branches + pull_request + - workflow_dispatch. Both wrappers run on post-merge pushes to - stable / providers branches; only AMD runs on PRs (per-PR ARM - coverage stays cron-driven via the canary). +3. Triggers — ARM = its own schedule + workflow_dispatch; AMD = its own + schedule (offset from ARM's cron) + pull_request + push (selective + tests on release-prep / providers branches) + workflow_dispatch. + ``release-branch-ci.yml`` dispatches the full runs of both on those + branches. Only AMD runs on PRs and pushes (ARM coverage stays + cron-driven via the canary and the dispatched release-branch runs). 4. ``concurrency.group`` prefix — ``ci-arm-`` vs ``ci-amd-`` 5. ``build-info`` outputs ``platform`` and ``runner-type`` — hardcoded per architecture (and the surrounding comment naming the "ARM/AMD copy") @@ -84,7 +84,10 @@ THIS_SCRIPT = Path(__file__).relative_to(REPO_ROOT) # AMD, append a rule here. LINE_RULES: list[tuple[str, str]] = [ (r"^name: Tests \((?:ARM|AMD)\)$", "name: Tests (PLACEHOLDER)"), - (r"^ group: ci-(?:arm|amd)-", " group: ci-PLACEHOLDER-"), + ( + r"^ ci-(?:arm|amd)-\$\{\{ github\.event\.pull_request\.number \|\| github\.ref \}\}\$\{\{$", + " ci-PLACEHOLDER-${{ github.event.pull_request.number || github.ref }}${{", + ), (r'^ platform: "linux/(?:arm64|amd64)"$', ' platform: "linux/PLACEHOLDER"'), (r"^ runner-type: '\[\"ubuntu-22\.04(?:-arm)?\"\]'$", " runner-type: 'PLACEHOLDER'"), (r'^ PLATFORM: "linux/(?:arm64|amd64)"$', ' PLATFORM: "linux/PLACEHOLDER"'), @@ -134,6 +137,16 @@ AMD_ONLY_BLOCK = """ schedule: - airflow-ctl/v[0-9]+-[0-9]+-test - airflow-ctl/v[0-9]+-[0-9]+-stable types: [opened, reopened, synchronize, ready_for_review] + push: + # Pushes to release-prep / providers branches run only the tests needed by the changes since the + # last successful run of the branch (selected as for a PR). Full runs of these branches, on AMD + # and ARM, are dispatched by `release-branch-ci.yml`. `main` does not build on push; its full + # runs are the canary above. + branches: + - v[0-9]+-[0-9]+-test + - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+ + - chart/v[0-9]+-[0-9]+x-test + - airflow-ctl/v[0-9]+-[0-9]+-test """
