This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch backport-ci-sync-v3-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 336be41b9cdd152c2f7a7ef9865dbcff92cd3e61 Author: Jarek Potiuk <[email protected]> AuthorDate: Sun Oct 25 19:41:01 2026 +0100 Do not fail CI passed for a cancelled duplicate CI run (#75578) * Do not fail CI passed for a cancelled duplicate CI run A run that concurrency cancels still runs the always() CI passed job, and that job failed the required check. When a stacked pull request's base and head are force-pushed together, GitHub sends two synchronize events for the same commit, so one run is cancelled seconds in and a failed CI passed sits next to the passing one of its twin. A cancelled run now reports under a name the rulesets do not require, so nothing is red, while a run that was only cancelled still never satisfies CI passed and the merge stays blocked. Generated-by: Claude Opus 5 * Accept the cancelled-run CI passed name in the fork CI tampering check A cancelled ci-amd.yml run now reports its CI passed job as "CI passed (run cancelled)". Without the name in the gated-run jobs, a run of a gated pull request that a newer push cancelled would be taken for project CI work and close the pull request as tampering with the gate. Generated-by: Claude Opus 5 * Keep failing CI passed when a CI job times out A job that hits its timeout is cancelled while the run is not, so the cancelled-run rename alone would turn a hung job into a passing check and drop it from the scheduled canary's Slack report. Generated-by: Claude Opus 5 * Make the Slack canary report wait for the CI passed gate job Generated-by: Claude Opus 5 (cherry picked from commit 1b9a32b528300a545ca71daee3d161da1468347c) The README conflict was resolved by keeping the release branch's "one check" wording. --- .github/workflows/ci-amd.yml | 34 ++++++++++++++++++---- .github/workflows/ci-arm.yml | 34 ++++++++++++++++++---- dev/README_AIRFLOW3_DEV.md | 10 +++++-- .../src/airflow_breeze/utils/fork_ci_reconcile.py | 9 +++--- dev/breeze/tests/test_fork_ci_reconcile.py | 10 +++++++ 5 files changed, 80 insertions(+), 17 deletions(-) diff --git a/.github/workflows/ci-amd.yml b/.github/workflows/ci-amd.yml index 7fd1452b3cf..49f08dbd367 100644 --- a/.github/workflows/ci-amd.yml +++ b/.github/workflows/ci-amd.yml @@ -1499,11 +1499,20 @@ jobs: # pull request to CI in the contributor's fork, and every job here is skipped. Reporting # "CI passed" then would let it merge on no CI, so the job reports under another name and # the fork CI reconciler posts "CI passed" from the fork run instead. + # A run with cancelled jobs and no failed one -- usually a duplicate run of the same commit that + # concurrency cancelled, such as the two `synchronize` events GitHub sends when a stacked pull + # request's base and head are force-pushed together -- reports under another name too, so it + # never puts a failed "CI passed" next to the passing one of its twin. "CI passed" is then + # missing for that run, which keeps the merge blocked until another run of the commit reports it. + # A job cancelled by its timeout gets that name too, since `name` cannot test whether the run + # itself was cancelled, but the job still fails. name: >- - ${{ needs.build-info.result == 'skipped' && 'CI passed (project CI not run)' || 'CI passed' }} + ${{ (!contains(needs.*.result, 'failure') && contains(needs.*.result, 'cancelled') && + 'CI passed (run cancelled)') || + (needs.build-info.result == 'skipped' && 'CI passed (project CI not run)') || 'CI passed' }} # The one check main's ruleset requires, so auto-merge waits for CI to finish. GitHub treats a # skipped required check as passed, so this job always runs and fails itself when any job it - # needs failed or was cancelled. Jobs skipped by selective checks count as passed. + # needs failed or timed out. Jobs skipped by selective checks count as passed. if: always() needs: - additional-ci-image-checks @@ -1546,12 +1555,25 @@ jobs: - tests-with-lowest-direct-resolution-providers runs-on: ["ubuntu-slim"] steps: - - name: "Fail if any CI job failed or was cancelled" - if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') + - name: "Fail if any CI job failed" + if: contains(needs.*.result, 'failure') run: | - echo "At least one CI job failed or was cancelled." + echo "At least one CI job failed." exit 1 + # A job that hits its timeout is cancelled while the run is not, and must fail this check, as + # the scheduled canary's Slack report lists only failed jobs. + - name: "Fail if a CI job was cancelled without the run being cancelled" + if: contains(needs.*.result, 'cancelled') && !cancelled() + run: | + echo "A CI job was cancelled, likely by its timeout, while the run was not cancelled." + exit 1 + - name: "Report that CI jobs were cancelled" + if: contains(needs.*.result, 'cancelled') && cancelled() + run: | + echo "At least one CI job was cancelled, so this run does not report CI passed." + echo 'CI jobs were cancelled: this run does not report `CI passed`.' >> "${GITHUB_STEP_SUMMARY}" - name: "All CI jobs passed or were skipped" + if: "!contains(needs.*.result, 'cancelled')" run: echo "All CI jobs passed or were skipped." notify-slack: @@ -1559,6 +1581,8 @@ jobs: needs: - build-info - check-run-needed + # Waits for the gate job, which is the only failed job when a CI job timed out. + - ci-passed - finalize-tests - tests-agent-frameworks # A no-op canary (`run-needed` is 'false') does not post; a failed check does. A failing gate diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml index ce349310799..e6299a05ece 100644 --- a/.github/workflows/ci-arm.yml +++ b/.github/workflows/ci-arm.yml @@ -1381,11 +1381,20 @@ jobs: # pull request to CI in the contributor's fork, and every job here is skipped. Reporting # "CI passed" then would let it merge on no CI, so the job reports under another name and # the fork CI reconciler posts "CI passed" from the fork run instead. + # A run with cancelled jobs and no failed one -- usually a duplicate run of the same commit that + # concurrency cancelled, such as the two `synchronize` events GitHub sends when a stacked pull + # request's base and head are force-pushed together -- reports under another name too, so it + # never puts a failed "CI passed" next to the passing one of its twin. "CI passed" is then + # missing for that run, which keeps the merge blocked until another run of the commit reports it. + # A job cancelled by its timeout gets that name too, since `name` cannot test whether the run + # itself was cancelled, but the job still fails. name: >- - ${{ needs.build-info.result == 'skipped' && 'CI passed (project CI not run)' || 'CI passed' }} + ${{ (!contains(needs.*.result, 'failure') && contains(needs.*.result, 'cancelled') && + 'CI passed (run cancelled)') || + (needs.build-info.result == 'skipped' && 'CI passed (project CI not run)') || 'CI passed' }} # The one check main's ruleset requires, so auto-merge waits for CI to finish. GitHub treats a # skipped required check as passed, so this job always runs and fails itself when any job it - # needs failed or was cancelled. Jobs skipped by selective checks count as passed. + # needs failed or timed out. Jobs skipped by selective checks count as passed. if: always() needs: - additional-ci-image-checks @@ -1427,12 +1436,25 @@ jobs: - tests-with-lowest-direct-resolution-providers runs-on: ["ubuntu-slim"] steps: - - name: "Fail if any CI job failed or was cancelled" - if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') + - name: "Fail if any CI job failed" + if: contains(needs.*.result, 'failure') run: | - echo "At least one CI job failed or was cancelled." + echo "At least one CI job failed." exit 1 + # A job that hits its timeout is cancelled while the run is not, and must fail this check, as + # the scheduled canary's Slack report lists only failed jobs. + - name: "Fail if a CI job was cancelled without the run being cancelled" + if: contains(needs.*.result, 'cancelled') && !cancelled() + run: | + echo "A CI job was cancelled, likely by its timeout, while the run was not cancelled." + exit 1 + - name: "Report that CI jobs were cancelled" + if: contains(needs.*.result, 'cancelled') && cancelled() + run: | + echo "At least one CI job was cancelled, so this run does not report CI passed." + echo 'CI jobs were cancelled: this run does not report `CI passed`.' >> "${GITHUB_STEP_SUMMARY}" - name: "All CI jobs passed or were skipped" + if: "!contains(needs.*.result, 'cancelled')" run: echo "All CI jobs passed or were skipped." notify-slack: @@ -1440,6 +1462,8 @@ jobs: needs: - build-info - check-run-needed + # Waits for the gate job, which is the only failed job when a CI job timed out. + - ci-passed - finalize-tests - tests-agent-frameworks # A no-op canary (`run-needed` is 'false') does not post; a failed check does. A failing gate diff --git a/dev/README_AIRFLOW3_DEV.md b/dev/README_AIRFLOW3_DEV.md index af75c46866f..c13904dbbec 100644 --- a/dev/README_AIRFLOW3_DEV.md +++ b/dev/README_AIRFLOW3_DEV.md @@ -196,9 +196,13 @@ Auto-merge is enabled for the whole repository, so every branch that PRs target then, its PRs never report the check, and a committer merges them by bypassing the rule. The `main` ruleset requires one check, **CI passed**. It is a job in the `Tests (AMD)` workflow that always -runs and fails when any other CI job failed or was cancelled; jobs that selective checks skip count as -passed, so PRs that run only part of the tests still get a result. The `check-ci-gate-needs` prek hook -makes sure every new CI job is added to it. `Tests (ARM)`, which runs on pushes and on a schedule rather +runs and fails when any other CI job failed; jobs that selective checks skip count as passed, so PRs that +run only part of the tests still get a result. A run with cancelled jobs and no failed one reports as +**CI passed (run cancelled)** instead, so a duplicate run of the same commit that concurrency cancelled +does not show a failed **CI passed** next to its twin's result. Such a run does not report **CI passed** +at all, so the PR stays blocked until another run of the commit, or a re-run, reports it. A job that +hits its timeout is cancelled while the run is not, so that run's check fails, under the same name. The +`check-ci-gate-needs` prek hook makes sure every new CI job is added to it. `Tests (ARM)`, which runs on pushes and on a schedule rather than on PRs, has the same job, kept identical by the `check-ci-workflows-in-sync` hook. For a PR that the fork CI gate (AIP-120) leaves to CI in the contributor's fork, project CI does not run, so the job reports as **CI passed (project CI not run)** instead, and the fork CI reconciler posts **CI passed** once the diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py index 35189f1a8f4..3c4eeafb70d 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py @@ -79,12 +79,13 @@ GATE_JOB_NAME = "Fork CI gate" # Jobs that also run, without needing the gate, in a run the gate left to fork CI: main's # check-run-needed (no steps outside schedule and push runs) and the "CI passed" gate job of the # merge ruleset, which reports as "CI passed (project CI not run)" when project CI was skipped and -# as "CI passed" when a newer push cancelled the run before build-info started. Like the gate, each -# must finish within GATE_MAX_DURATION, so project CI work moved under one of these names counts. -# Each runs once, so the names of one job are grouped: repeating short jobs under them counts too. +# as "CI passed (run cancelled)" when a newer push cancelled the run before build-info started -- +# "CI passed" in a workflow from before that name. Like the gate, each must finish within +# GATE_MAX_DURATION, so project CI work moved under one of these names counts. Each runs once, so +# the names of one job are grouped: repeating short jobs under them counts too. GATED_RUN_JOBS = ( frozenset({"Check if the run is needed"}), - frozenset({"CI passed (project CI not run)", "CI passed"}), + frozenset({"CI passed (project CI not run)", "CI passed (run cancelled)", "CI passed"}), ) GATED_RUN_JOB_NAMES = frozenset[str]().union(*GATED_RUN_JOBS) BOT_LOGINS = frozenset({"github-actions", "github-actions[bot]"}) diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py b/dev/breeze/tests/test_fork_ci_reconcile.py index be4e1dcb15f..3f7d56f7c14 100644 --- a/dev/breeze/tests/test_fork_ci_reconcile.py +++ b/dev/breeze/tests/test_fork_ci_reconcile.py @@ -719,6 +719,16 @@ def _job( False, id="cancelled-before-the-gate-started", ), + pytest.param( + [ + _job("Check if the run is needed", "cancelled", started=False), + _job("Fork CI gate", "cancelled", started=False), + _job("Build info", "cancelled", started=False), + _job("CI passed (run cancelled)", "success"), + ], + False, + id="cancelled-run-reporting-ci-passed-run-cancelled", + ), pytest.param( [_job("Fork CI gate", "cancelled", minutes=30, started=False)], False,
