This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 433ef3ebd6f0312bfc71746b184ee5e9fe1fd6a6 Author: Jarek Potiuk <[email protected]> AuthorDate: Sat Nov 14 09:46:37 2026 +0100 Report fork CI to pull requests sooner and with failing jobs (#76836) * List the jobs that did not pass in the fork CI state comment When the fork run of a gated pull request failed, the state comment only said "failed" and linked the run, so the author and reviewers had to open the fork's run to see what broke, and with job statuses turned off or deferred by the write budget the pull request showed nothing more. The state comment of a failed run now lists the jobs that failed, timed out or were cancelled, linking each job in the fork (at most 10). The jobs are reused from the job statuses when they were read in the same run, and a failed read leaves the comment for the next run instead of dropping the list. Generated-by: Claude Opus 5 * Run the fork CI monitor every 5 minutes Fork CI reaches a pull request only when the monitor runs: the fork run is in another repository, so no event in apache/airflow reports its start or end. At a 15-minute cadence plus GitHub's delay of scheduled workflows, the start, failure or success of a fork run showed up on the pull request 10 to 20 minutes late, and so did `CI run started`, which every pull request waits for before it can merge. The monitor now runs every 5 minutes. The per-run write budgets and the number of runs before a Slack note are derived from MONITOR_INTERVAL, so their hourly totals stay as they were; a test checks that the interval matches the workflow's cron. Unchanged pull requests cost no writes; the cost is about three times the monitor's runner time and reads. Generated-by: Claude Opus 5 * Read a failed fork run's jobs once and post the failed state without them The state comment's marker records the fork run and attempt its failed-job list belongs to, so later monitor runs do not read the same jobs again while the comment would stay unchanged. When the jobs cannot be read, the comment is held only if it already shows this head commit as failed; otherwise the failed state is posted without the list. Failed jobs are listed before timed-out and cancelled ones, so the cap of 10 does not hide them. Generated-by: Claude Opus 5 * List the failed job of fork CI jobs sharing a name When a cancelled or timed-out job came before a failed one of the same name, the state comment linked the cancelled one and sorted it after the real failures. Generated-by: Claude Opus 5 --- .github/workflows/fork-ci-monitor.yml | 5 +- contributing-docs/05_pull_requests.rst | 2 +- dev/AGENTS.md | 4 +- dev/breeze/doc/08_ci_tasks.rst | 2 +- .../doc/ci/08_fork_ci_for_external_contributors.md | 79 +++++--- .../src/airflow_breeze/utils/fork_ci_messages.py | 53 ++++- .../src/airflow_breeze/utils/fork_ci_reconcile.py | 149 +++++++++++--- dev/breeze/tests/test_fork_ci_messages.py | 42 ++++ dev/breeze/tests/test_fork_ci_reconcile.py | 220 +++++++++++++++++++-- 9 files changed, 483 insertions(+), 73 deletions(-) diff --git a/.github/workflows/fork-ci-monitor.yml b/.github/workflows/fork-ci-monitor.yml index e95becc9e27..a2d09f59fb0 100644 --- a/.github/workflows/fork-ci-monitor.yml +++ b/.github/workflows/fork-ci-monitor.yml @@ -27,7 +27,8 @@ name: "Fork CI monitor" on: # yamllint disable-line rule:truthy schedule: - - cron: '*/15 * * * *' + # MONITOR_INTERVAL in fork_ci_reconcile.py, whose per-run write budgets assume this cadence. + - cron: '*/5 * * * *' workflow_dispatch: inputs: dry-run: @@ -59,7 +60,7 @@ jobs: timeout-minutes: 20 steps: # What earlier runs reported, so that a problem that persists is posted to Slack once - # rather than every 15 minutes. Losing it, for example to cache eviction, re-posts once. + # rather than on every run. Losing it, for example to cache eviction, re-posts once. # Restored before anything that can fail, so a failing setup is reported once too, and # outside the checkout, where nothing but the steps below reads it. - name: "Restore what earlier runs reported" diff --git a/contributing-docs/05_pull_requests.rst b/contributing-docs/05_pull_requests.rst index 7320241b18c..d927b218f7c 100644 --- a/contributing-docs/05_pull_requests.rst +++ b/contributing-docs/05_pull_requests.rst @@ -272,7 +272,7 @@ and approves them automatically when all of these hold: If your PR's CI runs in your fork (it carries the ``fork ci required`` label), the CI run in ``apache/airflow`` only checks that, so none of the rules above apply. The ``CI run started`` check -waits until the fork CI monitor sees your fork's run, usually within 15 minutes, and then links it. Its runs are approved +waits until the fork CI monitor sees your fork's run, usually within 5 to 10 minutes, and then links it. Its runs are approved automatically unless the PR changes files that decide which CI jobs run and on which runners: anything under ``.github/``, ``dev/breeze/src/`` or ``scripts/ci/prek/``, or ``dev/breeze/pyproject.toml``, ``dev/breeze/uv.lock`` or ``scripts/ci/install_breeze.sh``. Then diff --git a/dev/AGENTS.md b/dev/AGENTS.md index f9e999acfc4..c2ac0cbbb2f 100644 --- a/dev/AGENTS.md +++ b/dev/AGENTS.md @@ -54,9 +54,9 @@ workflow of `apache/airflow` shares (see "Write budget" in If the budget is exceeded, turn the job statuses off: set `job_statuses = false` in `.github/fork-ci-gate.toml`. Signs that it is exceeded are the monitor's Slack note that job -statuses "have not fit in the budget" (posted only after three runs in a row deferred them) +statuses "have not fit in the budget" (posted only after job statuses were deferred for about 45 minutes in a row) staying for hours, pull requests listed because their fork run finished more than an hour ago with -job statuses still not posted, its Slack note that GitHub rate-limited it in two runs in a row, the +job statuses still not posted, its Slack note that GitHub rate-limited it for about 30 minutes in a row, the "Rate limits left" line of its run log approaching zero, or other workflows of the repository failing on secondary rate limits. The aggregate `Fork CI / Tests (AMD)` status and `CI passed` are posted either way, so turning job statuses off loses detail, not gating. diff --git a/dev/breeze/doc/08_ci_tasks.rst b/dev/breeze/doc/08_ci_tasks.rst index 03d61fed826..c4001b658c0 100644 --- a/dev/breeze/doc/08_ci_tasks.rst +++ b/dev/breeze/doc/08_ci_tasks.rst @@ -185,7 +185,7 @@ Reconciling pull requests gated to fork CI ------------------------------------------ Pull requests from external contributors can run their CI in the contributor's fork instead of in -``apache/airflow`` (AIP-120). The ``fork-ci-reconcile`` command, run every 15 minutes by the +``apache/airflow`` (AIP-120). The ``fork-ci-reconcile`` command, run every 5 minutes by the ``fork-ci-monitor.yml`` workflow, is the only place that changes those pull requests: it labels newly gated pull requests ``fork ci required``, keeps them in draft until the fork run is green, maintains one state comment and a ``Fork CI / Tests (AMD)`` commit status linking to the run in the fork, closes diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index 266665d9af5..5220187e34b 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -87,7 +87,7 @@ Non-goals: │ ├─ build-info: fork mode │ │ (build-info and everything after │ │ │ changed files from │ │ are skipped for gated PRs) │ │ │ compare API vs upstream │ │ │ - │ └─ public runners only │ │ fork-ci-monitor.yml (every 15 minutes) │ + │ └─ public runners only │ │ fork-ci-monitor.yml (every 5 minutes) │ └──────────────────────────────────┘ │ └─ breeze ci fork-ci-reconcile │ ▲ │ ├─ discover new gated PRs -> label │ │ GraphQL: checkSuites │ ├─ per labelled PR: fork state -> │ @@ -490,7 +490,7 @@ reads too: the default branch's comes from the reconciler's checkout, any other from the contents API once per run. Those with a gated reason get the label. A base branch with neither gate file is never gated, as its gate job runs project CI (`base_without_gate`), and one with only one of the two files is reported as unknown. The -24-hour window overlaps many 15-minute cadences, so a lost scheduled run changes nothing. A +24-hour window overlaps many monitor cadences, so a lost scheduled run changes nothing. A pull request whose author later becomes a committer keeps the label until a maintainer removes it or adds `use project ci`; the reconciler does not un-gate on its own. @@ -612,7 +612,7 @@ since the draft-state job runs after the close and GitHub refuses to draft a clo | **setup_required**: workflow state is `disabled_fork`, `disabled_manually` or `disabled_inactivity`, or the fork has no `ci-amd.yml` | `error` (red), link to the fork's Actions page for `ci-amd.yml` | create or update: how to enable, with the exact `gh repo sync <fork> --branch main` and `gh workflow enable ci-amd.yml -R <fork>` lines, and `breeze ci audit`; on transition from any other state but `closed` (the workflow stopped running after it worked): also post a **new** comment sayin [...] | **awaiting_push**: enabled, no run for the head SHA | `pending` (yellow), link to the fork's Actions page | on transition from setup_required: post a **new** comment asking to push again (an edit would not notify) | | **running**: run in progress | `pending` (yellow), link to the run | update state and run link | -| **failed**: conclusion is failure, cancelled, timed out or startup failure | `failure` (red), link to the run; for a startup failure the description says the run could not start | update with the run link; for a startup failure the text says no job ran and points to the fork's Actions settings | +| **failed**: conclusion is failure, cancelled, timed out or startup failure | `failure` (red), link to the run; for a startup failure the description says the run could not start | update with the run link and the jobs that did not pass; for a startup failure the text says no job ran and points to the fork's Actions settings | | **green**: conclusion success and the pull request does not change CI-defining files | `success` (green), link to the run | update: ready for review, run link; on the author's draft: mark it ready for review when it is | | **green_unverified**: conclusion success, but the pull request changes CI-defining files (see "Trusting a green fork run") | `pending` (yellow), link to the run | update: a maintainer must run project CI; on the author's draft: mark it ready for review, then a maintainer runs project CI | | **closed**: setup_required or awaiting_push for `close_after_days` | unchanged | close with a new comment; reopening restarts the clock | @@ -726,7 +726,7 @@ every event that runs the workflow files of a pull request's merge commit: `pull requests by head repository and branch. Looking at every recent run of the branch, rather than only at the head commit, is what stops a contributor from pushing a tampered commit and then a clean one to hide the tampered run. The 6-hour window spans many -15-minute cadences. Runs created before the most recent removal of `use project ci` are +monitor cadences. Runs created before the most recent removal of `use project ci` are ignored, because they ran legitimately while the label was on. So are runs created before the most recent reopen of the pull request by someone other than its author, which is how a maintainer brings back a pull request closed for tampering; reopening one whose gate is still @@ -877,6 +877,22 @@ are no job statuses either, and GitHub's error is only on the run page. It is ha but `Fork CI / Tests (AMD)` says "Fork CI run could not start - check your fork's Actions settings", and the state comment explains what to check instead of asking to fix failures. +For a failed run that started, the state comment lists the jobs that did not pass, each linking +to the job in the fork, at most 10 of them, failed jobs before timed-out and cancelled ones; the +job statuses list them all. The jobs come from +`GET /repos/<fork>/actions/runs/<id>/jobs?filter=latest`, reused from the job statuses when they +read them in the same run. A finished run's jobs do not change, so the marker records the run and +attempt the list belongs to (`failed_run`), and later runs do not read the jobs again while the +comment would otherwise stay the same: a failed pull request costs one jobs read per fork run, not +one per reconciler run. Job names come from the contributor's workflow, so each is shown in a code +span with backticks and line breaks replaced. When the jobs cannot be read and the state comment +already shows this head commit as `failed`, it is left as it is until the next run, rather than +dropping the list; otherwise the `failed` state is posted without the list, so the comment does +not keep describing a running or earlier run, and the next run tries the read again. A read that +failed after its retries is not repeated in +the same run, whether the job statuses or the state comment made it, so persistent server errors +cost each failed pull request one sequence of retries a run, not two. + ### Commit status The reconciler posts a commit status with context `Fork CI / Tests (AMD)` on the pull @@ -914,7 +930,7 @@ suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=lat - Only finished jobs are reported: a successful one as `success`, and a failed, timed-out, cancelled or unstartable one as `failure`. Queued and running jobs are not, so each job costs one write instead of two; `Fork CI / Tests (AMD)` shows the run as running, linking it, and with a - 15-minute cadence a per-job "running" status was behind the fork's own run page anyway. A job of + 5-minute cadence a per-job "running" status was behind the fork's own run page anyway. A job of a re-run in the fork keeps the result of the earlier attempt until it finishes. Skipped and neutral jobs, and the jobs every run has (`Fork CI gate`, `Check if the run is needed`, `Platform: AMD` and `CI passed`), are not reported. @@ -922,12 +938,12 @@ suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=lat (`GET /repos/apache/airflow/commits/<sha>/status`), and only differences are posted, failures first. GraphQL has no mutation for commit statuses (`createCheckRun` exists, but see above for why check runs do not fit), so each one is a REST call, and they count towards GitHub's limits on - content-creating requests: a reconciler run posts at most `MAX_JOB_STATUSES_PER_RUN` (40) of + content-creating requests: a reconciler run posts at most `MAX_JOB_STATUSES_PER_RUN` (13) of them, each write charged as it is made. What does not fit is posted by the next run, and once the budget is spent no further pull request's jobs or statuses are read in that run. One fork run of the full matrix (about 130 jobs) needs more than a run's budget on its own, which the next run catches up on, so deferring alone is not a problem: the run log lists the deferred pull - requests, and only when three runs in a row deferred, about 45 minutes of the budget not + requests, and only when nine runs in a row deferred, about 45 minutes of the budget not keeping up, is a note added to the Slack message. The count is kept in the record the monitor carries between runs (`--reported-file`); without a readable record the note is added at once. Its wording is fixed, so it is posted once while deferrals persist. A pull request whose fork run @@ -947,7 +963,7 @@ suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=lat classified, so for example a `CI passed` that no longer holds is still revoked. Reading a fork run's jobs fails now and then on GitHub server errors (HTTP 502, a reset HTTP/2 stream) that the next run gets past, so each failure is only logged, and the pull request is listed with the - unknown items for Slack once its job statuses failed in three runs in a row. The per pull request + unknown items for Slack once its job statuses failed in nine runs in a row, about 45 minutes. The per pull request count is kept in the same record as that of deferred job statuses; without a readable record it starts again. The item's wording is fixed whatever the error, so it is posted once while the failure persists, and the run log has each run's error. @@ -962,7 +978,7 @@ records this for agents. Each run logs what is left of the token's REST and GraphQL quotas (`GET /rate_limit`, which does not count against them) and how many of its calls GitHub rate-limited; the client retries those with -backoff. Rate limiting in two runs in a row adds a note to the Slack message, with fixed wording so +backoff. Rate limiting in six runs in a row, about 30 minutes, adds a note to the Slack message, with fixed wording so it is posted once while it persists; the count is kept in the same record as that of deferred job statuses. @@ -991,14 +1007,23 @@ requests a minute and 500 an hour. Assuming about 35 reported jobs for an averag monitor runs. - **Gate enabled**, assuming about half of the pull request runs come from gated pull requests (about 5-6 pushes an hour): about 200 writes an hour without a cap, 40% of the content-creating - limit and more at peaks. The cap of 40 writes a run, four runs an hour, holds job statuses to at - most 160 an hour: 32% of the content-creating limit, leaving the rest to the reconciler's other + limit and more at peaks. The cap of 13 writes a run, twelve runs an hour, holds job statuses to at + most 156 an hour: 31% of the content-creating limit, leaving the rest to the reconciler's other writes (labels, comments, the aggregate statuses) and to the other workflows that share it. Each - run posts at most 40 statuses at about one a second, well under the 80-a-minute limit. Job + run posts at most 13 statuses at about one a second, well under the 80-a-minute limit. Job statuses then lag behind at busy times, the aggregate status does not. The `CI run started` - statuses of pull requests whose CI runs in `apache/airflow` come out of the same 40 writes a run, - first and at most 20 of them (see "The `CI run started` check"), so they do not raise the total; - at about 11 pull request runs an hour they take about 3 writes a run. + statuses of pull requests whose CI runs in `apache/airflow` come out of the same 13 writes a run, + first and at most 6 of them (see "The `CI run started` check"), so they do not raise the total; + at about 11 pull request runs an hour they take about 1 write a run. + +The per-run caps and the run counts before a Slack note are derived from `MONITOR_INTERVAL` in +`fork_ci_reconcile.py`, which must match the monitor's cron (a test checks it), so changing the +cadence keeps the hourly totals above. The cadence was lowered from 15 to 5 minutes because fork CI +reaches a pull request only when the monitor runs: a fork run's start, failure or success showed up +to 15 minutes late, plus GitHub's delay of scheduled workflows. Unchanged pull requests cost no +writes, so the faster cadence adds reads, about 45 a run, well within the token's quotas, and +runner time: each run takes about two minutes on a GitHub-hosted runner, mostly installing +Breeze, so the monitor uses about 24 runner minutes an hour instead of 8. These are estimates: the share of gated runs, the jobs per push and the pushes per pull request are assumptions, not measurements. Watch the monitor's call counts in its run log after enabling @@ -1048,7 +1073,7 @@ run. job has not succeeded, a `success` is overwritten with `pending`; once it has, the monitor posts `success` ("Project CI passed for this commit"). Adding the label does not start project CI (see "Override labels"): the pull request stays blocked until a maintainer starts it and it passes, - and then until the next monitor run posts the `success`, which can take about 15 minutes. + and then until the next monitor run posts the `success`, which can take about 5 minutes. Removing the label returns the pull request to the normal flow above: removing it updates the pull request, so discovery (open pull requests updated within the last 24 hours) labels it `fork ci required` again and the next run reports the fork result, replacing the waiting @@ -1069,7 +1094,7 @@ run. - The protection is eventual, not atomic. Nothing reacts to the label when it is added (`pull_request_target` is not used, see "Goals and non-goals"), so a fork `success` posted before the label stays until the next monitor run that completes: normally within one cadence - (15 minutes), longer when GitHub delays or drops a scheduled run or a run fails, and one more + (5 minutes), longer when GitHub delays or drops a scheduled run or a run fails, and one more cycle when the label arrives after a run has taken its snapshot of the pull request. The pull request can merge on the fork result in that window. A maintainer who adds the label to a pull request whose fork CI is green waits for the `pending` before merging or enabling auto-merge, @@ -1087,7 +1112,7 @@ run. ### The `CI run started` check -Fork CI reports to the pull request only when the scheduled monitor runs, about every 15 minutes +Fork CI reports to the pull request only when the scheduled monitor runs, about every 5 minutes and later when GitHub delays scheduled workflows, and until then a gated pull request shows only skipped checks. The gate cannot say more itself: it runs on the contributor's `pull_request` event with a read-only token. Instead both CI rulesets in `.asf.yaml` also require a @@ -1113,8 +1138,8 @@ with state `success` linking the run: Folding the check into the scheduled monitor costs no runner per push, at the price of a pull request waiting for the next monitor run before it can merge, which a CI run takes longer than anyway. Each new head commit needs one write, taken first out of the per-run budget of job -statuses (see "Job statuses"), since merging waits for it; at most 20 such writes per run, so job -statuses keep at least half of the budget, and after the check became required the open pull +statuses (see "Job statuses"), since merging waits for it; at most `MAX_CI_RUN_STARTED_PER_RUN` (6) such +writes per run, so job statuses keep at least half of the budget, and after the check became required the open pull requests are caught up a batch at a time. A release branch has no monitor of its own, but `main`'s monitor covers pull requests to every base branch. @@ -1198,14 +1223,15 @@ change. Job statuses add a jobs call and a status call per pull request with a f progress or a completed run not yet reported, and up to two status mutations per job, at most `MAX_JOB_STATUSES_PER_RUN` per run; see "Write budget". Pull requests that pass 1 labelled and the search does not list yet add one GraphQL call per 25. Well inside -the `GITHUB_TOKEN` limits at a 15-minute cadence. The reconciler logs its call counts so the +the `GITHUB_TOKEN` limits at a 5-minute cadence. The reconciler logs its call counts so the budget stays observable. ## Monitor workflow `.github/workflows/fork-ci-monitor.yml`: -- `on: schedule: cron '*/15 * * * *'` and `workflow_dispatch` with a `dry-run` input. +- `on: schedule: cron '*/5 * * * *'` (`MONITOR_INTERVAL`) and `workflow_dispatch` with a `dry-run` + input. - Two jobs, each with its own `permissions` block and a justification comment; the workflow grants nothing by default. - The `reconcile` job: `actions: write, contents: read, pull-requests: write, statuses: write`. @@ -1377,10 +1403,11 @@ as a failed check rather than a crash. ## Known limitations -- Drafting, the setup comment and the commit status arrive within one cadence of the event - they reflect, not instantly. The cadence can be lowered, but the per-run write budgets are - sized for four runs an hour against the `GITHUB_TOKEN` limits shared by every workflow of the - repository, so they must be scaled down with it. +- Drafting, the setup comment and the commit status arrive within one cadence (5 minutes) of the + event they reflect, plus GitHub's delay of scheduled workflows, not instantly. The fork runs in + another repository, so no event in `apache/airflow` reports its start or end; a `workflow_run` + trigger on project CI's gate run would only cover a push, at the cost of a monitor run per + pull request push. - The workflows API lists workflows from the fork's default branch. A fork whose `main` predates `ci-amd.yml` reports it missing; the setup comment covers this with `gh repo sync`. diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py index 314609f52ac..845769d025f 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py @@ -30,8 +30,10 @@ from __future__ import annotations import json import re +from collections.abc import Sequence from dataclasses import dataclass from datetime import datetime +from typing import NamedTuple STATUS_CONTEXT = "Fork CI / Tests (AMD)" # Each job of the fork run is reported under this prefix. Job names come from the contributor's @@ -94,6 +96,8 @@ class Marker: behind: int | None = None # The completed fork run (``<run id>:<attempt>:<conclusion>``) whose jobs are all reported. jobs: str | None = None + # The fork run (``<run id>:<attempt>``) whose failed jobs the comment lists. + failed_run: str | None = None # Set only by a reconciler that ran the gate protection check for this head commit, so a # marker written before that check existed, or while it did not apply, never counts as one. gate_checked: bool = False @@ -105,6 +109,8 @@ def render_marker(marker: Marker) -> str: data["behind"] = marker.behind if marker.jobs is not None: data["jobs"] = marker.jobs + if marker.failed_run is not None: + data["failed_run"] = marker.failed_run if marker.gate_checked: data["gate_checked"] = True return f"{MARKER_PREFIX}{json.dumps(data, sort_keys=True)} -->" @@ -123,6 +129,7 @@ def parse_marker(body: str) -> Marker | None: since=datetime.fromisoformat(data["since"]), behind=data.get("behind"), jobs=data.get("jobs"), + failed_run=data.get("failed_run"), gate_checked=data.get("gate_checked") is True, ) except (json.JSONDecodeError, KeyError, TypeError, ValueError) as e: @@ -203,6 +210,44 @@ STARTUP_FAILURE_TEXT = ( ) +class FailedJob(NamedTuple): + """A job of a failed fork run that did not pass, as listed in the state comment.""" + + name: str + url: str + conclusion: str + + +# A full-matrix run can fail in dozens of jobs; the job statuses list them all. +MAX_LISTED_FAILED_JOBS = 10 +FAILED_JOB_LIST_HEADING = "Jobs that did not pass:" +_FAILED_JOB_OUTCOMES = { + "failure": "failed", + "timed_out": "timed out", + "cancelled": "cancelled", + "startup_failure": "could not start", +} + + +def build_failed_job_list(failed_jobs: Sequence[FailedJob]) -> str: + listed = [ + # Job names come from the contributor's workflow; a code span keeps them from posting + # Markdown, including mentions, as the bot. + f"- [`{_sanitize_path(job.name[:JOB_NAME_MAX_LENGTH])}`]({job.url}): " + f"{_FAILED_JOB_OUTCOMES.get(job.conclusion, 'failed')}" + for job in failed_jobs[:MAX_LISTED_FAILED_JOBS] + ] + if len(failed_jobs) > MAX_LISTED_FAILED_JOBS: + listed.append(f"- and {len(failed_jobs) - MAX_LISTED_FAILED_JOBS} more, see the run") + # One paragraph of the comment, so that ``strip_failed_job_list`` can find all of it. + return FAILED_JOB_LIST_HEADING + "\n" + "\n".join(listed) + + +def strip_failed_job_list(body: str) -> str: + """The state comment without its list of failed jobs.""" + return "\n\n".join(part for part in body.split("\n\n") if not part.startswith(FAILED_JOB_LIST_HEADING)) + + def behind_warning(behind: int, *, base_ref: str) -> str: return ( f"**This branch is {behind} commits behind `{base_ref}`.** Fork CI tests the branch as it is, " @@ -219,8 +264,12 @@ def state_comment( base_ref: str = "main", author_draft: bool = False, startup_failure: bool = False, + failed_jobs: Sequence[FailedJob] = (), ) -> str: - """:param startup_failure: the fork run of a failed state did not start, see ``STARTUP_FAILURE_TEXT``.""" + """ + :param startup_failure: the fork run of a failed state did not start, see ``STARTUP_FAILURE_TEXT``. + :param failed_jobs: the jobs of a failed state's fork run that did not pass. + """ text = _AUTHOR_DRAFT_TEXTS.get(marker.state) if author_draft else None if startup_failure: text = STARTUP_FAILURE_TEXT @@ -233,6 +282,8 @@ def state_comment( parts.append(behind_warning(marker.behind, base_ref=base_ref)) if run_url: parts.append(f"Latest run: {run_url}") + if marker.state == "failed" and failed_jobs: + parts.append(build_failed_job_list(failed_jobs)) if marker.state == "setup_required": parts.append(_setup_instructions(fork)) parts.append(f"How this works: [CI in your fork]({CONTRIBUTING_URL}).") diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py index da56f0c7310..f84e8ff2db2 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py @@ -71,6 +71,7 @@ from airflow_breeze.utils.fork_ci_messages import ( STALE_JOB_STATUS, STATUS_CONTEXT, STATUS_STATES, + FailedJob, Marker, fork_actions_url, gate_modified_close_comment, @@ -81,6 +82,7 @@ from airflow_breeze.utils.fork_ci_messages import ( released_comment, state_comment, status_description, + strip_failed_job_list, tamper_close_comment, ungated_comment, workflow_disabled_comment, @@ -129,8 +131,19 @@ BOT_LOGINS = frozenset({"github-actions", "github-actions[bot]"}) DISCOVERY_WINDOW = timedelta(hours=24) PASS_TWO_PAGE_SIZE = 25 FORK_BATCH_SIZE = 50 +# The cron interval of fork-ci-monitor.yml. Fork CI reaches a pull request only when the monitor +# runs, so this is how late it can be; the write budgets and alert thresholds below are per run and +# derived from it, to keep their hourly totals when it changes. +MONITOR_INTERVAL = timedelta(minutes=5) +RUNS_PER_HOUR = timedelta(hours=1) // MONITOR_INTERVAL + + +def count_runs_in(duration: timedelta) -> int: + return max(duration // MONITOR_INTERVAL, 1) + + # Project CI runs created this long ago or later are checked for gate tampering. Several hours -# overlap the 15-minute cadence, so a tampered run is seen even after the pull request was +# overlap the monitor's cadence, so a tampered run is seen even after the pull request was # pushed again with the gate restored. TAMPER_WINDOW = timedelta(hours=6) # A ci-amd.yml run in which only the gate executed finishes in about a minute; only longer runs @@ -171,30 +184,30 @@ UNREPORTED_JOB_NAMES = GATED_RUN_JOB_NAMES | {GATE_JOB_NAME, "Platform: AMD"} RUNNING_JOB_STATUSES = frozenset({"queued", "waiting", "pending", "requested", "in_progress"}) JOB_STATE_RANKS = {"failure": 0, "pending": 1, "success": 2} # Commit statuses count towards GitHub's limit of 500 content-creating requests per hour, shared by -# every workflow of the repository. At four runs an hour this keeps job statuses to at most about -# half of it; the rest follow on the next run, failures first. See "Job statuses" in the design doc. -MAX_JOB_STATUSES_PER_RUN = 40 +# every workflow of the repository. This keeps job statuses to about a third of it; the rest follow +# on the next run, failures first. See "Job statuses" in the design doc. +MAX_JOB_STATUSES_PER_RUN = 160 // RUNS_PER_HOUR # `CI run started` statuses of pull requests whose CI runs in apache/airflow come out of that budget # first: one write per new head commit, which merging waits for, while job statuses are informational. # At most this many, so job statuses keep the rest; the first runs after the check became required # catch up on the open pull requests a batch at a time, most recently updated first. -MAX_CI_RUN_STARTED_PER_RUN = 20 +MAX_CI_RUN_STARTED_PER_RUN = 80 // RUNS_PER_HOUR # `CI passed` successes for pull requests carrying `use project ci`, written once per new project CI # pass, so far fewer than this per run. Revoking a success is never limited. -MAX_PROJECT_CI_PASSED_PER_RUN = 5 +MAX_PROJECT_CI_PASSED_PER_RUN = max(20 // RUNS_PER_HOUR, 1) # The GitHub Actions app, whose check suites are the workflow runs of a commit. GITHUB_ACTIONS_APP_ID = 15368 # A single large fork run can need more writes than one run's budget, which the next run catches up # on, so deferring is only reported when it lasts this many consecutive runs: then the budget does # not keep up with the pull requests. -DEFERRED_RUNS_BEFORE_ALERT = 3 +DEFERRED_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=45)) # GitHub answering with rate limits in this many consecutive runs is reported: a single burst is # retried, while a persisting one means the repository's shared limits are close to exhausted. -RATE_LIMITED_RUNS_BEFORE_ALERT = 2 +RATE_LIMITED_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=30)) # Reading a fork run's jobs fails now and then on GitHub server errors (HTTP 502, a reset HTTP/2 # stream) that the next run gets past, so a pull request whose job statuses failed is only reported # when it failed in this many consecutive runs. -JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT = 3 +JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=45)) # Fixed text, so the item is posted to Slack once while the failure persists, whatever the error; # the run log has each run's error. JOB_STATUSES_FAILED_REASON = ( @@ -625,6 +638,7 @@ def classify( jobs_reported: str | None = None, head_shared_across_bases: bool = False, find_gate_change: Callable[[], str | None] = lambda: None, + find_failed_jobs: Callable[[], list[FailedJob] | None] = lambda: [], ) -> Plan: """ Decide the mutations for one gated pull request. @@ -646,6 +660,9 @@ def classify( :param find_gate_change: describes how the pull request changes the fork CI gate, or returns None; only called for pull requests that the gate tampering check applies to, once per head commit. + :param find_failed_jobs: the jobs of the fork run that did not pass, None when they cannot be + read; only called for a failed run that started. On None the state comment is left for the + next run, so that a failed read does not drop the list from it. """ plan = Plan() labels = set(pr.labels) @@ -846,25 +863,47 @@ def classify( _set_ci_passed(plan, pr, passed=state == ForkState.GREEN, target_url=target_url) _set_ci_run_started(plan, pr, fork) green = state in (ForkState.GREEN, ForkState.GREEN_UNVERIFIED) - _set_marker_comment( - plan, - pr, - state_comment( - Marker( - state.value, - pr.head_sha, - since, - behind=behind if green else None, - jobs=jobs_reported, - gate_checked=gate_checked, - ), + lists_failed_jobs = state == ForkState.FAILED and not startup_failure and fork.run_id is not None + new_marker = Marker( + state.value, + pr.head_sha, + since, + behind=behind if green else None, + jobs=jobs_reported, + failed_run=f"{fork.run_id}:{fork.run_attempt}" if lists_failed_jobs else None, + gate_checked=gate_checked, + ) + + def render(marker: Marker, failed_jobs: list[FailedJob]) -> str: + return state_comment( + marker, fork=pr.head_repo, run_url=fork.run_url, base_ref=pr.base_ref, author_draft=_is_author_draft(pr), startup_failure=startup_failure, - ), - ) + failed_jobs=failed_jobs, + ) + + failed_jobs: list[FailedJob] | None = [] + if lists_failed_jobs: + # A finished run's jobs do not change, so the list already in the comment still holds. + if ( + marker == new_marker + and pr.marker_comment_body is not None + and strip_failed_job_list(pr.marker_comment_body) == render(new_marker, []) + ): + return plan + failed_jobs = find_failed_jobs() + if failed_jobs is None: + # Only a comment that already shows this head as failed is worth keeping; anything else + # describes a state the pull request is no longer in. + if marker and marker.state == ForkState.FAILED.value and marker.sha == pr.head_sha: + return plan + failed_jobs = [] + # Without a list, the next run reads the jobs again. + new_marker = replace(new_marker, failed_run=None) + _set_marker_comment(plan, pr, render(new_marker, failed_jobs)) return plan @@ -1748,6 +1787,53 @@ def build_job_statuses(jobs: Iterable[dict[str, Any]]) -> dict[str, tuple[str, s return statuses +def find_failed_jobs(jobs: Iterable[dict[str, Any]]) -> list[FailedJob]: + """The finished jobs of a fork run that did not pass, by name, leaving out the jobs every run has.""" + failed: dict[str, FailedJob] = {} + for job in jobs: + if job["name"] in UNREPORTED_JOB_NAMES or job.get("status") in RUNNING_JOB_STATUSES: + continue + conclusion = job.get("conclusion") or "" + if JOB_STATUSES.get(conclusion, ("",))[0] != "failure": + continue + listed = failed.get(job["name"]) + # Of jobs sharing a name, one that failed is listed rather than one cancelled or timed out. + if listed is None or (listed.conclusion != "failure" and conclusion == "failure"): + failed[job["name"]] = FailedJob(job["name"], job["html_url"], conclusion) + # Failures first: the cap on the listed jobs must not hide them behind cancelled or timed-out ones. + return sorted(failed.values(), key=lambda job: (job.conclusion != "failure", job.name)) + + +# The jobs of fork runs read in this reconciler run, by run id. None records a read that failed after +# its retries, so the next reader gives up at once instead of repeating them; the next run reads again. +ReadJobs = dict[int, list[dict[str, Any]] | None] + + +def read_failed_jobs( + client: GitHubClient, + pr: PullRequestSnapshot, + fork: ForkObservation, + read_jobs: ReadJobs, +) -> list[FailedJob] | None: + """ + ``find_failed_jobs`` of the pull request's fork run; None when its jobs cannot be read. + + :param read_jobs: the jobs already read in this reconciler run; a read made here is recorded too. + """ + if fork.run_id is None or not pr.head_repo: + return [] + if fork.run_id in read_jobs: + jobs = read_jobs[fork.run_id] + else: + try: + jobs = client.get_run_jobs(pr.head_repo, fork.run_id) + except GitHubError as e: + console_print(f"[warning]#{pr.number}: could not read the jobs of {fork.run_url}: {e}[/]") + jobs = None + read_jobs[fork.run_id] = jobs + return None if jobs is None else find_failed_jobs(jobs) + + class JobStatusReporter: """Reports each job of a pull request's fork run as a commit status, within a per-run budget.""" @@ -1758,12 +1844,17 @@ class JobStatusReporter: repo: str, budget: int = MAX_JOB_STATUSES_PER_RUN, now: datetime | None = None, + read_jobs: ReadJobs | None = None, ): - """:param now: when given, deferring the statuses of a run that finished long ago is reported.""" + """ + :param now: when given, deferring the statuses of a run that finished long ago is reported. + :param read_jobs: where the jobs read, or a failed read, are kept, for the state comment to reuse. + """ self._client = client self._repo = repo self._now = now self.remaining = budget + self.read_jobs = {} if read_jobs is None else read_jobs self.deferred: list[int] = [] # (pull request, attention item) for runs whose statuses are deferred long after they finished. self.delayed: list[tuple[int, str]] = [] @@ -1808,7 +1899,12 @@ class JobStatusReporter: if self.remaining == 0: self._defer(pr, fork) return None - jobs = self._client.get_run_jobs(pr.head_repo, fork.run_id) + try: + jobs = self._client.get_run_jobs(pr.head_repo, fork.run_id) + except Exception: + self.read_jobs[fork.run_id] = None + raise + self.read_jobs[fork.run_id] = jobs # By the case-folded context, as GitHub compares them. current = { status["context"].casefold(): ( @@ -2682,11 +2778,13 @@ def run_reconcile( ci_passed_notes.append( f"listing the pull requests with `{LABEL_USE_PROJECT_CI}` failed: {type(e).__name__}" ) + read_jobs: ReadJobs = {} job_reporter = JobStatusReporter( client, repo=repo, budget=max(MAX_JOB_STATUSES_PER_RUN - ci_run_started_writes - ci_passed_reserved, 0), now=now, + read_jobs=read_jobs, ) previous_job_status_failure_runs = read_job_status_failure_runs(reported_file) @@ -2736,6 +2834,7 @@ def run_reconcile( jobs_reported=jobs_reported, head_shared_across_bases=pr.head_sha in shared_heads, find_gate_change=functools.partial(gate_change_finder.find, pr), + find_failed_jobs=functools.partial(read_failed_jobs, client, pr, fork, read_jobs), ) # A revocation to `pending` always goes through: it is idempotent and must not depend on # `report_project_ci_passed` succeeding later in this run. diff --git a/dev/breeze/tests/test_fork_ci_messages.py b/dev/breeze/tests/test_fork_ci_messages.py index 3b5b327593b..37b874b56d8 100644 --- a/dev/breeze/tests/test_fork_ci_messages.py +++ b/dev/breeze/tests/test_fork_ci_messages.py @@ -36,6 +36,8 @@ from airflow_breeze.utils.fork_ci_messages import ( JOB_NAME_MAX_LENGTH, JOB_STATUS_CONTEXT_PREFIX, MAX_LISTED_CI_FILES, + MAX_LISTED_FAILED_JOBS, + FailedJob, Marker, ci_change_approval_comment, ci_modification_note_comment, @@ -47,6 +49,7 @@ from airflow_breeze.utils.fork_ci_messages import ( released_comment, render_marker, state_comment, + strip_failed_job_list, tamper_close_comment, ungated_comment, workflow_disabled_comment, @@ -61,6 +64,7 @@ MARKER = Marker(state="awaiting_push", sha="abc123", since=datetime(2026, 10, 15 pytest.param(MARKER, id="without-behind"), pytest.param(replace(MARKER, behind=12), id="behind"), pytest.param(replace(MARKER, jobs="123:SUCCESS"), id="jobs"), + pytest.param(replace(MARKER, failed_run="123:2"), id="failed-run"), pytest.param(replace(MARKER, gate_checked=True), id="gate-checked"), ], ) @@ -121,6 +125,44 @@ def test_state_comment_links_the_contributor_documentation(startup_failure): assert text.endswith(FOOTER) +def test_failed_job_list_is_safe_markdown_and_capped(): + jobs = [ + FailedJob("Tests `evil`\n\n@someone", "https://job/0", "timed_out"), + *( + FailedJob(f"Job {i}", f"https://job/{i + 1}", "failure") + for i in range(MAX_LISTED_FAILED_JOBS + 1) + ), + ] + + text = state_comment( + Marker("failed", "a", MARKER.since), fork="u/airflow", run_url=None, failed_jobs=jobs + ) + + assert "- [`Tests ?evil???@someone`](https://job/0): timed out" in text + assert "\n@someone" not in text + assert "- [`Job 0`](https://job/1): failed" in text + assert "- and 2 more, see the run" in text + + +def test_failed_jobs_are_listed_only_for_a_failed_state(): + jobs = [FailedJob("Static checks", "https://job/1", "failure")] + + text = state_comment( + Marker("running", "a", MARKER.since), fork="u/airflow", run_url=None, failed_jobs=jobs + ) + + assert "Static checks" not in text + + +def test_strip_failed_job_list_leaves_the_comment_without_the_list(): + marker = Marker("failed", "a", MARKER.since) + jobs = [FailedJob(f"Job {i}", f"https://job/{i}", "failure") for i in range(MAX_LISTED_FAILED_JOBS + 1)] + + listed = state_comment(marker, fork="u/airflow", run_url="https://run", failed_jobs=jobs) + + assert strip_failed_job_list(listed) == state_comment(marker, fork="u/airflow", run_url="https://run") + + def test_parse_marker_absent(): assert parse_marker("just a comment") is None diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py b/dev/breeze/tests/test_fork_ci_reconcile.py index b9714bb70c0..519294739f3 100644 --- a/dev/breeze/tests/test_fork_ci_reconcile.py +++ b/dev/breeze/tests/test_fork_ci_reconcile.py @@ -24,6 +24,7 @@ from unittest import mock from urllib.parse import quote import pytest +import yaml from click.testing import CliRunner from airflow_breeze.commands.ci_commands import fork_ci_reconcile @@ -40,14 +41,22 @@ from airflow_breeze.utils.fork_ci_messages import ( JOB_STATUS_CONTEXT_PREFIX, STALE_JOB_STATUS, STATUS_CONTEXT, + FailedJob, Marker, parse_marker, render_marker, ) from airflow_breeze.utils.fork_ci_reconcile import ( + DEFERRED_RUNS_BEFORE_ALERT, + JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT, JOB_STATUSES_DEFERRED_NOTE, JOB_STATUSES_FAILED_REASON, + MAX_CI_RUN_STARTED_PER_RUN, + MAX_JOB_STATUSES_PER_RUN, + MAX_PROJECT_CI_PASSED_PER_RUN, + MONITOR_INTERVAL, RATE_LIMITED_NOTE, + RATE_LIMITED_RUNS_BEFORE_ALERT, BaseGateConfigs, BypassFinder, DraftChange, @@ -72,6 +81,7 @@ from airflow_breeze.utils.fork_ci_reconcile import ( fetch_open_pull_requests, fetch_workflow_state, find_ci_run_since, + find_failed_jobs, find_heads_shared_across_bases, find_project_ci_label_heads, find_project_ci_passed_url, @@ -87,6 +97,7 @@ from airflow_breeze.utils.fork_ci_reconcile import ( parse_pull_request, pull_request_changes_ci_files, read_ci_run_at, + read_failed_jobs, read_head_status, read_job_status_failure_runs, read_project_ci_passed_url, @@ -2929,6 +2940,9 @@ def test_run_reconcile_reports_job_statuses_before_classifying( (pr, fork), _ = mock_reporter.return_value.report.call_args assert pr.number == 1 assert mock_classify.call_args.kwargs["jobs_reported"] == "7:SUCCESS" + find_failed = mock_classify.call_args.kwargs["find_failed_jobs"] + assert find_failed.func is read_failed_jobs + assert find_failed.args == (client, pr, fork, mock_reporter.call_args.kwargs["read_jobs"]) @pytest.mark.parametrize("status", ["queued", "in_progress", "waiting"]) @@ -3035,23 +3049,23 @@ def test_run_reconcile_reports_failing_job_statuses_once_when_failing_persists( reported_file = tmp_path / "reported.json" reported_file.write_text("{}") alerts = [] + failures = [GitHubError("HTTP 502"), GitHubError("stream ID 1; CANCEL")] errors = [ GitHubError("HTTP 502"), GitHubError("HTTP 502"), None, - GitHubError("HTTP 502"), - GitHubError("stream ID 1; CANCEL"), - GitHubError("HTTP 502"), - GitHubError("stream ID 1; CANCEL"), + *(failures[run % 2] for run in range(JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT + 1)), ] for error in errors: mock_reporter.return_value.report.side_effect = error mock_reporter.return_value.report.return_value = None alerts.append(_reconcile_with_reporter(client, tmp_path, reported_file=reported_file)) - assert alerts == [False, False, False, False, False, True, False] + assert alerts == [*[False] * (JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT + 2), True, False] assert JOB_STATUSES_FAILED_REASON in (tmp_path / "s.json").read_text() - assert json.loads(reported_file.read_text())["job_status_failure_runs"] == {"1": 4} + assert json.loads(reported_file.read_text())["job_status_failure_runs"] == { + "1": JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT + 1 + } @pytest.mark.parametrize( @@ -3126,14 +3140,14 @@ def test_run_reconcile_reports_deferred_job_statuses_only_when_deferring_persist reported_file = tmp_path / "reported.json" reported_file.write_text("{}") alerts = [] - for deferred in ([1], [1], [], [1], [1], [1]): + for deferred in ([1], [1], [], *[[1]] * DEFERRED_RUNS_BEFORE_ALERT): mock_reporter.return_value.deferred = deferred mock_reporter.return_value.delayed = [] alerts.append(_reconcile_with_reporter(client, tmp_path, reported_file=reported_file)) - assert alerts == [False, False, False, False, False, True] + assert alerts == [*[False] * (DEFERRED_RUNS_BEFORE_ALERT + 2), True] assert JOB_STATUSES_DEFERRED_NOTE in (tmp_path / "s.json").read_text() - assert json.loads(reported_file.read_text())["deferred_runs"] == 3 + assert json.loads(reported_file.read_text())["deferred_runs"] == DEFERRED_RUNS_BEFORE_ALERT def test_build_job_statuses_treats_contexts_case_insensitively(): @@ -3745,7 +3759,10 @@ def test_run_reconcile_retries_a_pull_request_whose_head_moved(client, tmp_path) @pytest.mark.parametrize( ("fails", "job_status_budget"), - [pytest.param(False, 38, id="reported"), pytest.param(True, 20, id="fails")], + [ + pytest.param(False, MAX_JOB_STATUSES_PER_RUN - 2, id="reported"), + pytest.param(True, MAX_JOB_STATUSES_PER_RUN - MAX_CI_RUN_STARTED_PER_RUN, id="fails"), + ], ) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.report_project_ci_runs_started", autospec=True) @@ -3782,7 +3799,7 @@ def test_run_reconcile_marks_older_pilot_pull_requests_and_reports_project_ci_ru assert mock_classify.call_args.args[0].ci_run_at == datetime(2026, 11, 2, tzinfo=UTC) mock_report.assert_called_once_with(client, repo=REPO, skip={1}) - mock_reporter.assert_called_once_with(client, repo=REPO, budget=job_status_budget, now=NOW) + mock_reporter.assert_called_once_with(client, repo=REPO, budget=job_status_budget, now=NOW, read_jobs={}) assert has_alerts is fails if fails: assert "reporting `CI run started` for project CI failed: GitHubError" in output.read_text() @@ -3863,13 +3880,13 @@ def test_run_reconcile_reports_rate_limiting_only_when_it_persists( reported_file = tmp_path / "reported.json" reported_file.write_text("{}") alerts = [] - for rate_limited in (1, 0, 3, 2): + for rate_limited in (1, 0, *[3] * RATE_LIMITED_RUNS_BEFORE_ALERT): client.rate_limited = rate_limited alerts.append(_reconcile_with_reporter(client, tmp_path, reported_file=reported_file)) - assert alerts == [False, False, False, True] + assert alerts == [*[False] * (RATE_LIMITED_RUNS_BEFORE_ALERT + 1), True] assert RATE_LIMITED_NOTE in (tmp_path / "s.json").read_text() - assert json.loads(reported_file.read_text())["rate_limited_runs"] == 2 + assert json.loads(reported_file.read_text())["rate_limited_runs"] == RATE_LIMITED_RUNS_BEFORE_ALERT @pytest.mark.parametrize( @@ -4428,7 +4445,7 @@ def test_run_reconcile_reports_project_ci_passed_after_the_loop( names = [c[0] for c in order.mock_calls] assert names.index("report") > names.index("set_status") - assert mock_report_project_ci_passed.call_args.kwargs["budget"] == 5 + assert mock_report_project_ci_passed.call_args.kwargs["budget"] == MAX_PROJECT_CI_PASSED_PER_RUN def test_run_reconcile_reports_project_ci_passed_failure(mock_report_project_ci_passed, client, tmp_path): @@ -4462,3 +4479,176 @@ def test_report_project_ci_passed_counts_only_the_successes_that_were_written(cl client.set_status.side_effect = [GitHubError("HTTP 502"), None, None] assert report_project_ci_passed(client, repo=REPO, attention=[], budget=1) == [2] + + +def test_find_failed_jobs_lists_finished_jobs_that_did_not_pass(): + jobs = [ + _fork_job("Static checks"), + _fork_job("Postgres tests", conclusion="failure"), + _fork_job("Helm tests", status="in_progress", conclusion=None), + _fork_job("Docs", conclusion="timed_out"), + _fork_job("Java SDK tests", conclusion="skipped"), + _fork_job("Platform: AMD", conclusion="failure"), + _fork_job("Postgres tests", conclusion="cancelled", url="https://job/second"), + _fork_job("MySQL tests", conclusion="cancelled", url="https://job/cancelled"), + _fork_job("MySQL tests", conclusion="failure"), + ] + + assert find_failed_jobs(jobs) == [ + FailedJob("MySQL tests", "https://job/MySQL tests", "failure"), + FailedJob("Postgres tests", "https://job/Postgres tests", "failure"), + FailedJob("Docs", "https://job/Docs", "timed_out"), + ] + + [email protected]( + ("read_jobs", "side_effect", "expected", "reads"), + [ + pytest.param({1: [_fork_job("A", conclusion="failure")]}, None, ["A"], False, id="read-this-run"), + pytest.param({}, [[_fork_job("B", conclusion="failure")]], ["B"], True, id="not-read-yet"), + pytest.param({}, GitHubError("HTTP 502"), None, True, id="unreadable"), + pytest.param({1: None}, None, None, False, id="read-failed-this-run"), + ], +) +def test_read_failed_jobs(client, read_jobs, side_effect, expected, reads): + client.get_run_jobs.side_effect = side_effect + + failed = read_failed_jobs(client, _pr(), COMPLETED_RUN, read_jobs) + + assert (None if failed is None else [job.name for job in failed]) == expected + assert client.get_run_jobs.called is reads + assert 1 in read_jobs + + +def test_a_failed_jobs_read_is_not_repeated_for_the_state_comment(client): + client.rest_get.return_value = [] + client.get_run_jobs.side_effect = GitHubError("HTTP 502") + read_jobs: dict = {} + reporter = JobStatusReporter(client, repo=REPO, budget=10, read_jobs=read_jobs) + + with pytest.raises(GitHubError): + reporter.report(_pr(), COMPLETED_RUN) + + assert read_failed_jobs(client, _pr(), COMPLETED_RUN, read_jobs) is None + client.get_run_jobs.assert_called_once_with("someone/airflow", 1) + + +def test_job_status_reporter_keeps_the_jobs_it_read(client): + jobs = [_fork_job("Static checks", conclusion="failure")] + reporter = JobStatusReporter(_reporter_client(client, jobs), repo=REPO, budget=10) + + reporter.report(_pr(), COMPLETED_RUN) + + assert reporter.read_jobs == {1: jobs} + + +FAILED_JOB = FailedJob("Static checks", "https://job/1", "failure") + + [email protected]( + ("fork", "failed_jobs", "called", "listed"), + [ + pytest.param(COMPLETED_RUN, [FAILED_JOB], True, True, id="failed"), + pytest.param( + dataclasses.replace(COMPLETED_RUN, suite_conclusion="STARTUP_FAILURE"), + [FAILED_JOB], + False, + False, + id="startup-failure", + ), + pytest.param(RUNNING_RUN, [FAILED_JOB], False, False, id="running"), + ], +) +def test_state_comment_lists_the_jobs_of_a_failed_run(fork, failed_jobs, called, listed): + find_failed = mock.Mock(return_value=failed_jobs) + + plan = classify( + _pr(), + fork, + config=CONFIG, + now=NOW, + repo=REPO, + find_bypass_run=lambda: None, + changes_ci_files=lambda: False, + find_failed_jobs=find_failed, + ) + + assert find_failed.called is called + assert ("- [`Static checks`](https://job/1): failed" in plan.marker_comment) is listed + + [email protected]( + ("marker", "kept"), + [ + pytest.param(_marker("failed"), True, id="failed-this-head"), + pytest.param(_marker("failed", sha="older"), False, id="failed-earlier-head"), + pytest.param(_marker("running"), False, id="running"), + pytest.param(None, False, id="no-comment"), + ], +) +def test_state_comment_is_left_for_the_next_run_only_when_it_shows_the_head_failed(marker, kept): + plan = classify( + _pr(is_draft=False, labels=("fork ci required", "ready for maintainer review"), marker=marker), + COMPLETED_RUN, + config=CONFIG, + now=NOW, + repo=REPO, + find_bypass_run=lambda: None, + changes_ci_files=lambda: False, + find_failed_jobs=lambda: None, + ) + + assert plan.state == ForkState.FAILED + assert plan.status == ("failure", RUN_URL, "Fork CI failed - see Details") + assert plan.remove_labels == ["ready for maintainer review"] + if kept: + assert plan.marker_comment is None + else: + assert parse_marker(plan.marker_comment).state == "failed" + assert parse_marker(plan.marker_comment).failed_run is None + assert "Jobs that did not pass" not in plan.marker_comment + + +def _classify_failed(pr, fork=COMPLETED_RUN, find_failed=None): + return classify( + pr, + fork, + config=CONFIG, + now=NOW, + repo=REPO, + find_bypass_run=lambda: None, + changes_ci_files=lambda: False, + find_failed_jobs=find_failed or (lambda: [FAILED_JOB]), + ) + + [email protected]( + ("fork", "marker_overrides", "reads"), + [ + pytest.param(COMPLETED_RUN, {}, False, id="unchanged"), + pytest.param(dataclasses.replace(COMPLETED_RUN, run_attempt=2), {}, True, id="re-run"), + pytest.param(COMPLETED_RUN, {"failed_run": None}, True, id="not-listed-yet"), + ], +) +def test_jobs_of_a_listed_failed_run_are_not_read_again(fork, marker_overrides, reads): + pr = _pr(is_draft=False, labels=("fork ci required",)) + listed = _classify_failed(pr).marker_comment + marker = dataclasses.replace(parse_marker(listed), **marker_overrides) + find_failed = mock.Mock(return_value=[FAILED_JOB]) + + plan = _classify_failed( + dataclasses.replace(pr, marker=marker, marker_comment_body=listed), fork, find_failed + ) + + assert parse_marker(listed).failed_run == "1:1" + assert find_failed.called is reads + if not reads: + assert plan.marker_comment is None + + +def test_monitor_interval_is_the_monitor_workflow_cadence(): + workflow = yaml.safe_load((AIRFLOW_ROOT_PATH / ".github/workflows/fork-ci-monitor.yml").read_text()) + + crons = [entry["cron"] for entry in workflow[True]["schedule"]] + + assert crons == [f"*/{MONITOR_INTERVAL // timedelta(minutes=1)} * * * *"]
