This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 433ef3ebd6f0312bfc71746b184ee5e9fe1fd6a6
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sat Nov 14 09:46:37 2026 +0100

    Report fork CI to pull requests sooner and with failing jobs (#76836)
    
    * List the jobs that did not pass in the fork CI state comment
    
    When the fork run of a gated pull request failed, the state comment only
    said "failed" and linked the run, so the author and reviewers had to open
    the fork's run to see what broke, and with job statuses turned off or
    deferred by the write budget the pull request showed nothing more.
    
    The state comment of a failed run now lists the jobs that failed, timed
    out or were cancelled, linking each job in the fork (at most 10). The jobs
    are reused from the job statuses when they were read in the same run, and
    a failed read leaves the comment for the next run instead of dropping the
    list.
    
    Generated-by: Claude Opus 5
    
    * Run the fork CI monitor every 5 minutes
    
    Fork CI reaches a pull request only when the monitor runs: the fork run is
    in another repository, so no event in apache/airflow reports its start or
    end. At a 15-minute cadence plus GitHub's delay of scheduled workflows, the
    start, failure or success of a fork run showed up on the pull request 10 to
    20 minutes late, and so did `CI run started`, which every pull request
    waits for before it can merge.
    
    The monitor now runs every 5 minutes. The per-run write budgets and the
    number of runs before a Slack note are derived from MONITOR_INTERVAL, so
    their hourly totals stay as they were; a test checks that the interval
    matches the workflow's cron. Unchanged pull requests cost no writes; the
    cost is about three times the monitor's runner time and reads.
    
    Generated-by: Claude Opus 5
    
    * Read a failed fork run's jobs once and post the failed state without them
    
    The state comment's marker records the fork run and attempt its failed-job
    list belongs to, so later monitor runs do not read the same jobs again while
    the comment would stay unchanged. When the jobs cannot be read, the comment
    is held only if it already shows this head commit as failed; otherwise the
    failed state is posted without the list. Failed jobs are listed before
    timed-out and cancelled ones, so the cap of 10 does not hide them.
    
    Generated-by: Claude Opus 5
    
    * List the failed job of fork CI jobs sharing a name
    
    When a cancelled or timed-out job came before a failed one of the same
    name, the state comment linked the cancelled one and sorted it after the
    real failures.
    
    Generated-by: Claude Opus 5
---
 .github/workflows/fork-ci-monitor.yml              |   5 +-
 contributing-docs/05_pull_requests.rst             |   2 +-
 dev/AGENTS.md                                      |   4 +-
 dev/breeze/doc/08_ci_tasks.rst                     |   2 +-
 .../doc/ci/08_fork_ci_for_external_contributors.md |  79 +++++---
 .../src/airflow_breeze/utils/fork_ci_messages.py   |  53 ++++-
 .../src/airflow_breeze/utils/fork_ci_reconcile.py  | 149 +++++++++++---
 dev/breeze/tests/test_fork_ci_messages.py          |  42 ++++
 dev/breeze/tests/test_fork_ci_reconcile.py         | 220 +++++++++++++++++++--
 9 files changed, 483 insertions(+), 73 deletions(-)

diff --git a/.github/workflows/fork-ci-monitor.yml 
b/.github/workflows/fork-ci-monitor.yml
index e95becc9e27..a2d09f59fb0 100644
--- a/.github/workflows/fork-ci-monitor.yml
+++ b/.github/workflows/fork-ci-monitor.yml
@@ -27,7 +27,8 @@
 name: "Fork CI monitor"
 on:  # yamllint disable-line rule:truthy
   schedule:
-    - cron: '*/15 * * * *'
+    # MONITOR_INTERVAL in fork_ci_reconcile.py, whose per-run write budgets 
assume this cadence.
+    - cron: '*/5 * * * *'
   workflow_dispatch:
     inputs:
       dry-run:
@@ -59,7 +60,7 @@ jobs:
     timeout-minutes: 20
     steps:
       # What earlier runs reported, so that a problem that persists is posted 
to Slack once
-      # rather than every 15 minutes. Losing it, for example to cache 
eviction, re-posts once.
+      # rather than on every run. Losing it, for example to cache eviction, 
re-posts once.
       # Restored before anything that can fail, so a failing setup is reported 
once too, and
       # outside the checkout, where nothing but the steps below reads it.
       - name: "Restore what earlier runs reported"
diff --git a/contributing-docs/05_pull_requests.rst 
b/contributing-docs/05_pull_requests.rst
index 7320241b18c..d927b218f7c 100644
--- a/contributing-docs/05_pull_requests.rst
+++ b/contributing-docs/05_pull_requests.rst
@@ -272,7 +272,7 @@ and approves them automatically when all of these hold:
 
 If your PR's CI runs in your fork (it carries the ``fork ci required`` label), 
the CI run in
 ``apache/airflow`` only checks that, so none of the rules above apply. The 
``CI run started`` check
-waits until the fork CI monitor sees your fork's run, usually within 15 
minutes, and then links it. Its runs are approved
+waits until the fork CI monitor sees your fork's run, usually within 5 to 10 
minutes, and then links it. Its runs are approved
 automatically unless the PR changes files that decide which CI jobs run and on 
which runners:
 anything under ``.github/``, ``dev/breeze/src/`` or ``scripts/ci/prek/``, or
 ``dev/breeze/pyproject.toml``, ``dev/breeze/uv.lock`` or 
``scripts/ci/install_breeze.sh``. Then
diff --git a/dev/AGENTS.md b/dev/AGENTS.md
index f9e999acfc4..c2ac0cbbb2f 100644
--- a/dev/AGENTS.md
+++ b/dev/AGENTS.md
@@ -54,9 +54,9 @@ workflow of `apache/airflow` shares (see "Write budget" in
 
 If the budget is exceeded, turn the job statuses off: set `job_statuses = 
false` in
 `.github/fork-ci-gate.toml`. Signs that it is exceeded are the monitor's Slack 
note that job
-statuses "have not fit in the budget" (posted only after three runs in a row 
deferred them)
+statuses "have not fit in the budget" (posted only after job statuses were 
deferred for about 45 minutes in a row)
 staying for hours, pull requests listed because their fork run finished more 
than an hour ago with
-job statuses still not posted, its Slack note that GitHub rate-limited it in 
two runs in a row, the
+job statuses still not posted, its Slack note that GitHub rate-limited it for 
about 30 minutes in a row, the
 "Rate limits left" line of its run log approaching zero, or other workflows of 
the repository
 failing on secondary rate limits. The aggregate `Fork CI / Tests (AMD)`
 status and `CI passed` are posted either way, so turning job statuses off 
loses detail, not gating.
diff --git a/dev/breeze/doc/08_ci_tasks.rst b/dev/breeze/doc/08_ci_tasks.rst
index 03d61fed826..c4001b658c0 100644
--- a/dev/breeze/doc/08_ci_tasks.rst
+++ b/dev/breeze/doc/08_ci_tasks.rst
@@ -185,7 +185,7 @@ Reconciling pull requests gated to fork CI
 ------------------------------------------
 
 Pull requests from external contributors can run their CI in the contributor's 
fork instead of in
-``apache/airflow`` (AIP-120). The ``fork-ci-reconcile`` command, run every 15 
minutes by the
+``apache/airflow`` (AIP-120). The ``fork-ci-reconcile`` command, run every 5 
minutes by the
 ``fork-ci-monitor.yml`` workflow, is the only place that changes those pull 
requests: it labels newly
 gated pull requests ``fork ci required``, keeps them in draft until the fork 
run is green, maintains
 one state comment and a ``Fork CI / Tests (AMD)`` commit status linking to the 
run in the fork, closes
diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md 
b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
index 266665d9af5..5220187e34b 100644
--- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
+++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
@@ -87,7 +87,7 @@ Non-goals:
    │      ├─ build-info: fork mode    │      │         (build-info and 
everything after  │
    │      │  changed files from       │      │          are skipped for gated 
PRs)       │
    │      │  compare API vs upstream  │      │                                 
          │
-   │      └─ public runners only      │      │ fork-ci-monitor.yml (every 15 
minutes)    │
+   │      └─ public runners only      │      │ fork-ci-monitor.yml (every 5 
minutes)     │
    └──────────────────────────────────┘      │   └─ breeze ci 
fork-ci-reconcile          │
                      ▲                       │      ├─ discover new gated PRs 
-> label   │
                      │ GraphQL: checkSuites  │      ├─ per labelled PR: fork 
state ->    │
@@ -490,7 +490,7 @@ reads too: the default branch's comes from the reconciler's 
checkout, any other
 from the contents API once per run. Those with a gated reason get the label. A 
base branch with
 neither gate file is never gated, as its gate job runs project CI 
(`base_without_gate`), and
 one with only one of the two files is reported as unknown. The
-24-hour window overlaps many 15-minute cadences, so a lost scheduled run 
changes nothing. A
+24-hour window overlaps many monitor cadences, so a lost scheduled run changes 
nothing. A
 pull request whose author later becomes a committer keeps the label until a 
maintainer
 removes it or adds `use project ci`; the reconciler does not un-gate on its 
own.
 
@@ -612,7 +612,7 @@ since the draft-state job runs after the close and GitHub 
refuses to draft a clo
 | **setup_required**: workflow state is `disabled_fork`, `disabled_manually` 
or `disabled_inactivity`, or the fork has no `ci-amd.yml` | `error` (red), link 
to the fork's Actions page for `ci-amd.yml` | create or update: how to enable, 
with the exact `gh repo sync <fork> --branch main` and `gh workflow enable 
ci-amd.yml -R <fork>` lines, and `breeze ci audit`; on transition from any 
other state but `closed` (the workflow stopped running after it worked): also 
post a **new** comment sayin [...]
 | **awaiting_push**: enabled, no run for the head SHA | `pending` (yellow), 
link to the fork's Actions page | on transition from setup_required: post a 
**new** comment asking to push again (an edit would not notify) |
 | **running**: run in progress | `pending` (yellow), link to the run | update 
state and run link |
-| **failed**: conclusion is failure, cancelled, timed out or startup failure | 
`failure` (red), link to the run; for a startup failure the description says 
the run could not start | update with the run link; for a startup failure the 
text says no job ran and points to the fork's Actions settings |
+| **failed**: conclusion is failure, cancelled, timed out or startup failure | 
`failure` (red), link to the run; for a startup failure the description says 
the run could not start | update with the run link and the jobs that did not 
pass; for a startup failure the text says no job ran and points to the fork's 
Actions settings |
 | **green**: conclusion success and the pull request does not change 
CI-defining files | `success` (green), link to the run | update: ready for 
review, run link; on the author's draft: mark it ready for review when it is |
 | **green_unverified**: conclusion success, but the pull request changes 
CI-defining files (see "Trusting a green fork run") | `pending` (yellow), link 
to the run | update: a maintainer must run project CI; on the author's draft: 
mark it ready for review, then a maintainer runs project CI |
 | **closed**: setup_required or awaiting_push for `close_after_days` | 
unchanged | close with a new comment; reopening restarts the clock |
@@ -726,7 +726,7 @@ every event that runs the workflow files of a pull 
request's merge commit: `pull
 requests by head repository and branch. Looking at every recent run of
 the branch, rather than only at the head commit, is what stops a contributor 
from pushing a
 tampered commit and then a clean one to hide the tampered run. The 6-hour 
window spans many
-15-minute cadences. Runs created before the most recent removal of `use 
project ci` are
+monitor cadences. Runs created before the most recent removal of `use project 
ci` are
 ignored, because they ran legitimately while the label was on. So are runs 
created before the
 most recent reopen of the pull request by someone other than its author, which 
is how a
 maintainer brings back a pull request closed for tampering; reopening one 
whose gate is still
@@ -877,6 +877,22 @@ are no job statuses either, and GitHub's error is only on 
the run page. It is ha
 but `Fork CI / Tests (AMD)` says "Fork CI run could not start - check your 
fork's Actions
 settings", and the state comment explains what to check instead of asking to 
fix failures.
 
+For a failed run that started, the state comment lists the jobs that did not 
pass, each linking
+to the job in the fork, at most 10 of them, failed jobs before timed-out and 
cancelled ones; the
+job statuses list them all. The jobs come from
+`GET /repos/<fork>/actions/runs/<id>/jobs?filter=latest`, reused from the job 
statuses when they
+read them in the same run. A finished run's jobs do not change, so the marker 
records the run and
+attempt the list belongs to (`failed_run`), and later runs do not read the 
jobs again while the
+comment would otherwise stay the same: a failed pull request costs one jobs 
read per fork run, not
+one per reconciler run. Job names come from the contributor's workflow, so 
each is shown in a code
+span with backticks and line breaks replaced. When the jobs cannot be read and 
the state comment
+already shows this head commit as `failed`, it is left as it is until the next 
run, rather than
+dropping the list; otherwise the `failed` state is posted without the list, so 
the comment does
+not keep describing a running or earlier run, and the next run tries the read 
again. A read that
+failed after its retries is not repeated in
+the same run, whether the job statuses or the state comment made it, so 
persistent server errors
+cost each failed pull request one sequence of retries a run, not two.
+
 ### Commit status
 
 The reconciler posts a commit status with context `Fork CI / Tests (AMD)` on 
the pull
@@ -914,7 +930,7 @@ suites); its jobs come from `GET 
/repos/<fork>/actions/runs/<id>/jobs?filter=lat
 - Only finished jobs are reported: a successful one as `success`, and a 
failed, timed-out,
   cancelled or unstartable one as `failure`. Queued and running jobs are not, 
so each job costs one
   write instead of two; `Fork CI / Tests (AMD)` shows the run as running, 
linking it, and with a
-  15-minute cadence a per-job "running" status was behind the fork's own run 
page anyway. A job of
+  5-minute cadence a per-job "running" status was behind the fork's own run 
page anyway. A job of
   a re-run in the fork keeps the result of the earlier attempt until it 
finishes. Skipped and
   neutral jobs, and the jobs every run has (`Fork CI gate`, `Check if the run 
is needed`,
   `Platform: AMD` and `CI passed`), are not reported.
@@ -922,12 +938,12 @@ suites); its jobs come from `GET 
/repos/<fork>/actions/runs/<id>/jobs?filter=lat
   (`GET /repos/apache/airflow/commits/<sha>/status`), and only differences are 
posted, failures
   first. GraphQL has no mutation for commit statuses (`createCheckRun` exists, 
but see above for
   why check runs do not fit), so each one is a REST call, and they count 
towards GitHub's limits on
-  content-creating requests: a reconciler run posts at most 
`MAX_JOB_STATUSES_PER_RUN` (40) of
+  content-creating requests: a reconciler run posts at most 
`MAX_JOB_STATUSES_PER_RUN` (13) of
   them, each write charged as it is made. What does not fit is posted by the 
next run, and once
   the budget is spent no further pull request's jobs or statuses are read in 
that run. One fork run
   of the full matrix (about 130 jobs) needs more than a run's budget on its 
own, which the next
   run catches up on, so deferring alone is not a problem: the run log lists 
the deferred pull
-  requests, and only when three runs in a row deferred, about 45 minutes of 
the budget not
+  requests, and only when nine runs in a row deferred, about 45 minutes of the 
budget not
   keeping up, is a note added to the Slack message. The count is kept in the 
record the monitor
   carries between runs (`--reported-file`); without a readable record the note 
is added at once.
   Its wording is fixed, so it is posted once while deferrals persist. A pull 
request whose fork run
@@ -947,7 +963,7 @@ suites); its jobs come from `GET 
/repos/<fork>/actions/runs/<id>/jobs?filter=lat
   classified, so for example a `CI passed` that no longer holds is still 
revoked. Reading a fork
   run's jobs fails now and then on GitHub server errors (HTTP 502, a reset 
HTTP/2 stream) that the
   next run gets past, so each failure is only logged, and the pull request is 
listed with the
-  unknown items for Slack once its job statuses failed in three runs in a row. 
The per pull request
+  unknown items for Slack once its job statuses failed in nine runs in a row, 
about 45 minutes. The per pull request
   count is kept in the same record as that of deferred job statuses; without a 
readable record it
   starts again. The item's wording is fixed whatever the error, so it is 
posted once while the
   failure persists, and the run log has each run's error.
@@ -962,7 +978,7 @@ records this for agents.
 
 Each run logs what is left of the token's REST and GraphQL quotas (`GET 
/rate_limit`, which does
 not count against them) and how many of its calls GitHub rate-limited; the 
client retries those with
-backoff. Rate limiting in two runs in a row adds a note to the Slack message, 
with fixed wording so
+backoff. Rate limiting in six runs in a row, about 30 minutes, adds a note to 
the Slack message, with fixed wording so
 it is posted once while it persists; the count is kept in the same record as 
that of deferred job
 statuses.
 
@@ -991,14 +1007,23 @@ requests a minute and 500 an hour. Assuming about 35 
reported jobs for an averag
   monitor runs.
 - **Gate enabled**, assuming about half of the pull request runs come from 
gated pull requests
   (about 5-6 pushes an hour): about 200 writes an hour without a cap, 40% of 
the content-creating
-  limit and more at peaks. The cap of 40 writes a run, four runs an hour, 
holds job statuses to at
-  most 160 an hour: 32% of the content-creating limit, leaving the rest to the 
reconciler's other
+  limit and more at peaks. The cap of 13 writes a run, twelve runs an hour, 
holds job statuses to at
+  most 156 an hour: 31% of the content-creating limit, leaving the rest to the 
reconciler's other
   writes (labels, comments, the aggregate statuses) and to the other workflows 
that share it. Each
-  run posts at most 40 statuses at about one a second, well under the 
80-a-minute limit. Job
+  run posts at most 13 statuses at about one a second, well under the 
80-a-minute limit. Job
   statuses then lag behind at busy times, the aggregate status does not. The 
`CI run started`
-  statuses of pull requests whose CI runs in `apache/airflow` come out of the 
same 40 writes a run,
-  first and at most 20 of them (see "The `CI run started` check"), so they do 
not raise the total;
-  at about 11 pull request runs an hour they take about 3 writes a run.
+  statuses of pull requests whose CI runs in `apache/airflow` come out of the 
same 13 writes a run,
+  first and at most 6 of them (see "The `CI run started` check"), so they do 
not raise the total;
+  at about 11 pull request runs an hour they take about 1 write a run.
+
+The per-run caps and the run counts before a Slack note are derived from 
`MONITOR_INTERVAL` in
+`fork_ci_reconcile.py`, which must match the monitor's cron (a test checks 
it), so changing the
+cadence keeps the hourly totals above. The cadence was lowered from 15 to 5 
minutes because fork CI
+reaches a pull request only when the monitor runs: a fork run's start, failure 
or success showed up
+to 15 minutes late, plus GitHub's delay of scheduled workflows. Unchanged pull 
requests cost no
+writes, so the faster cadence adds reads, about 45 a run, well within the 
token's quotas, and
+runner time: each run takes about two minutes on a GitHub-hosted runner, 
mostly installing
+Breeze, so the monitor uses about 24 runner minutes an hour instead of 8.
 
 These are estimates: the share of gated runs, the jobs per push and the pushes 
per pull request
 are assumptions, not measurements. Watch the monitor's call counts in its run 
log after enabling
@@ -1048,7 +1073,7 @@ run.
   job has not succeeded, a `success` is overwritten with `pending`; once it 
has, the monitor posts
   `success` ("Project CI passed for this commit"). Adding the label does not 
start project CI (see
   "Override labels"): the pull request stays blocked until a maintainer starts 
it and it passes,
-  and then until the next monitor run posts the `success`, which can take 
about 15 minutes.
+  and then until the next monitor run posts the `success`, which can take 
about 5 minutes.
   Removing the label returns the pull request to the normal flow above: 
removing it updates the
   pull request, so discovery (open pull requests updated within the last 24 
hours) labels it
   `fork ci required` again and the next run reports the fork result, replacing 
the waiting
@@ -1069,7 +1094,7 @@ run.
 - The protection is eventual, not atomic. Nothing reacts to the label when it 
is added
   (`pull_request_target` is not used, see "Goals and non-goals"), so a fork 
`success` posted
   before the label stays until the next monitor run that completes: normally 
within one cadence
-  (15 minutes), longer when GitHub delays or drops a scheduled run or a run 
fails, and one more
+  (5 minutes), longer when GitHub delays or drops a scheduled run or a run 
fails, and one more
   cycle when the label arrives after a run has taken its snapshot of the pull 
request. The pull
   request can merge on the fork result in that window. A maintainer who adds 
the label to a pull
   request whose fork CI is green waits for the `pending` before merging or 
enabling auto-merge,
@@ -1087,7 +1112,7 @@ run.
 
 ### The `CI run started` check
 
-Fork CI reports to the pull request only when the scheduled monitor runs, 
about every 15 minutes
+Fork CI reports to the pull request only when the scheduled monitor runs, 
about every 5 minutes
 and later when GitHub delays scheduled workflows, and until then a gated pull 
request shows only
 skipped checks. The gate cannot say more itself: it runs on the contributor's 
`pull_request`
 event with a read-only token. Instead both CI rulesets in `.asf.yaml` also 
require a
@@ -1113,8 +1138,8 @@ with state `success` linking the run:
 Folding the check into the scheduled monitor costs no runner per push, at the 
price of a pull
 request waiting for the next monitor run before it can merge, which a CI run 
takes longer than
 anyway. Each new head commit needs one write, taken first out of the per-run 
budget of job
-statuses (see "Job statuses"), since merging waits for it; at most 20 such 
writes per run, so job
-statuses keep at least half of the budget, and after the check became required 
the open pull
+statuses (see "Job statuses"), since merging waits for it; at most 
`MAX_CI_RUN_STARTED_PER_RUN` (6) such
+writes per run, so job statuses keep at least half of the budget, and after 
the check became required the open pull
 requests are caught up a batch at a time. A release branch has no monitor of 
its own, but `main`'s monitor covers pull
 requests to every base branch.
 
@@ -1198,14 +1223,15 @@ change. Job statuses add a jobs call and a status call 
per pull request with a f
 progress or a completed run not yet reported, and up to two status mutations 
per job, at most
 `MAX_JOB_STATUSES_PER_RUN` per run; see "Write budget". Pull requests that 
pass 1 labelled and
 the search does not list yet add one GraphQL call per 25. Well inside
-the `GITHUB_TOKEN` limits at a 15-minute cadence. The reconciler logs its call 
counts so the
+the `GITHUB_TOKEN` limits at a 5-minute cadence. The reconciler logs its call 
counts so the
 budget stays observable.
 
 ## Monitor workflow
 
 `.github/workflows/fork-ci-monitor.yml`:
 
-- `on: schedule: cron '*/15 * * * *'` and `workflow_dispatch` with a `dry-run` 
input.
+- `on: schedule: cron '*/5 * * * *'` (`MONITOR_INTERVAL`) and 
`workflow_dispatch` with a `dry-run`
+  input.
 - Two jobs, each with its own `permissions` block and a justification comment; 
the workflow
   grants nothing by default.
 - The `reconcile` job: `actions: write, contents: read, pull-requests: write, 
statuses: write`.
@@ -1377,10 +1403,11 @@ as a failed check rather than a crash.
 
 ## Known limitations
 
-- Drafting, the setup comment and the commit status arrive within one cadence 
of the event
-  they reflect, not instantly. The cadence can be lowered, but the per-run 
write budgets are
-  sized for four runs an hour against the `GITHUB_TOKEN` limits shared by 
every workflow of the
-  repository, so they must be scaled down with it.
+- Drafting, the setup comment and the commit status arrive within one cadence 
(5 minutes) of the
+  event they reflect, plus GitHub's delay of scheduled workflows, not 
instantly. The fork runs in
+  another repository, so no event in `apache/airflow` reports its start or 
end; a `workflow_run`
+  trigger on project CI's gate run would only cover a push, at the cost of a 
monitor run per
+  pull request push.
 - The workflows API lists workflows from the fork's default branch. A fork 
whose `main`
   predates `ci-amd.yml` reports it missing; the setup comment covers this with
   `gh repo sync`.
diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py 
b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
index 314609f52ac..845769d025f 100644
--- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
+++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
@@ -30,8 +30,10 @@ from __future__ import annotations
 
 import json
 import re
+from collections.abc import Sequence
 from dataclasses import dataclass
 from datetime import datetime
+from typing import NamedTuple
 
 STATUS_CONTEXT = "Fork CI / Tests (AMD)"
 # Each job of the fork run is reported under this prefix. Job names come from 
the contributor's
@@ -94,6 +96,8 @@ class Marker:
     behind: int | None = None
     # The completed fork run (``<run id>:<attempt>:<conclusion>``) whose jobs 
are all reported.
     jobs: str | None = None
+    # The fork run (``<run id>:<attempt>``) whose failed jobs the comment 
lists.
+    failed_run: str | None = None
     # Set only by a reconciler that ran the gate protection check for this 
head commit, so a
     # marker written before that check existed, or while it did not apply, 
never counts as one.
     gate_checked: bool = False
@@ -105,6 +109,8 @@ def render_marker(marker: Marker) -> str:
         data["behind"] = marker.behind
     if marker.jobs is not None:
         data["jobs"] = marker.jobs
+    if marker.failed_run is not None:
+        data["failed_run"] = marker.failed_run
     if marker.gate_checked:
         data["gate_checked"] = True
     return f"{MARKER_PREFIX}{json.dumps(data, sort_keys=True)} -->"
@@ -123,6 +129,7 @@ def parse_marker(body: str) -> Marker | None:
             since=datetime.fromisoformat(data["since"]),
             behind=data.get("behind"),
             jobs=data.get("jobs"),
+            failed_run=data.get("failed_run"),
             gate_checked=data.get("gate_checked") is True,
         )
     except (json.JSONDecodeError, KeyError, TypeError, ValueError) as e:
@@ -203,6 +210,44 @@ STARTUP_FAILURE_TEXT = (
 )
 
 
+class FailedJob(NamedTuple):
+    """A job of a failed fork run that did not pass, as listed in the state 
comment."""
+
+    name: str
+    url: str
+    conclusion: str
+
+
+# A full-matrix run can fail in dozens of jobs; the job statuses list them all.
+MAX_LISTED_FAILED_JOBS = 10
+FAILED_JOB_LIST_HEADING = "Jobs that did not pass:"
+_FAILED_JOB_OUTCOMES = {
+    "failure": "failed",
+    "timed_out": "timed out",
+    "cancelled": "cancelled",
+    "startup_failure": "could not start",
+}
+
+
+def build_failed_job_list(failed_jobs: Sequence[FailedJob]) -> str:
+    listed = [
+        # Job names come from the contributor's workflow; a code span keeps 
them from posting
+        # Markdown, including mentions, as the bot.
+        f"- [`{_sanitize_path(job.name[:JOB_NAME_MAX_LENGTH])}`]({job.url}): "
+        f"{_FAILED_JOB_OUTCOMES.get(job.conclusion, 'failed')}"
+        for job in failed_jobs[:MAX_LISTED_FAILED_JOBS]
+    ]
+    if len(failed_jobs) > MAX_LISTED_FAILED_JOBS:
+        listed.append(f"- and {len(failed_jobs) - MAX_LISTED_FAILED_JOBS} 
more, see the run")
+    # One paragraph of the comment, so that ``strip_failed_job_list`` can find 
all of it.
+    return FAILED_JOB_LIST_HEADING + "\n" + "\n".join(listed)
+
+
+def strip_failed_job_list(body: str) -> str:
+    """The state comment without its list of failed jobs."""
+    return "\n\n".join(part for part in body.split("\n\n") if not 
part.startswith(FAILED_JOB_LIST_HEADING))
+
+
 def behind_warning(behind: int, *, base_ref: str) -> str:
     return (
         f"**This branch is {behind} commits behind `{base_ref}`.** Fork CI 
tests the branch as it is, "
@@ -219,8 +264,12 @@ def state_comment(
     base_ref: str = "main",
     author_draft: bool = False,
     startup_failure: bool = False,
+    failed_jobs: Sequence[FailedJob] = (),
 ) -> str:
-    """:param startup_failure: the fork run of a failed state did not start, 
see ``STARTUP_FAILURE_TEXT``."""
+    """
+    :param startup_failure: the fork run of a failed state did not start, see 
``STARTUP_FAILURE_TEXT``.
+    :param failed_jobs: the jobs of a failed state's fork run that did not 
pass.
+    """
     text = _AUTHOR_DRAFT_TEXTS.get(marker.state) if author_draft else None
     if startup_failure:
         text = STARTUP_FAILURE_TEXT
@@ -233,6 +282,8 @@ def state_comment(
         parts.append(behind_warning(marker.behind, base_ref=base_ref))
     if run_url:
         parts.append(f"Latest run: {run_url}")
+    if marker.state == "failed" and failed_jobs:
+        parts.append(build_failed_job_list(failed_jobs))
     if marker.state == "setup_required":
         parts.append(_setup_instructions(fork))
     parts.append(f"How this works: [CI in your fork]({CONTRIBUTING_URL}).")
diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py 
b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
index da56f0c7310..f84e8ff2db2 100644
--- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
+++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
@@ -71,6 +71,7 @@ from airflow_breeze.utils.fork_ci_messages import (
     STALE_JOB_STATUS,
     STATUS_CONTEXT,
     STATUS_STATES,
+    FailedJob,
     Marker,
     fork_actions_url,
     gate_modified_close_comment,
@@ -81,6 +82,7 @@ from airflow_breeze.utils.fork_ci_messages import (
     released_comment,
     state_comment,
     status_description,
+    strip_failed_job_list,
     tamper_close_comment,
     ungated_comment,
     workflow_disabled_comment,
@@ -129,8 +131,19 @@ BOT_LOGINS = frozenset({"github-actions", 
"github-actions[bot]"})
 DISCOVERY_WINDOW = timedelta(hours=24)
 PASS_TWO_PAGE_SIZE = 25
 FORK_BATCH_SIZE = 50
+# The cron interval of fork-ci-monitor.yml. Fork CI reaches a pull request 
only when the monitor
+# runs, so this is how late it can be; the write budgets and alert thresholds 
below are per run and
+# derived from it, to keep their hourly totals when it changes.
+MONITOR_INTERVAL = timedelta(minutes=5)
+RUNS_PER_HOUR = timedelta(hours=1) // MONITOR_INTERVAL
+
+
+def count_runs_in(duration: timedelta) -> int:
+    return max(duration // MONITOR_INTERVAL, 1)
+
+
 # Project CI runs created this long ago or later are checked for gate 
tampering. Several hours
-# overlap the 15-minute cadence, so a tampered run is seen even after the pull 
request was
+# overlap the monitor's cadence, so a tampered run is seen even after the pull 
request was
 # pushed again with the gate restored.
 TAMPER_WINDOW = timedelta(hours=6)
 # A ci-amd.yml run in which only the gate executed finishes in about a minute; 
only longer runs
@@ -171,30 +184,30 @@ UNREPORTED_JOB_NAMES = GATED_RUN_JOB_NAMES | 
{GATE_JOB_NAME, "Platform: AMD"}
 RUNNING_JOB_STATUSES = frozenset({"queued", "waiting", "pending", "requested", 
"in_progress"})
 JOB_STATE_RANKS = {"failure": 0, "pending": 1, "success": 2}
 # Commit statuses count towards GitHub's limit of 500 content-creating 
requests per hour, shared by
-# every workflow of the repository. At four runs an hour this keeps job 
statuses to at most about
-# half of it; the rest follow on the next run, failures first. See "Job 
statuses" in the design doc.
-MAX_JOB_STATUSES_PER_RUN = 40
+# every workflow of the repository. This keeps job statuses to about a third 
of it; the rest follow
+# on the next run, failures first. See "Job statuses" in the design doc.
+MAX_JOB_STATUSES_PER_RUN = 160 // RUNS_PER_HOUR
 # `CI run started` statuses of pull requests whose CI runs in apache/airflow 
come out of that budget
 # first: one write per new head commit, which merging waits for, while job 
statuses are informational.
 # At most this many, so job statuses keep the rest; the first runs after the 
check became required
 # catch up on the open pull requests a batch at a time, most recently updated 
first.
-MAX_CI_RUN_STARTED_PER_RUN = 20
+MAX_CI_RUN_STARTED_PER_RUN = 80 // RUNS_PER_HOUR
 # `CI passed` successes for pull requests carrying `use project ci`, written 
once per new project CI
 # pass, so far fewer than this per run. Revoking a success is never limited.
-MAX_PROJECT_CI_PASSED_PER_RUN = 5
+MAX_PROJECT_CI_PASSED_PER_RUN = max(20 // RUNS_PER_HOUR, 1)
 # The GitHub Actions app, whose check suites are the workflow runs of a commit.
 GITHUB_ACTIONS_APP_ID = 15368
 # A single large fork run can need more writes than one run's budget, which 
the next run catches up
 # on, so deferring is only reported when it lasts this many consecutive runs: 
then the budget does
 # not keep up with the pull requests.
-DEFERRED_RUNS_BEFORE_ALERT = 3
+DEFERRED_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=45))
 # GitHub answering with rate limits in this many consecutive runs is reported: 
a single burst is
 # retried, while a persisting one means the repository's shared limits are 
close to exhausted.
-RATE_LIMITED_RUNS_BEFORE_ALERT = 2
+RATE_LIMITED_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=30))
 # Reading a fork run's jobs fails now and then on GitHub server errors (HTTP 
502, a reset HTTP/2
 # stream) that the next run gets past, so a pull request whose job statuses 
failed is only reported
 # when it failed in this many consecutive runs.
-JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT = 3
+JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=45))
 # Fixed text, so the item is posted to Slack once while the failure persists, 
whatever the error;
 # the run log has each run's error.
 JOB_STATUSES_FAILED_REASON = (
@@ -625,6 +638,7 @@ def classify(
     jobs_reported: str | None = None,
     head_shared_across_bases: bool = False,
     find_gate_change: Callable[[], str | None] = lambda: None,
+    find_failed_jobs: Callable[[], list[FailedJob] | None] = lambda: [],
 ) -> Plan:
     """
     Decide the mutations for one gated pull request.
@@ -646,6 +660,9 @@ def classify(
     :param find_gate_change: describes how the pull request changes the fork 
CI gate, or returns
         None; only called for pull requests that the gate tampering check 
applies to, once per
         head commit.
+    :param find_failed_jobs: the jobs of the fork run that did not pass, None 
when they cannot be
+        read; only called for a failed run that started. On None the state 
comment is left for the
+        next run, so that a failed read does not drop the list from it.
     """
     plan = Plan()
     labels = set(pr.labels)
@@ -846,25 +863,47 @@ def classify(
     _set_ci_passed(plan, pr, passed=state == ForkState.GREEN, 
target_url=target_url)
     _set_ci_run_started(plan, pr, fork)
     green = state in (ForkState.GREEN, ForkState.GREEN_UNVERIFIED)
-    _set_marker_comment(
-        plan,
-        pr,
-        state_comment(
-            Marker(
-                state.value,
-                pr.head_sha,
-                since,
-                behind=behind if green else None,
-                jobs=jobs_reported,
-                gate_checked=gate_checked,
-            ),
+    lists_failed_jobs = state == ForkState.FAILED and not startup_failure and 
fork.run_id is not None
+    new_marker = Marker(
+        state.value,
+        pr.head_sha,
+        since,
+        behind=behind if green else None,
+        jobs=jobs_reported,
+        failed_run=f"{fork.run_id}:{fork.run_attempt}" if lists_failed_jobs 
else None,
+        gate_checked=gate_checked,
+    )
+
+    def render(marker: Marker, failed_jobs: list[FailedJob]) -> str:
+        return state_comment(
+            marker,
             fork=pr.head_repo,
             run_url=fork.run_url,
             base_ref=pr.base_ref,
             author_draft=_is_author_draft(pr),
             startup_failure=startup_failure,
-        ),
-    )
+            failed_jobs=failed_jobs,
+        )
+
+    failed_jobs: list[FailedJob] | None = []
+    if lists_failed_jobs:
+        # A finished run's jobs do not change, so the list already in the 
comment still holds.
+        if (
+            marker == new_marker
+            and pr.marker_comment_body is not None
+            and strip_failed_job_list(pr.marker_comment_body) == 
render(new_marker, [])
+        ):
+            return plan
+        failed_jobs = find_failed_jobs()
+    if failed_jobs is None:
+        # Only a comment that already shows this head as failed is worth 
keeping; anything else
+        # describes a state the pull request is no longer in.
+        if marker and marker.state == ForkState.FAILED.value and marker.sha == 
pr.head_sha:
+            return plan
+        failed_jobs = []
+        # Without a list, the next run reads the jobs again.
+        new_marker = replace(new_marker, failed_run=None)
+    _set_marker_comment(plan, pr, render(new_marker, failed_jobs))
     return plan
 
 
@@ -1748,6 +1787,53 @@ def build_job_statuses(jobs: Iterable[dict[str, Any]]) 
-> dict[str, tuple[str, s
     return statuses
 
 
+def find_failed_jobs(jobs: Iterable[dict[str, Any]]) -> list[FailedJob]:
+    """The finished jobs of a fork run that did not pass, by name, leaving out 
the jobs every run has."""
+    failed: dict[str, FailedJob] = {}
+    for job in jobs:
+        if job["name"] in UNREPORTED_JOB_NAMES or job.get("status") in 
RUNNING_JOB_STATUSES:
+            continue
+        conclusion = job.get("conclusion") or ""
+        if JOB_STATUSES.get(conclusion, ("",))[0] != "failure":
+            continue
+        listed = failed.get(job["name"])
+        # Of jobs sharing a name, one that failed is listed rather than one 
cancelled or timed out.
+        if listed is None or (listed.conclusion != "failure" and conclusion == 
"failure"):
+            failed[job["name"]] = FailedJob(job["name"], job["html_url"], 
conclusion)
+    # Failures first: the cap on the listed jobs must not hide them behind 
cancelled or timed-out ones.
+    return sorted(failed.values(), key=lambda job: (job.conclusion != 
"failure", job.name))
+
+
+# The jobs of fork runs read in this reconciler run, by run id. None records a 
read that failed after
+# its retries, so the next reader gives up at once instead of repeating them; 
the next run reads again.
+ReadJobs = dict[int, list[dict[str, Any]] | None]
+
+
+def read_failed_jobs(
+    client: GitHubClient,
+    pr: PullRequestSnapshot,
+    fork: ForkObservation,
+    read_jobs: ReadJobs,
+) -> list[FailedJob] | None:
+    """
+    ``find_failed_jobs`` of the pull request's fork run; None when its jobs 
cannot be read.
+
+    :param read_jobs: the jobs already read in this reconciler run; a read 
made here is recorded too.
+    """
+    if fork.run_id is None or not pr.head_repo:
+        return []
+    if fork.run_id in read_jobs:
+        jobs = read_jobs[fork.run_id]
+    else:
+        try:
+            jobs = client.get_run_jobs(pr.head_repo, fork.run_id)
+        except GitHubError as e:
+            console_print(f"[warning]#{pr.number}: could not read the jobs of 
{fork.run_url}: {e}[/]")
+            jobs = None
+        read_jobs[fork.run_id] = jobs
+    return None if jobs is None else find_failed_jobs(jobs)
+
+
 class JobStatusReporter:
     """Reports each job of a pull request's fork run as a commit status, 
within a per-run budget."""
 
@@ -1758,12 +1844,17 @@ class JobStatusReporter:
         repo: str,
         budget: int = MAX_JOB_STATUSES_PER_RUN,
         now: datetime | None = None,
+        read_jobs: ReadJobs | None = None,
     ):
-        """:param now: when given, deferring the statuses of a run that 
finished long ago is reported."""
+        """
+        :param now: when given, deferring the statuses of a run that finished 
long ago is reported.
+        :param read_jobs: where the jobs read, or a failed read, are kept, for 
the state comment to reuse.
+        """
         self._client = client
         self._repo = repo
         self._now = now
         self.remaining = budget
+        self.read_jobs = {} if read_jobs is None else read_jobs
         self.deferred: list[int] = []
         # (pull request, attention item) for runs whose statuses are deferred 
long after they finished.
         self.delayed: list[tuple[int, str]] = []
@@ -1808,7 +1899,12 @@ class JobStatusReporter:
         if self.remaining == 0:
             self._defer(pr, fork)
             return None
-        jobs = self._client.get_run_jobs(pr.head_repo, fork.run_id)
+        try:
+            jobs = self._client.get_run_jobs(pr.head_repo, fork.run_id)
+        except Exception:
+            self.read_jobs[fork.run_id] = None
+            raise
+        self.read_jobs[fork.run_id] = jobs
         # By the case-folded context, as GitHub compares them.
         current = {
             status["context"].casefold(): (
@@ -2682,11 +2778,13 @@ def run_reconcile(
         ci_passed_notes.append(
             f"listing the pull requests with `{LABEL_USE_PROJECT_CI}` failed: 
{type(e).__name__}"
         )
+    read_jobs: ReadJobs = {}
     job_reporter = JobStatusReporter(
         client,
         repo=repo,
         budget=max(MAX_JOB_STATUSES_PER_RUN - ci_run_started_writes - 
ci_passed_reserved, 0),
         now=now,
+        read_jobs=read_jobs,
     )
 
     previous_job_status_failure_runs = 
read_job_status_failure_runs(reported_file)
@@ -2736,6 +2834,7 @@ def run_reconcile(
                 jobs_reported=jobs_reported,
                 head_shared_across_bases=pr.head_sha in shared_heads,
                 find_gate_change=functools.partial(gate_change_finder.find, 
pr),
+                find_failed_jobs=functools.partial(read_failed_jobs, client, 
pr, fork, read_jobs),
             )
             # A revocation to `pending` always goes through: it is idempotent 
and must not depend on
             # `report_project_ci_passed` succeeding later in this run.
diff --git a/dev/breeze/tests/test_fork_ci_messages.py 
b/dev/breeze/tests/test_fork_ci_messages.py
index 3b5b327593b..37b874b56d8 100644
--- a/dev/breeze/tests/test_fork_ci_messages.py
+++ b/dev/breeze/tests/test_fork_ci_messages.py
@@ -36,6 +36,8 @@ from airflow_breeze.utils.fork_ci_messages import (
     JOB_NAME_MAX_LENGTH,
     JOB_STATUS_CONTEXT_PREFIX,
     MAX_LISTED_CI_FILES,
+    MAX_LISTED_FAILED_JOBS,
+    FailedJob,
     Marker,
     ci_change_approval_comment,
     ci_modification_note_comment,
@@ -47,6 +49,7 @@ from airflow_breeze.utils.fork_ci_messages import (
     released_comment,
     render_marker,
     state_comment,
+    strip_failed_job_list,
     tamper_close_comment,
     ungated_comment,
     workflow_disabled_comment,
@@ -61,6 +64,7 @@ MARKER = Marker(state="awaiting_push", sha="abc123", 
since=datetime(2026, 10, 15
         pytest.param(MARKER, id="without-behind"),
         pytest.param(replace(MARKER, behind=12), id="behind"),
         pytest.param(replace(MARKER, jobs="123:SUCCESS"), id="jobs"),
+        pytest.param(replace(MARKER, failed_run="123:2"), id="failed-run"),
         pytest.param(replace(MARKER, gate_checked=True), id="gate-checked"),
     ],
 )
@@ -121,6 +125,44 @@ def 
test_state_comment_links_the_contributor_documentation(startup_failure):
     assert text.endswith(FOOTER)
 
 
+def test_failed_job_list_is_safe_markdown_and_capped():
+    jobs = [
+        FailedJob("Tests `evil`\n\n@someone", "https://job/0";, "timed_out"),
+        *(
+            FailedJob(f"Job {i}", f"https://job/{i + 1}", "failure")
+            for i in range(MAX_LISTED_FAILED_JOBS + 1)
+        ),
+    ]
+
+    text = state_comment(
+        Marker("failed", "a", MARKER.since), fork="u/airflow", run_url=None, 
failed_jobs=jobs
+    )
+
+    assert "- [`Tests ?evil???@someone`](https://job/0): timed out" in text
+    assert "\n@someone" not in text
+    assert "- [`Job 0`](https://job/1): failed" in text
+    assert "- and 2 more, see the run" in text
+
+
+def test_failed_jobs_are_listed_only_for_a_failed_state():
+    jobs = [FailedJob("Static checks", "https://job/1";, "failure")]
+
+    text = state_comment(
+        Marker("running", "a", MARKER.since), fork="u/airflow", run_url=None, 
failed_jobs=jobs
+    )
+
+    assert "Static checks" not in text
+
+
+def test_strip_failed_job_list_leaves_the_comment_without_the_list():
+    marker = Marker("failed", "a", MARKER.since)
+    jobs = [FailedJob(f"Job {i}", f"https://job/{i}";, "failure") for i in 
range(MAX_LISTED_FAILED_JOBS + 1)]
+
+    listed = state_comment(marker, fork="u/airflow", run_url="https://run";, 
failed_jobs=jobs)
+
+    assert strip_failed_job_list(listed) == state_comment(marker, 
fork="u/airflow", run_url="https://run";)
+
+
 def test_parse_marker_absent():
     assert parse_marker("just a comment") is None
 
diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py 
b/dev/breeze/tests/test_fork_ci_reconcile.py
index b9714bb70c0..519294739f3 100644
--- a/dev/breeze/tests/test_fork_ci_reconcile.py
+++ b/dev/breeze/tests/test_fork_ci_reconcile.py
@@ -24,6 +24,7 @@ from unittest import mock
 from urllib.parse import quote
 
 import pytest
+import yaml
 from click.testing import CliRunner
 
 from airflow_breeze.commands.ci_commands import fork_ci_reconcile
@@ -40,14 +41,22 @@ from airflow_breeze.utils.fork_ci_messages import (
     JOB_STATUS_CONTEXT_PREFIX,
     STALE_JOB_STATUS,
     STATUS_CONTEXT,
+    FailedJob,
     Marker,
     parse_marker,
     render_marker,
 )
 from airflow_breeze.utils.fork_ci_reconcile import (
+    DEFERRED_RUNS_BEFORE_ALERT,
+    JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT,
     JOB_STATUSES_DEFERRED_NOTE,
     JOB_STATUSES_FAILED_REASON,
+    MAX_CI_RUN_STARTED_PER_RUN,
+    MAX_JOB_STATUSES_PER_RUN,
+    MAX_PROJECT_CI_PASSED_PER_RUN,
+    MONITOR_INTERVAL,
     RATE_LIMITED_NOTE,
+    RATE_LIMITED_RUNS_BEFORE_ALERT,
     BaseGateConfigs,
     BypassFinder,
     DraftChange,
@@ -72,6 +81,7 @@ from airflow_breeze.utils.fork_ci_reconcile import (
     fetch_open_pull_requests,
     fetch_workflow_state,
     find_ci_run_since,
+    find_failed_jobs,
     find_heads_shared_across_bases,
     find_project_ci_label_heads,
     find_project_ci_passed_url,
@@ -87,6 +97,7 @@ from airflow_breeze.utils.fork_ci_reconcile import (
     parse_pull_request,
     pull_request_changes_ci_files,
     read_ci_run_at,
+    read_failed_jobs,
     read_head_status,
     read_job_status_failure_runs,
     read_project_ci_passed_url,
@@ -2929,6 +2940,9 @@ def 
test_run_reconcile_reports_job_statuses_before_classifying(
     (pr, fork), _ = mock_reporter.return_value.report.call_args
     assert pr.number == 1
     assert mock_classify.call_args.kwargs["jobs_reported"] == "7:SUCCESS"
+    find_failed = mock_classify.call_args.kwargs["find_failed_jobs"]
+    assert find_failed.func is read_failed_jobs
+    assert find_failed.args == (client, pr, fork, 
mock_reporter.call_args.kwargs["read_jobs"])
 
 
 @pytest.mark.parametrize("status", ["queued", "in_progress", "waiting"])
@@ -3035,23 +3049,23 @@ def 
test_run_reconcile_reports_failing_job_statuses_once_when_failing_persists(
     reported_file = tmp_path / "reported.json"
     reported_file.write_text("{}")
     alerts = []
+    failures = [GitHubError("HTTP 502"), GitHubError("stream ID 1; CANCEL")]
     errors = [
         GitHubError("HTTP 502"),
         GitHubError("HTTP 502"),
         None,
-        GitHubError("HTTP 502"),
-        GitHubError("stream ID 1; CANCEL"),
-        GitHubError("HTTP 502"),
-        GitHubError("stream ID 1; CANCEL"),
+        *(failures[run % 2] for run in 
range(JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT + 1)),
     ]
     for error in errors:
         mock_reporter.return_value.report.side_effect = error
         mock_reporter.return_value.report.return_value = None
         alerts.append(_reconcile_with_reporter(client, tmp_path, 
reported_file=reported_file))
 
-    assert alerts == [False, False, False, False, False, True, False]
+    assert alerts == [*[False] * (JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT + 2), 
True, False]
     assert JOB_STATUSES_FAILED_REASON in (tmp_path / "s.json").read_text()
-    assert json.loads(reported_file.read_text())["job_status_failure_runs"] == 
{"1": 4}
+    assert json.loads(reported_file.read_text())["job_status_failure_runs"] == 
{
+        "1": JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT + 1
+    }
 
 
 @pytest.mark.parametrize(
@@ -3126,14 +3140,14 @@ def 
test_run_reconcile_reports_deferred_job_statuses_only_when_deferring_persist
     reported_file = tmp_path / "reported.json"
     reported_file.write_text("{}")
     alerts = []
-    for deferred in ([1], [1], [], [1], [1], [1]):
+    for deferred in ([1], [1], [], *[[1]] * DEFERRED_RUNS_BEFORE_ALERT):
         mock_reporter.return_value.deferred = deferred
         mock_reporter.return_value.delayed = []
         alerts.append(_reconcile_with_reporter(client, tmp_path, 
reported_file=reported_file))
 
-    assert alerts == [False, False, False, False, False, True]
+    assert alerts == [*[False] * (DEFERRED_RUNS_BEFORE_ALERT + 2), True]
     assert JOB_STATUSES_DEFERRED_NOTE in (tmp_path / "s.json").read_text()
-    assert json.loads(reported_file.read_text())["deferred_runs"] == 3
+    assert json.loads(reported_file.read_text())["deferred_runs"] == 
DEFERRED_RUNS_BEFORE_ALERT
 
 
 def test_build_job_statuses_treats_contexts_case_insensitively():
@@ -3745,7 +3759,10 @@ def 
test_run_reconcile_retries_a_pull_request_whose_head_moved(client, tmp_path)
 
 @pytest.mark.parametrize(
     ("fails", "job_status_budget"),
-    [pytest.param(False, 38, id="reported"), pytest.param(True, 20, 
id="fails")],
+    [
+        pytest.param(False, MAX_JOB_STATUSES_PER_RUN - 2, id="reported"),
+        pytest.param(True, MAX_JOB_STATUSES_PER_RUN - 
MAX_CI_RUN_STARTED_PER_RUN, id="fails"),
+    ],
 )
 @mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", 
autospec=True)
 
@mock.patch("airflow_breeze.utils.fork_ci_reconcile.report_project_ci_runs_started",
 autospec=True)
@@ -3782,7 +3799,7 @@ def 
test_run_reconcile_marks_older_pilot_pull_requests_and_reports_project_ci_ru
 
     assert mock_classify.call_args.args[0].ci_run_at == datetime(2026, 11, 2, 
tzinfo=UTC)
     mock_report.assert_called_once_with(client, repo=REPO, skip={1})
-    mock_reporter.assert_called_once_with(client, repo=REPO, 
budget=job_status_budget, now=NOW)
+    mock_reporter.assert_called_once_with(client, repo=REPO, 
budget=job_status_budget, now=NOW, read_jobs={})
     assert has_alerts is fails
     if fails:
         assert "reporting `CI run started` for project CI failed: GitHubError" 
in output.read_text()
@@ -3863,13 +3880,13 @@ def 
test_run_reconcile_reports_rate_limiting_only_when_it_persists(
     reported_file = tmp_path / "reported.json"
     reported_file.write_text("{}")
     alerts = []
-    for rate_limited in (1, 0, 3, 2):
+    for rate_limited in (1, 0, *[3] * RATE_LIMITED_RUNS_BEFORE_ALERT):
         client.rate_limited = rate_limited
         alerts.append(_reconcile_with_reporter(client, tmp_path, 
reported_file=reported_file))
 
-    assert alerts == [False, False, False, True]
+    assert alerts == [*[False] * (RATE_LIMITED_RUNS_BEFORE_ALERT + 1), True]
     assert RATE_LIMITED_NOTE in (tmp_path / "s.json").read_text()
-    assert json.loads(reported_file.read_text())["rate_limited_runs"] == 2
+    assert json.loads(reported_file.read_text())["rate_limited_runs"] == 
RATE_LIMITED_RUNS_BEFORE_ALERT
 
 
 @pytest.mark.parametrize(
@@ -4428,7 +4445,7 @@ def 
test_run_reconcile_reports_project_ci_passed_after_the_loop(
 
     names = [c[0] for c in order.mock_calls]
     assert names.index("report") > names.index("set_status")
-    assert mock_report_project_ci_passed.call_args.kwargs["budget"] == 5
+    assert mock_report_project_ci_passed.call_args.kwargs["budget"] == 
MAX_PROJECT_CI_PASSED_PER_RUN
 
 
 def 
test_run_reconcile_reports_project_ci_passed_failure(mock_report_project_ci_passed,
 client, tmp_path):
@@ -4462,3 +4479,176 @@ def 
test_report_project_ci_passed_counts_only_the_successes_that_were_written(cl
     client.set_status.side_effect = [GitHubError("HTTP 502"), None, None]
 
     assert report_project_ci_passed(client, repo=REPO, attention=[], budget=1) 
== [2]
+
+
+def test_find_failed_jobs_lists_finished_jobs_that_did_not_pass():
+    jobs = [
+        _fork_job("Static checks"),
+        _fork_job("Postgres tests", conclusion="failure"),
+        _fork_job("Helm tests", status="in_progress", conclusion=None),
+        _fork_job("Docs", conclusion="timed_out"),
+        _fork_job("Java SDK tests", conclusion="skipped"),
+        _fork_job("Platform: AMD", conclusion="failure"),
+        _fork_job("Postgres tests", conclusion="cancelled", 
url="https://job/second";),
+        _fork_job("MySQL tests", conclusion="cancelled", 
url="https://job/cancelled";),
+        _fork_job("MySQL tests", conclusion="failure"),
+    ]
+
+    assert find_failed_jobs(jobs) == [
+        FailedJob("MySQL tests", "https://job/MySQL tests", "failure"),
+        FailedJob("Postgres tests", "https://job/Postgres tests", "failure"),
+        FailedJob("Docs", "https://job/Docs";, "timed_out"),
+    ]
+
+
[email protected](
+    ("read_jobs", "side_effect", "expected", "reads"),
+    [
+        pytest.param({1: [_fork_job("A", conclusion="failure")]}, None, ["A"], 
False, id="read-this-run"),
+        pytest.param({}, [[_fork_job("B", conclusion="failure")]], ["B"], 
True, id="not-read-yet"),
+        pytest.param({}, GitHubError("HTTP 502"), None, True, id="unreadable"),
+        pytest.param({1: None}, None, None, False, id="read-failed-this-run"),
+    ],
+)
+def test_read_failed_jobs(client, read_jobs, side_effect, expected, reads):
+    client.get_run_jobs.side_effect = side_effect
+
+    failed = read_failed_jobs(client, _pr(), COMPLETED_RUN, read_jobs)
+
+    assert (None if failed is None else [job.name for job in failed]) == 
expected
+    assert client.get_run_jobs.called is reads
+    assert 1 in read_jobs
+
+
+def test_a_failed_jobs_read_is_not_repeated_for_the_state_comment(client):
+    client.rest_get.return_value = []
+    client.get_run_jobs.side_effect = GitHubError("HTTP 502")
+    read_jobs: dict = {}
+    reporter = JobStatusReporter(client, repo=REPO, budget=10, 
read_jobs=read_jobs)
+
+    with pytest.raises(GitHubError):
+        reporter.report(_pr(), COMPLETED_RUN)
+
+    assert read_failed_jobs(client, _pr(), COMPLETED_RUN, read_jobs) is None
+    client.get_run_jobs.assert_called_once_with("someone/airflow", 1)
+
+
+def test_job_status_reporter_keeps_the_jobs_it_read(client):
+    jobs = [_fork_job("Static checks", conclusion="failure")]
+    reporter = JobStatusReporter(_reporter_client(client, jobs), repo=REPO, 
budget=10)
+
+    reporter.report(_pr(), COMPLETED_RUN)
+
+    assert reporter.read_jobs == {1: jobs}
+
+
+FAILED_JOB = FailedJob("Static checks", "https://job/1";, "failure")
+
+
[email protected](
+    ("fork", "failed_jobs", "called", "listed"),
+    [
+        pytest.param(COMPLETED_RUN, [FAILED_JOB], True, True, id="failed"),
+        pytest.param(
+            dataclasses.replace(COMPLETED_RUN, 
suite_conclusion="STARTUP_FAILURE"),
+            [FAILED_JOB],
+            False,
+            False,
+            id="startup-failure",
+        ),
+        pytest.param(RUNNING_RUN, [FAILED_JOB], False, False, id="running"),
+    ],
+)
+def test_state_comment_lists_the_jobs_of_a_failed_run(fork, failed_jobs, 
called, listed):
+    find_failed = mock.Mock(return_value=failed_jobs)
+
+    plan = classify(
+        _pr(),
+        fork,
+        config=CONFIG,
+        now=NOW,
+        repo=REPO,
+        find_bypass_run=lambda: None,
+        changes_ci_files=lambda: False,
+        find_failed_jobs=find_failed,
+    )
+
+    assert find_failed.called is called
+    assert ("- [`Static checks`](https://job/1): failed" in 
plan.marker_comment) is listed
+
+
[email protected](
+    ("marker", "kept"),
+    [
+        pytest.param(_marker("failed"), True, id="failed-this-head"),
+        pytest.param(_marker("failed", sha="older"), False, 
id="failed-earlier-head"),
+        pytest.param(_marker("running"), False, id="running"),
+        pytest.param(None, False, id="no-comment"),
+    ],
+)
+def 
test_state_comment_is_left_for_the_next_run_only_when_it_shows_the_head_failed(marker,
 kept):
+    plan = classify(
+        _pr(is_draft=False, labels=("fork ci required", "ready for maintainer 
review"), marker=marker),
+        COMPLETED_RUN,
+        config=CONFIG,
+        now=NOW,
+        repo=REPO,
+        find_bypass_run=lambda: None,
+        changes_ci_files=lambda: False,
+        find_failed_jobs=lambda: None,
+    )
+
+    assert plan.state == ForkState.FAILED
+    assert plan.status == ("failure", RUN_URL, "Fork CI failed - see Details")
+    assert plan.remove_labels == ["ready for maintainer review"]
+    if kept:
+        assert plan.marker_comment is None
+    else:
+        assert parse_marker(plan.marker_comment).state == "failed"
+        assert parse_marker(plan.marker_comment).failed_run is None
+        assert "Jobs that did not pass" not in plan.marker_comment
+
+
+def _classify_failed(pr, fork=COMPLETED_RUN, find_failed=None):
+    return classify(
+        pr,
+        fork,
+        config=CONFIG,
+        now=NOW,
+        repo=REPO,
+        find_bypass_run=lambda: None,
+        changes_ci_files=lambda: False,
+        find_failed_jobs=find_failed or (lambda: [FAILED_JOB]),
+    )
+
+
[email protected](
+    ("fork", "marker_overrides", "reads"),
+    [
+        pytest.param(COMPLETED_RUN, {}, False, id="unchanged"),
+        pytest.param(dataclasses.replace(COMPLETED_RUN, run_attempt=2), {}, 
True, id="re-run"),
+        pytest.param(COMPLETED_RUN, {"failed_run": None}, True, 
id="not-listed-yet"),
+    ],
+)
+def test_jobs_of_a_listed_failed_run_are_not_read_again(fork, 
marker_overrides, reads):
+    pr = _pr(is_draft=False, labels=("fork ci required",))
+    listed = _classify_failed(pr).marker_comment
+    marker = dataclasses.replace(parse_marker(listed), **marker_overrides)
+    find_failed = mock.Mock(return_value=[FAILED_JOB])
+
+    plan = _classify_failed(
+        dataclasses.replace(pr, marker=marker, marker_comment_body=listed), 
fork, find_failed
+    )
+
+    assert parse_marker(listed).failed_run == "1:1"
+    assert find_failed.called is reads
+    if not reads:
+        assert plan.marker_comment is None
+
+
+def test_monitor_interval_is_the_monitor_workflow_cadence():
+    workflow = yaml.safe_load((AIRFLOW_ROOT_PATH / 
".github/workflows/fork-ci-monitor.yml").read_text())
+
+    crons = [entry["cron"] for entry in workflow[True]["schedule"]]
+
+    assert crons == [f"*/{MONITOR_INTERVAL // timedelta(minutes=1)} * * * *"]

Reply via email to