This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch fork-ci-time-based-status-budget in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6cc47d7a4f05d9d475676aed856875db8527e92e Author: Jarek Potiuk <[email protected]> AuthorDate: Sun Nov 15 23:15:28 2026 +0900 Count passed fork jobs instead of posting a status for each A full-matrix fork run has about 130 jobs, and posting each passed one as its own commit status spent most of the shared write budget on what the aggregate status already says. Failures still get their own status, which is what a reviewer looks for. The time since the previous monitor run is read from GitHub, because the record carried between runs stays unchanged after a failed run or Slack post, which would have let the next run spend that budget again. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq --- dev/AGENTS.md | 3 +- .../doc/ci/08_fork_ci_for_external_contributors.md | 47 +++-- .../src/airflow_breeze/utils/fork_ci_messages.py | 21 +- .../src/airflow_breeze/utils/fork_ci_reconcile.py | 103 +++++++--- dev/breeze/tests/test_fork_ci_messages.py | 35 ++++ dev/breeze/tests/test_fork_ci_reconcile.py | 217 +++++++++++++++++---- 6 files changed, 351 insertions(+), 75 deletions(-) diff --git a/dev/AGENTS.md b/dev/AGENTS.md index d0e6889b90e..166f7592d6f 100644 --- a/dev/AGENTS.md +++ b/dev/AGENTS.md @@ -47,7 +47,8 @@ Document `uv run` (not `python`) as the invocation method in READMEs and instruc ## Fork CI job statuses and GitHub's rate limits The fork CI monitor (`breeze ci fork-ci-reconcile`, run by `.github/workflows/fork-ci-monitor.yml`) -reports every job of a gated pull request's fork run as its own commit status. Those writes are the +reports every failed job of a gated pull request's fork run as its own commit status, and counts the +passed jobs in `Fork CI / Tests (AMD)`. Those writes are the largest share of what the monitor does with the repository's `GITHUB_TOKEN`, whose rate limits every workflow of `apache/airflow` shares (see "Write budget" in `dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md`). diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index ed71a62c95b..f9ca257532c 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -880,7 +880,12 @@ and `gate_bypassed` (`error`) and `failed` (`failure`), yellow for `awaiting_pus when one exists and the fork's Actions page for `ci-amd.yml` otherwise, so a maintainer can open the fork run in one click; for `gate_bypassed` it is the project CI run that was observed. The `description` is a short sentence under 140 characters that always ends with -where to look next. +where to look next. For `running`, `failed`, `green` and `green_unverified` it is followed by the +fork run's passed jobs, such as "Fork CI passed - see Details (127 jobs passed)", since passed jobs +are not posted as statuses of their own (see "Job statuses"). The count is taken from the jobs the +run read for its job statuses; a run that did not read them keeps the count the status shows, and +the count is never carried over from another state. The reconciler reads the state back from the +description without the count, and accepts a count only on those four states. Statuses are immutable and per SHA. The reconciler compares the desired `(state, target_url, description)` with the current one fetched in the pass 2 query and posts only on @@ -894,8 +899,9 @@ monitor run rather than `details_url`. A status honours `target_url`. ### Job statuses -Next to the aggregate status, each job of the fork run is reported as its own commit status, so -the pull request shows which jobs run, failed or passed, each linking to the job in the fork. +Next to the aggregate status, each failed job of the fork run is reported as its own commit +status, so the pull request shows which jobs failed, each linking to the job in the fork; passed +jobs are counted in `Fork CI / Tests (AMD)` instead. The run is the one `Fork CI / Tests (AMD)` reports (its `databaseId` is fetched with the check suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=latest`. @@ -904,9 +910,13 @@ suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=lat the prefix is what keeps a job from posting `Fork CI / Tests (AMD)`, from which the reconciler reads its decisions back, or `CI passed`, which the merge ruleset requires. Jobs that share a context, which GitHub compares case-insensitively, are reported as the worst of them. -- Only finished jobs are reported: a successful one as `success`, and a failed, timed-out, - cancelled or unstartable one as `failure`. Queued and running jobs are not, so each job costs one - write instead of two; `Fork CI / Tests (AMD)` shows the run as running, linking it, and with a +- Only finished jobs are reported: a failed, timed-out, cancelled or unstartable one as + `failure`, and a successful one as `success` only over a status of the same job that is not a + success, such as its failure in an earlier run of the commit, which would otherwise keep + showing. A full-matrix run has about 130 jobs and each status is a write, so posting every + passed job would take most of the budget for what the aggregate status already says; on + 2026-11-15 four such runs at once took about six hours to show. Queued and running jobs are not + reported, so each job costs at most one write; `Fork CI / Tests (AMD)` shows the run as running, linking it, and with a 5-minute cadence a per-job "running" status was behind the fork's own run page anyway. A job of a re-run in the fork keeps the result of the earlier attempt until it finishes. Skipped and neutral jobs, and the jobs every run has (`Fork CI gate`, `Check if the run is needed`, @@ -918,17 +928,17 @@ suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=lat content-creating requests: a reconciler run posts at most its share of `JOB_STATUSES_PER_HOUR` (240) for the time since the previous run, each write charged as it is made (see "Write budget"). What does not fit is posted by a later run, and once the budget is spent no further - pull request's jobs or statuses are read in that run. One fork run of the full matrix (about - 130 jobs) needs several runs' budget on its own, and a few of them at once keep the monitor - deferring for hours while every run makes progress, so deferring alone is not reported: the run - log lists the deferred pull requests. A pull request whose fork run finished more than an hour + pull request's jobs or statuses are read in that run. Deferring alone is not reported, since + later runs catch up and many failing runs at once can keep the monitor deferring while every + run makes progress: the run log lists the deferred pull requests. A pull request whose fork run finished more than an hour ago (the `updatedAt` of its completed check suite) and still has statuses waiting is listed on its own as an attention item, so a delay that a busy backlog keeps pushing back shows which pull request it holds up; the item names the run, so it is posted once for that run. - Commit statuses cannot be deleted and have no neutral state. When a run has completed, a job context posted for an earlier run of the same commit that the completed run does not have, or skipped, is overwritten with `success` and the description "Not in the latest fork run - see - Fork CI / Tests (AMD)", so it no longer shows as running or failed. + Fork CI / Tests (AMD)", so it no longer shows as running or failed; one that is already a + success is left as it is. - Once every job of a completed run is reported, the run is recorded in the marker as `jobs: "<run id>:<attempt>:<conclusion>"` (`runAttempt` is fetched with the check suites), so later runs do not read its jobs again. A run in progress is read on every run, and a re-run in @@ -970,7 +980,9 @@ jobs whose conclusion is not `skipped`, minus the four jobs that are not reporte | CI change (full tests) | [40207881565](https://github.com/apache/airflow/actions/runs/40207881565) | 131 | 127 | `apache/airflow` had 1898 `Tests (AMD)` pull request runs from 2026-10-18 to 2026-10-24, about 270 a -day or 11 an hour. Each reported job costs one write, its result. +day or 11 an hour. Each reported job costs one write, its result. Since passed jobs are only counted +in `Fork CI / Tests (AMD)`, a push costs one write per failed job (plus one per job that passes +after failing), and the estimates below, which assume every job is posted, are an upper bound. The limits that matter are GitHub's documented ones for the `GITHUB_TOKEN` of `apache/airflow`, shared by every workflow of the repository: 1000 REST requests an hour, and 80 content-creating @@ -989,15 +1001,18 @@ requests a minute and 500 an hour. Assuming about 35 reported jobs for an averag run (see "The `CI run started` check"), so they do not raise the total; at about 11 pull request runs an hour they take about 1 write a run. -A run gets the hourly budget for the time since the previous run, which it reads from the record -the monitor carries between runs (`run_at` in `--reported-file`), not one twelfth of it. GitHub +A run gets the hourly budget for the time since the previous monitor run started, not one twelfth +of it. That time is read from the monitor's runs (`GET +/repos/apache/airflow/actions/workflows/fork-ci-monitor.yml/runs`, runs still waiting in its +concurrency group left out), not from the record the monitor carries between runs, which a failed +run or a failed Slack post leaves unchanged although that run may have posted statuses. GitHub starts scheduled workflows later than their cron: on 2026-11-15 the 5-minute monitor ran every 9 to 10 minutes, so a fixed 13 writes a run (160 an hour at the cron's cadence) posted only about 82 an hour, and four full-matrix fork runs at once took about six hours to catch up. Runs queued behind a slow one get little, since the time between them is short. At most `MAX_JOB_STATUS_BUDGET_INTERVAL` (10 minutes) is counted, so a run after a gap, such as a -disabled workflow or a lost record, posts at most 40 statuses at about one a second: half of the -80-a-minute limit. Without a readable record a run gets one cron interval's share, 20 writes. +disabled workflow, posts at most 40 statuses at about one a second: half of the 80-a-minute +limit. When the previous run cannot be read, a run gets one cron interval's share, 20 writes. The per-run caps and the run counts before a Slack note are derived from `MONITOR_INTERVAL` in `fork_ci_reconcile.py`, which must match the monitor's cron (a test checks it), so changing the diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py index 7378ad7f368..b313399a798 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py @@ -562,5 +562,22 @@ STATUS_STATES = { } -def status_description(state: str) -> str: - return _STATUS_DESCRIPTIONS[state] +# States whose `Fork CI / Tests (AMD)` description counts the fork run's passed jobs, which are not +# posted as job statuses of their own. +STATES_WITH_PASSED_JOBS = frozenset({"running", "failed", "green", "green_unverified"}) +_PASSED_JOBS_SUFFIX = re.compile(r" \((\d+) jobs? passed\)\Z") + + +def status_description(state: str, passed_jobs: int | None = None) -> str: + description = _STATUS_DESCRIPTIONS[state] + if passed_jobs is None or state not in STATES_WITH_PASSED_JOBS: + return description + return f"{description} ({passed_jobs} job{'' if passed_jobs == 1 else 's'} passed)" + + +def split_passed_jobs(description: str) -> tuple[str, int | None]: + """A status description without its passed-jobs count, and the count; None without one.""" + match = _PASSED_JOBS_SUFFIX.search(description) + if match is None: + return description, None + return description[: match.start()], int(match[1]) diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py index bd25b5469ec..d1501093389 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py @@ -68,6 +68,7 @@ from airflow_breeze.utils.fork_ci_messages import ( JOB_STATUS_CONTEXT_PREFIX, JOB_STATUSES, STALE_JOB_STATUS, + STATES_WITH_PASSED_JOBS, STATUS_CONTEXT, STATUS_STATES, BaseCheck, @@ -83,6 +84,7 @@ from airflow_breeze.utils.fork_ci_messages import ( parse_marker, rebase_needed_description, released_comment, + split_passed_jobs, state_comment, status_description, strip_failed_job_list, @@ -203,10 +205,15 @@ JOB_STATE_RANKS = {"failure": 0, "pending": 1, "success": 2} # later run, failures first. See "Write budget" in the design doc. JOB_STATUSES_PER_HOUR = 240 # GitHub starts scheduled runs later than their cron, the monitor often only every 9-10 minutes, so a -# run gets the hourly budget for the time since the previous run rather than for MONITOR_INTERVAL. At +# run gets the hourly budget for the time since the previous run started rather than for +# MONITOR_INTERVAL. At # most this long is counted, so a run after a gap (a disabled workflow, a lost record) stays at half # of GitHub's limit of 80 content-creating requests a minute. MAX_JOB_STATUS_BUDGET_INTERVAL = timedelta(minutes=10) +MONITOR_WORKFLOW_FILE = "fork-ci-monitor.yml" +# Enough to find the previous started run behind the runs queued in the monitor's concurrency group. +MONITOR_RUNS_READ = 10 +NOT_STARTED_RUN_STATUSES = frozenset({"queued", "waiting", "pending", "requested"}) def compute_job_status_budget(since_previous_run: timedelta | None) -> int: @@ -606,20 +613,31 @@ def status_state(status: tuple[str, str, str] | None) -> str | None: """The state the reconciler recorded in a commit status it posted, if it posted it.""" if status is None: return None + description, passed_jobs = split_passed_jobs(status[2]) return next( ( state for state, github_state in STATUS_STATES.items() - if (github_state, status_description(state)) == (status[0], status[2]) + if (github_state, status_description(state)) == (status[0], description) + and (passed_jobs is None or state in STATES_WITH_PASSED_JOBS) ), None, ) def _set_status( - plan: Plan, pr: PullRequestSnapshot, state: str, target_url: str, *, force: bool = False + plan: Plan, + pr: PullRequestSnapshot, + state: str, + target_url: str, + *, + force: bool = False, + passed_jobs: int | None = None, ) -> None: - desired = (STATUS_STATES[state], target_url, status_description(state)) + """:param passed_jobs: the fork run's passed jobs; None keeps the count the status already shows.""" + if passed_jobs is None and pr.current_status is not None and status_state(pr.current_status) == state: + passed_jobs = split_passed_jobs(pr.current_status[2])[1] + desired = (STATUS_STATES[state], target_url, status_description(state, passed_jobs)) if force or desired != pr.current_status: plan.status = desired @@ -671,6 +689,7 @@ def classify( find_gate_change: Callable[[], str | None] = lambda: None, find_failed_jobs: Callable[[], list[FailedJob] | None] = lambda: [], verify_head_repo_unavailable: Callable[[], bool] = lambda: False, + passed_jobs: int | None = None, ) -> Plan: """ Decide the mutations for one gated pull request. @@ -696,6 +715,8 @@ def classify( :param find_failed_jobs: the jobs of the fork run that did not pass, None when they cannot be read; only called for a failed run that started. On None the state comment is left for the next run, so that a failed read does not drop the list from it. + :param passed_jobs: the passed jobs of the fork run, shown in `Fork CI / Tests (AMD)`; None when + this run did not read its jobs, which keeps the count the status shows. """ plan = Plan() labels = set(pr.labels) @@ -912,7 +933,12 @@ def classify( target_url = fork.run_url or fork_actions_url(pr.head_repo) startup_failure = state == ForkState.FAILED and fork.suite_conclusion == "STARTUP_FAILURE" _set_status( - plan, pr, "startup_failure" if startup_failure else state.value, target_url, force=refresh_status + plan, + pr, + "startup_failure" if startup_failure else state.value, + target_url, + force=refresh_status, + passed_jobs=passed_jobs, ) _set_ci_passed(plan, pr, state, base, target_url=target_url) _set_ci_run_started(plan, pr, fork) @@ -1854,6 +1880,23 @@ def fetch_fork_observations( return observations +def count_passed_jobs(jobs: Iterable[dict[str, Any]]) -> int: + """The finished, successful jobs of a fork run that job statuses would report.""" + return sum( + 1 + for job in jobs + if job["name"] not in UNREPORTED_JOB_NAMES + and job.get("status") not in RUNNING_JOB_STATUSES + and job.get("conclusion") == "success" + ) + + +def find_passed_jobs(fork: ForkObservation, read_jobs: ReadJobs) -> int | None: + """The passed jobs of the fork run, when this reconciler run read its jobs.""" + jobs = read_jobs.get(fork.run_id) if fork.run_id is not None else None + return count_passed_jobs(jobs) if jobs is not None else None + + def build_job_statuses(jobs: Iterable[dict[str, Any]]) -> dict[str, tuple[str, str, str]]: """The ``(state, target URL, description)`` to report for each finished fork job, by context.""" statuses: dict[str, tuple[str, str, str]] = {} @@ -2015,6 +2058,15 @@ class JobStatusReporter: for folded, (context, (_, target_url, _)) in current.items(): if context.startswith(JOB_STATUS_CONTEXT_PREFIX) and folded not in wanted: desired[context] = (STALE_JOB_STATUS[0], target_url, STALE_JOB_STATUS[1]) + # Passed jobs are only counted in `Fork CI / Tests (AMD)`: a full-matrix run has about 130, and + # each status is a write. A success is posted only over a job's failure from an earlier run of + # the commit, which would otherwise keep showing. + desired = { + context: status + for context, status in desired.items() + if status[0] != "success" + or current.get(context.casefold(), ("", ("success", "", "")))[1][0] != "success" + } changes = sorted( ( (context, status) @@ -2726,7 +2778,6 @@ def drop_reported( attention: list[tuple[int, str]], notes: list[str], record: bool = True, - run_at: datetime | None = None, rate_limited_runs: int = 0, job_status_failure_runs: dict[int, int] | None = None, ) -> tuple[list[tuple[int, str]], list[tuple[int, str]], list[str]]: @@ -2736,7 +2787,6 @@ def drop_reported( Only the previous run's items are kept, so an item that went away and comes back is posted again. A missing or unreadable file reports everything. - :param run_at: when this run started, recorded for the next run's job status budget. :param rate_limited_runs: the consecutive runs GitHub rate-limited, recorded for the next run. :param job_status_failure_runs: per pull request, the consecutive runs whose job statuses failed, recorded for the next run. @@ -2745,7 +2795,6 @@ def drop_reported( "unknown": [list(item) for item in unknown], "attention": [list(item) for item in attention], "notes": notes, - "run_at": run_at.isoformat() if run_at is not None else None, "rate_limited_runs": rate_limited_runs, "job_status_failure_runs": { str(number): runs for number, runs in (job_status_failure_runs or {}).items() @@ -2782,22 +2831,30 @@ def read_run_count(reported_file: Path | None, key: str) -> int | None: return count if isinstance(count, int) and not isinstance(count, bool) else 0 -def read_previous_run_at(reported_file: Path | None) -> datetime | None: - """When the previous run started, as it recorded; None without a readable record of it.""" - if reported_file is None: - return None - try: - previous = json.loads(reported_file.read_text()) - except (OSError, ValueError): - return None - run_at = previous.get("run_at") if isinstance(previous, dict) else None - if not isinstance(run_at, str): +def find_previous_monitor_run_start(client: GitHubClient, repo: str, *, run_url: str) -> datetime | None: + """ + When the monitor's latest run other than this one started; None when it cannot be told. + + Read from GitHub rather than from the record the monitor carries between runs, which a failed run + or a failed Slack post leaves unchanged although that run may have posted statuses. + """ + match = re.search(r"/actions/runs/(\d+)", run_url) + if match is None: return None try: - parsed = datetime.fromisoformat(run_at) - except ValueError: + runs = client.rest_get( + f"repos/{repo}/actions/workflows/{MONITOR_WORKFLOW_FILE}/runs?per_page={MONITOR_RUNS_READ}" + )["workflow_runs"] + except Exception as e: + console_print(f"[warning]Could not read the monitor's earlier runs: {e}[/]") return None - return parsed if parsed.tzinfo is not None else None + # Runs waiting in the concurrency group have not started, so they posted nothing. + starts = [ + parse_github_datetime(run.get("run_started_at")) + for run in runs + if run["id"] != int(match[1]) and run.get("status") not in NOT_STARTED_RUN_STATUSES + ] + return max((start for start in starts if start is not None), default=None) def read_job_status_failure_runs(reported_file: Path | None) -> dict[int, int]: @@ -2928,7 +2985,7 @@ def run_reconcile( f"listing the pull requests with `{LABEL_USE_PROJECT_CI}` failed: {type(e).__name__}" ) read_jobs: ReadJobs = {} - previous_run_at = read_previous_run_at(reported_file) + previous_run_at = find_previous_monitor_run_start(client, repo, run_url=run_url) job_status_budget = compute_job_status_budget( now - previous_run_at if previous_run_at is not None else None ) @@ -2991,6 +3048,7 @@ def run_reconcile( verify_head_repo_unavailable=functools.partial( check_head_repo_unavailable, client, repo, pr.number ), + passed_jobs=find_passed_jobs(fork, read_jobs), ) # A revocation to `pending` always goes through: it is idempotent and must not depend on # `report_project_ci_passed` succeeding later in this run. @@ -3063,7 +3121,6 @@ def run_reconcile( attention=attention, notes=notes, record=not client.dry_run, - run_at=now, rate_limited_runs=rate_limited_runs, job_status_failure_runs=job_status_failure_runs, ) diff --git a/dev/breeze/tests/test_fork_ci_messages.py b/dev/breeze/tests/test_fork_ci_messages.py index d945dfe3576..9077a9e16a7 100644 --- a/dev/breeze/tests/test_fork_ci_messages.py +++ b/dev/breeze/tests/test_fork_ci_messages.py @@ -38,6 +38,7 @@ from airflow_breeze.utils.fork_ci_messages import ( MAX_LISTED_FAILED_JOBS, OPEN_PR_LIMIT_URL, OVERLAP_LISTED, + STATES_WITH_PASSED_JOBS, BaseCheck, FailedJob, Marker, @@ -53,7 +54,9 @@ from airflow_breeze.utils.fork_ci_messages import ( rebase_needed_description, released_comment, render_marker, + split_passed_jobs, state_comment, + status_description, strip_failed_job_list, tamper_close_comment, ungated_comment, @@ -298,10 +301,42 @@ def test_setup_texts_give_the_exact_command(text): assert "gh workflow enable ci-amd.yml -R u/fork" in text [email protected]( + ("state", "passed_jobs", "expected"), + [ + pytest.param("green", 130, "Fork CI passed - see Details (130 jobs passed)", id="counted"), + pytest.param("running", 1, "Fork CI running - see Details (1 job passed)", id="one-job"), + pytest.param("green", None, "Fork CI passed - see Details", id="not-counted"), + pytest.param( + "setup_required", 3, "Fork CI not set up - see the pull request comment", id="state-without-a-run" + ), + ], +) +def test_status_description_counts_passed_jobs(state, passed_jobs, expected): + assert status_description(state, passed_jobs) == expected + + [email protected]( + ("description", "expected"), + [ + pytest.param("Fork CI passed - see Details (130 jobs passed)", ("Fork CI passed - see Details", 130)), + pytest.param("Fork CI running - see Details (1 job passed)", ("Fork CI running - see Details", 1)), + pytest.param("Fork CI passed - see Details", ("Fork CI passed - see Details", None)), + pytest.param( + "Fork CI passed (2 jobs passed) - see Details", + ("Fork CI passed (2 jobs passed) - see Details", None), + ), + ], +) +def test_split_passed_jobs(description, expected): + assert split_passed_jobs(description) == expected + + def test_status_descriptions_fit_github_limit(): base_ref = "providers-cncf-kubernetes/v10-22" descriptions = [ *_STATUS_DESCRIPTIONS.values(), + *(status_description(state, 99999) for state in STATES_WITH_PASSED_JOBS), CI_PASSED_REVOKED_DESCRIPTION, ci_passed_description(BaseCheck("d" * 40, 199), base_ref=base_ref), *( diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py b/dev/breeze/tests/test_fork_ci_reconcile.py index 79f177df73c..51221295180 100644 --- a/dev/breeze/tests/test_fork_ci_reconcile.py +++ b/dev/breeze/tests/test_fork_ci_reconcile.py @@ -77,6 +77,7 @@ from airflow_breeze.utils.fork_ci_reconcile import ( compare_with_base, compute_job_status_budget, compute_projection, + count_passed_jobs, discover, drop_reported, fetch_fork_observations, @@ -85,6 +86,8 @@ from airflow_breeze.utils.fork_ci_reconcile import ( find_ci_run_since, find_failed_jobs, find_heads_shared_across_bases, + find_passed_jobs, + find_previous_monitor_run_start, find_project_ci_label_heads, find_project_ci_passed_url, find_project_ci_run_url, @@ -102,12 +105,12 @@ from airflow_breeze.utils.fork_ci_reconcile import ( read_failed_jobs, read_head_status, read_job_status_failure_runs, - read_previous_run_at, read_project_ci_passed_url, read_project_ci_run_url, report_project_ci_passed, report_project_ci_runs_started, run_reconcile, + status_state, ) from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH @@ -655,6 +658,20 @@ def test_green_records_the_base_it_tested(pr_overrides, compared, expected_base) ), pytest.param(None, _marker("green"), ForkState.GREEN, True, id="marker-is-not-trusted"), pytest.param(("success", RUN_URL, "forged"), None, ForkState.GREEN, True, id="unknown-status"), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed)"), + None, + ForkState.GREEN, + False, + id="status-green-with-passed-jobs", + ), + pytest.param( + (UNVERIFIED_STATUS[0], RUN_URL, f"{UNVERIFIED_STATUS[2]} (1 job passed)"), + None, + ForkState.GREEN_UNVERIFIED, + False, + id="status-unverified-with-passed-jobs", + ), ], ) def test_green_trust_is_decided_once_per_commit(current_status, marker, expected_state, lists_files): @@ -699,6 +716,82 @@ def test_green_is_not_trusted_for_a_head_shared_across_bases(): changes.assert_not_called() [email protected]( + ("status", "expected"), + [ + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (12 jobs passed)"), "green", id="green" + ), + pytest.param( + ("pending", RUN_URL, "Fork CI running - see Details (1 job passed)"), "running", id="running" + ), + pytest.param( + ("error", RUN_URL, "Project CI ran for a gated pull request - see Details (3 jobs passed)"), + None, + id="count-on-a-state-without-one", + ), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (many jobs passed)"), None, id="no-number" + ), + ], +) +def test_status_state_reads_a_passed_jobs_count(status, expected): + assert status_state(status) == expected + + [email protected]( + ("current_status", "passed_jobs", "expected_status"), + [ + pytest.param( + None, 130, ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed)"), id="counted" + ), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed)"), + None, + None, + id="count-kept-when-jobs-not-read", + ), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (129 jobs passed)"), + 130, + ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed)"), + id="count-updated", + ), + ], +) +def test_fork_status_shows_the_passed_jobs(current_status, passed_jobs, expected_status): + plan = classify( + _pr(current_status=current_status), + GREEN, + config=CONFIG, + now=NOW, + repo=REPO, + find_bypass_run=lambda: None, + changes_ci_files=lambda: False, + compare_base=lambda: CURRENT_BASE, + passed_jobs=passed_jobs, + ) + + assert plan.state == ForkState.GREEN + assert plan.status == expected_status + + +def test_fork_status_does_not_carry_a_count_into_another_state(): + running = ("pending", RUN_URL, "Fork CI running - see Details (40 jobs passed)") + plan = classify( + _pr(current_status=running), + GREEN, + config=CONFIG, + now=NOW, + repo=REPO, + find_bypass_run=lambda: None, + changes_ci_files=lambda: False, + compare_base=lambda: CURRENT_BASE, + ) + + assert plan.status == GREEN_STATUS + + def test_find_heads_shared_across_bases(): prs = [ _pr(number=1, head_sha="a", base_ref="main"), @@ -2790,7 +2883,6 @@ def test_drop_reported_reports_everything_without_a_readable_previous_run(tmp_pa "unknown": [[1, "a"]], "attention": [], "notes": [], - "run_at": None, "rate_limited_runs": 0, "job_status_failure_runs": {}, } @@ -3030,16 +3122,16 @@ def _reporter_client(client, jobs, current=()): def test_job_status_reporter_posts_changed_statuses_failures_first(client): - already = { - "context": f"{JOB_STATUS_CONTEXT_PREFIX}Static checks", - "state": "success", - "target_url": "https://job/Static checks", - "description": "Passed in the fork - see Details", + failed_before = { + "context": f"{JOB_STATUS_CONTEXT_PREFIX}Helm tests", + "state": "failure", + "target_url": "https://job/Helm tests", + "description": "Failed in the fork - see Details", } _reporter_client( client, [_fork_job("Static checks"), _fork_job("Helm tests"), _fork_job("Unit", conclusion="failure")], - [already], + [failed_before], ) key = JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) @@ -3052,7 +3144,7 @@ def test_job_status_reporter_posts_changed_statuses_failures_first(client): def test_job_status_reporter_leaves_what_does_not_fit_in_the_budget_for_the_next_run(client): - _reporter_client(client, [_fork_job("A", conclusion="failure"), _fork_job("B"), _fork_job("C")]) + _reporter_client(client, [_fork_job(name, conclusion="failure") for name in "ABC"]) reporter = JobStatusReporter(client, repo=REPO, budget=2) assert reporter.report(_pr(), COMPLETED_RUN) is None @@ -3065,7 +3157,9 @@ def test_job_status_reporter_leaves_what_does_not_fit_in_the_budget_for_the_next def test_job_status_reporter_does_not_record_a_run_in_progress(client): - _reporter_client(client, [_fork_job("A"), _fork_job("B", status="in_progress", conclusion=None)]) + _reporter_client( + client, [_fork_job("A", conclusion="failure"), _fork_job("B", status="in_progress", conclusion=None)] + ) assert JobStatusReporter(client, repo=REPO).report(_pr(), RUNNING_RUN) is None client.set_status.assert_called_once() @@ -3195,7 +3289,7 @@ def test_job_status_reporter_leaves_other_contexts_alone(client): def test_job_status_reporter_charges_each_write_even_when_a_later_one_fails(client): - _reporter_client(client, [_fork_job("A", conclusion="failure"), _fork_job("B"), _fork_job("C")]) + _reporter_client(client, [_fork_job(name, conclusion="failure") for name in "ABC"]) client.set_status.side_effect = [None, GitHubError("HTTP 500")] reporter = JobStatusReporter(client, repo=REPO, budget=10) @@ -3226,6 +3320,21 @@ def _reconcile_with_reporter(client, tmp_path, config=CONFIG, reported_file=None ) [email protected]("airflow_breeze.utils.fork_ci_reconcile.find_passed_jobs", autospec=True) [email protected]("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) [email protected]("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) +def test_run_reconcile_shows_the_passed_jobs(mock_classify, mock_reporter, mock_passed, client, tmp_path): + mock_classify.return_value = Plan(state=ForkState.RUNNING) + mock_reporter.return_value.report.return_value = None + mock_reporter.return_value.deferred = [] + mock_reporter.return_value.delayed = [] + mock_passed.return_value = 17 + + _reconcile_with_reporter(client, tmp_path) + + assert mock_classify.call_args.kwargs["passed_jobs"] == 17 + + @mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) def test_run_reconcile_classifies_when_reporting_job_statuses_fails( @@ -3350,50 +3459,92 @@ def test_job_status_budget_after_a_gap_is_capped(): ) +MONITOR_RUN_URL = "https://github.com/apache/airflow/actions/runs/100" + + +def _monitor_run(run_id, started, status="completed"): + return {"id": run_id, "status": status, "run_started_at": started} + + @pytest.mark.parametrize( - ("content", "expected"), + ("runs", "expected"), [ - pytest.param(None, None, id="missing"), - pytest.param("not json", None, id="not-json"), - pytest.param("[]", None, id="not-a-dict"), - pytest.param("{}", None, id="recorded-before-the-time"), - pytest.param('{"run_at": 5}', None, id="not-a-string"), - pytest.param('{"run_at": "yesterday"}', None, id="not-a-time"), - pytest.param('{"run_at": "2026-11-10T11:51:00"}', None, id="without-a-timezone"), - pytest.param( - '{"run_at": "2026-11-10T11:51:00+00:00"}', + pytest.param( + [ + _monitor_run(102, "2026-11-10T11:59:00Z", status="pending"), + _monitor_run(100, "2026-11-10T11:58:00Z", status="in_progress"), + _monitor_run(99, "2026-11-10T11:51:00Z"), + _monitor_run(98, "2026-11-10T11:42:00Z"), + ], datetime(2026, 11, 10, 11, 51, tzinfo=UTC), - id="recorded", + id="previous-started-run", ), + pytest.param([_monitor_run(100, "2026-11-10T11:58:00Z")], None, id="first-run"), ], ) -def test_read_previous_run_at(tmp_path, content, expected): - reported_file = tmp_path / "reported.json" - if content is not None: - reported_file.write_text(content) +def test_find_previous_monitor_run_start(client, runs, expected): + client.rest_get.return_value = {"workflow_runs": runs} + + assert find_previous_monitor_run_start(client, REPO, run_url=MONITOR_RUN_URL) == expected + client.rest_get.assert_called_once_with( + f"repos/{REPO}/actions/workflows/fork-ci-monitor.yml/runs?per_page=10" + ) + + +def test_find_previous_monitor_run_start_without_a_run_id(client): + assert find_previous_monitor_run_start(client, REPO, run_url="https://run") is None + client.rest_get.assert_not_called() - assert read_previous_run_at(reported_file) == expected +def test_find_previous_monitor_run_start_when_the_runs_cannot_be_read(client): + client.rest_get.side_effect = GitHubError("gh api failed: HTTP 502") + assert find_previous_monitor_run_start(client, REPO, run_url=MONITOR_RUN_URL) is None + + [email protected]("airflow_breeze.utils.fork_ci_reconcile.find_previous_monitor_run_start", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.report_project_ci_runs_started", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) def test_run_reconcile_budgets_job_statuses_for_the_time_since_the_previous_run( - mock_classify, mock_report, mock_reporter, client, tmp_path + mock_classify, mock_report, mock_reporter, mock_previous, client, tmp_path ): mock_classify.return_value = Plan(state=ForkState.RUNNING) mock_report.return_value = [] mock_reporter.return_value.report.return_value = None mock_reporter.return_value.deferred = [] mock_reporter.return_value.delayed = [] - client.dry_run = False - reported_file = tmp_path / "reported.json" - reported_file.write_text(json.dumps({"run_at": (NOW - timedelta(minutes=9)).isoformat()})) + mock_previous.return_value = NOW - timedelta(minutes=9) - _reconcile_with_reporter(client, tmp_path, reported_file=reported_file) + _reconcile_with_reporter(client, tmp_path) assert mock_reporter.call_args.kwargs["budget"] == compute_job_status_budget(timedelta(minutes=9)) - assert read_previous_run_at(reported_file) == NOW + + +def test_count_passed_jobs(): + jobs = [ + _fork_job("A"), + _fork_job("B"), + _fork_job("C", conclusion="failure"), + _fork_job("D", conclusion="skipped"), + _fork_job("E", status="in_progress", conclusion=None), + _fork_job("Platform: AMD"), + ] + + assert count_passed_jobs(jobs) == 2 + + [email protected]( + ("fork", "read_jobs", "expected"), + [ + pytest.param(COMPLETED_RUN, {1: [_fork_job("A")]}, 1, id="read"), + pytest.param(COMPLETED_RUN, {}, None, id="not-read"), + pytest.param(COMPLETED_RUN, {1: None}, None, id="read-failed"), + pytest.param(ForkObservation(), {1: [_fork_job("A")]}, None, id="no-run"), + ], +) +def test_find_passed_jobs(fork, read_jobs, expected): + assert find_passed_jobs(fork, read_jobs) == expected def test_build_job_statuses_treats_contexts_case_insensitively():
