shahar1 opened a new pull request, #76690: URL: https://github.com/apache/airflow/pull/76690
### Why Five jobs run on every `Tests (AMD)` PR run, whatever the PR changes. Selective checks do not gate them, or gate them with `run-generic-basic-tests`, which is always `true`. The data comes from the jobs of 3754 failed `Tests (AMD)` PR runs (3137 attempts) from 2026-09-08 to 2026-11-07, and from 14,451 PR runs in that window (237 a day). Durations are medians of successful jobs. | Job | Runs in % of attempts | Median duration | Runner minutes a day | |---|---|---|---| | `Check that image builds quickly` | 83% | 408 s | About 1,340 | | `Publish documentation and validate versions` | 68% | 369 s | About 990 | | `Shared distributions tests` | 96% | 43 s | About 160 | | `Check translation completeness` | 96% | 27 s | About 100 | | `Platform: AMD` | 96% | 4 s | About 15 | Most of these runs test nothing new: - `Check that image builds quickly` fails only when a change breaks the Docker layer cache. Only the files that the image copies before the sources can do that. Breeze lists them in `FILES_FOR_REBUILD_CHECK`. - `Publish documentation` can fail only on a documentation, provider-version or docs-tooling change. A docstring change rebuilds the docs but cannot change what is published. - `Check translation completeness` ends with `|| true`. It never fails. These jobs failed 39 times in 61 days. I read the diff and the log of each failure. The gates below skip 24 of them: 21 came from a broken `main` that failed the same job in 3 to 8 unrelated PRs on the same day, and 3 were network errors. The gates skip no failure that only the PR caused. ### How Each job runs when a file it depends on changed, and on every full-test run, which includes the scheduled canary runs. | Output | The job runs when a changed file is... | |---|---| | `run-image-build-time-check` | an `ENVIRONMENT_FILES` match, `Dockerfile*`, `.dockerignore`, `scripts/docker/`, `hatch_build.py`, `generated/`, `uv.lock`, a `pyproject.toml` or the prek script that regenerates the provider dependencies before the build, and the CI image is built | | `run-docs-publish` | a docs tree, a `provider.yaml`, a `pyproject.toml`, a distribution's version file (`airflow/__init__.py`, `airflow/sdk/__init__.py`, `airflowctl/__init__.py`, `chart/Chart.yaml`), a changelog, the release notes or `devel-common/src/sphinx_exts/`, and the docs are built | | `run-shared-distributions-tests` | `shared/`, `devel-common/`, `dev/mypy/` (a local test dependency of `devel-common`), the root `pyproject.toml` or `uv.lock` | | `run-translation-check` | `airflow-core/src/airflow/ui/public/i18n/` | `run-generic-basic-tests` now controls only the standalone Airflow test, which still runs for every change. The `Platform: AMD` and `Platform: ARM` jobs become one step of `build-info`. `breeze verify` lists the shared tests through the new output instead of always. A test checks that every file in breeze's `FILES_FOR_REBUILD_CHECK` matches a gate. ### Expected result Over the 2175 commits merged to `main` in the window, the gates skip the image build-time check for 83% of the commits, the docs publish for 60%, the shared tests for 82% and the translation check for 80%. Full-test runs are counted as runs. That is about 28 runner-hours a day and about 820 jobs a day. The job count matters as much as the minutes: the ASF policy caps a workflow at 20 concurrent jobs and the project at 250,000 runner minutes a week. The change removes jobs and adds none. ### Risk Two jobs that run on every PR stay as they are. `System Tests` starts a full api-server, so any runtime code, including every provider that registers CLI commands, can break it; the data showed no safe gate for it. `Breeze unit tests` read many repository files, and the import-graph tests of #76686 assert facts about the live provider sources, so a gate would skip a Breeze run that a provider test change can fail. A gate for the Breeze tests (about 3.6 runner-hours a day) needs those tests to use fixtures first. A gate that is too narrow skips a job that a PR would have failed. The 61 days of data show no such case, but the rare jobs failed only 3 to 9 times each in that window. The scheduled canary runs keep every job, so a skipped failure surfaces there. ### Verification | Check | Result | |---|---| | `test_selective_checks.py`, `test_verification_plan.py` and `test_import_graph.py` | 478 passed, including 15 new cases | | Adversarial review, Copilot CLI with `gpt-6.1-sol` | 7 rounds; GO on the final commit. Rounds 1 to 6 widened the shared, docs-publish and image build-time gates and removed the System Tests and Breeze tests gates | | `prek run --from-ref main --stage pre-commit` | Passed | | The four changed workflow files | Parse as YAML; `ci-amd.yml` and `ci-arm.yml` carry the same wiring | The workflow changes have not run in CI. This PR changes `.github/workflows/*.yml`, so its own run uses the full matrix and must confirm them. Stacked on #76686 and #76685. The measurement scripts and the full report are not part of the PR. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Fable 5.1) Generated-by: Claude Code (Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01BVx6g9TNhjph6ctJDkRTLk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
