This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch backport-ci-sync-v3-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4e69211a777e5efc1fc672cc6a179dc16f7ba653 Author: Jarek Potiuk <[email protected]> AuthorDate: Sun Oct 25 19:40:34 2026 +0100 Add breeze ci audit for the fork CI setup (#74823) * Add breeze ci audit for the fork CI setup Fork CI only works when the contributor's remotes, fork and fork workflow are set up the way the gate and the reconciler expect, and committers must keep fork CI off so their pushes do not run twice. A read-only audit lets contributors verify all of that before pushing and offers the exact fix for anything missing. Remote parsing moves out of ci upgrade so both commands agree on what a correct setup is, and it now matches apache/airflow exactly instead of any URL containing it. Generated-by: Claude Opus 5 * Address review of breeze ci audit Decide the role before the remote and fork checks, so a committer working from an apache/airflow clone is not told to set up a fork their pull requests do not use; those checks, and an enabled fork workflow, are informational for roles that run project CI, since a committer trying use fork ci needs the workflow enabled. Report a failed API call as could-not-check instead of a wrong fork or a missing workflow, and recognise ssh host aliases from ~/.ssh/config as GitHub remotes again, a [...] Generated-by: Claude Opus 5 * Classify only public apache members as committers in breeze ci audit The gate decides from the pull request's author association, which reports a private member as CONTRIBUTOR, so their pull requests are gated. The audit asked for any membership, accepted disabled fork CI for them and recommended disabling it. It now checks public membership, and tells a private member to make it public or ask for the allowlist. Generated-by: Claude Opus 5 (cherry picked from commit ce04458488873834bbc2c13b8819ba5e32be31db) --- dev/breeze/doc/08_ci_tasks.rst | 18 + .../doc/ci/08_fork_ci_for_external_contributors.md | 42 +- dev/breeze/doc/images/output_ci.svg | 12 +- dev/breeze/doc/images/output_ci.txt | 2 +- dev/breeze/doc/images/output_ci_audit.svg | 116 ++++++ dev/breeze/doc/images/output_ci_audit.txt | 1 + .../output_setup_check-all-params-in-groups.svg | 28 +- .../output_setup_check-all-params-in-groups.txt | 2 +- .../output_setup_regenerate-command-images.svg | 2 +- .../output_setup_regenerate-command-images.txt | 2 +- .../src/airflow_breeze/commands/ci_commands.py | 93 +++-- .../airflow_breeze/commands/ci_commands_config.py | 9 + dev/breeze/src/airflow_breeze/utils/ci_audit.py | 403 +++++++++++++++++++ dev/breeze/src/airflow_breeze/utils/git_remotes.py | 79 ++++ dev/breeze/tests/test_ci_audit.py | 440 +++++++++++++++++++++ dev/breeze/tests/test_git_remotes.py | 103 +++++ 16 files changed, 1283 insertions(+), 69 deletions(-) diff --git a/dev/breeze/doc/08_ci_tasks.rst b/dev/breeze/doc/08_ci_tasks.rst index 5c85fced90c..ca16b4483fe 100644 --- a/dev/breeze/doc/08_ci_tasks.rst +++ b/dev/breeze/doc/08_ci_tasks.rst @@ -161,6 +161,24 @@ If it cannot determine which milestone should be added, it also adds a comment t :width: 100% :alt: Milestone Tag Assistant +Checking your fork CI setup +--------------------------- + +Pull requests from external contributors can run their CI in the contributor's fork (AIP-120). +``breeze ci audit`` checks, read-only, whether your checkout and fork are set up for that: ``gh`` +installed and authenticated, the ``upstream`` remote pointing at ``apache/airflow`` and ``origin`` at your +fork, whether you are a committer, collaborator, member of an exempted organisation, pilot participant or +external contributor, whether the ``Tests (AMD)`` workflow in your fork is enabled (or disabled, for those +whose pull requests run project CI), how far your fork's ``main`` is behind, and the branch you are on. +Each failing check prints the command that fixes it; ``--fix`` runs those commands after asking. + +These are all available flags of ``audit`` command: + +.. image:: ./images/output_ci_audit.svg + :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_audit.svg + :width: 100% + :alt: Breeze ci audit + Reconciling pull requests gated to fork CI ------------------------------------------ diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index a0ab1a81ada..6321692fc97 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -772,21 +772,39 @@ prints one row: status, finding, and the command that fixes it. | # | Check | Passing condition | Offered fix | |---|---|---|---| -| 1 | `gh` present and authenticated | `gh auth status` succeeds, version at least `MIN_GH_VERSION` | install / `gh auth login` | +| 1 | `gh` present and authenticated | `gh api user` succeeds, version at least `MIN_GH_VERSION` | install / `gh auth login` | | 2 | Remote naming | a remote named `upstream` points at `apache/airflow` and `origin` does not | `git remote rename`/`add` per `contributing-docs/10_working_with_git.rst` | -| 3 | Fork exists | `origin` is a fork of `apache/airflow` owned by the authenticated user | `gh repo fork apache/airflow --remote=false` then add `origin` | -| 4 | Who you are | classifies the user: committer (`GET /user/memberships/orgs/apache` active), collaborator (`.asf.yaml`), exempted (`[allowlist]`), pilot participant (`[pilot]`), external | informational | -| 5 | Fork workflows | external or pilot participant: `ci-amd.yml` state on the fork is `active`; committer/collaborator/exempted: state is not `active` | `gh workflow enable ci-amd.yml -R <fork>` or `gh workflow disable ...` | -| 6 | Fork `main` freshness | fork `main` is not behind `upstream/main` by more than the configured threshold | `gh repo sync <fork> --branch main` | +| 3 | Fork exists | `origin` is a fork of `apache/airflow` owned by the authenticated user | `gh repo fork apache/airflow --remote=false` then add `origin`, unless check 2's fix already adds or repoints it | +| 4 | Who you are | classifies the user in the gate's order: pilot participant (`[pilot]`, which gates whatever the role), committer (public `apache` membership, `GET /orgs/apache/public_members/<login>`, since the gate cannot see private membership), collaborator (`.asf.yaml`), exempted (`[allowlist]`), external; an external user whose `apache` membership is private is told to make it public or ask for `[allowlist]`; when the membership call fails for any reason but a 404, the role is r [...] +| 5 | Fork workflows | external or pilot participant: `ci-amd.yml` state on the fork is `active`; committer/collaborator/exempted: informational, with a hint when it is `active` | `gh workflow enable ci-amd.yml -R <fork>`, or `gh workflow disable ...` shown but not run by `--fix` | +| 6 | Fork `main` freshness | fork `main` is not behind `upstream/main` by more than 200 commits (`MAX_FORK_MAIN_BEHIND`) | `gh repo sync <fork> --branch main` | | 7 | Current branch | not `main`, has an upstream tracking `origin` | informational | -Committers and collaborators are told to keep fork workflows disabled because their pull -requests run on project CI; an enabled fork would run every push twice and attach a second, -confusing status. Forks of a public repository cannot be made private on GitHub, so that is -the only recommendation available. - -The remote-parsing block currently inside `breeze ci upgrade` is factored into -`utils/git_remotes.py` and reused, so both commands agree on what a correct setup is. +The role is determined before checks 2 and 3, though printed as check 4. For a committer, +collaborator or exempted contributor, whose pull requests run project CI, checks 2 and 3 are +informational: working from an `apache/airflow` clone without a fork is fine for them, and +`--fix` only runs the fixes of failed checks. The same goes for check 5: an enabled fork +workflow runs every push twice and attaches a second status, so they are told they can disable +it, but it is not a failure, because a committer trying fork CI with the `use fork ci` label +needs it enabled. Forks of a public repository cannot be made private on GitHub, so disabling +the workflow is the only recommendation available. + +A failed API call is reported as "could not check" rather than as a finding: a failed fork +lookup is not "not a fork of apache/airflow", and a failed workflow lookup is not "missing". +A 404 is a finding: the fork does not exist, or the workflow is missing. An `upstream` remote +that points somewhere else gets `git remote set-url upstream`, since `git remote add` would +fail on the existing name. + +The remote parsing that `breeze ci upgrade` had inline is factored into +`utils/git_remotes.py` and reused, so both commands agree on what a correct setup is. A remote +is recognised as a GitHub repository in every form git accepts (`http` or `https` with optional +credentials, `www.` or a port, scp-like ssh, `ssh://` with an optional port, any case of the +host, and an ssh host alias from `~/.ssh/config` such as `git@github-personal:apache/airflow.git`, +counted when the alias names `github` and has no dot, or is `github.com` with a `-` suffix such +as `github.com-work`, since any other dotted host such as a GitHub Enterprise server is a +different GitHub), and only when the repository is the whole path, so `apache/airflow-site` is never taken +for `apache/airflow`. A command the audit runs that is not installed, `gh` included, is reported +as a failed check rather than a crash. ## Documentation changes diff --git a/dev/breeze/doc/images/output_ci.svg b/dev/breeze/doc/images/output_ci.svg index bee7d61e074..7158eac2e10 100644 --- a/dev/breeze/doc/images/output_ci.svg +++ b/dev/breeze/doc/images/output_ci.svg @@ -1,4 +1,4 @@ -<svg class="rich-terminal" viewBox="0 0 1482 538.0" xmlns="http://www.w3.org/2000/svg";> +<svg class="rich-terminal" viewBox="0 0 1482 562.4" xmlns="http://www.w3.org/2000/svg";> <!-- Generated with Rich https://www.textualize.io --> <style> @@ -42,7 +42,7 @@ <defs> <clipPath id="breeze-ci-clip-terminal"> - <rect x="0" y="0" width="1463.0" height="487.0" /> + <rect x="0" y="0" width="1463.0" height="511.4" /> </clipPath> <clipPath id="breeze-ci-line-0"> <rect x="0" y="1.5" width="1464" height="24.65"/> @@ -101,9 +101,12 @@ <clipPath id="breeze-ci-line-18"> <rect x="0" y="440.7" width="1464" height="24.65"/> </clipPath> +<clipPath id="breeze-ci-line-19"> + <rect x="0" y="465.1" width="1464" height="24.65"/> + </clipPath> </defs> - <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="536" rx="8"/><text class="breeze-ci-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci</text> + <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="560.4" rx="8"/><text class="breeze-ci-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci</text> <g transform="translate(26,22)"> <circle cx="0" cy="0" r="7" fill="#ff5f57"/> <circle cx="22" cy="0" r="7" fill="#febc2e"/> @@ -132,7 +135,8 @@ </text><text class="breeze-ci-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-ci-line-16)">│</text><text class="breeze-ci-r4" x="24.4" y="410.4" textLength="207.4" clip-path="url(#breeze-ci-line-16)">set-milestone    </text><text class="breeze-ci-r1" x="256.2" y="410.4" textLength="1183.4" clip-path="url(#breeze-ci-line-16)">Set milestone on a merged PR if it doesn't have one. Used by t [...] </text><text class="breeze-ci-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-ci-line-17)">│</text><text class="breeze-ci-r1" x="256.2" y="434.8" textLength="1183.4" clip-path="url(#breeze-ci-line-17)">workflow.                                             [...] </text><text class="breeze-ci-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-ci-line-18)">│</text><text class="breeze-ci-r4" x="24.4" y="459.2" textLength="207.4" clip-path="url(#breeze-ci-line-18)">fork-ci-reconcile</text><text class="breeze-ci-r1" x="256.2" y="459.2" textLength="1183.4" clip-path="url(#breeze-ci-line-18)">Reconcile pull requests whose CI runs in contributor forks (AIP-120). Run on a schedule [...] -</text><text class="breeze-ci-r5" x="0" y="483.6" textLength="1464" clip-path="url(#breeze-ci-line-19)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-r1" x="1464" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-line-19)"> +</text><text class="breeze-ci-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-ci-line-19)">│</text><text class="breeze-ci-r4" x="24.4" y="483.6" textLength="207.4" clip-path="url(#breeze-ci-line-19)">audit            </text><text class="breeze-ci-r1" x="256.2" y="483.6" textLength="1183.4" clip-path="url(#breeze-ci-line-19)">Check that your checkout and fork are set up fo [...] +</text><text class="breeze-ci-r5" x="0" y="508" textLength="1464" clip-path="url(#breeze-ci-line-20)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-r1" x="1464" y="508" textLength="12.2" clip-path="url(#breeze-ci-line-20)"> </text> </g> </g> diff --git a/dev/breeze/doc/images/output_ci.txt b/dev/breeze/doc/images/output_ci.txt index 86605d72f01..1ebecd3df3e 100644 --- a/dev/breeze/doc/images/output_ci.txt +++ b/dev/breeze/doc/images/output_ci.txt @@ -1 +1 @@ -037d52dccd1cf22e81b335f1319ecb9c +ceb5f9dfbcb79c72545427568d9b7dbc diff --git a/dev/breeze/doc/images/output_ci_audit.svg b/dev/breeze/doc/images/output_ci_audit.svg new file mode 100644 index 00000000000..b52f659d46d --- /dev/null +++ b/dev/breeze/doc/images/output_ci_audit.svg @@ -0,0 +1,116 @@ +<svg class="rich-terminal" viewBox="0 0 1482 391.59999999999997" xmlns="http://www.w3.org/2000/svg";> + <!-- Generated with Rich https://www.textualize.io --> + <style> + + @font-face { + font-family: "Fira Code"; + src: local("FiraCode-Regular"), + url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Regular.woff2";) format("woff2"), + url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Regular.woff";) format("woff"); + font-style: normal; + font-weight: 400; + } + @font-face { + font-family: "Fira Code"; + src: local("FiraCode-Bold"), + url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff2/FiraCode-Bold.woff2";) format("woff2"), + url("https://cdnjs.cloudflare.com/ajax/libs/firacode/6.2.0/woff/FiraCode-Bold.woff";) format("woff"); + font-style: bold; + font-weight: 700; + } + + .breeze-ci-audit-matrix { + font-family: Fira Code, monospace; + font-size: 20px; + line-height: 24.4px; + font-variant-east-asian: full-width; + } + + .breeze-ci-audit-title { + font-size: 18px; + font-weight: bold; + font-family: arial; + } + + .breeze-ci-audit-r1 { fill: #c5c8c6 } +.breeze-ci-audit-r2 { fill: #d0b344 } +.breeze-ci-audit-r3 { fill: #c5c8c6;font-weight: bold } +.breeze-ci-audit-r4 { fill: #68a0b3;font-weight: bold } +.breeze-ci-audit-r5 { fill: #868887 } +.breeze-ci-audit-r6 { fill: #98a84b;font-weight: bold } +.breeze-ci-audit-r7 { fill: #8d7b39 } + </style> + + <defs> + <clipPath id="breeze-ci-audit-clip-terminal"> + <rect x="0" y="0" width="1463.0" height="340.59999999999997" /> + </clipPath> + <clipPath id="breeze-ci-audit-line-0"> + <rect x="0" y="1.5" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-1"> + <rect x="0" y="25.9" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-2"> + <rect x="0" y="50.3" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-3"> + <rect x="0" y="74.7" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-4"> + <rect x="0" y="99.1" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-5"> + <rect x="0" y="123.5" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-6"> + <rect x="0" y="147.9" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-7"> + <rect x="0" y="172.3" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-8"> + <rect x="0" y="196.7" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-9"> + <rect x="0" y="221.1" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-10"> + <rect x="0" y="245.5" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-11"> + <rect x="0" y="269.9" width="1464" height="24.65"/> + </clipPath> +<clipPath id="breeze-ci-audit-line-12"> + <rect x="0" y="294.3" width="1464" height="24.65"/> + </clipPath> + </defs> + + <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="389.6" rx="8"/><text class="breeze-ci-audit-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: ci audit</text> + <g transform="translate(26,22)"> + <circle cx="0" cy="0" r="7" fill="#ff5f57"/> + <circle cx="22" cy="0" r="7" fill="#febc2e"/> + <circle cx="44" cy="0" r="7" fill="#28c840"/> + </g> + + <g transform="translate(9, 41)" clip-path="url(#breeze-ci-audit-clip-terminal)"> + + <g class="breeze-ci-audit-matrix"> + <text class="breeze-ci-audit-r1" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-ci-audit-line-0)"> +</text><text class="breeze-ci-audit-r2" x="12.2" y="44.4" textLength="73.2" clip-path="url(#breeze-ci-audit-line-1)">Usage:</text><text class="breeze-ci-audit-r3" x="97.6" y="44.4" textLength="183" clip-path="url(#breeze-ci-audit-line-1)">breeze ci audit</text><text class="breeze-ci-audit-r1" x="292.8" y="44.4" textLength="12.2" clip-path="url(#breeze-ci-audit-line-1)">[</text><text class="breeze-ci-audit-r4" x="305" y="44.4" textLength="85.4" clip-path="url(#breeze-ci-audit-li [...] +</text><text class="breeze-ci-audit-r1" x="1464" y="68.8" textLength="12.2" clip-path="url(#breeze-ci-audit-line-2)"> +</text><text class="breeze-ci-audit-r1" x="12.2" y="93.2" textLength="1073.6" clip-path="url(#breeze-ci-audit-line-3)">Check that your checkout and fork are set up for fork CI (AIP-120), and offer the fixes.</text><text class="breeze-ci-audit-r1" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-ci-audit-line-3)"> +</text><text class="breeze-ci-audit-r1" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-ci-audit-line-4)"> +</text><text class="breeze-ci-audit-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-ci-audit-line-5)">╭─</text><text class="breeze-ci-audit-r5" x="24.4" y="142" textLength="158.6" clip-path="url(#breeze-ci-audit-line-5)"> Audit flags </text><text class="breeze-ci-audit-r5" x="183" y="142" textLength="1256.6" clip-path="url(#breeze-ci-audit-line-5)">───────────────────────────────────────────────────────────────────────────────────────────────────────</text><text [...] +</text><text class="breeze-ci-audit-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-ci-audit-line-6)">│</text><text class="breeze-ci-audit-r4" x="24.4" y="166.4" textLength="61" clip-path="url(#breeze-ci-audit-line-6)">--fix</text><text class="breeze-ci-audit-r1" x="109.8" y="166.4" textLength="658.8" clip-path="url(#breeze-ci-audit-line-6)">Run the offered fix commands, each after confirmation.</text><text class="breeze-ci-audit-r5" x="145 [...] +</text><text class="breeze-ci-audit-r5" x="0" y="190.8" textLength="1464" clip-path="url(#breeze-ci-audit-line-7)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-audit-r1" x="1464" y="190.8" textLength="12.2" clip-path="url(#breeze-ci-audit-line-7)"> +</text><text class="breeze-ci-audit-r5" x="0" y="215.2" textLength="24.4" clip-path="url(#breeze-ci-audit-line-8)">╭─</text><text class="breeze-ci-audit-r5" x="24.4" y="215.2" textLength="195.2" clip-path="url(#breeze-ci-audit-line-8)"> Common options </text><text class="breeze-ci-audit-r5" x="219.6" y="215.2" textLength="1220" clip-path="url(#breeze-ci-audit-line-8)">────────────────────────────────────────────────────────────────────────────────────────────────────</text [...] +</text><text class="breeze-ci-audit-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-ci-audit-line-9)">│</text><text class="breeze-ci-audit-r4" x="24.4" y="239.6" textLength="109.8" clip-path="url(#breeze-ci-audit-line-9)">--verbose</text><text class="breeze-ci-audit-r6" x="158.6" y="239.6" textLength="24.4" clip-path="url(#breeze-ci-audit-line-9)">-v</text><text class="breeze-ci-audit-r1" x="207.4" y="239.6" textLength="585.6" clip-path="url(#breeze-ci-audit-line-9)">Print&# [...] +</text><text class="breeze-ci-audit-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-ci-audit-line-10)">│</text><text class="breeze-ci-audit-r4" x="24.4" y="264" textLength="109.8" clip-path="url(#breeze-ci-audit-line-10)">--dry-run</text><text class="breeze-ci-audit-r6" x="158.6" y="264" textLength="24.4" clip-path="url(#breeze-ci-audit-line-10)">-D</text><text class="breeze-ci-audit-r1" x="207.4" y="264" textLength="719.8" clip-path="url(#breeze-ci-audit-line-10)">If dry [...] +</text><text class="breeze-ci-audit-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-ci-audit-line-11)">│</text><text class="breeze-ci-audit-r4" x="24.4" y="288.4" textLength="109.8" clip-path="url(#breeze-ci-audit-line-11)">--answer </text><text class="breeze-ci-audit-r6" x="158.6" y="288.4" textLength="24.4" clip-path="url(#breeze-ci-audit-line-11)">-a</text><text class="breeze-ci-audit-r1" x="207.4" y="288.4" textLength="329.4" clip-path="url(#breeze-ci-audit-line-11) [...] +</text><text class="breeze-ci-audit-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-ci-audit-line-12)">│</text><text class="breeze-ci-audit-r4" x="24.4" y="312.8" textLength="109.8" clip-path="url(#breeze-ci-audit-line-12)">--help   </text><text class="breeze-ci-audit-r6" x="158.6" y="312.8" textLength="24.4" clip-path="url(#breeze-ci-audit-line-12)">-h</text><text class="breeze-ci-audit-r1" x="207.4" y="312.8" textLength="329.4" clip-path="url(#breeze-ci-audi [...] +</text><text class="breeze-ci-audit-r5" x="0" y="337.2" textLength="1464" clip-path="url(#breeze-ci-audit-line-13)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-ci-audit-r1" x="1464" y="337.2" textLength="12.2" clip-path="url(#breeze-ci-audit-line-13)"> +</text> + </g> + </g> +</svg> diff --git a/dev/breeze/doc/images/output_ci_audit.txt b/dev/breeze/doc/images/output_ci_audit.txt new file mode 100644 index 00000000000..ebcacb0607d --- /dev/null +++ b/dev/breeze/doc/images/output_ci_audit.txt @@ -0,0 +1 @@ +9327d4e6fa36b9d7d08069ba29e77aa7 diff --git a/dev/breeze/doc/images/output_setup_check-all-params-in-groups.svg b/dev/breeze/doc/images/output_setup_check-all-params-in-groups.svg index 9e800dd4f5d..781cfd2a0df 100644 --- a/dev/breeze/doc/images/output_setup_check-all-params-in-groups.svg +++ b/dev/breeze/doc/images/output_setup_check-all-params-in-groups.svg @@ -215,20 +215,20 @@ </text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-5)">╭─</text><text class="breeze-setup-check-all-params-in-groups-r5" x="24.4" y="142" textLength="414.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-5)"> Check all params in groups flags </text><text class="breeze-setup-check-all-params-in-groups-r5" x="439.2" y="142" textLength="1 [...] </text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-6)">│</text><text class="breeze-setup-check-all-params-in-groups-r4" x="24.4" y="166.4" textLength="109.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-6)">--command</text><text class="breeze-setup-check-all-params-in-groups-r1" x="158.6" y="166.4" textLength="805.2" clip-path="url(#breeze-setup-check-all-params-in- [...] </text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-7)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="190.8" textLength="1134.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-7)">ci-image:build | ci-image:export-mount-cache | ci-image:import-mount-cache | ci-image:load | </text><text class="breeze [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-8)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="215.2" textLength="1281" clip-path="url(#breeze-setup-check-all-params-in-groups-line-8)">ci-image:pull | ci-image:save | ci-image:verify | ci:fix-ownership | ci:fork-ci-reconcile | ci:free-space</text [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-9)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="239.6" textLength="1281" clip-path="url(#breeze-setup-check-all-params-in-groups-line-9)">| ci:get-workflow-info | ci:resource-check | ci:selective-check | ci:set-milestone | ci:upgrade | cleanup< [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-10)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="264" textLength="1281" clip-path="url(#breeze-setup-check-all-params-in-groups-line-10)">| doctor | down | exec | generate-migration-file | issues | issues:unassign | k8s | k8s: [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-11)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="288.4" textLength="1281" clip-path="url(#breeze-setup-check-all-params-in-groups-line-11)">k8s:configure-cluster | k8s:create-cluster | k8s:delete-cluster | k8s:deploy-airflow | k8s:deploy-cluster</text><text c [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-12)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="312.8" textLength="1220" clip-path="url(#breeze-setup-check-all-params-in-groups-line-12)">| k8s:dev | k8s:k9s | k8s:logs | k8s:run-complete-tests | k8s:setup-env | k8s:setup-lang-sdk-test | [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-13)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="337.2" textLength="1195.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-13)">k8s:shell | k8s:smoke-test-overlay | k8s:status | k8s:tests | k8s:upload-k8s-image | prod-image |  [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-14)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="361.6" textLength="1268.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-14)">prod-image:build | prod-image:load | prod-image:pull | prod-image:save | prod-image:verify | registry  [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-15)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="386" textLength="1134.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-15)">registry:backfill | registry:extract-data | registry:publish-versions | release-management | </text><text class="breeze-s [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-16)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="410.4" textLength="1000.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-16)">release-management:add-back-references | release-management:check-release-files | </text><text class="breeze-setup-check-all-params-in-gr [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="434.8" textLength="1183.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">release-management:classify-provider-changes | release-management:clean-old-provider-artifacts | </text><text class="breeze-setup-check-a [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="459.2" textLength="1073.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">release-management:constraints-version-check | release-management:create-minor-branch | </text><text class="breeze-setup-check-all-params [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="483.6" textLength="1134.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">release-management:generate-airflowctl-changelog | release-management:generate-constraints | </text><text class="breeze-setup-check-all-p [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="508" textLength="1281" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">release-management:generate-issue-content-airflow-ctl | release-management:generate-issue-content-core | </text><text class="breeze-setup-check [...] -</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="532.4" textLength="671" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">release-management:generate-issue-content-helm-chart | </text><text class="breeze-setup-check-all-params-in-groups-r5" x="1451.8" y="532.4" textLength [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-8)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="215.2" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-8)">ci-image:pull | ci-image:save | ci-image:verify | ci:audit | ci:fix-ownership | ci:fork-ci-reconcile |&# [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-9)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="239.6" textLength="1207.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-9)">ci:free-space | ci:get-workflow-info | ci:resource-check | ci:selective-check | ci:set-milestone | </text><te [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-10)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="264" textLength="1281" clip-path="url(#breeze-setup-check-all-params-in-groups-line-10)">ci:upgrade | cleanup | doctor | down | exec | generate-migration-file | issues | issues:unass [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-11)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="288.4" textLength="1073.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-11)">k8s:build-k8s-image | k8s:configure-cluster | k8s:create-cluster | k8s:delete-cluster | </text><text class="breeze-se [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-12)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="312.8" textLength="1195.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-12)">k8s:deploy-airflow | k8s:deploy-cluster | k8s:dev | k8s:k9s | k8s:logs | k8s:run-complete-tests |  [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-13)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="337.2" textLength="1268.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-13)">k8s:setup-env | k8s:setup-lang-sdk-test | k8s:shell | k8s:smoke-test-overlay | k8s:status | k8s:tests  [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-14)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="361.6" textLength="1110.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-14)">k8s:upload-k8s-image | prod-image | prod-image:build | prod-image:load | prod-image:pull | </text><text cla [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-15)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="386" textLength="1134.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-15)">prod-image:save | prod-image:verify | registry | registry:backfill | registry:extract-data | </text><text class [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-16)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="410.4" textLength="1098" clip-path="url(#breeze-setup-check-all-params-in-groups-line-16)">registry:publish-versions | release-management | release-management:add-back-references | </text><text class="breeze-setup-check- [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="434.8" textLength="1073.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-17)">release-management:check-release-files | release-management:classify-provider-changes | </text><text class="breeze-setup-check-all-params [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="459.2" textLength="1183.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-18)">release-management:clean-old-provider-artifacts | release-management:constraints-version-check | </text><text class="breeze-setup-check-a [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="483.6" textLength="1122.4" clip-path="url(#breeze-setup-check-all-params-in-groups-line-19)">release-management:create-minor-branch | release-management:generate-airflowctl-changelog | </text><text class="breeze-setup-check-all-pa [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="508" textLength="1195.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-20)">release-management:generate-constraints | release-management:generate-issue-content-airflow-ctl | </text><text class="breeze-setup-check-all- [...] +</text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="532.4" textLength="1268.8" clip-path="url(#breeze-setup-check-all-params-in-groups-line-21)">release-management:generate-issue-content-core | release-management:generate-issue-content-helm-chart | </text><text class="breeze-setup- [...] </text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-22)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="556.8" textLength="1256.6" clip-path="url(#breeze-setup-check-all-params-in-groups-line-22)">release-management:generate-issue-content-providers | release-management:generate-providers-metadata | </text><text class="breeze-setup-c [...] </text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-23)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="581.2" textLength="1110.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-23)">release-management:install-provider-distributions | release-management:merge-prod-images | </text><text class="breeze-setup-check-all-par [...] </text><text class="breeze-setup-check-all-params-in-groups-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-setup-check-all-params-in-groups-line-24)">│</text><text class="breeze-setup-check-all-params-in-groups-r6" x="158.6" y="605.6" textLength="1159" clip-path="url(#breeze-setup-check-all-params-in-groups-line-24)">release-management:mirror-base-images | release-management:prepare-airflow-ctl-distributions | </text><text class="breeze-setup-check-all-p [...] diff --git a/dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt b/dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt index 76256ca029f..18ac0161089 100644 --- a/dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt +++ b/dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt @@ -1 +1 @@ -7f2c5e0d34a051e15d20ca31b995cebb +525c07613a5bcce29aeb394a631aee01 diff --git a/dev/breeze/doc/images/output_setup_regenerate-command-images.svg b/dev/breeze/doc/images/output_setup_regenerate-command-images.svg index 1c9a6a20d9f..0c8f45924ab 100644 --- a/dev/breeze/doc/images/output_setup_regenerate-command-images.svg +++ b/dev/breeze/doc/images/output_setup_regenerate-command-images.svg @@ -231,7 +231,7 @@ </text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="166.4" textLength="146.4" clip-path="url(#breeze-setup-regenerate-command-images-line-6)">--force     </text><text class="breeze-setup-regenerate-command-images-r1" x="195.2" y="166.4" textLength="402.6" clip-path="url(#breeze-set [...] </text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">│</text><text class="breeze-setup-regenerate-command-images-r4" x="24.4" y="190.8" textLength="146.4" clip-path="url(#breeze-setup-regenerate-command-images-line-7)">--command   </text><text class="breeze-setup-regenerate-command-images-r1" x="195.2" y="190.8" textLength="805.2" clip-path="url(#breeze-setup-regener [...] </text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="195.2" y="215.2" textLength="1134.6" clip-path="url(#breeze-setup-regenerate-command-images-line-8)">ci-image:build | ci-image:export-mount-cache | ci-image:import-mount-cache | ci-image:load | </text><text class="breeze-set [...] -</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="195.2" y="239.6" textLength="1122.4" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">ci-image:pull | ci-image:save | ci-image:verify | ci:fix-ownership | ci:fork-ci-reconcile | </text><text class="b [...] +</text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="195.2" y="239.6" textLength="1244.4" clip-path="url(#breeze-setup-regenerate-command-images-line-9)">ci-image:pull | ci-image:save | ci-image:verify | ci:audit | ci:fix-ownership | ci:fork-ci-reconcile |</text [...] </text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="195.2" y="264" textLength="1207.8" clip-path="url(#breeze-setup-regenerate-command-images-line-10)">ci:free-space | ci:get-workflow-info | ci:resource-check | ci:selective-check | ci:set-milestone | </text><text cla [...] </text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="195.2" y="288.4" textLength="1244.4" clip-path="url(#breeze-setup-regenerate-command-images-line-11)">ci:upgrade | cleanup | doctor | down | exec | generate-migration-file | issues | issues:una [...] </text><text class="breeze-setup-regenerate-command-images-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">│</text><text class="breeze-setup-regenerate-command-images-r6" x="195.2" y="312.8" textLength="1098" clip-path="url(#breeze-setup-regenerate-command-images-line-12)">| k8s:build-k8s-image | k8s:configure-cluster | k8s:create-cluster | k8s:delete-cluster | </text><text class="breeze-s [...] diff --git a/dev/breeze/doc/images/output_setup_regenerate-command-images.txt b/dev/breeze/doc/images/output_setup_regenerate-command-images.txt index 0b47fbd16cf..39e1abce9b2 100644 --- a/dev/breeze/doc/images/output_setup_regenerate-command-images.txt +++ b/dev/breeze/doc/images/output_setup_regenerate-command-images.txt @@ -1 +1 @@ -9ddeeee9e8c82e964380018f7103d1e1 +3588c0e4f1fda167e7f07b6d1b7843de diff --git a/dev/breeze/src/airflow_breeze/commands/ci_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_commands.py index 8c604278050..0dbc5027942 100644 --- a/dev/breeze/src/airflow_breeze/commands/ci_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/ci_commands.py @@ -41,6 +41,7 @@ from airflow_breeze.commands.common_options import ( option_verbose, ) from airflow_breeze.global_constants import ( + APACHE_AIRFLOW_GITHUB_REPOSITORY, CI_AMD_PLATFORM, CI_PLATFORMS, DEFAULT_PYTHON_MAJOR_MINOR_VERSION, @@ -61,6 +62,7 @@ from airflow_breeze.utils.docker_command_utils import ( perform_environment_checks, ) from airflow_breeze.utils.fork_compare import get_fork_changes, get_fork_pull_request_labels, is_fork_push +from airflow_breeze.utils.git_remotes import find_remote, find_remote_for_repo, get_remotes from airflow_breeze.utils.github import format_github_token_scope_guidance, retrieve_github_token from airflow_breeze.utils.path_utils import AIRFLOW_HOME_PATH, AIRFLOW_ROOT_PATH from airflow_breeze.utils.run_utils import run_command @@ -875,41 +877,12 @@ def upgrade( status_result = run_command(["git", "status", "--porcelain"], capture_output=True, text=True, check=False) is_clean = status_result.returncode == 0 and not status_result.stdout.strip() - # Check if we have the apache remote and get its name - remote_result = run_command(["git", "remote", "-v"], capture_output=True, text=True, check=False) - apache_remote_name = None - origin_remote_name = None - origin_repo = None # Store the user's fork repo (e.g., "username/airflow") - - if remote_result.returncode == 0: - # Parse remote output to find apache/airflow remote and origin remote - # Format: remote_name\turl (fetch|push) - for line in remote_result.stdout.splitlines(): - parts = line.split() - if len(parts) >= 2: - remote_name = parts[0] - remote_url = parts[1] - if "apache/airflow" in remote_url and apache_remote_name is None: - apache_remote_name = remote_name - # Also track origin remote for pushing - if remote_name == "origin" and origin_remote_name is None: - origin_remote_name = remote_name - # Extract repo from origin URL (supports both HTTPS and SSH formats) - # HTTPS: https://github.com/username/airflow.git - # SSH: [email protected]:username/airflow.git - if "github.com" in remote_url: - if "[email protected]:" in remote_url: - # SSH format - repo_part = remote_url.split("[email protected]:")[1] - elif "github.com/" in remote_url: - # HTTPS format - repo_part = remote_url.split("github.com/")[1] - else: - repo_part = None - - if repo_part: - # Remove .git suffix if present - origin_repo = repo_part.replace(".git", "").strip() + remotes = get_remotes() + apache_remote = find_remote_for_repo(remotes, APACHE_AIRFLOW_GITHUB_REPOSITORY) + apache_remote_name = apache_remote.name if apache_remote else None + origin_remote = find_remote(remotes, "origin") + origin_remote_name = origin_remote.name if origin_remote else None + origin_repo = origin_remote.repo if origin_remote else None has_apache_remote = apache_remote_name is not None @@ -1751,3 +1724,53 @@ def fork_ci_reconcile(config_path: Path | None, output_file: Path, run_url: str, if github_output: with open(github_output, "a") as f: f.write(get_ga_output("has-alerts", has_alerts) + "\n") + + +@ci_group.command( + name="audit", + help="Check that your checkout and fork are set up for fork CI (AIP-120), and offer the fixes.", +) [email protected]("--fix", is_flag=True, help="Run the offered fix commands, each after confirmation.") +@option_verbose +@option_dry_run +@option_answer +def audit(fix: bool): + import shlex + + import yaml + from rich.table import Table + + from airflow_breeze.utils.ci_audit import CheckStatus, run_audit, run_tool + from airflow_breeze.utils.fork_ci_gate import load_config + + asf_yaml = yaml.safe_load((AIRFLOW_ROOT_PATH / ".asf.yaml").read_text()) + collaborators = frozenset((asf_yaml.get("github") or {}).get("collaborators") or []) + config = load_config(AIRFLOW_ROOT_PATH / ".github" / "fork-ci-gate.toml") + + def report(): + results = run_audit(run=run_tool, config=config, collaborators=collaborators) + table = Table(title="Fork CI setup") + for column in ("#", "Check", "Status", "Finding", "Fix"): + table.add_column(column) + colors = {CheckStatus.OK: "green", CheckStatus.FAIL: "red", CheckStatus.INFO: "blue"} + for result in results: + table.add_row( + str(result.number), + result.name, + f"[{colors[result.status]}]{result.status.value}[/]", + result.finding, + "\n".join(result.fix), + ) + get_console().print(table) + return results + + failed = [result for result in report() if result.status == CheckStatus.FAIL] + if fix and any(result.fix for result in failed): + for result in failed: + if result.fix and user_confirm(f"Run the fix for '{result.name}': {'; '.join(result.fix)}?") == ( + Answer.YES + ): + for command in result.fix: + run_command(shlex.split(command), check=False) + failed = [result for result in report() if result.status == CheckStatus.FAIL] + sys.exit(1 if failed else 0) diff --git a/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py b/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py index e98b6fe04c3..cc99bcf1c14 100644 --- a/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py +++ b/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py @@ -27,6 +27,7 @@ CI_COMMANDS: dict[str, str | list[str]] = { "upgrade", "set-milestone", "fork-ci-reconcile", + "audit", ], } CI_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = { @@ -129,4 +130,12 @@ CI_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = { ], }, ], + "breeze ci audit": [ + { + "name": "Audit flags", + "options": [ + "--fix", + ], + }, + ], } diff --git a/dev/breeze/src/airflow_breeze/utils/ci_audit.py b/dev/breeze/src/airflow_breeze/utils/ci_audit.py new file mode 100644 index 00000000000..3b27ef8eb15 --- /dev/null +++ b/dev/breeze/src/airflow_breeze/utils/ci_audit.py @@ -0,0 +1,403 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Checks behind `breeze ci audit`: is this checkout and fork set up for fork CI (AIP-120)? + +Every check is read-only. A failing check offers the commands that fix it; `--fix` runs them after +confirmation. See "breeze ci audit" in dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md. +""" + +from __future__ import annotations + +import json +import re +import subprocess +from collections.abc import Callable, Sequence +from dataclasses import dataclass +from enum import Enum +from typing import TYPE_CHECKING + +from packaging.version import Version + +from airflow_breeze.global_constants import APACHE_AIRFLOW_GITHUB_REPOSITORY, MIN_GH_VERSION +from airflow_breeze.utils.git_remotes import find_remote, find_remote_for_repo, parse_remotes + +if TYPE_CHECKING: + from airflow_breeze.utils.fork_ci_gate import GateConfig + +UPSTREAM_URL = f"https://github.com/{APACHE_AIRFLOW_GITHUB_REPOSITORY}.git"; +CI_WORKFLOW_FILE = "ci-amd.yml" +# Commits a fork's main may lag behind upstream main before a sync is suggested. +MAX_FORK_MAIN_BEHIND = 200 + +Runner = Callable[[Sequence[str]], subprocess.CompletedProcess] + + +class CheckStatus(str, Enum): + OK = "ok" + FAIL = "fail" + INFO = "info" + + +class Role(str, Enum): + COMMITTER = "committer" + COLLABORATOR = "collaborator" + EXEMPTED = "exempted organisation member" + PILOT = "fork CI pilot participant" + EXTERNAL = "external contributor" + UNKNOWN = "contributor whose apache membership could not be checked" + + +@dataclass(frozen=True) +class CheckResult: + number: int + name: str + status: CheckStatus + finding: str + fix: tuple[str, ...] = () + + +def run_tool(command: Sequence[str]) -> subprocess.CompletedProcess: + """Run a command for the audit; a missing executable is a failed command, not a crash.""" + try: + return subprocess.run(list(command), capture_output=True, text=True, check=False) + except OSError as e: + return subprocess.CompletedProcess(list(command), 127, stdout="", stderr=str(e)) + + +def _ok(result: subprocess.CompletedProcess) -> bool: + return result.returncode == 0 + + +def _out(result: subprocess.CompletedProcess) -> str: + return (result.stdout or "").strip() + + +def redact_url(url: str) -> str: + """Drop the credentials an http(s) remote URL may carry, so a finding never prints a token.""" + return re.sub(r"^(https?://)[^@/]+@", r"\1", url) + + +def _is_not_found(result: subprocess.CompletedProcess) -> bool: + return "404" in (result.stderr or "") or "Not Found" in (result.stderr or "") + + +def is_public_apache_member(*, login: str, run: Runner) -> bool | None: + """Whether ``login`` is a public apache member; None when the API call failed.""" + result = run(["gh", "api", f"orgs/apache/public_members/{login}"]) + if _ok(result): + return True + return False if _is_not_found(result) else None + + +def is_private_apache_member(*, run: Runner) -> bool: + membership = run(["gh", "api", "user/memberships/orgs/apache", "--jq", ".state"]) + return _ok(membership) and _out(membership) == "active" + + +def get_role(*, login: str, run: Runner, config: GateConfig, collaborators: frozenset[str]) -> Role: + # In the gate's order: a pilot entry gates a pull request whatever its author's role. + if config.pilot_joined_at(login) is not None: + return Role.PILOT + # The gate only sees public membership: GitHub reports a private member's pull requests as + # `CONTRIBUTOR`, so they are gated like an external contributor's. + public_member = is_public_apache_member(login=login, run=run) + if public_member: + return Role.COMMITTER + if login.casefold() in {c.casefold() for c in collaborators}: + return Role.COLLABORATOR + if config.is_allowlisted(login): + return Role.EXEMPTED + # A failed call must not make a committer look external: the fixes would then turn on fork CI + # in their fork, and every push would run CI twice. + return Role.UNKNOWN if public_member is None else Role.EXTERNAL + + +def fork_ci_required(role: Role, config: GateConfig) -> bool | None: + """True when fork CI must be on, False when it must be off, None when either is fine.""" + if role == Role.PILOT: + return True + if role == Role.EXTERNAL: + return True if config.enabled else None + if role == Role.UNKNOWN: + return None + return False + + +def run_audit(*, run: Runner, config: GateConfig, collaborators: frozenset[str]) -> list[CheckResult]: + results: list[CheckResult] = [] + + gh_version = run(["gh", "--version"]) + if not _ok(gh_version): + results.append( + CheckResult( + 1, "gh", CheckStatus.FAIL, "`gh` is not installed", ("echo See https://cli.github.com";,) + ) + ) + return results + version_match = re.search(r"gh version (\d+\.\d+\.\d+)", _out(gh_version)) + if version_match is None or Version(version_match.group(1)) < Version(MIN_GH_VERSION): + results.append( + CheckResult(1, "gh", CheckStatus.FAIL, f"`gh` is older than {MIN_GH_VERSION}; upgrade it") + ) + return results + user = run(["gh", "api", "user", "--jq", ".login"]) + if not _ok(user): + results.append( + CheckResult(1, "gh", CheckStatus.FAIL, "`gh` is not authenticated", ("gh auth login",)) + ) + return results + login = _out(user) + results.append(CheckResult(1, "gh", CheckStatus.OK, f"authenticated as {login}")) + + # The role decides whether the fork setup matters at all: a committer working from an + # apache/airflow clone without a fork is fine while their pull requests run project CI. + role = get_role(login=login, run=run, config=config, collaborators=collaborators) + required = fork_ci_required(role, config) + setup_status = CheckStatus.INFO if required is False or role == Role.UNKNOWN else CheckStatus.FAIL + if required is False: + setup_note = "; only needed for fork CI, which your pull requests do not use" + elif role == Role.UNKNOWN: + setup_note = "; only needed for fork CI, which committers do not use" + else: + setup_note = "" + + remotes = parse_remotes(_out(run(["git", "remote", "-v"]))) + upstream = find_remote(remotes, "upstream") + apache = find_remote_for_repo(remotes, APACHE_AIRFLOW_GITHUB_REPOSITORY) + origin = find_remote(remotes, "origin") + # Whether the remotes fix already adds or repoints `origin`, so the fork fix must not add it again. + adds_origin = False + if ( + upstream is not None + and upstream.repo + and upstream.repo.casefold() == APACHE_AIRFLOW_GITHUB_REPOSITORY + ): + if origin is not None and origin.repo and origin.repo.casefold() == APACHE_AIRFLOW_GITHUB_REPOSITORY: + results.append( + CheckResult( + 2, + "remotes", + setup_status, + f"`origin` points at apache/airflow; it should be your fork{setup_note}", + (f"git remote set-url origin https://github.com/{login}/airflow.git";,), + ) + ) + adds_origin = True + else: + results.append(CheckResult(2, "remotes", CheckStatus.OK, "`upstream` is apache/airflow")) + elif apache is not None and upstream is None: + fix = [f"git remote rename {apache.name} upstream"] + if apache.name == "origin": + fix.append(f"git remote add origin https://github.com/{login}/airflow.git";) + adds_origin = True + results.append( + CheckResult( + 2, + "remotes", + setup_status, + f"apache/airflow is the remote `{apache.name}`; Airflow's convention is `upstream`{setup_note}", + tuple(fix), + ) + ) + elif upstream is not None: + results.append( + CheckResult( + 2, + "remotes", + CheckStatus.FAIL, + f"`upstream` points at {redact_url(upstream.url)}, not apache/airflow", + (f"git remote set-url upstream {UPSTREAM_URL}",), + ) + ) + else: + results.append( + CheckResult( + 2, + "remotes", + CheckStatus.FAIL, + "no `upstream` remote pointing at apache/airflow", + (f"git remote add upstream {UPSTREAM_URL}",), + ) + ) + + fork = origin.repo if origin is not None and origin.repo else None + fork_ok = False + if fork is None or fork.casefold() == APACHE_AIRFLOW_GITHUB_REPOSITORY: + results.append( + CheckResult( + 3, + "fork", + setup_status, + f"no `origin` remote pointing at your fork of apache/airflow{setup_note}", + ( + f"gh repo fork {APACHE_AIRFLOW_GITHUB_REPOSITORY} --remote=false", + *( + () + if adds_origin + else (f"git remote add origin https://github.com/{login}/airflow.git";,) + ), + ), + ) + ) + else: + repo_info = run( + ["gh", "api", f"repos/{fork}", "--jq", "{owner: .owner.login, parent: .parent.full_name}"] + ) + info = json.loads(_out(repo_info)) if _ok(repo_info) else None + if info is None and _is_not_found(repo_info): + results.append( + CheckResult( + 3, + "fork", + setup_status, + f"{fork} does not exist{setup_note}", + (f"gh repo fork {APACHE_AIRFLOW_GITHUB_REPOSITORY} --remote=false",), + ) + ) + elif info is None: + results.append( + CheckResult( + 3, + "fork", + CheckStatus.INFO, + f"could not check {fork}: {(repo_info.stderr or '').strip() or 'gh api failed'}", + ) + ) + elif (info.get("parent") or "").casefold() != APACHE_AIRFLOW_GITHUB_REPOSITORY: + results.append( + CheckResult(3, "fork", CheckStatus.FAIL, f"{fork} is not a fork of apache/airflow") + ) + elif (info.get("owner") or "").casefold() != login.casefold(): + results.append(CheckResult(3, "fork", CheckStatus.FAIL, f"{fork} is not owned by {login}")) + else: + fork_ok = True + results.append(CheckResult(3, "fork", CheckStatus.OK, f"{fork} is your fork of apache/airflow")) + + if role == Role.EXTERNAL and is_private_apache_member(run=run): + results.append( + CheckResult( + 4, + "role", + CheckStatus.INFO, + f"you are a {role.value}: your apache membership is private, so the gate cannot see it; " + "make it public, or ask to be added to [allowlist] in .github/fork-ci-gate.toml", + (f"gh api -X PUT orgs/apache/public_members/{login}",), + ) + ) + else: + results.append(CheckResult(4, "role", CheckStatus.INFO, f"you are a {role.value}")) + + if fork_ok and fork is not None: + workflow = run(["gh", "api", f"repos/{fork}/actions/workflows/{CI_WORKFLOW_FILE}", "--jq", ".state"]) + state = _out(workflow) if _ok(workflow) else "missing" if _is_not_found(workflow) else None + if state is None: + results.append( + CheckResult( + 5, + "fork workflows", + CheckStatus.INFO, + f"could not check `Tests (AMD)` in {fork}: {(workflow.stderr or '').strip() or 'gh api failed'}", + ) + ) + elif required is True and state != "active": + enable: tuple[str, ...] = (f"gh workflow enable {CI_WORKFLOW_FILE} -R {fork}",) + if state == "missing": + enable = (f"gh repo sync {fork} --branch main", *enable) + results.append( + CheckResult( + 5, + "fork workflows", + CheckStatus.FAIL, + f"`Tests (AMD)` is {state} in {fork}; enable it", + enable, + ) + ) + elif required is False and state == "active": + # Not a failure: a committer trying fork CI with the `use fork ci` label needs it on. + results.append( + CheckResult( + 5, + "fork workflows", + CheckStatus.INFO, + f"`Tests (AMD)` is enabled in {fork}, but your pull requests run project CI, so each " + "push runs CI twice; disable it unless you use the `use fork ci` label", + (f"gh workflow disable {CI_WORKFLOW_FILE} -R {fork}",), + ) + ) + elif required is None: + results.append( + CheckResult( + 5, + "fork workflows", + CheckStatus.INFO, + f"`Tests (AMD)` is {state} in {fork}; " + + ( + "whether you need fork CI depends on your apache membership" + if role == Role.UNKNOWN + else "fork CI is not required yet" + ), + ) + ) + else: + results.append( + CheckResult(5, "fork workflows", CheckStatus.OK, f"`Tests (AMD)` is {state} in {fork}") + ) + + compare = run( + [ + "gh", + "api", + f"repos/{APACHE_AIRFLOW_GITHUB_REPOSITORY}/compare/{fork.split('/')[0]}:main...main", + "--jq", + ".ahead_by", + ] + ) + behind = int(_out(compare)) if _ok(compare) and _out(compare).isdigit() else None + if behind is None: + results.append( + CheckResult(6, "fork main", CheckStatus.INFO, "could not compare your fork's main") + ) + elif behind > MAX_FORK_MAIN_BEHIND: + results.append( + CheckResult( + 6, + "fork main", + CheckStatus.FAIL, + f"your fork's main is {behind} commits behind apache/airflow", + (f"gh repo sync {fork} --branch main",), + ) + ) + else: + results.append( + CheckResult(6, "fork main", CheckStatus.OK, f"{behind} commits behind apache/airflow") + ) + + branch = _out(run(["git", "rev-parse", "--abbrev-ref", "HEAD"])) + tracking = run(["git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}"]) + if branch in ("main", "HEAD", ""): + finding = "create a branch for your change; pull requests from your fork's main are hard to update" + elif not _ok(tracking): + finding = f"`{branch}` has no upstream; push it with `git push -u origin {branch}`" + elif not _out(tracking).startswith("origin/"): + finding = ( + f"`{branch}` tracks {_out(tracking)}; push it to your fork with `git push -u origin {branch}`" + ) + else: + finding = f"`{branch}` tracks {_out(tracking)}" + results.append(CheckResult(7, "branch", CheckStatus.INFO, finding)) + return results diff --git a/dev/breeze/src/airflow_breeze/utils/git_remotes.py b/dev/breeze/src/airflow_breeze/utils/git_remotes.py new file mode 100644 index 00000000000..281bcbb0aa4 --- /dev/null +++ b/dev/breeze/src/airflow_breeze/utils/git_remotes.py @@ -0,0 +1,79 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Git remotes of the local checkout, as `breeze ci upgrade` and `breeze ci audit` understand them.""" + +from __future__ import annotations + +import re +from collections.abc import Sequence +from dataclasses import dataclass + +from airflow_breeze.utils.run_utils import run_command + +# Every form git and GitHub accept for a GitHub repository: http or https with optional +# credentials, `www.` or a port, scp-like ssh and ssh URLs with an optional port. The host is +# case-insensitive. An ssh host may also be an alias from ~/.ssh/config, such as +# `git@github-personal:apache/airflow.git` (a host without a dot that names `github`) or +# `[email protected]:apache/airflow.git` (github.com with an undotted `-` suffix). Any other dotted host, +# such as a GitHub Enterprise server, is a different GitHub. The +# repository must be the whole path, so `apache/airflow-site` never matches `apache/airflow`. +_SSH_GITHUB_HOST = r"(?:(?:www\.)?github\.com(?:-[^@\s/:.]+)?|[^@\s/:.]*github[^@\s/:.]*)" +GITHUB_REMOTE_URL = re.compile( + r"^(?:https?://(?:[^@/]+@)?(?:www\.)?github\.com(?::\d+)?/" + rf"|(?:[^@\s/:]+@)?{_SSH_GITHUB_HOST}:" + rf"|ssh://(?:[^@\s/]+@)?{_SSH_GITHUB_HOST}(?::\d+)?/)" + r"(?P<repo>[^/\s]+/[^/\s]+?)(?:\.git)?/?$", + re.IGNORECASE, +) + + +@dataclass(frozen=True) +class Remote: + name: str + url: str + repo: str | None + + +def get_github_repo(url: str) -> str | None: + """Return ``owner/name`` for a GitHub remote URL, None for any other URL.""" + match = GITHUB_REMOTE_URL.match(url) + return match.group("repo") if match else None + + +def parse_remotes(git_remote_v_output: str) -> list[Remote]: + """Parse ``git remote -v`` output into one entry per remote, using its fetch URL.""" + remotes: dict[str, Remote] = {} + for line in git_remote_v_output.splitlines(): + parts = line.split() + if len(parts) < 2 or parts[0] in remotes: + continue + remotes[parts[0]] = Remote(name=parts[0], url=parts[1], repo=get_github_repo(parts[1])) + return list(remotes.values()) + + +def find_remote_for_repo(remotes: Sequence[Remote], repo: str) -> Remote | None: + wanted = repo.casefold() + return next((remote for remote in remotes if (remote.repo or "").casefold() == wanted), None) + + +def find_remote(remotes: Sequence[Remote], name: str) -> Remote | None: + return next((remote for remote in remotes if remote.name == name), None) + + +def get_remotes() -> list[Remote]: + result = run_command(["git", "remote", "-v"], capture_output=True, text=True, check=False) + return parse_remotes(result.stdout) if result.returncode == 0 else [] diff --git a/dev/breeze/tests/test_ci_audit.py b/dev/breeze/tests/test_ci_audit.py new file mode 100644 index 00000000000..2dbc0d8a3a5 --- /dev/null +++ b/dev/breeze/tests/test_ci_audit.py @@ -0,0 +1,440 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import json +import subprocess +from datetime import UTC, datetime +from unittest import mock + +import pytest +from click.testing import CliRunner + +from airflow_breeze.commands.ci_commands import audit +from airflow_breeze.utils.ci_audit import CheckResult, CheckStatus, Role, run_audit, run_tool +from airflow_breeze.utils.confirm import Answer +from airflow_breeze.utils.fork_ci_gate import GateConfig + +REMOTES_OK = ( + "origin\[email protected]:someone/airflow.git (fetch)\n" + "upstream\thttps://github.com/apache/airflow.git (fetch)\n" +) + + +# (return code, stdout) or (return code, stdout, stderr) of a command. +Output = tuple[int, str] | tuple[int, str, str] + +NOT_A_MEMBER: Output = (1, "", "gh: Not Found (HTTP 404)") + + +def _machine(**overrides: Output | None) -> dict[str, Output | None]: + """Outputs of every command the audit runs, keyed by a short name.""" + machine: dict[str, Output | None] = { + "gh_version": (0, "gh version 2.80.0 (2026-09-01)"), + "login": (0, "someone"), + "remotes": (0, REMOTES_OK), + "repo": (0, json.dumps({"owner": "someone", "parent": "apache/airflow"})), + "membership": NOT_A_MEMBER, + "private_membership": (1, ""), + "workflow": (0, "active"), + "behind": (0, "12"), + "branch": (0, "my-change"), + "tracking": (0, "origin/my-change"), + } + machine.update(overrides) + return machine + + +def _runner(machine): + keys = { + ("gh", "--version"): "gh_version", + ("gh", "api", "user", "--jq", ".login"): "login", + ("git", "remote", "-v"): "remotes", + ("gh", "api", "orgs/apache/public_members/someone"): "membership", + ("gh", "api", "user/memberships/orgs/apache", "--jq", ".state"): "private_membership", + ("git", "rev-parse", "--abbrev-ref", "HEAD"): "branch", + ("git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}"): "tracking", + } + + def run(command): + command = tuple(command) + key = keys.get(command) + if key is None and command[:2] == ("gh", "api"): + if "/actions/workflows/" in command[2]: + key = "workflow" + elif "/compare/" in command[2]: + key = "behind" + elif command[2].startswith("repos/"): + key = "repo" + if key is None: + raise AssertionError(f"unexpected command {command}") + returncode, stdout, *stderr = machine[key] + return subprocess.CompletedProcess(list(command), returncode, stdout=stdout, stderr="".join(stderr)) + + return run + + +ENABLED = GateConfig(enabled=True) + + +def _audit(machine, config=ENABLED, collaborators=frozenset()): + return { + result.number: result + for result in run_audit(run=_runner(machine), config=config, collaborators=collaborators) + } + + +def test_healthy_external_setup(): + results = _audit(_machine()) + + assert [r.status for r in results.values()] == [ + CheckStatus.OK, + CheckStatus.OK, + CheckStatus.OK, + CheckStatus.INFO, + CheckStatus.OK, + CheckStatus.OK, + CheckStatus.INFO, + ] + assert results[4].finding == f"you are a {Role.EXTERNAL.value}" + + [email protected]( + ("overrides", "message"), + [ + pytest.param({"gh_version": (1, "")}, "not installed", id="no-gh"), + pytest.param({"gh_version": (0, "gh version 2.10.0")}, "older than", id="old-gh"), + pytest.param({"login": (1, "")}, "not authenticated", id="not-logged-in"), + ], +) +def test_gh_problems_stop_the_audit(overrides, message): + results = _audit(_machine(**overrides)) + + assert list(results) == [1] + assert results[1].status == CheckStatus.FAIL + assert message in results[1].finding + + [email protected]( + ("remotes", "fix"), + [ + pytest.param( + "origin\thttps://github.com/apache/airflow.git (fetch)\n", + ( + "git remote rename origin upstream", + "git remote add origin https://github.com/someone/airflow.git";, + ), + id="cloned-upstream-as-origin", + ), + pytest.param( + "origin\[email protected]:someone/airflow.git (fetch)\napache\thttps://github.com/apache/airflow (fetch)\n", + ("git remote rename apache upstream",), + id="upstream-named-apache", + ), + pytest.param( + "origin\[email protected]:someone/airflow.git (fetch)\n", + ("git remote add upstream https://github.com/apache/airflow.git";,), + id="no-upstream", + ), + pytest.param( + "origin\thttps://github.com/apache/airflow.git (fetch)\n" + "upstream\thttps://github.com/apache/airflow.git (fetch)\n", + ("git remote set-url origin https://github.com/someone/airflow.git";,), + id="origin-also-upstream", + ), + pytest.param( + "origin\[email protected]:someone/airflow.git (fetch)\n" + "upstream\thttps://github.com/someone-else/airflow.git (fetch)\n", + ("git remote set-url upstream https://github.com/apache/airflow.git";,), + id="upstream-points-elsewhere", + ), + ], +) +def test_remote_fixes(remotes, fix): + results = _audit(_machine(remotes=(0, remotes))) + + assert results[2].status == CheckStatus.FAIL + assert results[2].fix == fix + + +def test_origin_is_not_added_when_the_remotes_fix_repoints_it(): + results = _audit( + _machine( + remotes=( + 0, + "origin\thttps://github.com/apache/airflow.git (fetch)\n" + "upstream\thttps://github.com/apache/airflow.git (fetch)\n", + ) + ) + ) + + assert results[3].fix == ("gh repo fork apache/airflow --remote=false",) + + +def test_origin_is_added_once_when_upstream_was_cloned_as_origin(): + results = _audit(_machine(remotes=(0, "origin\thttps://github.com/apache/airflow.git (fetch)\n"))) + + assert results[3].fix == ("gh repo fork apache/airflow --remote=false",) + + [email protected]( + ("repo", "message"), + [ + pytest.param(json.dumps({"owner": "someone", "parent": None}), "is not a fork", id="not-a-fork"), + pytest.param( + json.dumps({"owner": "other", "parent": "apache/airflow"}), "is not owned", id="not-owned" + ), + ], +) +def test_fork_problems(repo, message): + results = _audit(_machine(repo=(0, repo))) + + assert results[3].status == CheckStatus.FAIL + assert message in results[3].finding + assert 5 not in results + + [email protected]( + ("config", "membership", "collaborators", "workflow", "role", "status", "fix"), + [ + pytest.param( + GateConfig(enabled=True), + NOT_A_MEMBER, + frozenset(), + (0, "disabled_fork"), + Role.EXTERNAL, + CheckStatus.FAIL, + ("gh workflow enable ci-amd.yml -R someone/airflow",), + id="external-disabled", + ), + pytest.param( + GateConfig(enabled=True), + NOT_A_MEMBER, + frozenset(), + (1, "", "gh: Not Found (HTTP 404)"), + Role.EXTERNAL, + CheckStatus.FAIL, + ( + "gh repo sync someone/airflow --branch main", + "gh workflow enable ci-amd.yml -R someone/airflow", + ), + id="external-missing-workflow", + ), + pytest.param( + GateConfig(enabled=False), + NOT_A_MEMBER, + frozenset(), + (0, "disabled_fork"), + Role.EXTERNAL, + CheckStatus.INFO, + (), + id="external-before-switch-on", + ), + pytest.param( + GateConfig(enabled=False, pilot={"someone": datetime(2026, 10, 20, tzinfo=UTC)}), + NOT_A_MEMBER, + frozenset(), + (0, "disabled_fork"), + Role.PILOT, + CheckStatus.FAIL, + ("gh workflow enable ci-amd.yml -R someone/airflow",), + id="pilot-disabled", + ), + pytest.param( + GateConfig(enabled=True), + (0, ""), + frozenset(), + (0, "active"), + Role.COMMITTER, + CheckStatus.INFO, + ("gh workflow disable ci-amd.yml -R someone/airflow",), + id="committer-enabled-is-not-a-failure", + ), + pytest.param( + GateConfig(enabled=True), + NOT_A_MEMBER, + frozenset(), + (1, "", "HTTP 502: Bad Gateway"), + Role.EXTERNAL, + CheckStatus.INFO, + (), + id="workflow-call-failed", + ), + pytest.param( + GateConfig(enabled=True, pilot={"someone": datetime(2026, 10, 20, tzinfo=UTC)}), + (0, ""), + frozenset(), + (0, "disabled_fork"), + Role.PILOT, + CheckStatus.FAIL, + ("gh workflow enable ci-amd.yml -R someone/airflow",), + id="committer-in-the-pilot", + ), + pytest.param( + GateConfig(enabled=True), + NOT_A_MEMBER, + frozenset({"SomeOne"}), + (0, "disabled_manually"), + Role.COLLABORATOR, + CheckStatus.OK, + (), + id="collaborator-disabled", + ), + pytest.param( + GateConfig(enabled=True, allowlist={"Org": ("someone",)}), + NOT_A_MEMBER, + frozenset(), + (0, "disabled_fork"), + Role.EXEMPTED, + CheckStatus.OK, + (), + id="exempted-disabled", + ), + pytest.param( + GateConfig(enabled=True), + (1, "", "HTTP 502: Bad Gateway"), + frozenset(), + (0, "disabled_fork"), + Role.UNKNOWN, + CheckStatus.INFO, + (), + id="membership-not-checked", + ), + ], +) +def test_fork_workflows_by_role(config, membership, collaborators, workflow, role, status, fix): + results = _audit(_machine(membership=membership, workflow=workflow), config, collaborators) + + assert results[4].finding == f"you are a {role.value}" + assert results[5].status == status + assert results[5].fix == fix + + [email protected]( + ("behind", "status"), + [ + pytest.param((0, "200"), CheckStatus.OK, id="at-threshold"), + pytest.param((0, "201"), CheckStatus.FAIL, id="too-far-behind"), + pytest.param((1, ""), CheckStatus.INFO, id="compare-failed"), + ], +) +def test_fork_main_freshness(behind, status): + results = _audit(_machine(behind=behind)) + + assert results[6].status == status + + [email protected]("airflow_breeze.commands.ci_commands.run_command", autospec=True) [email protected]("airflow_breeze.commands.ci_commands.user_confirm", autospec=True, return_value=Answer.YES) [email protected]("airflow_breeze.utils.ci_audit.run_audit", autospec=True) [email protected]( + ("args", "second_report", "exit_code", "expected_commands"), + [ + pytest.param([], None, 1, [], id="report-only"), + pytest.param(["--fix"], [], 0, [["git", "remote", "add", "upstream", "u"]], id="fixed"), + ], +) +def test_audit_command( + mock_run_audit, mock_confirm, mock_run_command, args, second_report, exit_code, expected_commands +): + failing = [CheckResult(2, "remotes", CheckStatus.FAIL, "no upstream", ("git remote add upstream u",))] + mock_run_audit.side_effect = [failing] if second_report is None else [failing, second_report] + + result = CliRunner().invoke(audit, args) + + assert result.exit_code == exit_code, result.output + assert [c.args[0] for c in mock_run_command.call_args_list] == expected_commands + + [email protected]( + ("branch", "tracking", "expected"), + [ + pytest.param((0, "main"), (0, "origin/main"), "create a branch", id="on-main"), + pytest.param((0, "fix"), (1, ""), "has no upstream", id="untracked"), + pytest.param((0, "fix"), (0, "upstream/fix"), "tracks upstream/fix; push it", id="tracks-upstream"), + pytest.param((0, "fix"), (0, "origin/fix"), "`fix` tracks origin/fix", id="tracks-origin"), + ], +) +def test_branch(branch, tracking, expected): + results = _audit(_machine(branch=branch, tracking=tracking)) + + assert expected in results[7].finding + + +def test_run_tool_reports_a_missing_executable(): + result = run_tool(["breeze-audit-no-such-executable", "--version"]) + + assert result.returncode == 127 + assert "breeze-audit-no-such-executable" in result.stderr + + +def test_missing_fork_fails(): + results = _audit(_machine(repo=(1, "", "gh: Not Found (HTTP 404)"))) + + assert results[3].status == CheckStatus.FAIL + assert results[3].finding == "someone/airflow does not exist" + assert 5 not in results + + +def test_failed_fork_lookup_is_not_reported_as_a_wrong_fork(): + results = _audit(_machine(repo=(1, "", "HTTP 502: Bad Gateway"))) + + assert results[3].status == CheckStatus.INFO + assert results[3].finding.startswith("could not check someone/airflow") + assert 5 not in results + + +def test_private_apache_member_is_gated_like_an_external_contributor(): + results = _audit(_machine(private_membership=(0, "active"), workflow=(0, "disabled_fork"))) + + assert results[4].finding.startswith( + f"you are a {Role.EXTERNAL.value}: your apache membership is private" + ) + assert results[4].fix == ("gh api -X PUT orgs/apache/public_members/someone",) + assert results[5].status == CheckStatus.FAIL + assert results[5].fix == ("gh workflow enable ci-amd.yml -R someone/airflow",) + + [email protected]( + ("membership", "status"), + [ + pytest.param((0, ""), CheckStatus.INFO, id="committer"), + pytest.param(NOT_A_MEMBER, CheckStatus.FAIL, id="external"), + pytest.param((1, "", "HTTP 502: Bad Gateway"), CheckStatus.INFO, id="membership-not-checked"), + ], +) +def test_fork_setup_matters_only_for_fork_ci(membership, status): + results = _audit( + _machine( + membership=membership, remotes=(0, "origin\thttps://github.com/apache/airflow.git (fetch)\n") + ) + ) + + assert results[2].status == status + assert results[3].status == status + + +def test_wrong_upstream_finding_does_not_print_credentials(): + remotes = ( + "origin\[email protected]:someone/airflow.git (fetch)\n" + "upstream\thttps://someone:[email protected]/someone-else/airflow.git (fetch)\n" + ) + + results = _audit(_machine(remotes=(0, remotes))) + + assert "ghp_secret" not in results[2].finding + assert "https://github.com/someone-else/airflow.git"; in results[2].finding diff --git a/dev/breeze/tests/test_git_remotes.py b/dev/breeze/tests/test_git_remotes.py new file mode 100644 index 00000000000..0b4d2b1a799 --- /dev/null +++ b/dev/breeze/tests/test_git_remotes.py @@ -0,0 +1,103 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import pytest + +from airflow_breeze.utils.git_remotes import ( + Remote, + find_remote, + find_remote_for_repo, + get_github_repo, + parse_remotes, +) + + [email protected]( + ("url", "expected"), + [ + pytest.param("https://github.com/apache/airflow.git";, "apache/airflow", id="https"), + pytest.param("https://github.com/apache/airflow";, "apache/airflow", id="https-no-suffix"), + pytest.param( + "https://[email protected]/someone/airflow.git";, "someone/airflow", id="https-credentials" + ), + pytest.param("[email protected]:someone/my-airflow.git", "someone/my-airflow", id="ssh"), + pytest.param("ssh://[email protected]/someone/airflow.git", "someone/airflow", id="ssh-url"), + pytest.param("http://github.com/apache/airflow.git";, "apache/airflow", id="http"), + pytest.param("https://www.github.com/apache/airflow";, "apache/airflow", id="www"), + pytest.param("https://GitHub.com/apache/airflow.git";, "apache/airflow", id="upper-case-host"), + pytest.param( + "ssh://[email protected]:22/someone/airflow.git", "someone/airflow", id="ssh-url-with-port" + ), + pytest.param( + "git@github-personal:apache/airflow.git", "apache/airflow", id="ssh-host-alias-from-ssh-config" + ), + pytest.param("github-work:someone/airflow.git", "someone/airflow", id="ssh-host-alias-without-user"), + pytest.param( + "ssh://git@github-personal:2222/apache/airflow.git", "apache/airflow", id="ssh-url-alias" + ), + pytest.param( + "[email protected]:apache/airflow.git", "apache/airflow", id="ssh-github-com-suffixed-alias" + ), + pytest.param( + "ssh://[email protected]/apache/airflow.git", + "apache/airflow", + id="ssh-url-github-com-suffixed-alias", + ), + pytest.param( + "[email protected]:apache/airflow.git", None, id="ssh-dotted-github-com-prefix" + ), + pytest.param("[email protected]:someone/airflow.git", None, id="ssh-not-github"), + pytest.param("[email protected]:apache/airflow.git", None, id="ssh-github-enterprise-host"), + pytest.param("https://gitlab.com/someone/airflow.git";, None, id="not-github"), + pytest.param("/local/path/airflow", None, id="local-path"), + ], +) +def test_get_github_repo(url, expected): + assert get_github_repo(url) == expected + + +REMOTE_V = """\ +origin\[email protected]:someone/airflow.git (fetch) +origin\[email protected]:someone/airflow.git (push) +upstream\thttps://github.com/apache/airflow.git (fetch) +upstream\thttps://github.com/apache/airflow.git (push) +site\thttps://github.com/apache/airflow-site.git (fetch) +site\thttps://github.com/apache/airflow-site.git (push) +""" + + +def test_parse_remotes(): + assert parse_remotes(REMOTE_V) == [ + Remote("origin", "[email protected]:someone/airflow.git", "someone/airflow"), + Remote("upstream", "https://github.com/apache/airflow.git";, "apache/airflow"), + Remote("site", "https://github.com/apache/airflow-site.git";, "apache/airflow-site"), + ] + + +def test_find_remote_for_repo_matches_the_exact_repository(): + remotes = parse_remotes("site\thttps://github.com/apache/airflow-site.git (fetch)\n" + REMOTE_V) + + assert find_remote_for_repo(remotes, "Apache/Airflow") == remotes[2] + assert find_remote_for_repo(remotes, "other/repo") is None + + +def test_find_remote(): + remotes = parse_remotes(REMOTE_V) + + assert find_remote(remotes, "origin") == remotes[0] + assert find_remote(remotes, "missing") is None
