This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch backport-ci-sync-v3-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 24f1e785c3618aca228fe598995779d06c136a46 Author: Jarek Potiuk <[email protected]> AuthorDate: Sun Oct 25 19:40:43 2026 +0100 Document fork CI for contributors and maintainers (#74825) * Document fork CI for contributors and maintainers Contributors need to know how to set up CI in their fork, how to check that setup, and how to join or leave the pilot, and maintainers need to know which parts of a gated pull request's lifecycle the reconciler owns so triage does not fight it. The previous one-line advice to enable GitHub Actions in the fork did not say any of that. Generated-by: Claude Opus 5 * Say when fork CI applies in the contributor docs While the gate is off, a contributor outside the pilot who enabled fork CI as the docs said got CI twice on every pull request. The docs now say fork CI applies to pilot participants and use fork ci pull requests until it is switched on for everyone, and the maintainer docs cover the needs project ci label, the CI passed status and why removing fork ci required is not enough. Generated-by: Claude Opus 5 * Describe apache/airflow push runs after release-branch CI moved to a schedule ci-arm.yml no longer lists the push branches (#75420). Point to CANONICAL_PUSH_BRANCH_FILTERS and say that the incremental push base is only used in apache/airflow. Generated-by: Claude Opus 5 * Fix the fork CI section title level and label wording in the contributor docs The new subsection used a title style the file had not established, which made Newsfragments a level-4 section and broke links to it. Also say when a pull request the author drafted gets ready for maintainer review, use a placeholder pilot instant, and describe the fork compare base as breeze's default branch rather than the branch name. Generated-by: Claude Opus 5 (cherry picked from commit bfc288222a33c697c7354cc919d3ed373a1fc646) --- contributing-docs/03_contributors_quick_start.rst | 13 +++++ .../03a_contributors_quick_start_beginners.rst | 5 ++ contributing-docs/05_pull_requests.rst | 10 +++- contributing-docs/10_working_with_git.rst | 3 + contributing-docs/18_contribution_workflow.rst | 65 +++++++++++++++++++++- contributing-docs/25_maintainer_pr_triage.md | 33 +++++++++++ dev/breeze/doc/ci/05_workflows.md | 25 +++++++++ .../doc/ci/08_fork_ci_for_external_contributors.md | 1 - 8 files changed, 151 insertions(+), 4 deletions(-) diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index 5adc87ad9dc..61b1af39148 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -237,6 +237,19 @@ Forking and cloning Project to clone the repo locally (you can also do it in your IDE - see the `Using your IDE`_ chapter below) +4. Add ``apache/airflow`` as the ``upstream`` remote, following the + `Git remote naming conventions <10_working_with_git.rst>`__: + + .. code-block:: bash + + git remote add upstream https://github.com/apache/airflow.git + + If you use SSH with GitHub, add ``[email protected]:apache/airflow.git`` instead; ``breeze ci audit`` + recognises both forms. + + Once Breeze is installed, run ``breeze ci audit`` before your first push to check your remotes + and the CI in your fork; see `CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__. + .. note:: For windows based machines, on cloning, the Git line endings may be different from unix based systems and might lead to unexpected behaviour on running breeze tooling. Manually setting a property will mitigate this issue. diff --git a/contributing-docs/03a_contributors_quick_start_beginners.rst b/contributing-docs/03a_contributors_quick_start_beginners.rst index a8c333233ad..fdbea14df69 100644 --- a/contributing-docs/03a_contributors_quick_start_beginners.rst +++ b/contributing-docs/03a_contributors_quick_start_beginners.rst @@ -71,6 +71,11 @@ Option A – Breeze on Your Laptop git clone https://github.com/<you>/airflow.git cd airflow + git remote add upstream https://github.com/apache/airflow.git + +Before your first push, run ``breeze ci audit`` (after installing Breeze below) to check that your +remotes and the CI in your fork are set up; see +`CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__. 2. **Set up Git hooks and install Breeze** diff --git a/contributing-docs/05_pull_requests.rst b/contributing-docs/05_pull_requests.rst index 4052986d8ba..c7fee6d2677 100644 --- a/contributing-docs/05_pull_requests.rst +++ b/contributing-docs/05_pull_requests.rst @@ -94,8 +94,14 @@ these guidelines: The Airflow repo uses `GitHub Actions <https://help.github.com/en/actions>`_ to run the tests and `codecov <https://codecov.io/gh/apache/airflow>`_ to track - coverage. You can set up both for free on your fork. It will help you make sure you do not - break the build with your PR and that you help increase coverage. + coverage. Fork CI is being piloted: it applies to pilot participants and to pull requests a + maintainer labels ``use fork ci``, and to every external contributor only once it is switched + on for everyone. Until then, enabling the ``Tests (AMD)`` workflow in your fork outside the + pilot runs CI twice for each pull request, in your fork and in apache/airflow. When fork CI + applies to you, enable that workflow and every push to your fork runs CI for free; pull + requests labelled ``fork ci required`` take their CI result from your fork and stay in draft + until that run is green. See `CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__ + and run ``breeze ci audit``, which tells you whether fork CI applies to you yet. Also we advise to install locally `prek hooks <08_static_code_checks.rst#prek-hooks>`_ to apply various checks, code generation and formatting at the time you make a local commit - which gives you near-immediate feedback on things you need to fix before you push your code to the PR, or in diff --git a/contributing-docs/10_working_with_git.rst b/contributing-docs/10_working_with_git.rst index e51cb1a1f21..edb6b54188d 100644 --- a/contributing-docs/10_working_with_git.rst +++ b/contributing-docs/10_working_with_git.rst @@ -37,6 +37,9 @@ you should configure your local checkout to match: * ``origin`` — your personal fork of ``apache/airflow`` (where you push branches for PRs). +``breeze ci audit`` checks these names, offers the commands below when they differ, and also checks +the CI in your fork; see `CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__. + Always push PR branches to ``origin``; don't push to ``upstream`` (the branch protection on ``apache/airflow`` will reject it anyway). Working on dedicated branches is recommended, though developing directly on your fork's ``main`` is diff --git a/contributing-docs/18_contribution_workflow.rst b/contributing-docs/18_contribution_workflow.rst index 0ce8075e9f0..89a73df9712 100644 --- a/contributing-docs/18_contribution_workflow.rst +++ b/contributing-docs/18_contribution_workflow.rst @@ -43,7 +43,8 @@ In general, your contribution includes the following stages: 2. Create a `local virtualenv <07_local_virtualenv.rst>`_, initialize the `Breeze environment <../dev/breeze/doc/README.rst>`__, and install `prek tool <08_static_code_checks.rst#prek-hooks>`__. - If you want to add more changes in the future, set up your fork and enable GitHub Actions. + Set up the ``upstream`` and ``origin`` remotes and check them with ``breeze ci audit``; if + fork CI applies to you (see `CI in your fork`_), enable CI in your fork as described there. 3. Join `devlist <https://lists.apache.org/[email protected]>`__ and set up a `Slack account <https://s.apache.org/airflow-slack>`__. @@ -126,6 +127,68 @@ to make them immediately visible in the environment. 6. Open your IDE (for example, PyCharm) and select the virtual environment you created as the project's default virtual environment in your IDE. +CI in your fork +~~~~~~~~~~~~~~~ + +Pull requests from contributors who are not committers or collaborators are moving to running +their CI in the contributor's own fork, on GitHub's free runners, instead of on the shared CI +capacity of the Apache Software Foundation (`AIP-120 +<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711/AIP-120+Run+CI+for+external+contributions+in+contributor+forks>`__). +CI on a public fork costs you nothing and has no minute limit; it runs on every push to your +fork, so you get feedback before a maintainer looks at your pull request. The change is being +piloted: until it is switched on for everyone, it applies to pilot participants (see below) and +to pull requests a maintainer labels ``use fork ci``. + +Set up your checkout and fork once: + +1. Name your remotes as described in `Git remote naming conventions <10_working_with_git.rst>`__: + ``upstream`` for ``apache/airflow`` and ``origin`` for your fork. + +2. If fork CI applies to you, enable the ``Tests (AMD)`` workflow in your fork: today that means + you joined the pilot or a maintainer labelled your pull request ``use fork ci``; once fork CI is + switched on for everyone, it means you are not a committer, a collaborator or a member of an + exempted organisation. GitHub disables workflows in new forks. Otherwise keep it disabled, so + your pushes do not run CI twice: + + .. code-block:: bash + + gh workflow enable ci-amd.yml -R <you>/airflow + + If the workflow is missing, your fork's ``main`` is older than it; sync it first with + ``gh repo sync <you>/airflow --branch main``. + +3. Check everything at once. ``breeze ci audit`` prints one row per check and the command that fixes + anything missing; ``--fix`` runs those commands after asking: + + .. code-block:: bash + + breeze ci audit + +When fork CI applies to your pull request, it is labelled ``fork ci required`` and kept in draft +until the ``Tests (AMD)`` run in your fork for its latest commit is green. A +``Fork CI / Tests (AMD)`` status on the pull request links to that run, and one comment on the pull +request explains the current state. Once the run is green, a pull request the monitor drafted is +marked ready for review and labelled ``ready for maintainer review``; a pull request you drafted +yourself stays a draft, and gets the label, once you mark it ready. A pull request whose fork CI +does not run for 7 days is closed with an explanation; you can reopen it after setting up fork CI. If your pull request changes +files that define what CI runs, such as ``.github/``, ``dev/breeze/`` or ``scripts/ci/``, a green +fork run is not enough: a maintainer reviews the change and runs project CI for it. + +**Joining the pilot.** Open a pull request that adds your GitHub login to the ``[pilot]`` table in +``.github/fork-ci-gate.toml``, with a join instant after the expected merge, for example the next day +at 00:00 UTC: + +.. code-block:: toml + + [pilot] + "your-login" = YYYY-MM-DDT00:00:00Z + +Every pull request you open after that instant uses fork CI; your earlier pull requests keep project +CI. To leave the pilot, open a pull request that removes the line. A pilot entry gates your pull +requests whatever your role, so a committer in the pilot also enables the workflow; +``breeze ci audit`` tells you which applies to you. The design is described in +`Fork CI for external contributors <../dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md>`__. + Step 3: Connect with People --------------------------- diff --git a/contributing-docs/25_maintainer_pr_triage.md b/contributing-docs/25_maintainer_pr_triage.md index 046a1b20fa2..ab12c1f877d 100644 --- a/contributing-docs/25_maintainer_pr_triage.md +++ b/contributing-docs/25_maintainer_pr_triage.md @@ -242,6 +242,39 @@ The triage workflow uses the following labels and states: | `suspicious changes detected` | PR (and all open PRs by the same author) was closed because the diff contained suspicious patterns (secret exfiltration, malicious CI modifications, etc.). | | `closed because of open PR limit` | PR was closed in the one-time closure that introduced the [open PR limit](32_open_pull_request_limit.rst). Applied by `dev/close_prs_over_open_pr_limit.py`, never by the triage skill. If the author reopens the PR, triage it like any other. | | **Draft status** | PR was converted to draft because it does not meet quality criteria. The contributor is asked to fix the listed issues and mark the PR as "Ready for review" once done. | +| `fork ci required` | PR runs its CI in the contributor's fork (AIP-120). Applied and managed by the fork CI reconciler, never by the triage skill. | +| `needs project ci` | Fork CI is green, but the PR changes CI-defining files, so a maintainer must review them and add `use project ci`. Applied and removed by the fork CI reconciler. Filter on it to find PRs waiting only on a maintainer. | +| `use project ci` / `use fork ci` | Set by a maintainer to override the fork CI decision for one PR: run project CI, or use fork CI for a PR that would otherwise run project CI. | + +## Pull requests gated to fork CI + +Pull requests labelled `fork ci required` run their CI in the contributor's fork. The scheduled +fork CI reconciler (`breeze ci fork-ci-reconcile`, run by `fork-ci-monitor.yml`) owns four things +for them, and the triage skill must treat those pull requests as out of scope for each: + +- **Draft state.** The reconciler keeps the pull request in draft until the fork run for its + latest commit is green, and undrafts only what it drafted. +- **`ready for maintainer review`.** The reconciler adds it when the fork run is green and the + pull request is not a draft the author made (it adds it once the author marks it ready), and + removes it when a new push starts a new run. +- **`needs project ci`.** The reconciler adds it when the fork run is green but the pull request + changes CI-defining files, and removes it once the state is anything else. These pull requests + stay in draft and wait for a maintainer to review the CI change and add `use project ci`. +- **Closing after inactivity.** The reconciler closes a pull request whose fork CI has not run for + `close_after_days`. + +Everything else the skill does (quality criteria, pings, suspicious-change flags) applies as +usual. The `Fork CI / Tests (AMD)` commit status shows the fork CI state and links to the run in +the fork. To move a single pull request back to project CI, add `use project ci`; the reconciler +releases it on its next run; removing `fork ci required` by hand is not enough, the next run +labels a gated pull request again. Adding the label does not start project CI by itself: re-run +the pull request's latest `Tests (AMD)` run (`gh run rerun <run-id> -R apache/airflow`, or +**Re-run all jobs** on the run's page), or ask the author to push. Pull requests that change CI-defining files stay in draft even +when the fork run is green and are labelled `needs project ci`, for a maintainer to review and run +project CI with `use project ci`. When the fork run is green and trusted, the reconciler also posts +the `CI passed` status the merge ruleset requires, so these pull requests can be merged or +auto-merged without project CI. See +[Fork CI for external contributors](../dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md). ## For contributors diff --git a/dev/breeze/doc/ci/05_workflows.md b/dev/breeze/doc/ci/05_workflows.md index 0842e313d40..7ec7871993c 100644 --- a/dev/breeze/doc/ci/05_workflows.md +++ b/dev/breeze/doc/ci/05_workflows.md @@ -23,6 +23,7 @@ - [CI run types](#ci-run-types) - [Pull request run](#pull-request-run) + - [Fork push run](#fork-push-run) - [Canary run](#canary-run) - [Workflows](#workflows) - [Workflow Architecture Overview](#workflow-architecture-overview) @@ -92,6 +93,30 @@ all dependencies to latest versions and build images with them. This way we chec dependencies are upgraded. This can also be forced by setting the `upgrade to newer dependencies` label in the PR if you are a committer and want to force dependency upgrade. +Every pull request run of `ci-amd.yml` starts with the `Fork CI gate` job. For pull requests +whose CI runs in the contributor's fork (AIP-120), the gate skips `build-info` and therefore +everything after it, so the run finishes in seconds; see +[Fork CI for external contributors](08_fork_ci_for_external_contributors.md). + +## Fork push run + +These runs are triggered by a push to any branch of a contributor's fork in which the +`Tests (AMD)` workflow is enabled. They run in the fork, on GitHub's public runners and with the +fork's token, and are how pull requests labelled `fork ci required` get their CI result. They are +selected like a pull request run: the changed files come from comparing the pushed commit with +`apache/airflow` (`main`, or the release branch the pushed code was cut from, as breeze's default +branch at that commit records it), so a fork whose `main` is behind does not make the run bigger, +and they never use self-hosted runners. The `schedule` trigger is skipped in forks, so enabling +the workflow does not start the canary matrix there. The fork CI reconciler reads the result of these runs and reports it on the pull +request as the `Fork CI / Tests (AMD)` status. + +In `apache/airflow` itself, pushes still only run CI for the release-prep and providers branches +(`CANONICAL_PUSH_BRANCH_FILTERS` in `fork_ci_gate.py`, the branches `release-branch-ci.yml` starts +full runs for), and they test only the changes since the last successful run of the branch. That +incremental base is looked up only in `apache/airflow`: a fork's push is always compared with +`apache/airflow` as described above. A push to any other branch produces a run in which only the +gate executes. + ## Canary run This workflow runs on a schedule on the `main` branch, and a few times a day diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index 6321692fc97..6e352fea115 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -818,7 +818,6 @@ as a failed check rather than a crash. - `contributing-docs/10_working_with_git.rst`: cross-link from the naming conventions. - `contributing-docs/25_maintainer_pr_triage.md`: the ownership split with the reconciler. - `dev/breeze/doc/ci/05_workflows.md`: add the fork push run type. -- `dev/breeze/doc/ci/03_github_variables.md`: the new gate job outputs. - `dev/breeze/doc/08_ci_tasks.rst`: `breeze ci audit` and `breeze ci fork-ci-reconcile`. - `dev/breeze/doc/ci/README.md`: index this document.
