Now that we aren’t using ASF runners anyway, what is the driver for this
change? Are we worried about how quickly we are using our AWS credits?
---
More fundamentally: I don’t get what problem we are actually hoping to solve
with this change?
—
And also, the bigger question above, -1 to the flow **as it stands today**
(even assuming the 15min fork reconciler is moved to a quicker interval later).
This flow feels clunky, and until my above question is answered clearly and
succinctly, I do not see the value in this change.
Some of these may be fixable, but here are the things I don’t like about this
flow:
- It wasn’t clear to me from this vote thread, that by design, we will allow
some PRs to pass based on the CI status reported from the fork.
This makes the next point _even more important to fix_.
- This tests something different to the CI in the apache/airflow repo on PR in
one very important way.
When the CI is run on a PR in apache/airflow, this is running the CI against
the merge commit of the PR to the target branch (`refs/pull/N/merge`), but the
run ci in fork is just against the tip of the branch.
This one is probably fixable by changing the ci-amd.yaml to notice when it’s
on a fork and finding the matching PR somehow.
- Frankly I find the “set the PR to draft” instantly on a CI failure abusive
and hostile.
Doubly so as once it is put in to draft, it stays there. It didn’t get
automatically marked as not draft for me. Maybe I didn’t wait long enough? This
PR stayed in draft for the whole fork run, which was about 45 minutes here,
plus up to one reconciler cycle, so roughly an hour from push to ready.
To fix this, do it after 1 day of failing CI status, not as soon as we notice
it.
- No granular feedback report, or even obvious link to view the actions runs on
the fork.
Ahh, the delayedreconciler makes this behaviour non-obvious. It does handle
it by reporting the status checks, this was cos the reconcile jobs runs every
15 mins right now
- Adding "use project ci" doesn't stop a PR merging on the fork CI result, and
with auto-merge enabled that's wide enough to drive a
bus through.
Once fork CI is green, the reconciler posts the required "CI passed" status
as success.
Adding "use project ci" never withdraws that success state, and doesn't
start project CI either; for that you have to close and reopen the PR (or push).
So the sequence is:
fork CI goes green
-> add "use project ci"
-> close/reopen
-> approve with auto-merge on, or merge, before project CI finishes
-> it merges on the fork's "CI passed".
I tested this: with project CI queued, the PR showed as mergeable, and the
fork's status was the only "CI passed" on the commit.
So to make sure a PR is validated by project CI, you have to remember to
add the label, close and reopen, and then not merge or enable auto-merge until
project CI has finished and passed.
My tl:dr: this makes our CI more complex. And I don’t see the complexity cost
outweighs the benefits.
[^1] It currently lives at
dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md - is that it’s final
home?
> On 5 Nov 2026, at 10:51, Ash Berlin-Taylor via dev <[email protected]>
> wrote:
>
> Thanks, testing it here https://github.com/apache/airflow/pull/76536
>
> The main thing I want to see around is the experience of “fork CI is green,
> now what do we have to do to merge it”.
>
>> On 5 Nov 2026, at 04:17, Jarek Potiuk <[email protected]> wrote:
>>
>> Yes.
>>
>> What Shahar wrote - that was also from the beginning part of the AIP-120
>> proposal - having labels, way to test it - and we already tried it over
>> last 2 weeks, made a number of improvements, have a few edge cases to
>> handle (all described in the VOTE email) and thread open in #contributors
>> where we discussed all that. The email in the devlist above is just a
>> summary of it.
>>
>> So - absolutely, Ash - add "use fork ci" on your PRs and you will get the
>> experience. Even more - you can actually, review, rebase, re-run, review,
>> and approve any of the PRs from the people listed in
>> https://github.com/apache/airflow/blob/main/.github/fork-ci-gate.toml - who
>> offered to test it during the pilot. Report issues or raise concerns if you
>> want, and we (and even some of the contributors who volunteered to pilot
>> it) are addressing any issues and ironing the wrinkles within hours. We
>> have a whole team of people who are able to help address any of your
>> concerns there (and they did so over the last 2 weeks).
>>
>> J.
>>
>> On Wed, Nov 4, 2026 at 12:40 PM Shahar Epstein via dev <
>> [email protected]> wrote:
>>
>>> Ash - to test it out, simply label your own PR with "use fork ci" and
>>> trigger the workflow (push any change).
>>> The participants of the pilot get this auto-labeled, but maintianers could
>>> apply it manually as well.
>>>
>>>
>>> https://github.com/apache/airflow/blob/0ae09c32182e2fd0f4f50b1ce423dd1b42f77343/contributing-docs/18_contribution_workflow.rst#ci-in-your-fork
>>>
>>> ---
>>>
>>> +1 (binding) from me, this is really needed at this point of time.
>>>
>>>
>>> Shahar
>>>
>>> On Wed, Nov 4, 2026 at 2:52 PM Ash Berlin-Taylor via dev <
>>> [email protected]> wrote:
>>>
>>>> Before I vote on this, is there a way I can test out the proposed
>>>> model/changes so I can see what the flow looks like, both from a
>>> maintainer
>>>> and a contributor?
>>>>
>>>>> On 4 Nov 2026, at 11:06, Jarek Potiuk <[email protected]> wrote:
>>>>>
>>>>> Hello everyone,
>>>>>
>>>>> I am calling a vote on AIP-120, "Your PR, your CI" [1]. As I said when
>>>>> I withdrew the earlier [RESULT] and reopened the discussion [2], this
>>>>> goes to a vote rather than [LAZY CONSENSUS], because not everyone who
>>>>> objected has withdrawn their objection.
>>>>>
>>>>> AIP-120 will change to ADIP-12 if the separate [LAZY CONSENSUS] on the
>>>>> new ADIP wiki section passes [3]. That only changes where the page
>>>>> lives and what it is called. This vote does not depend on it.
>>>>>
>>>>> What we are voting on
>>>>> ---------------------
>>>>>
>>>>> Whether to switch on CI in contributors' forks for all external
>>>>> contributors, as described in AIP-120 [1] and the design [4]:
>>>>>
>>>>> * PRs from external contributors run CI in the contributor's own fork,
>>>>> on GitHub's free runners, instead of on the ASF runners. Committers,
>>>>> collaborators and exempted organisations keep project CI, unchanged.
>>>>>
>>>>> * A contributor enables one workflow in their fork, once. After that a
>>>>> plain "git push" runs CI. No PR against their own fork is needed, and
>>>>> "breeze ci audit" checks the setup.
>>>>>
>>>>> * The PR shows the fork run as a commit status linking to the run. A PR
>>>>> is drafted when the fork needs setup or the run failed, and undrafted
>>>>> when the run is green. A run in progress does not draft it.
>>>>>
>>>>> * A green fork run is not enough when the PR changes what decides which
>>>>> jobs run and on which runners (workflows, breeze sources and similar).
>>>>> Such PRs stay visible for review, but wait for a maintainer to run
>>>>> project CI with the "use project ci" label. Maintainers can use that
>>>>> label on any PR they want verified in project CI before merging.
>>>>>
>>>>> * A gated PR with no fork CI activity for 7 days is closed, and can be
>>>>> reopened.
>>>>>
>>>>> Not part of this vote:
>>>>>
>>>>> * The one-time reset of the open PR backlog. It is not proposed here.
>>>>> * Which organisations are exempted. That is agreed separately on this
>>>>> list.
>>>>>
>>>>> Where things stand
>>>>> ------------------
>>>>>
>>>>> The implementation is merged and switched off for everyone except the
>>>>> pilot: "enabled = false" in .github/fork-ci-gate.toml. If the vote
>>>>> passes, I will post a final pilot update on this list and then switch
>>>>> it on in a separate PR. Rollback is setting "enabled = false" again. If
>>>>> the vote does not pass, it is not switched on for all external
>>>>> contributors.
>>>>>
>>>>> What the pilot showed
>>>>> ---------------------
>>>>>
>>>>> Seven external contributors joined the opt-in pilot on 2026-10-31. So
>>>>> far 16 new PRs carry the "fork ci required" label, 9 of them merged,
>>>>> plus some older PRs that were rebased. Feedback is in the
>>>>> #internal-airflow-ci-cd Slack channel. Some quotes:
>>>>>
>>>>> "So far, this has been far less inconvenient that I expected. The
>>>>> only downside is that the CI workflow takes a bit longer on my own
>>>>> fork that when using the projects runners." - Sameer Mesiah
>>>>>
>>>>> "Total run time was ~30 minutes, which is basically on par with how
>>>>> long it was taking on the shared Apache CI (for this PR
>>>>> specifically)." - Justin Pakzad
>>>>>
>>>>> "I was actually quite surprised that the runtimes were pretty close
>>>>> (in some cases quicker) on my fork. I only tested a provider/task-sdk
>>>>> PRs, I'm assuming larger core changes will def take longer." - Justin
>>>>> Pakzad
>>>>>
>>>>> "I love it that I can finally rerun failed jobs since they're running
>>>>> on the fork" - Iliya Romm
>>>>>
>>>>> That last one was a benefit we did not plan: contributors can restart
>>>>> a single failed job themselves, without asking a maintainer.
>>>>>
>>>>> It was not all smooth, and that is what the pilot was for:
>>>>>
>>>>> "this had a green (main) CI run, with two maintainer approvals, then
>>>>> I rebased and the fork CI failed, which auto-marked the PR as draft.
>>>>> [...] I think the auto-draft rules should be more careful." - Iliya
>>>>> Romm
>>>>>
>>>>> "My PR only changes documentation [...] but fork CI converted it to
>>>>> draft automatically." - Aaron Chen
>>>>>
>>>>> "GitHub just disabled it without any notification, so my workflows on
>>>>> fork silently stopped" - Andrew Chang
>>>>>
>>>>> Improvements made after the pilot started
>>>>> -----------------------------------------
>>>>>
>>>>> * Rebased pilot PRs are now gated, not only new ones, and a required
>>>>> "CI run started" check shows from the moment of the push (#76148).
>>>>> * Cancelled project CI runs no longer leave a mixed red/green status
>>>>> on the PR (#76148).
>>>>> * A run in progress no longer drafts a PR, so rebasing an approved PR
>>>>> does not draft it unless the new run fails (#76151).
>>>>> * One pending status links to the fork run, replacing a wall of pending
>>>>> jobs. This also spends fewer GitHub API writes, and we now watch the
>>>>> rate limits (#76154).
>>>>> * Scheduled workflows are skipped in forks. They were the likely reason
>>>>> GitHub disabled Actions on some forks (#76243).
>>>>> * "breeze ci audit" checks that the fork's GitHub Actions can actually
>>>>> run (#76136, by Andrew).
>>>>> * Only changes that can steer CI are held for a maintainer, and they
>>>>> are no longer drafted, so docs under dev/breeze/ pass normally
>>>>> (#76282).
>>>>> * Fork CI PRs that change CI files no longer get the misleading "need
>>>>> workflow approval" label (#76289, by Andrew).
>>>>> * A re-run that passes now replaces the failed job statuses on the PR
>>>>> (#76327).
>>>>> * Fork image names are lowercased in the PROD image example tests
>>>>> (#76143), the contributor docs were improved (#76272, by Aaron), and
>>>>> transient GitHub errors no longer cause repeated alerts (#76185).
>>>>>
>>>>> Still open:
>>>>>
>>>>> * Status on the PR lags, because the monitor runs every 15 minutes.
>>>>> Aaron saw the first bot action after about 23 minutes.
>>>>> * The monitor's budget for job status writes still fills up with
>>>>> several large fork runs active. We need to watch this as more
>>>>> contributors join.
>>>>> * A failed run after a rebase still drafts the PR, which can cancel
>>>>> auto-merge when the failure is a flaky test. Iliya also suggested that
>>>>> maintainer approvals could count towards running project CI.
>>>>> * The labels are confusing and could use better names.
>>>>> * Pilot PRs were mostly small to medium. Large core changes have had
>>>>> little testing.
>>>>>
>>>>> The objections
>>>>> --------------
>>>>>
>>>>> You should vote knowing what was raised. Please read the original
>>>>> thread [5] and the [RESULT] thread [6] rather than rely on my summary.
>>>>>
>>>>> * Ash's main objection is that this optimises for the wrong thing: the
>>>>> problem is PRs not being reviewed, and making contributors run CI
>>>>> does nothing about that backlog. Ash also pointed out that allowing
>>>>> contributors to run CI in forks and forcing them to are separate
>>>>> things. This vote is about requiring it. Ash has not withdrawn this
>>>>> objection.
>>>>>
>>>>> My answer: it does not add review capacity, and I do not claim it
>>>>> does. The aim is to slow down iteration on big PRs, so that fewer of
>>>>> them compete for review, and to move the cost of getting a PR green
>>>>> to its author. How much that helps is my expectation, not a measured
>>>>> result.
>>>>>
>>>>> * Ash and Dennis both raised the effect on new contributors. Dennis
>>>>> described it as a big barrier to entry, and Ash the case of someone
>>>>> doing all the setup only to have the PR rejected afterwards.
>>>>>
>>>>> My answer: the setup is now one step, with no second PR and no
>>>>> sorting through workflow files, but it is still an extra step that a
>>>>> new contributor did not have before. That cost is real. The pilot
>>>>> participants are experienced contributors, so the pilot does not tell
>>>>> us how a first-time contributor will find it.
>>>>>
>>>>> * Slower CI for big PRs is intended, but it should be stated plainly.
>>>>> On 2026-10-05 I estimated about 85 minutes or more for a full matrix
>>>>> run in a fork. The first real full run (#75695) took about 2 hours 40
>>>>> minutes.
>>>>>
>>>>> On the other side, Arrow, DataFusion and SeaTunnel already run PR CI in
>>>>> contributors' forks, or are moving to it, mainly to reduce pressure on
>>>>> the shared ASF runners. For us that is a side benefit.
>>>>>
>>>>> How the vote works
>>>>> ------------------
>>>>>
>>>>> This is a procedural vote, for the reasons in [2]: where CI runs for
>>>>> contributors' PRs is a decision about how the project operates, not a
>>>>> change to Airflow itself or its public interfaces. Hussein asked about
>>>>> this on the [RESULT] thread, and nobody objected to it in the reopened
>>>>> discussion.
>>>>>
>>>>> It passes with at least 3 binding +1 votes and more binding +1 than -1
>>>>> votes [7]. Everyone is welcome to vote; votes from PMC members are
>>>>> binding. A single -1 does not block it, but I will respond to every -1.
>>>>>
>>>>> The vote runs until 2026-11-09 12:00 UTC. That is longer than the
>>>>> 72-hour minimum because of the Summit Reconnect on 4-5 November and the
>>>>> weekend.
>>>>>
>>>>> [ ] +1 Switch on CI in contributors' forks for external contributors,
>>>>> after a final pilot update
>>>>> [ ] 0 No opinion
>>>>> [ ] -1 Do not do this, because...
>>>>>
>>>>> [1]
>>>>
>>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711/AIP-120+Run+CI+for+external+contributions+in+contributor+forks
>>>>> [2] https://lists.apache.org/thread/h5qsfqthmsqoh5j7wyy12r90mnnvz8kt
>>>>> [3] https://lists.apache.org/thread/qxjx3lnwm8gro0dstqr3cvgyj5jg6hyk
>>>>> [4]
>>>>
>>> https://github.com/apache/airflow/blob/main/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
>>>>> [5] https://lists.apache.org/thread/w74655o8y1jzrzcnygntyyovocf7zmzj
>>>>> [6] https://lists.apache.org/thread/krl1bzszlfgfz70m494rm8jq1ph6dr01
>>>>> [7] https://www.apache.org/foundation/voting.html
>>>>>
>>>>> J.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [email protected]
>>>>> For additional commands, e-mail: [email protected]
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [email protected]
>>>> For additional commands, e-mail: [email protected]
>>>>
>>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
