I concur with the vision and share the similar feeling that I often get overwhelmed trying to close noisy / low effort PRs that I do not have enough mindspace to provide good reviews where it matters.
I was not proposing using AI in CI, I was rather saying we build upon Ash's suggestion or any other such counter suggestion and validate and early-filter using one of our CI workflows. Thanks & Regards, Amogh Desai On Tue, Jun 9, 2026 at 7:25 PM Vincent Beck <[email protected]> wrote: > Without proposing a solution I also feel the pain of this overwhelming > data generated by AI. This makes it even harder to (humanly) review other > PRs given the number of low quality PRs out there. One consequence I could > notice is people are working more in silos nowadays and are reviewing only > PRs from people from their group/team. Even tagging or assigning someone as > code reviewer barely work (I include myself in that :)). > > Writing code and writing plain english has never been that easy and cheap, > I agree we should try to find a way to control that flow, the more is > definitely not the better. > > On 2026/06/09 13:24:09 Pierre Jeambrun wrote: > > "I want to be involved in a community of humans working to build > software. > > I do not want to see LLMs producing so much output that other people need > > LLMs to summarise it, with no humans looking at things" > > > > I definitely share this feeling and for me we're already there. So much > > output (most of it crap) to ingest that I need multiple agents to answer > > for me on comments/PR, etc... (I'm sure, generating more crap along the > way > > even if I read and double check everything my agents are producing). > > > > Nowadays it feels really difficult to have time to look at every piece, > in > > a good level of detail myself unless we're ok with queuing up a > significant > > number of code reviews. Also automatic code reviews from copilot and all > > are just making the review process longer and even more complicated - > Every > > PR has dozens of comments to address and to catch up with for context. > > > > > > On Tue, Jun 9, 2026 at 2:48 PM Tzu-ping Chung via dev < > > [email protected]> wrote: > > > > > While I agree that banning AI-opened PRs does not fundamentally > anything, > > > I suspect it would actually be quite useful. Ultimately, we are not > against > > > AI-assisted, or even GENERATED PRs. What we want to prevent is > low-effort > > > submissions, and the additional hurdle would help drive away a lot of > > > spams. > > > It’s similar to captcha. Sure, you can totally work around it with not > too > > > much effort, but most of the low effort actors would not even care to > go > > > through that, but instead choose to bother someone else. In the mean > time, > > > the effort is still low enough that legitimate contributions would not > be > > > bothered too much. It’s a reasonable and practical proof-of-effort IMO. > > > > > > I am in favour of this personally. > > > > > > TP > > > > > > > > > > On 9 Jun 2026, at 20:30, Ash Berlin-Taylor <[email protected]> wrote: > > > > > > > > I don’t care one way or another about using AI as a tool in CI, that > is > > > secondary to my goal which is to try and do something to make it clear > what > > > we expect from people wanting to contribute to Airflow, namely: > > > > > > > > 1. Human involvement. > > > > > > > > By submitting a PR you are saying “yes I want to be a member of the > > > community”. Agents submitting without human interaction go against > this. > > > > > > > > 2. Human ownership. > > > > > > > > It is _your responsibility_ as the PR author to follow up on it, > address > > > comments, and request reviews. > > > > > > > > > > > > I frankly find the AI generated triage comments verbose, and a > waste of > > > time and pure noise even without the `@` spam. > > > > > > > > If the user doesn’t care enough about their own PR to follow up on > it: > > > close it after some time. We don’t need to baby sit them. Nor do I > need yet > > > more commit email messages to read through. > > > > > > > > > > > > So how does it sound: It sounds like hell to me and an even bigger > waste > > > of electricity in a climate crisis. > > > > > > > > I want to be involved in a community of humans working to build > > > software. I do not want to see LLMs producing so much output that other > > > people need LLMs to summarise it, with no humans looking at things. > > > > > > > > -ash > > > > > > > >> On 9 Jun 2026, at 13:18, Jarek Potiuk <[email protected]> wrote: > > > >> > > > >>> Why? Because AI “instructions” cannot be trusted. And I am after a > > > signal > > > >> that people are blindly using LLMs without enough human > introversion. > > > >> > > > >> But is not that what you are doing? This proposal is about adding > > > another > > > >> AI instruction (just hidden in HTML) - how is that going to help? > > > >> > > > >>> You already updated the instructions to not `@` the reviewer here > > > >> > > > >> Indeed, LLMs are not deterministic by nature. But they are > improvable. > > > >> Through iterations of refinement and adding more guardrails we can > > > improve > > > >> it—and this is exactly why I am running it manually to make it > better. > > > This > > > >> is the same as in regular breeze development in the past. Initially, > > > there > > > >> were many small issues - and I remember how you complained about > them > > > and > > > >> how unnecessary they seemed—yet we now perfected it over time. Now, > it > > > >> allows all contributors and maintainers to work much more > efficiently > > > and > > > >> lose less time. BTW. Thanks for notifying me; I must strengthen > this one > > > >> and see why, as there might be another improvement to implement. > This is > > > >> also why we are not "yet" doing CI analysis by AI - because I want > to > > > >> iterate on it and fix it in the way to know which parts are > > > deterministic. > > > >> > > > >>> I want to do anything and everything to reduce the drive by > > > contribution > > > >> with no human activity. I’m happy to spend my time helping humans, > but > > > if > > > >> they are just going to feed that back to an LLM and burn an > egregious > > > >> amount of carbon: no thank you. > > > >> > > > >> And again I am not sure how the proposal to add that instruction > would > > > >> address this particular issue? Are you just proposing to add another > > > >> instruction for the LLM (or am I wrong?). How does it solve the > problem? > > > >> > > > >> From what I understand we have two basic proposals here - that > > > contradict > > > >> each other: > > > >> > > > >> * Ash - do not use AI to fight with AI at all > > > >> * Amoght, Shahar - use AI in CI > > > >> > > > >> But I think, the triage I am running now shows a third way: > > > >> > > > >> * we use AI to try out and generate triage action and figure out > which > > > >> parts are practically 100% deterministic and can help with triage > (this > > > is > > > >> the stats I am gathering now) > > > >> * qe use AI to convert the SKILLS we have into deterministic CI code > > > that > > > >> does those triage steps (no AI used at all at runtime) > > > >> * we continue perfecting the manually-triggered AI SKILLS to get > more AI > > > >> heuristics that we can turn into deterministic CI code > > > >> > > > >> This seems to fulfill seemingly contradictory expectations that > > > different > > > >> people have in a nice way. I am about to produce stats from the > last run > > > >> and was just about to propose this approach. > > > >> > > > >> How does it sound Ash, Amogh, Shahar and others ? > > > >> > > > >> J. > > > >> > > > >> > > > >> On Tue, Jun 9, 2026 at 12:55 PM Ash Berlin-Taylor <[email protected]> > > > wrote: > > > >> > > > >>> Why? Because AI “instructions” cannot be trusted. And I am after a > > > signal > > > >>> that people are blindly using LLMs without enough human > introversion. > > > >>> > > > >>> Want a prime example? > > > >>> > > > >>> The pr triage skill. > > > >>> > > > >>> You already updated the instructions to not `@` the reviewer here > > > >>> > > > > https://github.com/apache/airflow-steward/blob/76cfa5e1d2e682b88df5205e9cda396df51a66b6/skills/pr-management-triage/comment-templates.md#reviewer-mention-policy > > > >>> > > > >>>> When a comment's only addressee is the PR author (the > > > >>> request-author-confirmation, reviewer-ping author-primary, and > > > review-nudge > > > >>> author-primary templates), the body references the reviewer without > > > >>> @-mentioning them > > > >>> > > > >>> And yet the LLM did it again: > > > >>> > https://github.com/apache/airflow/pull/66633#discussion_r3344849352 > > > >>> > > > >>>> @korex-f — A reviewer (@ashb) has requested changes on this PR, so > > > I've > > > >>> removed the ready for maintainer review label — the next step is on > > > your > > > >>> side. Could you address the review comments (push a fix, or reply > > > in-thread > > > >>> explaining why the feedback doesn't apply)? Once addressed, > re-request > > > >>> review from @ashb or re-mark the PR ready and it returns to the > > > maintainer > > > >>> queue. Thank you. > > > >>> > > > >>> And frankly I’m tired of all this shit. > > > >>> > > > >>> I want to do anything and everything to reduce the drive by > > > contribution > > > >>> with no human activity. I’m happy to spend my time helping humans, > but > > > if > > > >>> they are just going to feed that back to an LLM and burn an > egregious > > > >>> amount of carbon: no thank you. > > > >>> > > > >>> -ash > > > >>> > > > >>> > > > >>>> On 9 Jun 2026, at 10:38, Jarek Potiuk <[email protected]> wrote: > > > >>>> > > > >>>> Hi Ash, Amogh, and Shahar, > > > >>>> > > > >>>> Ash, I'm curious to learn more about how the "brown m&m test" > differs > > > >>> from > > > >>>> our current request for agents to identify themselves. Could you > help > > > me > > > >>>> understand the flow and the specific benefits you see? It feels > > > similar > > > >>> to > > > >>>> me, but I'd love to hear your perspective in case I'm missing a > > > nuance. > > > >>>> > > > >>>> Regarding the gh pr create --web approach, we included those > > > instructions > > > >>>> to ensure we meet ASF legal guidelines for Gen-AI headers, and to > > > support > > > >>>> contributors who might not have Copilot. That said, if you have > ideas > > > on > > > >>>> how to trim the context or improve the templates, we truly > appreciate > > > PRs > > > >>>> that improve them—and many people already have. AGENTS.md is a > team > > > >>> effort, > > > >>>> and we’re always looking for ways to make it better. Let's keep > our > > > >>>> collaboration positive as we refine these processes together. > > > >>>> > > > >>>> Amogh and Shahar, yep the idea of an validatio step in the CI for > > > >>>> first-time contributions is something we should implement sooner > or > > > >>> later. > > > >>>> I have actually been gathering stats on this for the last two > weeks. > > > I’ve > > > >>>> been preparing to see how manually triggered triage tasks can turn > > > into > > > >>>> automated ones—I'm gathering stats on when human judgment is > needed. I > > > >>>> shared some stats about this recently and will continue gathering > > > them. > > > >>> The > > > >>>> next step is discussing here what and how we can automate. > > > >>>> > > > >>>> Also, the current triage process already uses our Pull Request > > > criteria > > > >>> to > > > >>>> pre-classify the PRs and only marks them with "ready for > maintainer > > > >>> review" > > > >>>> if those criteria are met. So, if there are any specific criteria > > > you’d > > > >>>> like to see added to our "Pull request criteria," PRs are most > welcome > > > >>>> there as well. > > > >>>> > > > >>>> Best regards, > > > >>>> > > > >>>> Jarek > > > >>> > > > >>> > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
