Hello everyone, *TL;DR; I have a very concrete proposal and a complete PR (entirely Claude-coded) that I would like to try as soon as possible, provided the community responds positively. I believe it will allow as to very efficiently handle even way more AI generated PRs than we have now - by efficiently removing (Drafting or Closing) PRs that fail our quality checks and instructing authors what they should do to fix the issues.*
The PR is here: https://github.com/apache/airflow/pull/62682 - it adds a new command to breeze: "breeze pr auto-triage" This is an automated triage of pull requests (PRs) that have not yet been triaged—specifically, all PRs originating from non-collaborators. It's a mixture of deterministic checks (for failing tests, static checks, the number of open bad-quality PRs by the author etc.) and LLM-based assessments (currently Claude Code and Codex are supported) of whether the PR meets our quality criteria. From my initial checks more than 90% of problematic PRs can be assessed without LLMs via deterministic checks. The PR uses highly optimized GraphQL queries to GitHub to make all assessments. This allows for auto-assessment and suggested actions for hundreds of PRs in minutes. More time is needed if LLM assessments are triggered after all deterministic criteria pass, but I parallelised that as well to speed it up. The most important thing here is that it does NOT run automatically. This is a maintainer-driven tool that makes triaging fast and efficient, while absolutely leaving the maintainer in the driving seat. All relevant information is surfaced for each pull request (PR), including author details (number of PRs opened and closed, account age, contribution locations) with links available in the terminal that allow the maintainer to dig deeper into both the PR and the author. Maintainers can very quickly make decisions, allowing the tool to post generated comments and take actions such as converting the PR to a draft or closing it. A maintainer running the triage must trigger all those actions. While it mostly implements and automates our current processes, the important change is that I propose that we: * Automatically convert PRs that fail the criteria to Draft and explain the author that they should un-draft it only when the issues are fixed * We CLOSE all PRs from an author when we detect suspicious changes (when a maintainer needs to determine whether to run workflows for new users) or when the author has more than 3 open PRs failing quality checks. We apply the "closed because of suspicious changes" label or the "closed because of multiple quality violations" label. * We apply label "ready for maintainer review" when both deterministic and LLM checks pass for the PR It completely automates the flow of triager, including generating helpful comments that explain what needs to be fixed (with links to documentation), why the action is necessary, and that users are welcome to re-create PRs after fixing the quality issues. We cannot attach big screenshots here but here is the image that shows example assesment here https://ibb.co/ZprSsjWW I would love to try it **NOW** myself and see if it can help us all and perfect the tool - any suggestions are welcome. Eventually, we might build a group of triagers that will be doing it regularly, I believe this can be done very efficiently and quickly with tool like that. We already have built-in filtering criteria; for example we can triage PRs by "area", specific providers, etc. However, I think we can eventually figure out ways to share the triage effort, especially since I don't think it will be substantial. Let me know what you think. J. On Fri, Feb 27, 2026 at 8:59 PM Jarek Potiuk <[email protected]> wrote: > Hello everyone, > > While preparing for consensus on the assignment policy, I created PR > https://github.com/apache/airflow/pull/62585. This PR adds a new command > to Breeze, `breeze issues unassign`, which unassigns anyone who is not a > committer or collaborator. > > I want this to be the first of several Breeze commands I plan to add to > help manage the AI overhead and burden on maintainers. > > I got inspired bu Hugo van Kamerade's (my friend, Python release manager) > tool https://hugovk.dev/blog/2026/gh-triage/. He added the `gh` plugin > that helps him manage spam coming to Python. I hope we can have very > similar set of commands and regular process of performing cleanup with the > issues/prs we are getting. > > BTW. I am using Claude Code to add those commands (so this is a bit like > using AI to fight AI slop). But in a smart way. > > In our case we have `breeze` that we are already using for `ci upgrade` by > maintainers and I see no reason why we could not use our own CLI to make us > far more efficient with assessing and quickly and efficiently processing > incoming spam. > > Starting with AGENTS.md that describes what we expect (and instructs > agents to make good PRs) and changing our assignment process - I think we > should proceed to implement step-by-step handling of the incoming traffic: > > a) Quickly assess how well PRs implement our expectations, point out > problems, and close them > > b) automatically telling the collaborators what is wrong with their PRs if > they are incomplete (for example when tests are failing, or when they need > a rebase) > > c) automatically responding to issues that they are incomplete and need > more information > > d) Allow filtering by area (so that maintainers focusing on a particular > area can periodically review only the areas they are intereste > e) all that with some AI assistance (I plan to imlpement integration with > some modern AI LLMs so that it is seamless for those maintainers who > already use some of those (including Cloud Code, GH Copilot (maintainers > can apply for free access there), Codex and any models someone prefers - > including local models). > > f) all that with maintainer in the driver's seat—we won't do those things > fully automatically - but we will get reviewable action proposal in bulk > that the maintainer will be able to accept, modify or reject. > > .... more... > > All that will be open to contribution and I will be happy to leading > introduction and disseminating those CLI options between maintainers to > make sure those get incorporated in our daily work - relieving some of the > burden we are all experiencing and sharing it between people. > > I think this is a viable approach to address our current burden > proactively, rather than waiting for others to act. > > This is also somewhat experimental since we haven't seen it done before, > so suggestions, comments, ideas and PRs that could help us become more > efficient and better maintainers are most welcome. > > Let me know what you think. > > J. >
