This sounds great, I like the approach of give them a chance and then close the PR, and using labels for both will make it easier to automate some of the messaging and other work here and reduce the burden on maintainers
On Wed, 28 Oct 2026, 10:11 Raúl Cumplido, <[email protected]> wrote: > Hi, > > I found the new CPython AI usage guidelines [1] interesting. Having > one label to request the author of the PR to explain the contribution > in their own words (no AI usage) sounds like an interesting option. A > second label for expected AI usage not met is used there to close the > PR. > > Regards, > Raúl > > [1] > https://discuss.python.org/t/updated-triage-guidelines-and-process-for-our-ai-policy/109717 > > El mar, 27 oct 2026 a las 16:15, Nic Crane (<[email protected]>) > escribió: > > > > Hi folks, > > > > I spent some time today having a look at the impact of AI on the project, > > and also had AI (ironically) deep dive into some of this[1], looking at a > > few metrics around reviewer capacity, examining PR discussions and also > > looking at discussions here on the developer mailing list, with some > input > > from me in there as well. > > > > I've summarised the main problems and some potential solutions below, but > > I'd like to hear from other maintainers on whether this broadly reflects > > your experiences? > > > > Thanks, > > > > Nic > > > > > > Main problems currently: > > - increased maintainer time spent reviewing PRs which are ultimately > > abandoned after time spent reviewing, or are predominantly AI-generated > so > > take longer to merge as they need more feedback > > - maintainer time spend enforcing rules around AI-generated PRs, and > > non-committers reviewing PRs with little ability to flag them as > problematic > > - sheer volume of AI-generated PRs, as well as around a third of recent > PRs > > skipping AI disclosure at all > > > > Recommendations: > > 1. Move the PR cap to .asf.yaml creation_cap before pull_request_target > is > > disabled in November (#55164). (awaiting merge of > > https://github.com/apache/infrastructure-asfyaml/pull/135) > > 2. Make AI use disclosure mandatory, as numpy, pandas and matplotlib do, > > with a bot that converts PRs missing it to draft. > > 3. Adopt #55260’s rule against AI-written review replies, and rok’s > neutral > > needs-author-engagement label rather than an AI label. > > 4. Auto-draft PRs idle for 14 days with conflicts, failing CI or changes > > requested, and trial fork-hosted CI for non-committer PRs as DataFusion > and > > arrow-rs do. > > 5. Gate PR opening on triage state as pandas does, so PRs are only > expected > > against issues a maintainer has confirmed. > > 6. Track monthly: new-contributor PRs with no human response, and > reviewer > > messages on PRs that never merge. > > 7. Grow the merge group from the people already triaging: give the active > > non-committer reviewers the collaborator role and a label-triggered > > autoclose, and treat that as the path to committer. This is the only > > measure that adds capacity rather than cutting inflow. > > > > [1] Full report here: > > https://thisisnic.github.io/ai-arrow-impact/report.html > > > > On Mon, 26 Oct 2026 at 12:30, Raúl Cumplido <[email protected]> wrote: > > > > > Hi, > > > > > > I've followed up with a PR [1] based on the comments from Pedro Matias > > > and Rok so we have this documented. See that PR for the documentation > > > changes. > > > > > > Regards, > > > Raúl > > > > > > [1] https://github.com/apache/arrow/pull/55260 > > > > > > El mar, 22 sept 2026 a las 10:57, Jean-Baptiste Onofré > > > (<[email protected]>) escribió: > > > > > > > > Discussions must take place publicly, even between contributors from > the > > > > same company. > > > > > > > > At the ASF, we contribute as individuals, so company affiliation > does not > > > > matter here; we are all project contributors. > > > > > > > > All discussions and decisions should happen on the dev mailing list. > > > > Discussions regarding quality and implementation should also take > place > > > > publicly on the relevant PR or the dev mailing list. > > > > > > > > Regards, > > > > JB > > > > > > > > > > > > On Mon, Sep 21, 2026 at 8:04 PM Dmitry Chirkov <[email protected]> > > > wrote: > > > > > > > > > I like Rok's proposed approach. > > > > > > > > > > Pedro and I discussed this topic offline. While we can have > private, > > > > > internal conversations with authors regarding contribution quality > > > within > > > > > the company, open source contributions lack direct messaging > > > capabilities. > > > > > As a result, this interaction - unfortunately - needs to take place > > > > > publicly. > > > > > > > > > > Dmitry Chirkov, > > > > > Dremio > > > > > > > > > > On Mon, Sep 21, 2026 at 9:40 AM Rok Mihevc <[email protected]> > > > wrote: > > > > > > > > > > > Agreed with Pedro's proposal as well! > > > > > > > > > > > > I'd suggest another item here: let's introduce a GitHub label: > > > > > > "needs-author-engagement". Reviewer could use this label to flag > PRs > > > that > > > > > > require substantial responses from their authors before a > productive > > > > > review > > > > > > can continue. This label should not focus on AI use, but rather > on > > > the > > > > > > author's observable understanding of the PR's subject matter and > > > their > > > > > > engagement with reviewer feedback. Such label would also help > other > > > > > > reviewers avoid duplicate effort and serve as feedback to the > author > > > of > > > > > the > > > > > > PR. > > > > > > > > > > > > Rok > > > > > > > > > > > > On Mon, Sep 21, 2026 at 6:24 PM Nic Crane <[email protected]> > > > wrote: > > > > > > > > > > > > > Hi Pedro, > > > > > > > > > > > > > > Thanks for raising this issue, and I think it's very timely in > that > > > > > many > > > > > > of > > > > > > > us are experiencing similar issues. Your proposal sounds good > to > > > me. > > > > > > > > > > > > > > Nic > > > > > > > > > > > > > > On Fri, 18 Sept 2026 at 10:33, Pedro Matias < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > > > I'd like to revive this thread. My goal is simple: I want to > > > discuss > > > > > > > > establishing guidelines for reviewers, especially those of us > > > who are > > > > > > not > > > > > > > > Arrow committers/PMCs, on how to handle low-quality PRs with > weak > > > > > > > > engagement. The community added AI guidelines for > contributors, > > > but > > > > > not > > > > > > > for > > > > > > > > reviewers: > > > https://arrow.apache.org/docs/developers/reviewing.html > > > > > > > > > > > > > > > > Lately I've been reviewing a PR that I think fits this > > > description. > > > > > > I've > > > > > > > > pointed the person to the AI guidelines at > > > > > > > > > > > > > > > > > > https://arrow.apache.org/docs/developers/overview.html#ai-generated-code > > > > > > > , > > > > > > > > but the guidelines were ignored. I can continue steering the > code > > > > > with > > > > > > > > reviews, but I am afraid this might encourage others to > repeat > > > this > > > > > > > pattern > > > > > > > > of weak engagement. > > > > > > > > > > > > > > > > I repeat R. Tyler Croy's ask: "don't rely on everybody > following > > > the > > > > > > > rules, > > > > > > > > and come up with an agreed upon way to handle those that > don't." > > > > > > > > > > > > > > > > I looked into other projects to see if any of them have > something > > > > > > > similar. > > > > > > > > LLVM has guidance [0] on both how to warn the contributor and > > > when to > > > > > > > > escalate to someone with permission to lock the > conversation, as > > > well > > > > > > as > > > > > > > a > > > > > > > > label that can be added to a low effort PR. > > > > > > > > > > > > > > > > I propose we add a section titled "Handling violations of > the AI > > > > > > > > contribution guidelines" to the reviewer guidelines. I wrote > a > > > small > > > > > > > draft > > > > > > > > [1] of what it could look like. I'm happy to iterate on it if > > > people > > > > > > > think > > > > > > > > this is something worth adding. > > > > > > > > > > > > > > > > I'm particularly curious about what committers/PMC members > think > > > > > should > > > > > > > be > > > > > > > > the way for read-only reviewers to escalate to maintainers. > The > > > > > current > > > > > > > > proposal suggests pinging someone, which might be too noisy. > > > > > > > > > > > > > > > > 0- > https://llvm.org/docs/AIToolPolicy.html#handling-violations > > > > > > > > > > > > > > > > 1- When a reviewer finds that a contribution does not seem to > > > conform > > > > > > to > > > > > > > > the guidelines for AI usage, they should respond with the > > > following > > > > > > > > message: > > > > > > > > " > > > > > > > > This PR does not seem to meet the standards for AI generated > > > > > > > contributions. > > > > > > > > Please read the guidelines at > > > > > > > > > > > > > > > > > > https://arrow.apache.org/docs/developers/overview.html#ai-generated-code > > > > > > > > and ensure you modify your PR to conform to the rules. > > > > > > > > " > > > > > > > > If the contributor fails to adapt their work and/or > engagement > > > level > > > > > to > > > > > > > > meet the guidelines' standards, maintainers may close the PR. > > > > > Reviewers > > > > > > > > without permission to close the PR should escalate by > pinging a > > > > > > > maintainer > > > > > > > > via comment indicating that they do not believe the change > meets > > > the > > > > > > > > standards. > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Pedro Matias > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 13, 2026 at 3:53 PM Nic Crane < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > On a similar note, after conversations with folks around > what > > > > > appear > > > > > > to > > > > > > > > be > > > > > > > > > AI-generated mailing list responses, I've also opened a PR > > > > > suggesting > > > > > > > > > people disclose any AI-generated questions they post to > mailing > > > > > list > > > > > > > > > discussions; feel free to add any comments there (if > you're a > > > > > human! > > > > > > > ;) ) > > > > > > > > > > > > > > > > > > https://github.com/apache/arrow/pull/49277/changes > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 22 Jan 2026 at 20:42, Nic Crane < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > PR here for anyone interested: > > > > > > > > > https://github.com/apache/arrow/pull/48952 > > > > > > > > > > > > > > > > > > > > On Thu, 22 Jan 2026 at 09:56, Nic Crane < > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Thanks Andrew, I really like how you spell out the > reasoning > > > > > > around > > > > > > > > it, > > > > > > > > > I > > > > > > > > > >> will see how we can incorporate some of those ideas > > > > > > > > > >> > > > > > > > > > >> On Thu, 22 Jan 2026 at 09:23, Andrew Lamb < > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >>> > We have had repeated attempts at contributions by > some > > > folks > > > > > > who > > > > > > > > > simply > > > > > > > > > >>> do not understand their generated code and when asked > for > > > > > > > > > clarification, > > > > > > > > > >>> have the LLM generate more incorrect commentary. It's > very > > > > > > > > > >>> Dunning-Krueger > > > > > > > > > >>> and leads to lots of frustration all around. > > > > > > > > > >>> > > > > > > > > > >>> We saw this too in DataFusion and I was pleased with > what > > > we > > > > > came > > > > > > > up > > > > > > > > > with > > > > > > > > > >>> for rationale about why it is not helpful[1]. > Basically the > > > > > > > reviewers > > > > > > > > > are > > > > > > > > > >>> more efficient using the LLM tools directly and the > > > contributor > > > > > > > isn't > > > > > > > > > >>> learning anything either. > > > > > > > > > >>> > > > > > > > > > >>> Andrew > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> [1]: > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://datafusion.apache.org/contributor-guide/index.html#why-fully-ai-generated-prs-without-understanding-are-not-helpful > > > > > > > > > >>> > > > > > > > > > >>> On Mon, Jan 19, 2026 at 12:48 PM R Tyler Croy < > > > > > > [email protected]> > > > > > > > > > >>> wrote: > > > > > > > > > >>> > > > > > > > > > >>> > (replies inline) > > > > > > > > > >>> > > > > > > > > > > >>> > On Sunday, January 18th, 2026 at 7:43 PM, Gang Wu < > > > > > > > > [email protected]> > > > > > > > > > >>> > wrote: > > > > > > > > > >>> > > > > > > > > > > >>> > > - Summitters should review all lines of generated > code > > > > > before > > > > > > > > > >>> creating > > > > > > > > > >>> > the > > > > > > > > > >>> > > PR to > > > > > > > > > >>> > > understand every piece of detail just like they are > > > written > > > > > > by > > > > > > > > the > > > > > > > > > >>> > > submitters > > > > > > > > > >>> > > themselves. > > > > > > > > > >>> > > - AI tools are notorious for generating overly > verbose > > > > > > > comments, > > > > > > > > > >>> > unnecessary > > > > > > > > > >>> > > test cases, fixing test failures using wrong > > > approaches, > > > > > etc. > > > > > > > > Make > > > > > > > > > >>> sure > > > > > > > > > >>> > > these > > > > > > > > > >>> > > are checked and fixed. > > > > > > > > > >>> > > - Reviewers are humans, so please try to break down > > > large > > > > > PRs > > > > > > > > into > > > > > > > > > >>> > smaller > > > > > > > > > >>> > > ones to make reviewers' life easier to get PRs > promptly > > > > > > > reviewed. > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > Like others I think Nic's draft is a good one, I > would > > > like > > > > > to > > > > > > > > offer > > > > > > > > > >>> some > > > > > > > > > >>> > thoughts as a maintainer (delta-rs) which has > received > > > > > > increased > > > > > > > > > >>> > AI-assisted pull requests over the past six months. > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > The "PR may be closed without further review" > statement I > > > > > would > > > > > > > > > >>> strongly > > > > > > > > > >>> > encourage moving to the very beginning of the > policy. I > > > > > would > > > > > > > also > > > > > > > > > >>> > encourage labels being used like "ai-assisted" to > signal > > > to > > > > > > other > > > > > > > > > >>> > contributors who may or may not wish to engage in > > > reviewing > > > > > > > > potential > > > > > > > > > >>> slop. > > > > > > > > > >>> > > > > > > > > > > >>> > We have had repeated attempts at contributions by > some > > > folks > > > > > > who > > > > > > > > > >>> simply do > > > > > > > > > >>> > not understand their generated code and when asked > for > > > > > > > > clarification, > > > > > > > > > >>> have > > > > > > > > > >>> > the LLM generate more incorrect commentary. It's > very > > > > > > > > > Dunning-Krueger > > > > > > > > > >>> and > > > > > > > > > >>> > leads to lots of frustration all around. > > > > > > > > > >>> > > > > > > > > > > >>> > Like most policies it's important to speak to those > that > > > are > > > > > > > acting > > > > > > > > > in > > > > > > > > > >>> > good faith but don't rely on everybody following the > > > rules, > > > > > and > > > > > > > > come > > > > > > > > > up > > > > > > > > > >>> > with an agreed upon way to handle those that don't. > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > Either way I think it's good to ship! :) > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > Cheers > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
