Agreed with Pedro's proposal as well! I'd suggest another item here: let's introduce a GitHub label: "needs-author-engagement". Reviewer could use this label to flag PRs that require substantial responses from their authors before a productive review can continue. This label should not focus on AI use, but rather on the author's observable understanding of the PR's subject matter and their engagement with reviewer feedback. Such label would also help other reviewers avoid duplicate effort and serve as feedback to the author of the PR.
Rok On Mon, Sep 21, 2026 at 6:24 PM Nic Crane <[email protected]> wrote: > Hi Pedro, > > Thanks for raising this issue, and I think it's very timely in that many of > us are experiencing similar issues. Your proposal sounds good to me. > > Nic > > On Fri, 18 Sept 2026 at 10:33, Pedro Matias <[email protected]> > wrote: > > > Hello all, > > > > I'd like to revive this thread. My goal is simple: I want to discuss > > establishing guidelines for reviewers, especially those of us who are not > > Arrow committers/PMCs, on how to handle low-quality PRs with weak > > engagement. The community added AI guidelines for contributors, but not > for > > reviewers: https://arrow.apache.org/docs/developers/reviewing.html > > > > Lately I've been reviewing a PR that I think fits this description. I've > > pointed the person to the AI guidelines at > > https://arrow.apache.org/docs/developers/overview.html#ai-generated-code > , > > but the guidelines were ignored. I can continue steering the code with > > reviews, but I am afraid this might encourage others to repeat this > pattern > > of weak engagement. > > > > I repeat R. Tyler Croy's ask: "don't rely on everybody following the > rules, > > and come up with an agreed upon way to handle those that don't." > > > > I looked into other projects to see if any of them have something > similar. > > LLVM has guidance [0] on both how to warn the contributor and when to > > escalate to someone with permission to lock the conversation, as well as > a > > label that can be added to a low effort PR. > > > > I propose we add a section titled "Handling violations of the AI > > contribution guidelines" to the reviewer guidelines. I wrote a small > draft > > [1] of what it could look like. I'm happy to iterate on it if people > think > > this is something worth adding. > > > > I'm particularly curious about what committers/PMC members think should > be > > the way for read-only reviewers to escalate to maintainers. The current > > proposal suggests pinging someone, which might be too noisy. > > > > 0- https://llvm.org/docs/AIToolPolicy.html#handling-violations > > > > 1- When a reviewer finds that a contribution does not seem to conform to > > the guidelines for AI usage, they should respond with the following > > message: > > " > > This PR does not seem to meet the standards for AI generated > contributions. > > Please read the guidelines at > > https://arrow.apache.org/docs/developers/overview.html#ai-generated-code > > and ensure you modify your PR to conform to the rules. > > " > > If the contributor fails to adapt their work and/or engagement level to > > meet the guidelines' standards, maintainers may close the PR. Reviewers > > without permission to close the PR should escalate by pinging a > maintainer > > via comment indicating that they do not believe the change meets the > > standards. > > > > Best regards, > > Pedro Matias > > > > > > > > > > On Fri, Feb 13, 2026 at 3:53 PM Nic Crane <[email protected]> wrote: > > > > > On a similar note, after conversations with folks around what appear to > > be > > > AI-generated mailing list responses, I've also opened a PR suggesting > > > people disclose any AI-generated questions they post to mailing list > > > discussions; feel free to add any comments there (if you're a human! > ;) ) > > > > > > https://github.com/apache/arrow/pull/49277/changes > > > > > > > > > On Thu, 22 Jan 2026 at 20:42, Nic Crane <[email protected]> wrote: > > > > > > > PR here for anyone interested: > > > https://github.com/apache/arrow/pull/48952 > > > > > > > > On Thu, 22 Jan 2026 at 09:56, Nic Crane <[email protected]> wrote: > > > > > > > >> Thanks Andrew, I really like how you spell out the reasoning around > > it, > > > I > > > >> will see how we can incorporate some of those ideas > > > >> > > > >> On Thu, 22 Jan 2026 at 09:23, Andrew Lamb <[email protected]> > > wrote: > > > >> > > > >>> > We have had repeated attempts at contributions by some folks who > > > simply > > > >>> do not understand their generated code and when asked for > > > clarification, > > > >>> have the LLM generate more incorrect commentary. It's very > > > >>> Dunning-Krueger > > > >>> and leads to lots of frustration all around. > > > >>> > > > >>> We saw this too in DataFusion and I was pleased with what we came > up > > > with > > > >>> for rationale about why it is not helpful[1]. Basically the > reviewers > > > are > > > >>> more efficient using the LLM tools directly and the contributor > isn't > > > >>> learning anything either. > > > >>> > > > >>> Andrew > > > >>> > > > >>> > > > >>> [1]: > > > >>> > > > >>> > > > > > > https://datafusion.apache.org/contributor-guide/index.html#why-fully-ai-generated-prs-without-understanding-are-not-helpful > > > >>> > > > >>> On Mon, Jan 19, 2026 at 12:48 PM R Tyler Croy <[email protected]> > > > >>> wrote: > > > >>> > > > >>> > (replies inline) > > > >>> > > > > >>> > On Sunday, January 18th, 2026 at 7:43 PM, Gang Wu < > > [email protected]> > > > >>> > wrote: > > > >>> > > > > >>> > > - Summitters should review all lines of generated code before > > > >>> creating > > > >>> > the > > > >>> > > PR to > > > >>> > > understand every piece of detail just like they are written by > > the > > > >>> > > submitters > > > >>> > > themselves. > > > >>> > > - AI tools are notorious for generating overly verbose > comments, > > > >>> > unnecessary > > > >>> > > test cases, fixing test failures using wrong approaches, etc. > > Make > > > >>> sure > > > >>> > > these > > > >>> > > are checked and fixed. > > > >>> > > - Reviewers are humans, so please try to break down large PRs > > into > > > >>> > smaller > > > >>> > > ones to make reviewers' life easier to get PRs promptly > reviewed. > > > >>> > > > > >>> > > > > >>> > Like others I think Nic's draft is a good one, I would like to > > offer > > > >>> some > > > >>> > thoughts as a maintainer (delta-rs) which has received increased > > > >>> > AI-assisted pull requests over the past six months. > > > >>> > > > > >>> > > > > >>> > The "PR may be closed without further review" statement I would > > > >>> strongly > > > >>> > encourage moving to the very beginning of the policy. I would > also > > > >>> > encourage labels being used like "ai-assisted" to signal to other > > > >>> > contributors who may or may not wish to engage in reviewing > > potential > > > >>> slop. > > > >>> > > > > >>> > We have had repeated attempts at contributions by some folks who > > > >>> simply do > > > >>> > not understand their generated code and when asked for > > clarification, > > > >>> have > > > >>> > the LLM generate more incorrect commentary. It's very > > > Dunning-Krueger > > > >>> and > > > >>> > leads to lots of frustration all around. > > > >>> > > > > >>> > Like most policies it's important to speak to those that are > acting > > > in > > > >>> > good faith but don't rely on everybody following the rules, and > > come > > > up > > > >>> > with an agreed upon way to handle those that don't. > > > >>> > > > > >>> > > > > >>> > Either way I think it's good to ship! :) > > > >>> > > > > >>> > > > > >>> > > > > >>> > Cheers > > > >>> > > > > >>> > > > > >>> > > > >> > > > > > >
