On 10/1/26 14:21, Peter Maydell wrote:
On Wed, 30 Sept 2026 at 09:54, Paolo Bonzini <[email protected]> wrote:
+- large, heavily LLM-assisted contributions where LLMs write large parts
+ of functional code. These are only allowed if *pre-arranged*,
+ *high-quality* and *well-tested*.
+
+The last case (*"LLM-created contribution"*) is the one that requires
+a more careful approach. In particular:
+
+- "Pre-arranged" means that a maintainer has communicated ahead of time,
+ for example on `IRC or the QEMU mailing
+ list <https://www.qemu.org/contribute/>`__, that they are willing
+ to review an LLM-created contribution.
+
+- "High-quality" means that the contributor must apply the same judgment
+ that would be applied to other code changes. Contributors must invest
+ substantial time reviewing their contributions, curating them, and
+ understanding them in depth.
+
+- "Well-tested" means the LLM-created contributions will be held to a
+ higher standard than human-created ones, because LLMs make it easier
+ to write tests. There are no exceptions for "writing the tests seems
+ hard" or for `yak shaving <https://en.wiktionary.org/wiki/yak_shaving>`__.
...but I think fundamentally I am not in favour of having this
fourth category of permitted uses. It seems even to have lost the
idea of it being an "experimental" approach in this revision of the
series. The first three categories are fine, and I have no particular
problem with our relaxing our current policy to explicitly permit them.
Our problem as a project does not seem to be on the "writing
code" side of things, but has always been on the "code review"
side. This policy relaxation makes it much easier to write more
code. Even when there's a "pre-arranged reviewer" that isn't
necessarily helpful -- if you're one of the primary maintainers
for an area of QEMU then the fact that somebody else is reviewing
it doesn't make a big AI-generated code infusion any more palatable.
I understand where you're coming from.
The purpose of the fourth category is norming contributions that are
already happening. I apologize for the allegation in advance, but take
for example
https://lore.kernel.org/qemu-devel/[email protected]/;
on one hand it is very high quality, on the other hand I give it at
least a 70% likelihood of being heavily LLM-assisted for the
implementation and not just for the analysis.
The question is: do we want contributions like that one? If we decide
that their quality and usefulness are high enough to want them, what
process do we need around them? I think we all agree that, if that kind
of contribution is desirable, we want the submitters to declare AI usage
and the reviewers to be aware of it.
Allowing large heavily LLM-assisted contributions seems to me:
* the biggest risk legally speaking (though I think this is
in practice not all that big a risk at this point)
* the biggest risk of the codebase degenerating into
unmaintainability
* the biggest risk of finding ourselves reviewing contributions
that are in practice much more vibe-coded than the policy text
above is asking for
I agree completely with how you phrased the first and third. I agree
that the second bullet is a risk too, but it's mitigated by "LLM-created
contributions will be held to a higher standard than human-created ones,
because LLMs make it easier to write tests" (and by maintainers being
able to *refuse* specific contributions, which is the point of
pre-arrangement; note that pre-arrangement should exist today as a
matter of courtesy for any large change).
The third risk is probably the biggest of these. The heaviest toll that
AI has on maintainers may not be the review per se; if a contribution is
good and desirable, having to review it is all fine. Rather, the
problem is that maintainers have to be ready to say no more often to the
free puppies, and that is not necessarily easy. And dually, that
contributors may not realize it when they're not respecting the
maintainers' efforts.
I add a fourth risk: that new contributors using AI will not be willing
to do the final stretch of the work that *has* to be manual, and makes a
difference between work worth reviewing and work worth including. This
would be the biggest waste of time for the reviewer.
I look at our issue tracker where we see AI assisted contributions
that run the whole range from "an human making intelligent use of
the tool to provide high-quality bug reports with good analysis"
through to "absolute slop dumped on us with no apparent human
thought or intervention". I don't really want to expand that
from our issue tracker to our code review and codebase, please.
I would be much happier seeing the advocates of AI putting
effort into making the process of QEMU code review better and
less work, rather than in applying it to the code writing process.
To clarify, my goal with this policy change is not to advocate for AI
usage, but to advocate for transparency when AI is involved.
I agree that making QEMU code review better should be a goal.
Unfortunately it is an expensive goal (in real-world money).
Paolo
