On Tue, 1 Sept 2026 at 17:14, Paolo Bonzini <[email protected]> wrote:
>
> Until now QEMU's code provenance policy declined any contribution
> believed to include or derive from AI-generated content. A blanket ban
> was easy to maintain while LLM output was rarely usable on its own, but
> as the tools improved an absolute prohibition has become harder to
> justify and enforce.
Yes, but the "enforcement" part is just as hard to deal with for
any AI policy short of "we don't care whether AI was used or not
and don't care if it's labelled that way or not". A contributor
who was unaware of or chose to deliberately ignore our existing
AI policy is just as likely to be unaware of or to ignore an updated
policy that says "yes, but only with limitations and must be
pre-arranged for larger amounts of generated code". (A contributor
who deliberately ignores the policy is I hope going to be rare;
contributors who are simply unaware of it are something that we can
hopefully improve on by (a) having an AGENTS.md that will tell their
LLM about it and (b) asking in case of doubt on the maintainer side.)
I think fundamentally my issue here is that to me it doesn't seem
like the project has problems with "we aren't writing code fast
enough"; the problems are more "we don't have enough code review",
"our existing code is getting holes poked in it by automated
bug finding" and "we have too much old code that's barely used and
unmaintained". So an AI policy update whose major move is "allow
more generated code" feels bad to me, because it's inevitably
going to increase the workload on reviewers and maintainers.
I would be much happier seeing more use of AI in the project to
help us where we're struggling.
> This proposed policy seeks to implement the cautious experimentation
> approach, inspired mostly by the Rust project's policy and by Software
> Freedom Conservancy's own recommendations on LLMs and generative AI.
The question I had with the Rust policy was to what extent their
"experimentation" is actually going to be cautious experimentation,
and how much it turns out to be "this is a back door that lets the
people who want to write and review LLM-generated code do that anyway
as long as the people who don't want to can mostly ignore it".
If this is really experimental then we should have some clear
boundaries and criteria for our experimentation, covering for
instance
(a) when do we assess the success or failure of the experiment?
(b) what are we looking at to decide success/failure?
(c) what are our boundaries for what kinds of change we're willing
to make under this experiment and what we are not?
(d) what is our plan for rolling back or otherwise recovering if
the experiment seems to have failed?
The Rust policy makes some attempts at some of these e.g. with its
"circuit breaker" provision and the requirement that LLM changes are
"non-critical" ones.
> +LLM-assisted and LLM-created contributions
> +''''''''''''''''''''''''''''''''''''''''''
> +
> +Use of generative AI tools for code contributions generally falls into
> +four buckets:
> +
> +- "background" assistance, such as completion and code review. This does
> + not need disclosure, however you are expected to examine and understand
> + any code or any changes produced by the tool.
> +
> +- small LLM-assisted bugfixes (as a rule of thumb, below 10 lines of code)
> + are allowed with disclosure. Splitting a submission in multiple parts
> + is **not** a valid way to bypass this rule; repeated, frequent posting
> + of such fixes must follow the rule for full contributions.
> +
> +- use of LLMs to help generating parts of a larger patch---a test case, a
> + parser, boilerplate code for a new API, a tool to help performing
> + mechanical changes, etc. These are generally allowed, but disclosure
> + is recommended.
I guess I'm generally OK with these (though I might add an "If in doubt
about whether your use here is too extensive, ask" to the last one:
"a parser" is potentially a pretty big thing to be delegating to the LLM,
for instance, and might either be "mostly boilerplate" or to shade over
into the "writing large parts of functional code" category).
I do note that even for "small bugfixes below 10 lines of code" the
code review effort can still be pretty huge where it's touching something
like a device model, where you have to go and find the right datasheet
or spec and confirm whether the proposed change is really the right one
or if it just fixes whatever the assert/crash was but in the wrong way.
(Handwritten example from me: I tried to fix an IDE divide-by-zero
https://patchew.org/QEMU/[email protected]/
and even though I had read the IDE spec and the fix is pretty small and
it does fix the bug's test case, it turns out that it's a lot more
complicated than that and the proper fix is something entirely different.)
Having a hundred "fix minor bug in old code" patches on the list that
are unreviewed isn't a lot better than having a hundred issues in the
bug tracker (indeed, it's arguably worse, since we have no tracking
system for patches on the mailing list; at least the issues won't just
get lost in the deluge...)
> +- large, heavily LLM-assisted contributions where LLMs write large parts
> + of functional code. These are only allowed if *pre-arranged*,
> + *high-quality* and *well-tested*.
> +
> +The last case (*"LLM-created contribution"*) is the one that requires
> +a more careful approach. In particular:
> +
> +- "Pre-arranged" means that a maintainer has communicated ahead of time,
> + for example on `IRC or the QEMU mailing
> + list <https://www.qemu.org/contribute/>`__, that they are willing
> + to review an LLM-created contribution.
> +
> +- "High-quality" means that the contributor must apply the same judgment
> + that would be applied to other code changes. Contributors must invest
> + substantial time reviewing their contributions, curating them, and
> + understanding them in depth; in particular, you are still expected to
> + :ref:`understand and explain your changes
> + <write_a_meaningful_commit_message>` and the rationale behind them.
> +
> +- "Well-tested" means the LLM-created contributions will be held to a
> + higher standard than human-created ones, because LLMs make it easier
> + to write tests. There are no exceptions for "writing the tests seems
> + hard" or for `yak shaving <https://en.wiktionary.org/wiki/yak_shaving>`__.
This last part is the one that I am much less certain about, and where
I'm not sure we have sufficiently bounded and defined our experiment
(as noted above).
I'm also more willing to allow leeway and to trust the judgement on
LLM use for somebody who is already a regular contributor to the
project (and so has some idea of how the codebase works and better
ability to spot when generated code has gone off in the wrong
direction), versus patches from somebody who hasn't contributed
before. Is that something we want to try to encode in policy
(e.g. with limitations on the "pre-arranged larger contribution"
case) ?
-- PMM
