On 9/2/26 14:01, Peter Maydell wrote:
A contributor
who was unaware of or chose to deliberately ignore our existing
AI policy is just as likely to be unaware of or to ignore an updated
policy that says "yes, but only with limitations and must be
pre-arranged for larger amounts of generated code".
Oh, they won't be unaware of it. :) That's why I included the
AGENTS.md, even though it's technically separate. They'll be reminded
every two minutes of the need to pre-arrange review, until they tell the
LLM to shut up.
(A contributor who deliberately ignores the policy is I hope going to
be rare; contributors who are simply unaware of it are something that
we can hopefully improve on by (a) having an AGENTS.md that will tell
their LLM about it and (b) asking in case of doubt on the maintainer
side.)
Exactly. Also, we have contributors who are unware of it now and we
live with it or look at it on a case-by-case basis. People ignoring the
policy will be the same.
I think fundamentally my issue here is that to me it doesn't seem
like the project has problems with "we aren't writing code fast
enough"; the problems are more "we don't have enough code review",
"our existing code is getting holes poked in it by automated
bug finding" and "we have too much old code that's barely used and
unmaintained". So an AI policy update whose major move is "allow
more generated code" feels bad to me, because it's inevitably
going to increase the workload on reviewers and maintainers.
I would be much happier seeing more use of AI in the project to
help us where we're struggling.
I agree. The hope is to get a bit of that via better testing, and that
"we aren't writing code fast enough" is applied to stuff that helps long
term rather than to the itch of the day. Whether it works or not it
remains to be seen.
I certainly don't want an explosion of new device models and
unreviewable qtests.
The question I had with the Rust policy was to what extent their
"experimentation" is actually going to be cautious experimentation,
and how much it turns out to be "this is a back door that lets the
people who want to write and review LLM-generated code do that anyway
as long as the people who don't want to can mostly ignore it".
If this is really experimental then we should have some clear
boundaries and criteria for our experimentation, covering for
instance
(a) when do we assess the success or failure of the experiment?
(b) what are we looking at to decide success/failure?
(c) what are our boundaries for what kinds of change we're willing
to make under this experiment and what we are not?
(d) what is our plan for rolling back or otherwise recovering if
the experiment seems to have failed?
I agree these are good questions. They don't have obvious answers but
as a start:
(a) a year was more or less the distance between the old policy and the
time when we started noticing AI work on the mailing list, so probably
another 8-12 months? Such as from November's QEMU Summit to the one in
Sep-Nov 2027?
(b) Having to abort the experiment in advance is the obvious failure
mode. Everything goes unnoticed is success. I would say that pervasive
discussions on boundaries would be pretty bad too. Success/failure may
also be split across categories, say by forbidding the bugfix category.
(c) no idea. The concept of pre-arrangement leaves this to individual
maintainers. As Daniel points out "different rules for different
subsystems" is not a good thing in principle, but in the end there is
already a large subjective element in what maintainers are willing to
review and even from whom.
(d) worst case we could simply revert the policy, at least in effects
though probably not in language.
I'll add (e) what are our boundaries for AI patches covering multiple
maintainers? The extreme case is Alex's "make QOM parenting mandatory"
series
(https://lore.kernel.org/qemu-devel/[email protected]/),
but it's actually not the hardest---apart from being done with AI, we
had similarly large changes in the past such as the Meson switch and
qdev_new() refactoring. In those cases a pre-arrangement was needed
anyway with the community, more than with a specific sets of maintainers.
The Rust policy makes some attempts at some of these e.g. with its
"circuit breaker" provision and the requirement that LLM changes are
"non-critical" ones.
I didn't include a "circuit breaker" because someone would have to
implement it, but also because I don't expect any maintainer to be so
eager as to flood QEMU with AI-generated work. And we're a much smaller
community so I hope we would be able to sort it out among ourselves.
+LLM-assisted and LLM-created contributions
+''''''''''''''''''''''''''''''''''''''''''
+
+Use of generative AI tools for code contributions generally falls into
+four buckets:
+
+- "background" assistance
+- small LLM-assisted bugfixes
+- use of LLMs to help generating parts of a larger patch
I guess I'm generally OK with these (though I might add an "If in doubt
about whether your use here is too extensive, ask" to the last one:
"a parser" is potentially a pretty big thing to be delegating to the LLM,
for instance, and might either be "mostly boilerplate" or to shade over
into the "writing large parts of functional code" category).
Sure I can remove the parser example. The initial one I had was
"switching to a new API". That is more representative of the intent it
would basically put Alex's 137 patch series almost entirely under this
section and I didn't want to do that.
I can report that the LLM was very reluctant to let me use this third
category, and even less to let me do it without "AI-used-for". Even for
a one-line change to a "#define" in a 1000-line patch, it insisted that
perhaps I should have included the trailer. The AGENTS.md makes it very
meticulous. :)
I do note that even for "small bugfixes below 10 lines of code" the
code review effort can still be pretty huge where it's touching something
like a device model, where you have to go and find the right datasheet
or spec and confirm whether the proposed change is really the right one
or if it just fixes whatever the assert/crash was but in the wrong way.
[...] Having a hundred "fix minor bug in old code" patches on the list that
are unreviewed isn't a lot better than having a hundred issues in the
bug tracker (indeed, it's arguably worse, since we have no tracking
system for patches on the mailing list; at least the issues won't just
get lost in the deluge...)
Yes, and especially when we have them submitted to "Odd fixes" areas the
risk of maintainer DDOS is there. But if people start submitting too
much you *can* tell them according to the policy that you need them to
develop e.g. a test suite. Which yes, will *also* be more work to
review for maintainers, but probably something that was sorely needed.
The hope is that by not placing AI-generated code in a "don't ask, don't
tell" area we actually have tools to push back and reach an equilibrium
that is better for the project and for the maintainers. How it will
work, it remains to be seen.
I'm also more willing to allow leeway and to trust the judgement on
LLM use for somebody who is already a regular contributor to the
project (and so has some idea of how the codebase works and better
ability to spot when generated code has gone off in the wrong
direction), versus patches from somebody who hasn't contributed
before. Is that something we want to try to encode in policy
(e.g. with limitations on the "pre-arranged larger contribution"
case) ?
Pre-arrangement does not mean you're forced to say yes. Saying "I've
never done this and I don't want to give you false hopes, so I'll
decline" is fine.
Paolo
