Hi Zubin! Am Mo., 27. Juli 2026 um 21:39 Uhr schrieb Zubin Duggal <[email protected]>: > > I am strongly opposed to this. I also don't see the value in > it. If you want to ask an LLM to review your code, you can > probably do this much quicker by just asking one. Today you > are never more than a few clicks away from a text box that > will put you in direct contact with one. > > If you want to maintain a shared repository of common prompts, > skills, guidance etc for LLMs regarding work on GHC, I have > no objection to this (I would prefer to only have documentation > and prose targetting humans in the main GHC repository though). > > In fact I would like to propose a policy in the exact opposite > direction: > > Disallow _purely autonomous_ systems from posting on the GHC > Gitlab, unless such a system is cleared in advance by the > administrators of the instance and/or contributors to GHC. > > What this won't disallow: > > * Getting an LLM to isolate and create a minimal reproducer > or bug description and posting this. > > * Getting an LLM to propose a patch or fix for a particular > issue > > * Asking an LLM about the details of how some particular > external system works, and then posting the results. > > * Asking an LLM to catalogue a bunch of related issues and > extract a common pattern > > * etc. > > What this will disallow: > > Autonomous systems without humans in the loop writing and > interacting with the gitlab.
Interesting! I don't want to discuss your policy proposal in depth here, because IMHO it belongs on the LLM policy thread. However if there's a deep opposition to autonomous systems on GitLab (with the exception of Marge Bot I guess), I don't mind keeping this non-autonomous for a few months while we find out whether the LLM-generated reviews are helpful or not. My idea is somewhat inspired by the Sashiko system in the Linux kernel (see "Email from Sashiko" at https://lwn.net/SubscriberLink/1083275/d5518174ec4dcbcb/), and there it appears that at least some maintainers appreciate that it reduces their workload. > My reasons > > 1. Code review is a tool for building a shared understanding > and a common mental model of the codebase and problem > domain. An LLM model cannot do that, or even if you believe > it can, this internal model is lost as soon as that > particular session is garbage collected. At best you can try > to reconstruct some approximation of this model later by > feeding the same inputs into a fresh session on a probably > different model. Yes, I certainly don't want to take away these human reviews. I fully agree that they are very important. However I think that an initial LLM-generated review could be useful to filter out basic issues, broken conventions etc. And point out bugs. GHC is a complex system and judging by the current rate of bug reports, it's quite clear that human code review is insufficient to catch many issues. > 2. I would find it a very sad if the GHC Gitlab became a > place where bots talk to other bots who act upon and those > comments to make addtional changes in response to this in > an uncontrolled feedback loop. I predict that in the limit > any such feedback loop becomes quickly detached from the > original problem and devolves into a lot of noise totally > divorced from any real human concerns. I agree that this would be sad, and that's why I'm in favour of upholding the "human conversation" part of the current draft LLM policy. > 3. Prompt injection. Gitlab comments are an untrusted > input source and LLMs fundamentally cannot discriminate > between instructions and data. You can imagine prompt > injection attacks where hidden text in a gitlab comment, > attached bug reproducer, link etc. gets the model to do > various destructive things, like spamming the gitlab, > editing/deleting your prior comments, mining bitcoin on the > CI runners, attempting to inject malicious code into the > compiler, testcases, reproducers etc. I don't quite understand how this is connected to my offer. I may have to apply some safeguards in the review but it's still running on _my_ system. Regarding the "untrusted input source": accounts already need to be verified. Recent models are also increasingly resistant to prompt injection. > That doesn't mean that there isn't value for LLM assisted > autonomous processes on the gitlab, given that they are > desiged properly keeping in mind the concerns about human > centeredness, resource usage and prompt injection. > > Here are some examples of LLM assisted system/bots that > could actually be useful if audited and pre-approved with > the community: > > * A bot that automatically bisects regressions and identifies > the change responsible for it. > In principle this could be done with a purely deterministic > script, but often you run into problems with the toolchain > or commits not being individually buildable etc that require > some adjustements on the fly. > > * A bot that automatically minimises reproducers and posts > the results after the testcase can been verified on a deterministic > system Thanks, these are good suggestions. :) > > On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote: > >Hi devs! > > > >Inspired by the recent discussion on the LLM policy, I'd like to try > >introducing LLM-generated code reviews in GHC, as an opt-in service. > > > >My motivation: > > > >* Catch issues before human reviewers spend time on them. > >* Reduce the number of bugs merged. > > > > > >What this is — and what it isn't > >-------------------------------- > > > >This is tool output: requested on the MR, produced by a model, posted > >verbatim > >and clearly marked as machine-generated. This is not a review in the sense of > >draft LLM policy. > > > >I'm not vouching for any of it. Think of it as a linter the author could have > >run themselves, except on my LLM budget. Nobody is required to act on > >it even read it. > > > >I mention this explicitly because the policy asks reviewers to take full > >responsibility for each line of their review. I can't do that here — it would > >mean checking every finding myself, which defeats the purpose and would > >restrict the service to areas of the compiler I already know well. > > > > > >How to request one > >------------------ > > > >* Ping me on the MR with a comment like "@sjakobi llm-review please". > > * Feel free to request particular aspects the review should cover. > >* Authors can request a review for their own MRs. Maintainers can request one > > on any MR. > >* If I have enough usage left on my plan, I'll confirm that I'm on it. > >* I'll then prompt the model, and post the result on the MR. > >* Feedback on the quality of the review is of course very welcome. > > > > > >What I promise > >-------------- > > > >* Every review is labelled as machine-generated, with model, effort level and > > the exact prompt used. > >* Reviews are posted verbatim. I don't edit or curate them, so what you see > >is > > exactly what the model produced. > >* Only the public MR diff and public repository context go to the model. > > Nothing else. > >* I'll keep working on the prompts and the choice of model to improve the > > signal-to-noise ratio. > >* If the signal-to-noise ratio stays inacceptably bad, we can simply > >stop this service. > > > > > >Review coverage > >--------------- > > > >I'll probably start with a fairly basic review prompt, primarily aimed at > >correctness issues. > > > >Possible extensions: > > > >* Documentation consistency: check that documentation stays consistent, > > including Notes elsewhere that reference the changed code. > >* Performance: check for performance issues — potentially including checking > > the -ddump-simpl output for the changed code for unnecessary > > allocation etc. > >* Feel free to suggest anything else! > > > > > >Format > >------ > > > >Initially a single comment per review. Once we reach a good signal-to-noise > >ratio, we can consider inline comments on the diff. > > > > > >Caveats > >------- > > > >This is a volunteer service I intend to provide in my spare time. If I'm AFK > >or on vacation, reviews will take longer. If I can't keep up with requests, I > >will prioritise which MRs I run. I might take a break entirely. > > > > > >Getting involved > >---------------- > > > >Is anyone interested in joining this effort? If so we could form a team > >(@llm-reviewers?), and share prompting techniques, model choices, etc. > > > >If the service turns out to be genuinely useful, we can look into > >automating it and – if necessary – possible funding. > > > > > >Cheers, > >Simon > >_______________________________________________ > >ghc-devs mailing list -- [email protected] > >To unsubscribe send an email to [email protected] _______________________________________________ ghc-devs mailing list -- [email protected] To unsubscribe send an email to [email protected]
