Thank you for your thoughts, Moritz! At this point I feel like a have a fairly good idea of your position on the matter and your reasoning behind it.
I invite you to give GlaHiCo a try on your next MR, with a delivery method of your choosing, and at a stage of "MR-readiness" of your choice. I would be happy to receive more feedback from you then. Cheers, Simon Am So., 2. Aug. 2026 um 16:57 Uhr schrieb Moritz Angermann <[email protected]>: > > Dear Simon, > > Sorry for the late reply, I had been enjoying some time in Ha Long Bay with > my family. > Let me start by stating that I do not contest the potential value of LLM > reviews. I use them > myself as well. And I think providing this service to interested parties is > great! It's presumably > something you enjoy working on, and one more option for those who are > interested. That > seems like a win-win to me. > > I don't think I can quantify or qualify if, and how much a secondary review > by an LLM to > supplement one's own review would impact negatively. I believe everyone will > have to > experiment with this themselves and figure out what works best for them. It > might not even > be the same for everyone. > > My concern is that machine generated output ends up verbatim, or largely > verbatim in > issue trackers, and pull requests. As I outlined in the statement with > Julian, to both of > us (and this might be very different for others!) the participation in open > source projects > is valuable due to the human collaboration, which specifically includes the > fact that > humans end up getting into a conversation about a change. > > Reading a completely machine (auto)generated blurb in that conversation > feels, to some, > rather offensive, and may look like pure noise. Whether there was a human > acting as > conduit to a LLM or not is fairly irrelevant to that. > > This becomes substantially different (at least to me), if someone > (potentially) read some > secondary review, and after careful review of that, incorporated parts, of it > into their own > review. Sure maybe his own review, and the secondary might coincide, maybe > even > largely. But they would need to make these their own, verify the claims, ... > and then > post their review. This is a very subtle line, and fairly hard to properly > articulate I > believe. > > Let me try to bring up two common examples (CodeRabbit, and CoPilot) on > GitHub. > PR's at that point looks like: > > [some comment by a human] > [ lots of ] > [ noise ] > [ by ] > [ some ] > [ machine(s) ] > [ maybe another comment by a human ] > ... > > The (human) signal to noise ratio goes down fast. > > Now I do personally like to just ask CoPilot for a review of the code (on my > repos), > just as a quick cross validation. My personal preference would be that this > creates > comments only visible to me. But that's not how GH works, so I end up > requesting > a CoPilot review and go through the findings, then hide/delete all the > comments from > copilot to get rid of the noise, such that the PR discussion can focus on the > human > collaboration and not the tool noise interleaved. > > CodeRabbit, and other tools also have this annoying habit of posting > information > as comments, which makes them very disruptive (imo) to the human discourse > around a PR. If GH offered a better way to attach metadata or a separate > track > for these kinds of information, I might feel different about this. > > This (for me!) is the reason, why I'm very skeptical of automatically added > data into > the human conversation around a MR on gitlab am, and why I am suggesting to > keep this in a second medium (e.g. email, or external web interface) > completely > outside of the lane of human discussion around any PR. > > If we all were using git as intended :D, via email, this would be rather > simple, one > could simply forward the patch emails to a service email address, that would > respond > in private with the annotated LLM review. Similarly how one would facilitate > a private > review of a patch from a coworker, before sending ones own review to the > shared > mailing list. > > I hope this somewhat clarifies where _I_ stand on this topic, and my thinking > around > it. > > Best, > Moritz > > On Fri, 31 Jul 2026 at 00:14, Simon Jakobi <[email protected]> wrote: >> >> Moritz, >> >> let me loop in the mailing list again. >> >> I understand that my service really doesn't have the value of a proper >> human code review. However, judging from my private trial runs, it >> does seem to catch bugs that neither the testsuite nor human reviewers >> caught. MR authors could of course use LLMs themselves to find the >> same bugs, but this doesn't seem to be widespread practice yet. >> Potential reasons for this: cost or lack of access for other reasons, >> insufficient quality of the review output. >> >> I also agree that there is a potential risk that my service could >> reduce the current human review practice. My suggestion would be to >> make a small survey in three months or so where we evaluate how my >> service affects this project, including the social interactions. >> >> Potential tweaks on how to use this service: >> * MR authors can request a private review for their draft MRs in order >> to catch issues before human reviewers spend time on them. >> * Only once the "human review" has concluded, we do a second GlaHiCo >> run, and the result is shared publicly on the MR. >> >> Does this sound better to you? >> >> Best, >> Simon >> >> Am Do., 30. Juli 2026 um 02:38 Uhr schrieb Moritz Angermann >> <[email protected]>: >> > >> > Hi Simon, >> > >> > I want to highlight the following: >> >> >> >> >> >> 1. The information in the review is intended for all people who are >> >> involved in an MR: Both authors and reviewers. >> > >> > >> > I personally find this to be straddling the line where we force >> > exclusively LLM generated content on others and it starts to blur the line >> > between where the human is. >> > >> > If a human receives (on request) a LLM review, treats it as probabilistic >> > text output, and reviews the review for potential useful highlight, and >> > then selectively makes those highlights their own before fixing them >> > themselves or discussing them with others has a qualitatively difference >> > to “look at the LLM generated review at <url>.” The latter one (at least >> > to me), comes across as lacking respect towards others. >> > >> > If you use a tool to perform a review, I don’t really care, this is your >> > agency, your freedom to do as you please, but if you pass this off to me >> > as: here look at this output an llm generated, instead of reading the >> > output making it your own and then passing it on as your own with your own >> > comments, … who is it with whom I interact? >> > Why don’t I instead just ask the llm myself? >> > >> > >> > Now if everyone agrees that they want to collectively ask a llm, sure. >> > Even though I do use LLMs, I wouldn’t want unedited llm output pass on to >> > me. There is no value in that to me; I can do that myself. I want to >> > interact with humans and their ideas, intellect and curiosity. >> > >> > It’s the same fundamental conflict I have wirh coderabbit and copilot >> > reviews. I find them intellectually exceptionally boring. Distracting >> > even. And having them in PR discussions (especially as comments like on >> > GitHub), I feel myself losing interest extremely quickly. This is >> > distinctly different from me using a LLM on the side as assertive tool to >> > support my understanding, and form my opinion (and maybe word choice) when >> > writing a comment. >> > >> > My primary concern is that while I’m perfectly ok with using LLMs, this >> > opens the door towards verbatim llm output as a normalized way to interact >> > with others; something which I oppose. >> > >> > Best, >> > Moritz >> > >> >> >> >> Am Mo., 27. Juli 2026 um 19:38 Uhr schrieb Simon Jakobi >> >> <[email protected]>: >> >> > >> >> > Hi devs! >> >> > >> >> > Inspired by the recent discussion on the LLM policy, I'd like to try >> >> > introducing LLM-generated code reviews in GHC, as an opt-in service. >> >> > >> >> > My motivation: >> >> > >> >> > * Catch issues before human reviewers spend time on them. >> >> > * Reduce the number of bugs merged. >> >> > >> >> > >> >> > What this is — and what it isn't >> >> > -------------------------------- >> >> > >> >> > This is tool output: requested on the MR, produced by a model, posted >> >> > verbatim >> >> > and clearly marked as machine-generated. This is not a review in the >> >> > sense of >> >> > draft LLM policy. >> >> > >> >> > I'm not vouching for any of it. Think of it as a linter the author >> >> > could have >> >> > run themselves, except on my LLM budget. Nobody is required to act on >> >> > it even read it. >> >> > >> >> > I mention this explicitly because the policy asks reviewers to take full >> >> > responsibility for each line of their review. I can't do that here — it >> >> > would >> >> > mean checking every finding myself, which defeats the purpose and would >> >> > restrict the service to areas of the compiler I already know well. >> >> > >> >> > >> >> > How to request one >> >> > ------------------ >> >> > >> >> > * Ping me on the MR with a comment like "@sjakobi llm-review please". >> >> > * Feel free to request particular aspects the review should cover. >> >> > * Authors can request a review for their own MRs. Maintainers can >> >> > request one >> >> > on any MR. >> >> > * If I have enough usage left on my plan, I'll confirm that I'm on it. >> >> > * I'll then prompt the model, and post the result on the MR. >> >> > * Feedback on the quality of the review is of course very welcome. >> >> > >> >> > >> >> > What I promise >> >> > -------------- >> >> > >> >> > * Every review is labelled as machine-generated, with model, effort >> >> > level and >> >> > the exact prompt used. >> >> > * Reviews are posted verbatim. I don't edit or curate them, so what you >> >> > see is >> >> > exactly what the model produced. >> >> > * Only the public MR diff and public repository context go to the model. >> >> > Nothing else. >> >> > * I'll keep working on the prompts and the choice of model to improve >> >> > the >> >> > signal-to-noise ratio. >> >> > * If the signal-to-noise ratio stays inacceptably bad, we can simply >> >> > stop this service. >> >> > >> >> > >> >> > Review coverage >> >> > --------------- >> >> > >> >> > I'll probably start with a fairly basic review prompt, primarily aimed >> >> > at >> >> > correctness issues. >> >> > >> >> > Possible extensions: >> >> > >> >> > * Documentation consistency: check that documentation stays consistent, >> >> > including Notes elsewhere that reference the changed code. >> >> > * Performance: check for performance issues — potentially including >> >> > checking >> >> > the -ddump-simpl output for the changed code for unnecessary >> >> > allocation etc. >> >> > * Feel free to suggest anything else! >> >> > >> >> > >> >> > Format >> >> > ------ >> >> > >> >> > Initially a single comment per review. Once we reach a good >> >> > signal-to-noise >> >> > ratio, we can consider inline comments on the diff. >> >> > >> >> > >> >> > Caveats >> >> > ------- >> >> > >> >> > This is a volunteer service I intend to provide in my spare time. If >> >> > I'm AFK >> >> > or on vacation, reviews will take longer. If I can't keep up with >> >> > requests, I >> >> > will prioritise which MRs I run. I might take a break entirely. >> >> > >> >> > >> >> > Getting involved >> >> > ---------------- >> >> > >> >> > Is anyone interested in joining this effort? If so we could form a team >> >> > (@llm-reviewers?), and share prompting techniques, model choices, etc. >> >> > >> >> > If the service turns out to be genuinely useful, we can look into >> >> > automating it and – if necessary – possible funding. >> >> > >> >> > >> >> > Cheers, >> >> > Simon >> >> _______________________________________________ >> >> ghc-devs mailing list -- [email protected] >> >> To unsubscribe send an email to [email protected] _______________________________________________ ghc-devs mailing list -- [email protected] To unsubscribe send an email to [email protected]
