Thanks for all the feedback, everyone! Regarding the choice of channel, how about I upload the reviews as snippets to e.g. https://gitlab.haskell.org/sjakobi/llm-reviews/-/snippets/? The advantage of snippets is that people can comment on them, so I can easily collect feedback on the quality of the reviews.
It's important to me though that there is some visible connection from the MR to the review: 1. The information in the review is intended for all people who are involved in an MR: Both authors and reviewers. 2. Similarly, I'd really love to get feedback from everyone who cares to read the reviews. This way I can improve the service more quickly. 3. I want some visibility, so more people are likely to give the service a try. In case you're worried about your AI agents ingesting the generated reviews, I think it shouldn't be too hard to block access to that repo for them. In other news, Anthropic has generously accepted my application for their open source program (https://claude.com/contact-sales/claude-for-oss), so for the next 6 months, I do have a decent amount of usage that I can spend on reviews (including access to Fable 5!). Consequently, I have started vibe-coding a primitive "mini-sashiko" for GHC: GlaHiCo (https://gitlab.haskell.org/sjakobi/GlaHiCo/-/merge_requests/1). Right now there's just a single "finder" agent, and each finding gets checked separately by a "verifier" agent. The code is a mess already of course, but it's slowly getting useful too. If you have a feature request or want to contribute in other ways, please let me know! :) Cheers, Simon Am Mo., 27. Juli 2026 um 19:38 Uhr schrieb Simon Jakobi <[email protected]>: > > Hi devs! > > Inspired by the recent discussion on the LLM policy, I'd like to try > introducing LLM-generated code reviews in GHC, as an opt-in service. > > My motivation: > > * Catch issues before human reviewers spend time on them. > * Reduce the number of bugs merged. > > > What this is — and what it isn't > -------------------------------- > > This is tool output: requested on the MR, produced by a model, posted verbatim > and clearly marked as machine-generated. This is not a review in the sense of > draft LLM policy. > > I'm not vouching for any of it. Think of it as a linter the author could have > run themselves, except on my LLM budget. Nobody is required to act on > it even read it. > > I mention this explicitly because the policy asks reviewers to take full > responsibility for each line of their review. I can't do that here — it would > mean checking every finding myself, which defeats the purpose and would > restrict the service to areas of the compiler I already know well. > > > How to request one > ------------------ > > * Ping me on the MR with a comment like "@sjakobi llm-review please". > * Feel free to request particular aspects the review should cover. > * Authors can request a review for their own MRs. Maintainers can request one > on any MR. > * If I have enough usage left on my plan, I'll confirm that I'm on it. > * I'll then prompt the model, and post the result on the MR. > * Feedback on the quality of the review is of course very welcome. > > > What I promise > -------------- > > * Every review is labelled as machine-generated, with model, effort level and > the exact prompt used. > * Reviews are posted verbatim. I don't edit or curate them, so what you see is > exactly what the model produced. > * Only the public MR diff and public repository context go to the model. > Nothing else. > * I'll keep working on the prompts and the choice of model to improve the > signal-to-noise ratio. > * If the signal-to-noise ratio stays inacceptably bad, we can simply > stop this service. > > > Review coverage > --------------- > > I'll probably start with a fairly basic review prompt, primarily aimed at > correctness issues. > > Possible extensions: > > * Documentation consistency: check that documentation stays consistent, > including Notes elsewhere that reference the changed code. > * Performance: check for performance issues — potentially including checking > the -ddump-simpl output for the changed code for unnecessary > allocation etc. > * Feel free to suggest anything else! > > > Format > ------ > > Initially a single comment per review. Once we reach a good signal-to-noise > ratio, we can consider inline comments on the diff. > > > Caveats > ------- > > This is a volunteer service I intend to provide in my spare time. If I'm AFK > or on vacation, reviews will take longer. If I can't keep up with requests, I > will prioritise which MRs I run. I might take a break entirely. > > > Getting involved > ---------------- > > Is anyone interested in joining this effort? If so we could form a team > (@llm-reviewers?), and share prompting techniques, model choices, etc. > > If the service turns out to be genuinely useful, we can look into > automating it and – if necessary – possible funding. > > > Cheers, > Simon _______________________________________________ ghc-devs mailing list -- [email protected] To unsubscribe send an email to [email protected]
