Hi devs! Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation: * Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged. What this is — and what it isn't -------------------------------- This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy. I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it. I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well. How to request one ------------------ * Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome. What I promise -------------- * Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service. Review coverage --------------- I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues. Possible extensions: * Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else! Format ------ Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff. Caveats ------- This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely. Getting involved ---------------- Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc. If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding. Cheers, Simon _______________________________________________ ghc-devs mailing list -- [email protected] To unsubscribe send an email to [email protected]
