Hello, I think examples like Clinton's are very valuable as they help keep the discussion concrete. I use LLMs a fair bit, but I spend a lot of time thinking in what ways they can help me. As I was reading the example it occurred to me that I would have approached the situations a bit differently, which might be interesting to others too, so here are my thoughts. For context, the situation is I used an LLM which found a bug in a part of the code base I am not familiar with, what do I do?
(1) My first choice would be: (a) do I want to learn more about this part of the code and fix the bug, or (b) file a bug report, so that I or someone else can deal with it later. (2) Say I chose (a), and for the sake of the example let's assume the fix for this bug is not a one line typo somewhere. (2.1) I would typically NOT start by asking the LLM to fix it. Instead, I will ask it to analyze the code and explain the problem to me. I'd keep the discussion going, asking questions etc. until I think I have a good understanding of the problem. This is useful for me because it let's me get "warmed up with the code", and also clarify in my head on what I am trying to do. It helps the LLM because it gets it to build the right context for working on the code. (2.2) Next I would ask the LLM to outline a fix, but still not write code. Instead, I ask it to write a "plan" on how we'd modify the code, and save it in a fie. I read through the file and code, and keep modifying the plan, adjust thing the plan, until I am clear on what needs doing. In the process I also ask the LLM to check our work, by looking for things that we might be missing in the plan etc. Of course, I am also trying to think of such things. (2.3) The next step is start writing code. Typically, I ask the LLM to start implementing bits of the plan, and stop after each step, marking the in the plan as it goes along. This allows me to review each code modification separately, which I find much easier than reviewing a big edit that does everything at once. As I am reviewing the code, I also keep asking the LLM to fix things I don't like (e.g., usual programming stuff, factor stuff, code style, add/remove/modify comments). The plan is also a convenient way to track what still needs doing. I also make commits when I get to a point that seems logically consistent, and test various bits in the intermediate ways. I wrote my process up, because I think it is a bit different from other folks, based on the comments I read, and I hope it highlights that there are many ways to use LLMs while programming, and generating code is just one part of the whole exercise. The process works well, because while I am learning about the code base, the LLM's context is being refined and biased towards the kind of solution I'd like it to generate. So it is not as fast as just "one shot"-ing a solution, but it tends to generate the solution I want. I think this is more pragmatic than accepting a solution you don't understand, as such a solution would quite likely fix one thing, but break something else. Of course, one does not always have a bunch of time to spend fixing a bug, but if that's the case, the pragmatic thing would be to document the issue in the form of a bug report. Anyway, I hope this helpful, I find this discussion quiet interesting, and I really appreciate people sharing their experiences. I fully understand Simon's sentiment about how humbling it is when these things find so many bugs in code I've worked on for years. On the positive side, I find it a lot easier to motivate myself to fix them with LLM help, so I think on average code quality is improving. Cheers, Iavor On Fri, Oct 23, 2026 at 5:43 PM Clinton Mead via ghc-devs < [email protected]> wrote: > This thread is somewhat timely, as I've just pushed up > https://gitlab.haskell.org/ghc/ghc/-/work_items/28029, a LLM built fix to > a uninitialised memory access issue in the WASM backend (I think). > > I ended up spending a couple of working days on it, rejecting an RTS only > solution because I was pretty sure there was a problem in the code > generator also (as the LLM had pointed out it only occurred in certain > optimisation levels). If also directed it to write a variety of test cases > in the draft-MR that it can. But I totally get what people are saying about > the LLM generated prose in the comments, they are a bit painful to read. I > would generally use the word that rhymes with banker to describe someone > who wrote like that in real life. > > But the fact is, I haven't got either the time or the expertise to do much > more with this. I haven't worked on GHC internals, and whilst I try to take > the time to contribute back to the Haskell ecosystem so it remains and > grows as a viable language in industry, I'm also trying to encourage my > employer to continue to make the unusual decision of paying someone to keep > and extend a Haskell system in production. Going off on weeks/months long > study sessions to do proper study into how GHC works is not practical for > me if I want to keep getting paid to write Haskell (and if I want my job to > exist for others in the future as well). > > So the practical options are either a slop issue/draft-MR is submitted, or > just keep it to myself. I don't expect this to be merged in its current > state or even at all, I get that. But I still feel an AI slop issue with a > reproducer for a bug is better than none at all. > > I think the guidelines as they are written are fine. There is the escape > hatch of a draft-MR, to avoid the "understanding every line" requirement, > which I have used. And I think it's reasonable to need to attribute AI > usage. Unless you want to change the guidelines to just outright say NO to > AI generated bug reports. But if you're going to do that, be honest and say > "if an LLM finds a bug report and you can't explain it or write a fix > yourself, run your own private fork and don't tell us about it because we > don't want to know". > > If you don't want to go that way, sloppy issues/MRs are just going to > happen because people are going to run into these issues more often, even > when they're not looking for them (as in my case). The only difference now > is that whilst previously I may have just used a workaround, or maybe just > made a vague report "I think something is broke here but I'm not sure > what/why", now you've got a draft-MR with 2 commits, the first with test > cases that fail and the second with an AI slop fix with test cases that > pass. Surely that's a better state of affairs? There's no obligation to put > this slop straight into the codebase so I don't see how there's a downside > of the current AI contribution policy. > > On Sat, Oct 24, 2026 at 11:01 AM Julian Ospald via ghc-devs < > [email protected]> wrote: > >> I'm afraid this is going to get worse. >> >> In the age of LLMs, understanding something is not the priority anymore. >> Productivity, not being "blocked" and quick drive-by contributions are the >> new norm. >> >> The technical leadership committee could absolutely fund the work of >> essential library maintainers like Kazu. But as you say, the solution is to >> increase institutional knowledge and inspire people to take it slow and >> understand something deeply. I'm not quite sure how... a bounty program may >> in fact attract more slop coders than interested long term maintainers. >> >> If you have ideas, please reach out: >> [email protected] >> >> Sorry for the OT. >> >> On October 23, 2026 3:50:45 PM UTC, amindfv--- via ghc-devs < >> [email protected]> wrote: >> >> On 10/23/2026 12:58 PM CEST Wolfgang Jeltsch via ghc-devs < >> [email protected]> wrote: >> >> >> >> >> >> Hi! >> >> >> >> It seems that there is a general story shining through what Kazu wrote: >> >> >> >> 1. We accidentally introduce lots of security issues into our >> software. >> >> >> >> 2. We don’t find these security issues but use LLMs to find them. >> >> >> >> 3. We cannot fix these security issues but use LLMs to fix them. >> >> >> >> 4. We don’t understand our codebases but use LLMs to make changes. >> >> >> >> 5. We cannot take responsibility for the changes made by LLMs but hope >> >> that they don’t introduce new issues. >> >> >> >> 6. We include LLM-generated text into our documentation that we don’t >> >> understand but let LLMs explain its meaning to us. >> >> >> >> This seems like an absolutely worrying state of affairs. I wonder why >> we >> >> got there: >> >> >> >> • Did we care too little about design? >> >> >> >> • Did we introduce too many features? >> >> >> >> • Did we care too much about development speed and too litle about >> >> quality? >> >> >> >> • Did we pay too little attention to detail? >> > >> >I think it's the same old open source story: many, many people use >> Kazu's software for mission-critical work, but very few people have >> contributed back to its development. I know of many private companies that >> could really be in trouble if Kazu's code had security flaws, yet when >> critical bugs are discovered, I'm not aware of much industrial urgency to >> fix them. The assumption is that Kazu or another developer (or an LLM) will >> fix them for free. >> > >> >Maybe we need to think about ways to support the work of comprehending >> these vital but sometimes-boring (sorry, Kazu! :) ) libraries. I don't >> necessarily mean in terms of money, but to fix the problem that the number >> of people dependent on a library is so much larger than the number who >> understand enough to fix it. >> > >> >- Tom >> >_______________________________________________ >> >ghc-devs mailing list -- [email protected] >> >To unsubscribe send an email to [email protected] >> _______________________________________________ >> ghc-devs mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> > _______________________________________________ > ghc-devs mailing list -- [email protected] > To unsubscribe send an email to [email protected] >
_______________________________________________ ghc-devs mailing list -- [email protected] To unsubscribe send an email to [email protected]
