Perhaps I can add my perspective. This thread is largely about LLM generated tickets. Mikolaj in particular has been posting a series of a couple of dozen such tickets -- so many in fact that I started collecting them on a tab of my personal WIP spreadsheet. You can see it here <https://docs.google.com/spreadsheets/d/1o05HVgAnsSjuLNFyDZ78DJPIirw664i7i48g6aHan5w/edit?usp=sharing> (look at the Mikolaj tab). From this I have learned
- All of them that I have investigated so far have proved genuine -- they expose genuine shortcomings in GHC. - It's pretty overwhelming, even discouraging, to get so many in a short time. And yet, and yet.... I would rather know than not-know. - The explanations are often long, but I zoom in on the reproducer, and only then refer back to the explanation. As Mikolaj says, these reproducers are far, far, far better than the gigantic reproducers he used to offer. Provided the ticket has a short statement of what is wrong, and a reproducer, I don't mind if there is AI-generated goop to accompany it. Indeed, once I can reproduce it, the goop is often somewhat helpful. - I say "somewhat" because once I have understood the problem, I can often come up with a much crisper summary myself Here's a concrete example <https://gitlab.haskell.org/ghc/ghc/-/work_items/25526#note_705423>, with my summary belo. - It would be absurd to expect Mikolaj to write every word. He doesn't know enough about GHC's internals. If I insisted on that, he just wouldn't submit them, and that would be a shame. - *The biggest shortcoming is this: I have no idea how important the ticket is*. When all tickets were human generated, I knew that every single one was somewhat important to some human. I don't want to spend time fixing bugs that no one cares about, and I'm 100% against LLM's autonomously submitting tickets. Every ticket must have a human behind it. Meanwhile, I asked Mikolaj to prioritise his flood of tickets, which he has done as you'll see on the spreadsheet. - One way in which the flood can feel exhausting is simply because it exposes so many embarrassing shortcomings in the compiler on which I have lavished so much love and care. How can I have been so stupid in so many different ways? Why is GHC so over-complicated? It is humbling. But in a good way, I think. Finally, while tickets may have a lot of readers, they are ephemeral. What I *really *care about is the code and the Notes in the code. It will be read for decades, whereas no one in 10 years time is going to be reading today's tickets. (There are occasional exceptions.) For the code and Notes I want every line to be human written, or so well studied, audited and revised that it might as well be human written. TL;DR: for me, provided a ticket has a human author who actively cares about, clear signposting and attribution, and a concrete reproducer, I'm pretty relaxed about accompanying slop. Relatedly, let me add that I have also been requesting *LLM reviews* for my MRs, through Simon Jacobi's good offices, and I have been astonished how good they are. They often identify genuine issues with my lovingly crafted MRs, and ones which both I and human reviewers have missed. At the time it's a bit "Oh no, I wanted to land this and now I can't", but again it must be better to know about the bugs than not to know. Again it's a humbling experience. So thus far, I have had a pretty positive experience of LLM-generated bug reports and reviews. But as I say above, I am much, much more cautious about contributions to GHC's code base and documentation. Simon On Thu, 22 Oct 2026 at 21:45, Mikolaj Konarski via ghc-devs < [email protected]> wrote: > > "[E]very contribution must have an identifiable human author (or > authors), who takes full responsibility for their entire contribution" > > This is precisely why I advocate to "embrace the slop", that is, > treat LLM-generated content as raw data, as if the content was > a log, a crash message, a debug session transcript. > But let me start by describing my experience with LLMs and GHC > so that my position can be better understood. > > I'm recently posting many GHC issues with LLM-generated > data and, while I hope to over time improve the way I do it, > I see that their quality is much better than when I I did > not use LLM for two reasons: > > 1. Before, I was posting huge reproducers and bright minds > were spending weeks minimizing them or sometimes they > could not. Now LLMs miminize all my reproducers. > > 2. I have no clue about GHC internals, so I could only guess > if my complaint is intended and/or known behaviour, which bugs > are already reported in a way that I don't recognize, which defect > is just a known difficult trade-off inherent in GHC heuristics biting me > occasionally and so not worth a report if I don't have a body > of statistical evidence to support a heuristic overhaul proposal. > Today LLMs recognize each of these scenarios for me and I'm > withdrawing half of my attempted defect reports. > > As a bonus, when I'm reading the slop to summarize it in a sentence > or two at the issues' start, I'm learning GHC internals. I'm learning > in particular from the often correct root cause analysis by LLMs > and from the often incorrect fix proposals. This is one reason giving > me hope my future GHC LLM-assisted contributions will be higher quality. > Another is GHC devs' feedback. Yet another is, I hope, LLMs getting better. > > Back to embracing the slop: I can't "take full responsibility" for what > LLM writes, because I'm clueless. It's the same as if I was posting > a transcript of a clueless debug session --- I totally can't take > responsibility for my working hypotheses. Neither, when I minimize > a big performance regression, I can't take responsibility for > the guess that the 3% slower minimized repo represents > the same mechanism that slowed down my original big program > by 30% (which is the only argument why it's worth looking at). > > What I can take full responsibility for is my user story > and the dumb raw data. "This log is from commandline X". > "This gdb output is from my program interrupted at Y". > "This LLM output is from Z". > > I hope we can improve our understanding not only what > kind of LLM-generated data is helpful, but also which kind of "Z" > is most honest and most helpful to all people concerned. > Is it mixing LLM-content with human-content? Certainly > not for me, because I'm too clueless to correct an LLM > and by mixing I'm diluting the attribution and introducing errors. > Should Z be the prompt? A summary in my own often misguided > words clearly separated from the slop (this is what I do currently)? > > All the best, > Mikolaj > _______________________________________________ > ghc-devs mailing list -- [email protected] > To unsubscribe send an email to [email protected] >
_______________________________________________ ghc-devs mailing list -- [email protected] To unsubscribe send an email to [email protected]
