In your post, Russ, you say: “They are trained to produce fluent language, not to produce valid statements.“
Is that actually, operationally, what they are trained to do? I speak from a position of ignorance here, but my impression is that they are trained to effectively stitch together fragments of varying lengths, according to rules for what stitchings are “compatible”. My thinking here is metaphorical, to homologous recombination in DNA. Some regions that don’t start out contiguous can be concatenated by DNA repair machinery, because under the physics to which it responds, they have plausible enough overlap that it considers them “compatible” or eligible to be identified at the join region, their “mis-matches” edited out. Other pairs are so dissimilar that, under its operating physics, the repair machinery will effectively never join them. My metaphor isn’t great, in the sense that if what LLMs (for human speech) are doing is “next-word prediction”, that says that the notion of “joining” is reduced formally to appending next-words onto strings. Though, to the extent that certain substrings of next-words are extremely frequently attested across the corpus of all the training expressions, one would expect to see extended sequences essentially reproduced as fragments with large probability. If my previous two characterizations aren’t fundamentally wrong, it would follow that fluent speech-generation becomes possible because the compatible-joining relations are suffficiently strong in human languages that the attention structures or other feed-forward aspects of the architecture have no trouble capturing them in parameters, even though human linguists trying to write them as re-write rules from which a computer could generate native-like speech failed for decades to get anywhere close to that. My interpretation here would be consistent with what I believed was the main watershed change in the LLMs: that the parametric models would, ultimately, have terribly few parameters, whereas the LLMs can flood-fill a corpus with parameters, and then try to drip out the parts that don’t “stick to” some pattern in the data, and are regarded as the excess entropy from the sampling algorithm that the training is supposed to recognize and remove. It is easy to imagine that fluent speech has far more regularities than rule-book linguists captured parametrically, but still few enough that LLMs can have no trouble attaching to almost-all of them, with parameters to spare. Hence fluent speech could be epiphenomenal on what they are (operationally, mechanistically) being trained to do, but a natural summary statistic for the effectiveness of that training, and of course the one that drives market engagement. But if the above is the case, then the question of when they get “the syntax” right and “the semantics” wrong, would seem to turn on how much context from the training set is needed to identify semantically as well as syntactically appropriate “allowed joins” of fragments. When short fragments contain enough of their own context to constrain most of the semantics, the stitching training algorithm has no reason to perform any worse at revealing the semantic signal in the training set than the syntactic one. But if probability needs to be withheld for a long time in the prediction model, driving it to prioritize a much smaller number of longer or more remote assembled inputs from the training data, it could still do fine on syntax but fail to “find” and “render” the semantic signal in the training data, even if that signal is present in principal. I would not feel a need to use terms like “understanding” anywhere in the above, to make predictions of what kinds of successes or failures an LLM might deliver from the user’s perspective. It seems to me like something that all lives in the domain of hardness-of-search combinatorics in data-spaces with a lot of difficult structure. Eric > On Sep 10, 2025, at 7:02, Russ Abbott <[email protected]> wrote: > > > OpenAI just published a paper on hallucinations > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fcdn.openai.com%2fpdf%2fd04913be-3f6f-4d2b-b283-ff432ef4aaa5%2fwhy-language-models-hallucinate.pdf&c=E,1,IvBfvLzhn3L6LCNk3_ktKoEbc9NI2Oqq8vlFpNcIXCHElptIB-Fx-UxQYyTnCFW_ToeD5Kd4RjHkY-6fLxSBqZueOcvRqyHwpsHPK9ugMNcsOw,,&typo=1> > as well as a post summarizing the paper > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fopenai.com%2findex%2fwhy-language-models-hallucinate%2f&c=E,1,tEcctM28Lbt5XBi3gNiUX-RiFelMYHNq6K3VJBilGv1_Z8uAt34ta8FaU-FcW5i8V3-2tsjNPu_at8Es78G2_drdmykgOltvjRvvaw1hUgnXUsv3&typo=1>. > The two of them seem wrong-headed in such a simple and obvious way that I'm > surprised the issue they discuss is still alive. > > The paper and post point out that LLMs are trained to generate fluent > language--which they do extraordinarily well. The paper and post also point > out that LLMs are not trained to distinguish valid from invalid statements. > Given those facts about LLMs, it's not clear why one should expect LLMs to be > able to distinguish true statements from false statements--and hence why one > should expect to be able to prevent LLMs from hallucinating. > > In other words, LLMs are built to generate text; they are not built to > understand the texts they generate and certainly not to be able to determine > whether the texts they generate make factually correct or incorrect > statements. > > Please see my post > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2frussabbott.substack.com%2fp%2fwhy-language-models-hallucinate-according&c=E,1,zLy4H6KEpD5hDchYiBUjiH2J5dG2O9bmqa-jm1z6mGgRSqZgDKaVd2D2Xh_2Wuzi7FtZu2kjIOTNjQuk4iwsnfNUG68UPCxmZvD_IHTVUEPTcW6HDgpmcozzRQ,,&typo=1> > elaborating on this. > > Why is this not obvious, and why is OpenAI still talking about it? > > -- Russ Abbott > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2frussabbott.substack.com%2f&c=E,1,cKMmq0etz4RiUaE4G2F04re6Su0EnNyqR9j5Dx8RcccQVNOB2r5CMNBzxRL9EYmN3lG_11nhB4wP-5jPf7NR86Mb9VxP9Jn2YUdKPZQT&typo=1> > (Click for my Substack) > Professor Emeritus, Computer Science > California State University, Los Angeles > > > .- .-.. .-.. / ..-. --- --- - . .-. ... / .- .-. . / .-- .-. --- -. --. / ... > --- -- . / .- .-. . / ..- ... . ..-. ..- .-.. > FRIAM Applied Complexity Group listserv > Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom > https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fbit.ly%2fvirtualfriam&c=E,1,TGWUFxByQV3GAAU3oSRoMNfDJD6ptWzY73PWkEy6wjvRSnx8Mc4UYZvwNnCZtQTtnx4s1YQWhA5OFZgcHYsPOfh2UOY3Y08aOLzFbRROXd4isiXdoT93L5Ncgw,,&typo=1 > to (un)subscribe > https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,R8rvP64Y8Ojn7C4RmXsVaTwfI61-h--86QYAcdZfJB5b2Vma9UVdbCXCsDqLzWtC_TM9Ckm5LlRcoIn4_6mGC8c_WptkWvx_WtZA0PdtE8ViiUc,&typo=1 > FRIAM-COMIC > https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,JolQcZ2iD8sPfKhQE-npSBJtUmqqa8EaE0J19wBCnesx4rjYKUpByO5mwjwVUiEn91veQr1Bk3B0gvLNuTtgIkN8-2VZRSkQS61pFh_zro8Oe_g7&typo=1 > archives: 5/2017 thru present > https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fredfish.com%2fpipermail%2ffriam_redfish.com%2f&c=E,1,3J72bQm1T2SIdCaPyxSx4gitJ3Bt_OjLNAoKxcLa4u2f5Yw2m3gHImwAjCKE9RabMTMbzMedGiltpwWQ5w10fnNmDFvVkW9oQcfwHVezCQ,,&typo=1 > 1/2003 thru 6/2021 http://friam.383.s1.nabble.com/
.- .-.. .-.. / ..-. --- --- - . .-. ... / .- .-. . / .-- .-. --- -. --. / ... --- -- . / .- .-. . / ..- ... . ..-. ..- .-.. FRIAM Applied Complexity Group listserv Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom https://bit.ly/virtualfriam to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com FRIAM-COMIC http://friam-comic.blogspot.com/ archives: 5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ 1/2003 thru 6/2021 http://friam.383.s1.nabble.com/
