I agree. The top ranked text compressors don't model grammar at all. On Fri, May 24, 2024, 11:47 PM Rob Freeman <chaotic.langu...@gmail.com> wrote:
> Ah, I see. Yes, I saw that reference. But I interpreted it only to > mean the general forms of a grammar. Do you think he means the > mechanism must actually be a grammar? > > In the earlier papers I interpret him to be saying, if language is a > grammar, what kind of a grammar must it be? And, yes, it seemed he was > toying with actual physical mechanisms relating to levels of brain > structure. Thalamo-cortical loops? > > The problem with that is, language doesn't actually seem to be any > kind of grammar at all. > > It's like saying if the brain had to be an internal combustion engine, > it might be a Mazda rotary. BFD. It's not an engine at all. > > I don't know if the authors realized that. But surely that's the point > of the HNet paper. That something can generate the general forms of a > grammar, without actually being a grammar. > > I guess this goes back to your assertion in our prior thread that > "learning" needs to be constrained by "physical priors" of some kind > (was it?) Are there physical "objects" constraining the "learning", or > does the "learning" vaguely resolve as physical objects, but not > quite? > > I don't think vague resemblance to objects means the objects must exist, > at all. > > Take Kepler and the planets. If the orbits of planets are epicycles, > which epicycles would they be? The trouble is, it turns out they are > not epicycles. > > And at least epicycles work! That's the thing for natural language. > Formal grammar doesn't even work. None of them. Nested stacks, context > free, Chomsky hierarchy up, down, and sideways. They don't work. So > figuring out which formal grammar is best, is a pointless exercise. > None of them work. > > Yes, broadly human language seems to resolve itself into forms which > resemble formal grammar (it's probably designed to do that, so that it > can usefully represent the world.) And it might be generally useful to > decide which formal grammar it best (vaguely) resembles. > > But in detail it turns out human language does not obey the rules of > any formal grammar at all. > > It seems to be a bit like the way the output of a TV screen looks like > objects moving around in space. Yes, it looks like objects moving in > space. You might even generate a physics based on the objects which > appear to be there. It might work quite well until you came to Road > Runner cartoons. That doesn't mean the output of a TV screen is > actually objects moving around in space. If you insist on implementing > a TV screen as objects moving around in space, well, it might be a > puppet show similar enough to amuse the kids. But you won't make a TV > screen. You will always fail. And fail in ways very reminiscent of the > way formal grammars almost succeed... but fail, to represent human > language. > > Same thing with a movie. Also looks a lot like objects moving around > on a screen. But is it objects moving on a screen? Different again. > > Superficial forms do not always equate to mechanisms. > > That's what's good about the HNet paper for me. It discusses how those > general forms might emerge from something else. > > The history of AI in general, and natural language processing in > particular, has been a search for those elusive "grammars" we see > chasing around on the TV screens of our minds. And they all failed. > What has succeeded has been breaking the world into bits (pixels?) and > allowing them to come together in different ways. Then the game became > how to bring them together. Supervised "learning" spoon fed the > "objects" and bound the pixels together explicitly. Unsupervised > learning tried to resolve "objects" as some kind of similarity between > pixels. AI got a bump when, by surprise, letting the "objects" go > entirely turned out to generate text that was more natural than ever! > Who'd a thunk it? Letting "objects" go entirely works best! If it > hadn't been for the particular circumstances of language, pushing you > to a "prediction" conception of the problem, how long would it have > taken us to stumble on that? The downside to that was, letting > "objects" go entirely also doesn't totally fit with what we > experience. We do experience the world as "objects". And without those > "objects" at all, LLMs are kind of unhinged babblers. > > So where's the right balance? Is the solution as LeCun, and perhaps > you, suggest (or Ben, looking for "semantic primitives" two years > ago...), to forget about the success LLMs had by letting go of objects > entirely. To repeat our earlier failures and seek the "objects" > elsewhere. Some other data. Physics? I see the objects, dammit! Look! > There's a coyote, and there's a road runner, and... Oh, my physics > didn't allow for that... > > Or could it be the right balance is, yes, to ignore the exact > structure of the objects as LLMs have done, but no, not to do it as > LLMs do by totally ignoring "objects", but to ignore only the internal > structure of the "objects", by focusing on relations defining objects > in ways which allow their internal "pattern" to vary. > > That's what I see being presented in the HNet paper. Maybe I'm getting > ahead of its authors. Because that is the solution I'm presenting > myself. But I interpret the HNet paper to present that option also. > Cognitive objects, including "grammar", can emerge with a freedom > which resembles the LLM freedom of totally ignoring "objects" (which > seems to be necessary, both by the success of LLMs at generating text, > and by the observed failure of formal grammars historically) if you > specify them in terms of external relations. > > Maybe the paper authors don't see it. But the way they talk about > generating grammars based on external relations, opens the door to it. > > On Fri, May 24, 2024 at 10:12 PM James Bowery <jabow...@gmail.com> wrote: > > > > > > > > On Thu, May 23, 2024 at 9:19 PM Rob Freeman <chaotic.langu...@gmail.com> > wrote: > >> > >> ...(Regarding the HNet paper) > >> The ideas of relational category in that paper might really shift the > >> needle for current language models. > >> > >> That as distinct from the older "grammar of mammalian brain capacity" > >> paper, which I frankly think is likely a dead end. > > > > > > Quoting the HNet paper: > >> > >> We conjecture that ongoing hierarchical construction of > >> such entities can enable increasingly “symbol-like” repre- > >> sentations, arising from lower-level “statistic-like” repre- > >> sentations. Figure 9 illustrates construction of simple “face” > >> configuration representations, from exemplars constructed > >> within the CLEVR system consisting of very simple eyes, > >> nose, mouth features. Categories (¢) and sequential rela- > >> tions ($) exhibit full compositionality into sequential rela- > >> tions of categories of sequential relations, etc.; these define > >> formal grammars (Rodriguez & Granger 2016; Granger > >> 2020). Exemplars (a,b) and near misses (c,d) are presented, > >> initially yielding just instances, which are then greatly re- > >> duced via abductive steps (see Supplemental Figure 13). > > > > Artificial General Intelligence List / AGI / see discussions + > participants + delivery options Permalink ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T682a307a763c1ced-Mca3eb6ef6f8a4b6ebcbad2b5 Delivery options: https://agi.topicbox.com/groups/agi/subscription
