Addendum: another candidate for this variational model for finding distributions to replace back-prop (and consequently with the potential to capture predictive structure which is chaotic attractors. Though they don't appreciate the need yet.) There's Extropic, which is proposing using heat noise. And, another, LiquidAI. If it's true LiquidAI have nodes which are little reservoir computers, potentially that might work on a similar variational estimation/generation of distributions basis. Joscha Bach is involved with that. Though I don't know in what capacity.
James: "Physics Informed Machine Learning". "Building models from data using optimization and regression techniques". Fine. If you have a physics to constrain it to. We don't have that "physics" for language. Richard Granger you say? The brain is constrained to be a "nested stack"? https://www.researchgate.net/publication/343648662_Toward_the_quantification_of_cognition Language is a nested stack? Possibly. Certainly you get a (softish) ceiling of recursion starting level 3. The famous, level 2: "The rat the cat chased escaped" (OK) vs. level 3: "The rat the cat the dog bit chased escaped." (Borderline not OK.) How does that contradict my assertion that such nested structures must be formed on the fly, because they are chaotic attractors of predictive symmetry on a sequence network? On the other hand, can fixed, pre-structured, nested stacks explain contradictory (semantic) categories, like "strong tea" (OK) vs "powerful tea" (not OK)? Unless stacks form on the fly, and can contradict, how can we explain that "strong" can be a synonym (fit in the stack?) for "powerful" in some contexts, but not others? On the other hand, a constraint like an observation of limitations on nesting, might be a side effect of the other famous soft restriction, the one on dependency length. A restriction on dependency length is an easier explanation for nesting limits, and fits with the model that language is just a sequence network, which gets structured (into substitution groups/stacks?) on the fly. On Mon, May 6, 2024 at 11:06 PM James Bowery <jabow...@gmail.com> wrote: > > Let's give the symbolists their due: > > https://youtu.be/JoFW2uSd3Uo?list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa > > The problem isn't that symbolists have nothing to offer, it's just that > they're offering it at the wrong level of abstraction. > > Even in the extreme case of LLM's having "proven" that language modeling > needs no priors beyond the Transformer model and some hyperparameter > tweaking, there are language-specific priors acquired over the decades if not > centuries that are intractable to learn. > > The most important, if not conspicuous, one is Richard Granger's discovery > that Chomsky's hierarchy elides the one grammar category that human cognition > seems to use. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tb63883dd9d6b59cc-Me078486d3e7a407326e33a8a Delivery options: https://agi.topicbox.com/groups/agi/subscription
