See the recent DeepMind paper "Neural Networks and the Chomsky Hierarchy <https://arxiv.org/abs/2207.02098>" for the sense of "grammar" I'm using when talking about the HNet paper's connection to Granger's prior papers about "grammar", the most recent being "Toward the quantification of cognition <https://arxiv.org/abs/2008.05580>". Although the DeepMind paper doesn't refer to Granger's work on HOPDAs, it does at least illustrate a fact, long-recognized in the theory of computation:
Grammar, Computation Regular, Finite-state automaton Context-free, Non-deterministic pushdown automaton Context sensitive, Linear-bounded non-deterministic Turing machine Recursively enumerable, Turing machine Moreover, the DeepMind paper's empirical results support the corresponding hierarchy of computational power. Having said that, it is critical to recognize that everything in a finite universe reduces to finite-state automata in hardware -- it is only in our descriptive languages that the hierarchy exists. We don't describe all computer programs in terms of finite-state automata aka regular grammar languages. We don't describe all computer programs even in terms of Turing complete automata aka recursively enumerable grammar languages. And I *have* stated before (which I first linked to the HNet paper) HOPDAs are interesting as a heuristic because they *may* point the way to a prioritization if not restriction on the program search space that evolution has found useful in creating world models during an individual organism's lifetime. The choice of language, hence the level of grammar, depends on its utility in terms of the Algorithmic Information Criterion for model selection. I suppose one could assert that none of that matters so long as there is any portion of the "instruction set" that requires the Turing complete fiction, but that's a rather ham-handed critique of my nuanced point. On Sat, May 25, 2024 at 9:37 PM Rob Freeman <chaotic.langu...@gmail.com> wrote: > Thanks Matt. > > The funny thing is though, as I recall, finding semantic primitives > was the stated goal of Marcus Hutter when he instigated his prize. > > That's fine. A negative experimental result is still a result. > > I really want to emphasize that this is a solution, not a problem, though. > > As the HNet paper argued, using relational categories, like language > embeddings, decouples category from pattern. It means we can have > categories, grammar "objects" even, it is just that they may > constantly be new. And being constantly new, they can't be finitely > "learned". > > LLMs may have been failing to reveal structure, because there is too > much of it, an infinity, and it's all tangled up together. > > We might pick it apart, and have language models which expose rational > structure, the Holy Grail of a neuro-symbolic reconciliation, if we > just embrace the constant novelty, and seek it as some kind of > instantaneous energy collapse in the relational structure of the data. > Either using a formal "Hamiltonian", or, as I suggest, finding > prediction symmetries in a network of language sequences, by > synchronizing oscillations or spikes. > > On Sat, May 25, 2024 at 11:33 PM Matt Mahoney <mattmahone...@gmail.com> > wrote: > > > > I agree. The top ranked text compressors don't model grammar at all. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T682a307a763c1ced-Mcca9a6d522c416b1c95cd3d1 Delivery options: https://agi.topicbox.com/groups/agi/subscription
