It's also worth reiterating a point I made before <https://agi.topicbox.com/groups/agi/Teaac2c1a9c4f4ce3-M4a92b688c0804deb6a6a12a1/gproton-combinatorial-hierarchy-computational-irreducibility-and-other-things-that-just-dont-matter-to-reaching-agi> about the confusion between abstract grammar as a prior (heuristic) for grammar induction and the incorporation of so-induced grammars as priors, such as in "physics informed machine learning <https://www.youtube.com/playlist?list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa>".
In the case of physics informed machine learning, the language of physics is incorporated into the learning algorithm. This helps the machine learning algorithm learn things about the physical world without having to re-derive the body of physics knowledge. Don't confuse the two levels here: 1) My suspicion that natural language learning may benefit from prioritizing HOPDA as an *abstract* grammar to learn something about natural languages -- such as their grammars. 2) My suspicion (supported by "X informed machine learning" exemplified by the aforelinked work) that there may be prior knowledge about natural language more specific than the level of *abstract* grammar -- such as specific rules of thumb for, say, the English language that may greatly speed training time on English corpora. On Sun, May 26, 2024 at 9:40 AM James Bowery <jabow...@gmail.com> wrote: > See the recent DeepMind paper "Neural Networks and the Chomsky Hierarchy > <https://arxiv.org/abs/2207.02098>" for the sense of "grammar" I'm using > when talking about the HNet paper's connection to Granger's prior papers > about "grammar", the most recent being "Toward the quantification of > cognition <https://arxiv.org/abs/2008.05580>". Although the DeepMind > paper doesn't refer to Granger's work on HOPDAs, it does at least > illustrate a fact, long-recognized in the theory of computation: > > Grammar, Computation > Regular, Finite-state automaton > Context-free, Non-deterministic pushdown automaton > Context sensitive, Linear-bounded non-deterministic Turing machine > Recursively enumerable, Turing machine > > Moreover, the DeepMind paper's empirical results support the corresponding > hierarchy of computational power. > > Having said that, it is critical to recognize that everything in a finite > universe reduces to finite-state automata in hardware -- it is only in our > descriptive languages that the hierarchy exists. We don't describe all > computer programs in terms of finite-state automata aka regular grammar > languages. We don't describe all computer programs even in terms of Turing > complete automata aka recursively enumerable grammar languages. > > And I *have* stated before (which I first linked to the HNet paper) > HOPDAs are interesting as a heuristic because they *may* point the way to > a prioritization if not restriction on the program search space that > evolution has found useful in creating world models during an individual > organism's lifetime. > > The choice of language, hence the level of grammar, depends on its utility > in terms of the Algorithmic Information Criterion for model selection. > > I suppose one could assert that none of that matters so long as there is > any portion of the "instruction set" that requires the Turing complete > fiction, but that's a rather ham-handed critique of my nuanced point. > > > > On Sat, May 25, 2024 at 9:37 PM Rob Freeman <chaotic.langu...@gmail.com> > wrote: > >> Thanks Matt. >> >> The funny thing is though, as I recall, finding semantic primitives >> was the stated goal of Marcus Hutter when he instigated his prize. >> >> That's fine. A negative experimental result is still a result. >> >> I really want to emphasize that this is a solution, not a problem, though. >> >> As the HNet paper argued, using relational categories, like language >> embeddings, decouples category from pattern. It means we can have >> categories, grammar "objects" even, it is just that they may >> constantly be new. And being constantly new, they can't be finitely >> "learned". >> >> LLMs may have been failing to reveal structure, because there is too >> much of it, an infinity, and it's all tangled up together. >> >> We might pick it apart, and have language models which expose rational >> structure, the Holy Grail of a neuro-symbolic reconciliation, if we >> just embrace the constant novelty, and seek it as some kind of >> instantaneous energy collapse in the relational structure of the data. >> Either using a formal "Hamiltonian", or, as I suggest, finding >> prediction symmetries in a network of language sequences, by >> synchronizing oscillations or spikes. >> >> On Sat, May 25, 2024 at 11:33 PM Matt Mahoney <mattmahone...@gmail.com> >> wrote: >> > >> > I agree. The top ranked text compressors don't model grammar at all. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T682a307a763c1ced-M440c27b119465aeba2e4d2bb Delivery options: https://agi.topicbox.com/groups/agi/subscription
