Re: [agi] Can symbolic approach entirely replace NN approach?

[James Bowery]

It's also worth reiterating a point I made before

about the confusion between abstract grammar as a prior (heuristic) for
grammar induction and the incorporation of so-induced grammars as priors,
such as in "physics informed machine learning
".

In the case of physics informed machine learning, the language of physics
is incorporated into the learning algorithm.  This helps the machine
learning algorithm learn things about the physical world without having to
re-derive the body of physics knowledge.

Don't confuse the two levels here:

1) My suspicion that natural language learning may benefit from
prioritizing HOPDA as an *abstract* grammar to learn something about
natural languages -- such as their grammars.

2) My suspicion (supported by "X informed machine learning" exemplified by
the aforelinked work) that there may be prior knowledge about natural
language more specific than the level of *abstract* grammar -- such as
specific rules of thumb for, say, the English language that may greatly
speed training time on English corpora.

On Sun, May 26, 2024 at 9:40 AM James Bowery  wrote:

> See the recent DeepMind paper "Neural Networks and the Chomsky Hierarchy
> " for the sense of "grammar" I'm using
> when talking about the HNet paper's connection to Granger's prior papers
> about "grammar", the most recent being "Toward the quantification of
> cognition ".  Although the DeepMind
> paper doesn't refer to Granger's work on HOPDAs, it does at least
> illustrate a fact, long-recognized in the theory of computation:
>
> Grammar, Computation
> Regular, Finite-state automaton
> Context-free, Non-deterministic pushdown automaton
> Context sensitive, Linear-bounded non-deterministic Turing machine
> Recursively enumerable, Turing machine
>
> Moreover, the DeepMind paper's empirical results support the corresponding
> hierarchy of computational power.
>
> Having said that, it is critical to recognize that everything in a finite
> universe reduces to finite-state automata in hardware -- it is only in our
> descriptive languages that the hierarchy exists.  We don't describe all
> computer programs in terms of finite-state automata aka regular grammar
> languages.  We don't describe all computer programs even in terms of Turing
> complete automata aka recursively enumerable grammar languages.
>
> And I *have* stated before (which I first linked to the HNet paper)
> HOPDAs are interesting as a heuristic because they *may* point the way to
> a prioritization if not restriction on the program search space that
> evolution has found useful in creating world models during an individual
> organism's lifetime.
>
> The choice of language, hence the level of grammar, depends on its utility
> in terms of the Algorithmic Information Criterion for model selection.
>
> I suppose one could assert that none of that matters so long as there is
> any portion of the "instruction set" that requires the Turing complete
> fiction, but that's a rather ham-handed critique of my nuanced point.
>
>
>
> On Sat, May 25, 2024 at 9:37 PM Rob Freeman 
> wrote:
>
>> Thanks Matt.
>>
>> The funny thing is though, as I recall, finding semantic primitives
>> was the stated goal of Marcus Hutter when he instigated his prize.
>>
>> That's fine. A negative experimental result is still a result.
>>
>> I really want to emphasize that this is a solution, not a problem, though.
>>
>> As the HNet paper argued, using relational categories, like language
>> embeddings, decouples category from pattern. It means we can have
>> categories, grammar "objects" even, it is just that they may
>> constantly be new. And being constantly new, they can't be finitely
>> "learned".
>>
>> LLMs may have been failing to reveal structure, because there is too
>> much of it, an infinity, and it's all tangled up together.
>>
>> We might pick it apart, and have language models which expose rational
>> structure, the Holy Grail of a neuro-symbolic reconciliation, if we
>> just embrace the constant novelty, and seek it as some kind of
>> instantaneous energy collapse in the relational structure of the data.
>> Either using a formal "Hamiltonian", or, as I suggest, finding
>> prediction symmetries in a network of language sequences, by
>> synchronizing oscillations or spikes.
>>
>> On Sat, May 25, 2024 at 11:33 PM Matt Mahoney 
>> wrote:
>> >
>> > I agree. The top ranked text compressors don't model grammar at all.

--
Artificial General Intelligence List: AGI
Permalink: 
https://agi.topicbox.com/groups/agi/T682a307a763c1ced-M440c27b119465aeba2e4d2bb
Delivery options: https://agi.topicbox.com/groups/agi/subscrip

Re: [agi] Can symbolic approach entirely replace NN approach?

[James Bowery]

See the recent DeepMind paper "Neural Networks and the Chomsky Hierarchy
" for the sense of "grammar" I'm using
when talking about the HNet paper's connection to Granger's prior papers
about "grammar", the most recent being "Toward the quantification of
cognition ".  Although the DeepMind paper
doesn't refer to Granger's work on HOPDAs, it does at least illustrate a
fact, long-recognized in the theory of computation:

Grammar, Computation
Regular, Finite-state automaton
Context-free, Non-deterministic pushdown automaton
Context sensitive, Linear-bounded non-deterministic Turing machine
Recursively enumerable, Turing machine

Moreover, the DeepMind paper's empirical results support the corresponding
hierarchy of computational power.

Having said that, it is critical to recognize that everything in a finite
universe reduces to finite-state automata in hardware -- it is only in our
descriptive languages that the hierarchy exists.  We don't describe all
computer programs in terms of finite-state automata aka regular grammar
languages.  We don't describe all computer programs even in terms of Turing
complete automata aka recursively enumerable grammar languages.

And I *have* stated before (which I first linked to the HNet paper) HOPDAs
are interesting as a heuristic because they *may* point the way to a
prioritization if not restriction on the program search space that
evolution has found useful in creating world models during an individual
organism's lifetime.

The choice of language, hence the level of grammar, depends on its utility
in terms of the Algorithmic Information Criterion for model selection.

I suppose one could assert that none of that matters so long as there is
any portion of the "instruction set" that requires the Turing complete
fiction, but that's a rather ham-handed critique of my nuanced point.



On Sat, May 25, 2024 at 9:37 PM Rob Freeman 
wrote:

> Thanks Matt.
>
> The funny thing is though, as I recall, finding semantic primitives
> was the stated goal of Marcus Hutter when he instigated his prize.
>
> That's fine. A negative experimental result is still a result.
>
> I really want to emphasize that this is a solution, not a problem, though.
>
> As the HNet paper argued, using relational categories, like language
> embeddings, decouples category from pattern. It means we can have
> categories, grammar "objects" even, it is just that they may
> constantly be new. And being constantly new, they can't be finitely
> "learned".
>
> LLMs may have been failing to reveal structure, because there is too
> much of it, an infinity, and it's all tangled up together.
>
> We might pick it apart, and have language models which expose rational
> structure, the Holy Grail of a neuro-symbolic reconciliation, if we
> just embrace the constant novelty, and seek it as some kind of
> instantaneous energy collapse in the relational structure of the data.
> Either using a formal "Hamiltonian", or, as I suggest, finding
> prediction symmetries in a network of language sequences, by
> synchronizing oscillations or spikes.
>
> On Sat, May 25, 2024 at 11:33 PM Matt Mahoney 
> wrote:
> >
> > I agree. The top ranked text compressors don't model grammar at all.

--
Artificial General Intelligence List: AGI
Permalink: 
https://agi.topicbox.com/groups/agi/T682a307a763c1ced-Mcca9a6d522c416b1c95cd3d1
Delivery options: https://agi.topicbox.com/groups/agi/subscription