Matt Mahoney wrote:
One problem with some
connectionist models is trying to assign a 1-1 mapping between words and
neurons. The brain might have 10^8 neurons devoted to language, enough to
represent many copies of the different senses of a word and to learn new ones.
But most of the nets I am talking about do not assign 1 neuron to one
concept: they had three layers of roughly ten nodes each, and total
connectivity between layers (so 100 plus 100 connection weights). It
was the *weights* that stored the data, not the neurons. And the
concepts were stored across *all* of the weights.
Ditto for the brain. With a few thousand neurons, in three layers, we
could store ALL of the grapheme-phoneme correspondences in one entire
language.
Then you'll need to represent sequential information in such a way that
you can do something with it. Recurrent neural nets suck very badly if
you actually try to use them for anything, so don't get fooled by their
Soren Song.
Yes, but I think they are necessary. Lexical words, semantics, and grammar
all constrain each other. Recurrent networks can oscillate or become chaotic.
Even the human brain doesn't deal with this perfectly, so we have migraines
and epilepsy.
No: recurrent nets are terrible for all sorts of other reasons. I
don't see their ability to go into a chaotic regimes as virtue that
explains epilepsy! The recurrence by itself is not the bad part, BTW,
it is the whole style of the specific type of net called "recurrent".
Then you will need to represent layered representations: concepts
learned from conjunctikons of other concepts rather than layer-1
percepts. Then represent action, negation, operations, intentions,
variables.......
These are high level grammars, like learning how to convert word problems into
arithmetic or first order logic. I think anything learned at the level of
higher education is going to require a huge network (beyond what is practical
now), but I think the underlying learning principles are the same.
Oh, I disagree entirely: these are the basic things needed as the
*underpinning* of the grammar. You need action for verbs, negation for
everything, operations for abstraction, etc. etc.
It is just not procuctive to focus on the computaional complexity issues
at this stage: gotta get a lot of mechanisms tried out before we can
even begin to talk about such stuff (and, as I say, I don't believe we
will really care even then).
I think it is important to estimate these things. The analogy is that it is
useful to know that certain problems are hard or impossible regardless of any
proposed solution, like traveling salesman or recursive data compression. If
we can estimate the complexity of language modeling in a similar way, I see no
reason not to.
But you cannot do any estimates like that until the algorithm itself is
clear: there are no *algorithms* available for grammar learning,
nothing that describes the class of all possible algorithms that do
grammar learning. Complexity calculations mean nothing for handwaving
suggestions about (eg) representing numbers of neurons: they strictly
only apply to situations in which you can point to an algorithm and ask
how it behaves.
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=4007604&user_secret=8eb45b07
