Matt Mahoney wrote:
--- Richard Loosemore <[EMAIL PROTECTED]> wrote:
Matt Mahoney wrote:
I doubt you could model sentence structure usefully with a neural network
capable of only a 200 word vocabulary. By the time children learn to use
complete sentences they already know thousands of words after exposure to
hundreds of megabytes of language. The problem seems to be about O(n^2).
As
you double the training set size, you also need to double the number of
connections to represent what you learned.
-- Matt Mahoney, [EMAIL PROTECTED]
The problem does not need to be O(n^2).
And remember: I used a 200 word vocabulary in a program I wrote 16
years ago, on a machine with only one thousandth of today's power.
And besides, solving the problem of understanding sentences could easily
be done in principle with even a vocabulary as small as 200 words.
Richard Loosemore.
What did your simulation actually accomplish? What were the results? What do
you think you could achieve on a modern computer?
Oh, I hope there's no misunderstanding: I did not build networks to do
any kind of syntactic learning, they just learned relationships between
phonemic representations and graphemes. (They learned to spell). What
they showed was something already known for the learning of
pronunciation: that the system first learns spellings by rote, then
increases its level of accuracy and at the same time starts to pick up
regularities in the mapping. Then it starts to "regularize" the
spellings. For example: having learned to spell "height" correctly in
the early stages, it would then start to spell it incorrectly as "hite"
because it had learned many other words in which the spelling of the
phoneme sequence in "height" would involve "-ite". Then in the last
stages it would learn the correct spellings again.
Simple results (there were a few more tentative ideas, but not much).
My goal has always been to understand exactly what to put into those
kinds of mechanisms to get semantics and syntax to be learned in the
same powerful way. That is a task that has occupied me for 20 years
(since before those spelling networks, in fact).
What could I do today? Ask me in a year or so. My guess, given all the
experience I have had writing systems and thinking about the issues
involved, is that many of the puzzles involved in building systems that
learn in a powerful way are actually *much* easier than people think
they are, but to solve those puzzles we need to shake off a certain way
of thinking. The solutions are just around the corner, but ain't nobody
gonna see them if they won't actually believe me enough to go around the
corner and look.
I do my best to shake people out of that way of thinking, but I feel
like Stanislaw Lem's Ijon Tichy character, in the Seventh Voyage (in the
book 'Star Diaries') where he tries to shake past versions of himself
awake, but meets the stubborn resistance of people who really want to be
left alone to do some more sleeping.
That's how it feels to me, from my POV.
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=4007604&user_secret=8eb45b07
