If such neural systems can actually spit out sensible analyses of natural language, it would obviously be a huge discovery and could probably be sold to a good number of people as a commercial product. So why aren't more people investing in this, if you've already got working software that just needs a suitable supercomputer?
- Tom --- Matt Mahoney <[EMAIL PROTECTED]> wrote: > --- Tom McCabe <[EMAIL PROTECTED]> wrote: > > --- Matt Mahoney <[EMAIL PROTECTED]> wrote: > > > Personally, I would experiment with > > > neural language models that I can't currently > > > implement because I lack the > > > computing power. > > > > Could you please describe these models? > > Essentially models in which neurons (with time > delays) respond to increasingly > abstract language concepts: letters, syllables, > words, grammatical roles, > phrases, and sentence structures. This is not > really new. Models like these > have been proposed in the 1980's but were never > fully implemented due to lack > of computing power. These constraints resulted in > connectionist systems in > which each concept mapped to a single neuron. Such > models can't learn well. > There is no mechanism for adding to the vocabulary, > for instance. I believe > you need at least hundreds of neurons per concept, > where each neuron may > correlate weakly with hundreds of different > concepts. Exactly how many, I > don't know. That is why I need to experiment. > > One problem that bothers me is the disconnect > between the information > theoretic estimates of the size of a language model, > about 10^9 bits, and > models based on neuroanatomy, perhaps 10^14 bits. > Experiments might tell us > what's wrong with our neural models. But how to do > such experiments? A fully > connected network of 10^9 connections trained on > 10^9 bits of data would > require about 10^18 operations, about a year on a > PC. There are optimizations > I could do, such as activating only a small fraction > of the neurons at one > time, but if the model fails, is it because of these > optimizations or because > you really do need 10^14 connections, or the > training data is bad, or > something else? > > > -- Matt Mahoney, [EMAIL PROTECTED] > > ----- > This list is sponsored by AGIRI: > http://www.agiri.org/email > To unsubscribe or change your options, please go to: > http://v2.listbox.com/member/?&; > ____________________________________________________________________________________Choose the right car based on your needs. Check out Yahoo! Autos new Car Finder tool. http://autos.yahoo.com/carfinder/ ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=4007604&user_secret=8eb45b07
