--- Tom McCabe <[EMAIL PROTECTED]> wrote: > --- Matt Mahoney <[EMAIL PROTECTED]> wrote: > > Personally, I would experiment with > > neural language models that I can't currently > > implement because I lack the > > computing power. > > Could you please describe these models?
Essentially models in which neurons (with time delays) respond to increasingly abstract language concepts: letters, syllables, words, grammatical roles, phrases, and sentence structures. This is not really new. Models like these have been proposed in the 1980's but were never fully implemented due to lack of computing power. These constraints resulted in connectionist systems in which each concept mapped to a single neuron. Such models can't learn well. There is no mechanism for adding to the vocabulary, for instance. I believe you need at least hundreds of neurons per concept, where each neuron may correlate weakly with hundreds of different concepts. Exactly how many, I don't know. That is why I need to experiment. One problem that bothers me is the disconnect between the information theoretic estimates of the size of a language model, about 10^9 bits, and models based on neuroanatomy, perhaps 10^14 bits. Experiments might tell us what's wrong with our neural models. But how to do such experiments? A fully connected network of 10^9 connections trained on 10^9 bits of data would require about 10^18 operations, about a year on a PC. There are optimizations I could do, such as activating only a small fraction of the neurons at one time, but if the model fails, is it because of these optimizations or because you really do need 10^14 connections, or the training data is bad, or something else? -- Matt Mahoney, [EMAIL PROTECTED] ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=4007604&user_secret=8eb45b07
