--- Richard Loosemore <[EMAIL PROTECTED]> wrote: > Matt Mahoney wrote: > > --- "John G. Rose" <[EMAIL PROTECTED]> wrote: > >> Is there really a bit per synapse? Is representing a synapse with a bit > an > >> accurate enough simulation? One synapse is a very complicated system. > > > > A typical neural network simulation uses several bits per synapse. A > Hopfield > > net implementation of an associative memory stores 0.15 bits per synapse. > But > > cognitive models suggest the human brain stores .000001 bits per synapse. > > (There are 10^15 synapses but human long term memory capacity is 10^9 > bits). > > Sorry, I don't buy this at all. This makes profound assumptions about > how information is stored in memory, averagng out the "net" storage and > ignoring the immediate storage capacity. A typical synapse actually > stores a great deal more than a fraction of a bit, as far as we can > tell, but this information is stored in such a way that the system as a > whole can actually use the information in a meaningful way. > > In that context, quoting "0.000001 bits per synapse" is a completely > meaningless statement.
I was referring to Landauer's estimate of long term memory learning rate of about 2 bits per second. http://www.merkle.com/humanMemory.html This does not include procedural memory, things like visual perception and knowing how to walk. So 10^-6 bits is low. But how do we measure such things? > Also, "typical" neural network simulations use more than a few bits as > well. When I did a number of backprop NN studies in the early 90s, my > networks had to use floating point numbers because the behavior of the > net deteriorated badly if the numerical precision was reduced. This was > especially important on long training runs or large datasets. That's what I meant by "few". In the PAQ8 compressors I have to use at least 16 bits. -- Matt Mahoney, [EMAIL PROTECTED] ------------------------------------------- singularity Archives: http://www.listbox.com/member/archive/11983/=now RSS Feed: http://www.listbox.com/member/archive/rss/11983/ Modify Your Subscription: http://www.listbox.com/member/?member_id=4007604&id_secret=96140713-a54b2b Powered by Listbox: http://www.listbox.com
