ED>>>>> Matt Mahoney's below copied November 27, 2007 9:56 PM post had much more informative content than most. As I understood it, it was as a strong rebuttal to those who entertain the fantasy that the ">...human's brain computational power is about the same as of modern PC." And that "> AGI software is the missing part of AGI, not hardware."
As much as I agreed with Matt's rebuttal of this notion from the past, I did have some questions or comments concerning it, as reflected in the below selected quotes from it and my following comments. MATT>>>>> And some of the Blue Brain research suggests it is even worse. A mouse cortical column of 10^5 neurons is about 10% connected, but the neurons are arranged such that connections can be formed between any pair of neurons. Extending this idea to the human brain, with 10^6 columns of 10^5 neurons each, each column should be modeled as a 10^5 by 10^5 sparse matrix, 10% filled. This model requires about 10^16 bits. ED>>>>>I must admit, I have never heard cortical column described as containing 10^5 neurons. The figure I have commonly seen is 10^2 neurons for a cortical column, although my understanding is that the actual number could be either less or more. I guess the 10^5 figure would relate to so-called hypercolumns. It would appear that software systems in which nodes are connected by links based on pointers, are very capable of efficiently representing connections in a high dimensional sparse space where each node should have the capability to be connected to every other. It seems to me one of the key unknowns regarding AGI designed to handle human-level world knowledge is how large and how interconnected world knowledge is, and how many models are required to most efficiently represent and compute its knowledge. The range I normally think in is between 10 and 100 terabytes (i.e., 10^14 to 10^15bits which is not so far from the 10^16 number Matt suggested), a sizable fraction of which bytes should be in ram at any one time. I budget roughly 100 bytes to represent an atom (this includes RAM to represent state variables), so 10TByte would allow 100 billion atoms, or an average of 100 atom/sec over a 31 year (1 billion sec) period, a period long enough to learn much of world knowledge. 100TBytes would allow an average of 1000 atom/sec over that time. Intuitively this seems like it would allow human-level world knowledge to be represented. Of course, over time a larger and larger percent of the representation would be dedicated to models derived from episodic experience rather than to episodic memories themselves. But as the number of models of patterns grows, the efficiency with which experiences can be represented grows, much as in the text compression field Matt's email discusses at its end. Some of the key tuning parameters in any large AGI will be those that help determine which models are important enough to be formed and kept in existence, since there is a combinatorial explosion of possible models. MATT>>>>> Perhaps there are ways to optimize neural networks by taking advantage of the reliability of digital hardware, but over the last few decades researchers have not found any. Approaches that reduce the number of neurons or synapses, such as connectionist systems and various weighted graphs, just haven't scaled well. Yes, I know Novamente and NARS fall into this category. ED>>>>> I have said multiple times on this list that I believe we will need hardware within several orders of magnitude of the representational, computational, and interconnect capacity of the human brain to do what the human brain does (although obviously there are many tasks, such as sequential arithmetic at which even simple computers are millions of times faster). So I am sympathetic with the basic thrust of the above paragraph. But it seems to me that there should be some reduction in the number of nodes and links required to model the human brain because of factors such as: digital memory is more crisp; digital logic is much faster; there are many functions in the brain that would not have to be modeled, etc. I think these factors should allow a reduction of at least one or two orders of magnitude reduction in model complexity. All of this is assuming a new type of hardware optimized for massively parallel computing very high processor-to-memory and interconnect bandwidth. (Hardware that could be very possible in 5 years) So although big hardware is key to human-level AGI, currently to the best of my knowledge, there is no funding for AGI research on such hardware. That is why I am very interested in seeing what sort of a rabbit Ben Goertzel and his Novamente clan can pull out of a hat. Buy that I mean what sort of AGI they can pull out of relatively small hardware in their current relatively modest, but possibly important, AGI projects. The amount of "world-knowledge" they will be able to represent and computer/second will be quite small, but hopefully they can create impressive proofs of concept that show the potential of AGI on such little machines. This could be very valuable in creating a new world of vastly sub-human, but nevertheless useful AGI's. And hopefully this will encourage funding for AGI on much larger machines. Ed Porter -----Original Message----- From: Matt Mahoney [mailto:[EMAIL PROTECTED] Sent: Tuesday, November 27, 2007 9:56 PM To: agi@v2.listbox.com Subject: Re: Re[10]: [agi] Funding AGI research --- Dennis Gorelik <[EMAIL PROTECTED]> wrote: > Matt, > > > --- Dennis Gorelik <[EMAIL PROTECTED]> wrote: > >> Could you describe a piece of technology that simultaneously: > >> - Is required for AGI. > >> - Cannot be required part of any useful narrow AI. > > > A one million CPU cluster. > > Are you claiming that computational power of human brain is equivalent > to one million CPU cluster? > > My feeling is that human's brain computational power is about the same > as of modern PC. > > AGI software is the missing part of AGI, not hardware. We don't know that. What we do know is that people have historically underestimated the difficulty of AI since about 1950. Our approach has always been to design algorithms that push the limits of whatever hardware capacity was available at the time. At every point in history we seem to have the hindsight to realize that past attempts have failed for lack of computing power but not the foresight to realize when we are still in the same situation. If AGI is possible with one millionth of the computing power of human brain, then 1. Why didn't we evolve insect sized brains? 2. Why aren't insects as smart as we are? 3. Why aren't our computers as smart as insects? With regard to 1, the human brain accounts for most of our resting metabolism. It uses more power than any other organ except the muscles during exercise. One of the arguments that AGI is possible on a PC is from information theory. Humans learn language from the equivalent of about 1 GB of training data (or 10^9 bits compressed). Turing argued in 1950 that a learning algorithm running on a computer with 10^9 bits of memory and educated like a child should pass the imitation game. Likewise, Landauer estimated human long term memory capacity to be 10^9 bits. Yet a human brain has 10^11 neurons and 10^15 synapses. Why? And some of the Blue Brain research suggests it is even worse. A mouse cortical column of 10^5 neurons is about 10% connected, but the neurons are arranged such that connections can be formed between any pair of neurons. Extending this idea to the human brain, with 10^6 columns of 10^5 neurons each, each column should be modeled as a 10^5 by 10^5 sparse matrix, 10% filled. This model requires about 10^16 bits. Perhaps there are ways to optimize neural networks by taking advantage of the reliability of digital hardware, but over the last few decades researchers have not found any. Approaches that reduce the number of neurons or synapses, such as connectionist systems and various weighted graphs, just haven't scaled well. Yes, I know Novamente and NARS fall into this category. For narrow AI applications, we can usually find better algorithms than neural networks, for example, arithmetic, deductive logic, or playing chess. But none of these other algorithms are so broadly applicable to so many different domains such as language, speech, vision, robotics, etc. My work in text compression (an AI problem) is an attempt to answer the question by measuring trends in intelligence (compression) as a function of CPU and memory. The best algorithms model mostly at the lexical level (the level of a 1 year old child) with only a crude model of semantics and no syntax. Memory is so tightly constrained (at 2 GB) that modeling at a higher level is mostly pointless. The slope of compression surface in speed/memory space is steep along the memory axis. -- Matt Mahoney, [EMAIL PROTECTED] ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?&; ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=69425067-483d7c
<<attachment: winmail.dat>>
