Can humans keep superintelligences under control -- can superintelligence-augmented humans compete
Richard Loosemore (RL>>>>) wrote the following on Fri 11/2/2007 11:15 AM, in response to a post by Matt Mahoney. My comments are preceded by ED>>>> RL>>>> This is the worst possible summary of the situation, because instead of dealing with each issue as if there were many possibilities, it pretends that there is only one possible outcome to each issue. In this respect it is as bad as (or worse than) all the science fiction nonsense that has distorted AI since before AI even existed. ED>>>> You above statement is quite a put down. Is it justified? Yes Matt doesnt describe all the multiple possibilities, but are the possibilities he describes really worse than science fiction nonsense? I personally think Matt has raises some real and important issues. We have had go-rounds on this list before about similar subjects, such as under the thread Religion-free technical content. When we did a significant number of the people on this list seemed to agree with the notion that superhuman-level AGI poses a treat to mankind. I think even Eliezer Yudkowsky, who has spend a lot of time on this issue, does not yet consider it solved. That is one of the reasons so many of us believe in the need for some sort of transhumanist transformation so that our descendants (whatever they may be) have a chance to continue surviving in the presence of such superintelligences. RL>>>> Example 1: "...humans cannot predict -- and therefore cannot control -- machines that are vastly smarter." According to some interpretations of how AI systems will be built, this is simply not true at all. If AI systems are built with motivation systems that are stable, then we could predict that they will remain synchronized with the goals of the human race until the end of history. This does not mean that we could "predict" them in the sense of knowing everything they would say and do before they do it, but it would mean that we could know what their goals abd values were - and this would be the the only important sense of the word "predict". ED>>>> I thought it is far from clear one can create motivational systems that are stable in an intelligent, learning, human- or superhuman-level AGI in a complex changing world. Yes, I think you can make a Roomba have a stable motivation system and limited AGIs with them, but that does not mean you could make a superhuman-level AGI -- learning and acting with a considerable freedom in our complex world -- have one. First it is difficult to define goals in ways that cover all the situations in which they might have to be applied. Situations may come up when it is not at all clear what should be done to pursue the goal. This is particularly true when it comes to goals such as be friendly to people. Second, most systems of any complexity will have conflicting goals or be subject to situations in which application of the same goal to different parts of the reality can conflict. For example, should a human-friendly AGI kill a terrorist before he blows up a suicide bomb in a crowded market. If it could have, should it have shot down one or both of the airplanes that hit the World Trade Center on 9/11 as they were close to hitting the building. In an over crowded world, should it kill half of humanity to make more room and better lives in a for the billions of remaining human, since those left living would be conscious to appreciate the improved living conditions and those that were killed wouldnt know the difference. Third, an intelligence of any generality and power has to be able to modify its goals, such as by creating subgoals and by interpreting how goals apply in different situations, and these powers conflict with stability. I have read your paper on complexity cited by you, I think, when a somewhat similar discussion arose before. I came away with the feeling your paper did not even come close to answering the above questions. I can understand how through complexity and grounding a goal could be defined in ways that would be much more meaningful than if defined in just words as naked symbols. I also understand how you could hardwire certain emotional responses to certain aspects of reality and make achieving or avoiding such emotional responses top level goals or the top level goal. But human have similar features in their goal structure, and still occasionally act in ways that contradict instinctual goals and such emotionally built in biases. And it is not clear to what extent such hardware encoded biases can deal with all of the issues of possible conflicts and different possible interpretations that are bound to arise in a complex world. The world is likely to be more complex than any grounding and/or biasing system you are going to put into these machines too insure they stay friendly to people. So that is why it is essential for humans to stay in the loop during the transhuman transformation, for humans to use intelligence augmentation, and for humans to keep a watch on the machines. But as is discussed in the last portion of this email, it is not clear how much this is going to buy the human brain. RL>>>> Example 2: "This really is a fundamental problem, proved in a more formal sense by Shane Legg (http://www.vetta.org/documents/IDSIA-12-06-1.pdf). This paper "proves" nothing whatever about the issue! Example 3: "Recursive self improvement is a probabilistic, evolutionary process that favors rapid reproduction and acquisition of computing resources (aka intelligence), regardless of its initial goals." This is a statement about the goal system of an AGI, but it is extraordinarily presumptious. I can think of many, many types of non-goal-stack motivational systems for which this statement is a complete falsehood. I have described some of those systems on this list before, but this paragraph simply pretends that all such motivational systems just do not exist. ED>>>> Forgive me for my ignorance. What sort of non-goal-stack motivational system is applicable for allowing an AGI to learn and act in complex ways in a complex environment for some purpose or use that would cause us humans to want to build them. Rather than cite a book or a long paper, please give at least a summary in several sentences of what you are referring too. And if you are referred to the complexity paper I discussed above, please point out what I missed when reading it if you think it actually answers the type of problems I discussed above. RL>>>> Example 4: "Each successive generation gets smarter, faster, and less dependent on human cooperation." Absolutely not true. If "humans" take advantage of the ability to enhance their own intelligence up to the same level as the AGI systems, the amount of "dependence" between the two groups will stay exactly the same, for the simple reason that there will not be a sensible distinction between the two groups. ED>>>> This is why I envision a need for a trans-humanist transformation, i.e., for humans to increasingly become more machine (hopefully not too fast), so humans can try to keep up with machines. But there are real issues about the extent to which any intelligence which has the human brain at its top level of control can compete with machines that conceivably could have a top level decision process with hundreds or millions of times the bandwidth. There are also questions of how much bandwidth of machine intelligence can be pumped into, or shared, with a human consciousness and/or subconscious, i.e., how much of the superintelligence we humans could be conscious of and/or effectively use in our subconscious. So if the human brain is not at the top level of decision making, we are no longer in control. And if our consciousnesses are not capable of appreciating more than a small part of what the superintelligence we are part of is doing, we wont even be aware of exactly what most of the bionic entity were are part of is thinking. (Of course, this is somewhat similar to the way the subconscious affects us. It would not be that hard to have a system where the superintelligence only communicates to our brain its consciousness, or portions of its consciousness that its learing indicate will have a certain importance or interest, so that it would be acting somewhat like an extended subconsciousness to us that would occasionally pop ideas up into our subconsciousness or consciousness. This would greatly increase our mental powers, particularly if we had the capability to send information down to it to control it, give it sub-goals, or queries, etc. But this would not solve the limited bottle neck of the human brains top level decision making) So we would not be keeping up with the machines. They would be taking us along for the ride that is, for as long as they desired to continue doing so. OF COURSE IT IS AT LEAST CONCEIVABLE THAT WAYS COULD FOUND FOR HUMAN AND VASTLY SUPERHUMAN-MACHINE CONSCIOUSNESS TO BE LARGELY MERGED AND SHARED. I DONT KNOW OF ANY, BUT I WOULD BE INTERESTING IN ANY FRACTIONALLY SOLID IDEAS ABOUT THIS FROM READERS. I currently tend to think of consciousness as massive spreading activation in the human brain from certain sets of patterns (those in the mind theaters spotlight) to parts much of the subconscious the mind theaters audience). In this mind theater the audience is interactive. Different audience member had different thinks in their heads and respond to different activations in different ways. Certain activations might cause one or more audience members to shout out. I think of consciousness and subconsciousness in an AGI in a similar manner, but I do not know how much and in exactly which ways being inside such a machine consciousness would be like being inside my own. It would have self awareness and grounding for its qualia, but I dont know these things would seem like on the inside. Some brain scientist (I dont know what percent) think consciousness might be the result of waves in the Reticular Thalamic Nuclei (like those when a rock hits smooth water) that send a rippling roughly circular activation or dis-inhibition pattern throughout much of the cortex (like having the microphone in stadium, to extend the mind theater concept.) Some suggest that the interlaminar Nuclei of the Thalamus effect consciousness by selecting, probably under the influence of the prefrontal cortex and the basil ganglia, the spiking pattern of a first set of cortical columns representing one or more activations, and cause the cortico-thalamic feedback loop to tune in on cortical columns having spiking patterns with similar signal components, indicating they receive activation from or give activation to cortical columns of the first set and/or causing other cortical columns to be preferentially responsive to inputs having signal components corresponding to output by the first set. In any case, other than having certain number of electrical links between nodes and links in a superintelligence and neurons the brain, it is not clears how the two could meaningfully share their consciousnesses, and it is not clear what the bandwidth of such links could be, how much bandwidth the human brain is capable of making sense of, and how much of the human brain should be given over to such links. The questions is, how much better than a good video monitor and speaker system could such links be. Presumably they could communicate semantic knowledge much faster, but how much, I havent a clue. (The improvement in bandwidth could be much greater in the reverse direction, from the brain out. I WOULD BE INTERESTING IN OTHER PEOPLES THOUGHTS ON THIS ISSUE, BECAUSE IT SEEMS TO BE AN IMPORTANT ONE IN DETERMINING HOW IMPORTANT A ROLE HUMAN CONSCIOUSNESS CAN CONTINUE TO PLAY IN THE SUPERINTELLIGENCE FUTURE. Ed Porter -----Original Message----- From: Richard Loosemore [mailto:[EMAIL PROTECTED] Sent: Friday, November 02, 2007 11:15 AM To: agi@v2.listbox.com Subject: Re: [agi] popularizing & injecting sense of urgenc Matt Mahoney wrote: > AGI does not need promoting. AGI could potentially replace all human > labor, currently valued at US $66 trillion per year worldwide. Google > has gone from nothing to the fifth biggest company in the U.S. in 10 > years by solving just a little bit of of the AI problem better than > its competitors. > > We should be more concerned about the risks of AGI. When humans can > make machines smarter than themselves, then so can those machines. > The result will be an intelligence explosion. > http://mindstalk.net/vinge/vinge-sing.html > > The problem is that humans cannot predict -- and therefore cannot > control -- machines that are vastly smarter. The SIAI ( > http://www.singinst.org/ ) has tried to address these risks, so far > without success. This really is a fundamental problem, proved in a > more formal sense by Shane Legg ( > http://www.vetta.org/documents/IDSIA-12-06-1.pdf ). Recursive self > improvement is a probabilistic, evolutionary process that favors rapid > reproduction and acquisition of computing resources (aka > intelligence), regardless of its initial goals. Each successive > generation gets smarter, faster, and less dependent on human > cooperation. > > Whether this is good or bad is a philosophical question we can't > answer. It is what it is. The brain is a computer, programed through > evolution with goals that maximize fitness but limit our capacity for > rational introspection. Could your consciousness exist in a machine > with different goals or different memories? Do you become the godlike > intelligence that replaces the human race? This is the worst possible summary of the situation, because instead of dealing with each issue as if there were many possibilities, it pretends that there is only one possible outcome to each issue. In this respect it is as bad as (or worse than) all the science fiction nonsense that has distorted AI since before AI even existed. Example 1: "...humans cannot predict -- and therefore cannot control -- machines that are vastly smarter." According to some interpretations of how AI systems will be built, this is simply not true at all. If AI systems are built with motivation systems that are stable, then we could predict that they will remain synchronized with the goals of the human race until the end of history. This does not mean that we could "predict" them in the sense of knowing everything they would say and do before they do it, but it would mean that we could know what their goals abd values were - and this would be the the only important sense of the word "predict". Example 2: "This really is a fundamental problem, proved in a more formal sense by Shane Legg (http://www.vetta.org/documents/IDSIA-12-06-1.pdf). This paper "proves" nothing whatever about the issue! Example 3: "Recursive self improvement is a probabilistic, evolutionary process that favors rapid reproduction and acquisition of computing resources (aka intelligence), regardless of its initial goals." This is a statement about the goal system of an AGI, but it is extraordinarily presumptious. I can think of many, many types of non-goal-stack motivational systems for which this statement is a complete falsehood. I have described some of those systems on this list before, but this paragraph simply pretends that all such motivational systems just do not exist. Example 4: "Each successive generation gets smarter, faster, and less dependent on human cooperation." Absolutely not true. If "humans" take advantage of the ability to enhance their own intelligence up to the same level as the AGI systems, the amount of "dependence" between the two groups will stay exactly the same, for the simple reason that there will not be a sensible distinction between the two groups. Richard Loosemore ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?&; ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=60542526-6121d1
