On 10/1/07, Richard Loosemore <[EMAIL PROTECTED]> wrote: > Jef Allbright wrote: > > On 9/30/07, Richard Loosemore <[EMAIL PROTECTED]> wrote: > >>> The motivational system of some types of AI (the types you would > >>> classify as tainted by complexity) can be made so reliable that > >>> the likelihood of them becoming unfriendly would be similar to > >>> the likelihood of the molecules of an Ideal Gas suddenly deciding > >>> to split into two groups and head for opposite ends of their > >>> container. > > > > Richard, in the context of the foregoing, I'd like to know your > > thoughts on the effective differences between a powerful entity being > > "nice" like a friend versus doing the "right" thing in the bigger > > picture, much like a parent doing what they perceive best despite > > screams of pain and protest from their children. > > I need to begin by making a general comment. When trying to answer > questions like this, it is very important that we don't accidentally > assume that the AGI would be dumb. I think that misunderstandings often > arise because of that.
My baseline assumption was that a "powerful entity" is necessarily not dumb. > For example, in the scenario/situation you suggest above, there is a > hint of the idea that an AGI might feel the need to "do the right thing > in the bigger picture". But when we conceive of that "bigger picture" > in our minds, we have to be careful that we don't simply assume that the > AGI would have a *particular* bigger picture in mind, when in fact we > can see that an intelligent system would have to be rather stupid or > narrow minded if it did not see the flaws in that bigger picture. My baseline assumption was that a "powerful entity" is necessarily not narrow-minded. > So, to address your question directly: would it be nice like a friend, > or would it do what was "best" for us whether it hurt us or not? > > let's talk about the human case. > > When human parents try to raise their children, they have anxieties > about things that will happen to their children when they grow up, if > the kids make certain mistakes. > > We need a concrete example. I have a son who is learning to play cello, > and enjoys it very much, but is reluctant to pick it up spontaneously > and practice. He wants to have the pleasure and sense of achievement of > playing well, but would rather that this fun come without expense > (i.e. without cutting into valuable play time). So, of course, Dad > tries to make sure that he gets enough practice every day, even > though this leads to grumpiness, go-slows, big arguments, etc etc. > > Now you have to ask very carefully, why do I consider it to be in his > big-picture interest to push him, rather than just let him practice > whenever he wants (i.e. for a few minutes a week, just before lesson)? > > Well, the answer to that is that I have a whole bunch of agenda items > that drive me to do it, including wanting to make sure that he is going > to get along in life as well as any other kid, and wanting him to have > something that I missed. But these are all generated by specifics of > the way I, as a human being and as a particular individual, am built. > > So here is the first conclusion: when we imagine an AGI-Nanny, I think > we tend to assume that it WILL have the motivations that drive us when > we act as parents. No intrinsic reason to believe that assumption. Of course I explicitly presented the case of a large asymmetry between the "nanny" and *any* human assessors. > There are two things we are doing wrong when we imagine this situation. > One is to think that the AGI will experience the same motivations as us, > and (a quite separate assumption) we think that the motivations it > brings to the table could be completely out of our control, or become > completely alien to us. I brought neither of these assumptions to the table. > I am sure you will agree, on reflection, that we should not make mistake > number 1: we should not automatically assume the same motivations that > would drive us to be nannies. > > But the second mistake is more subtle. There is no good reason to > suppose that it will come to its own decisions about whether to nanny, > or what kind of nannying to do. WE get to choose, when we design it, > what kind of motivations lie behind any nannying behavior. Here I think it's apparent that you don't grasp the pragmatics of meaning within an expanding context. Motivations are meaningful only within context. As the context evolves, in a necessarily unpredictable way, it is increasingly unlikely that meaning will be preserved. This is separate from your valid point about the robustness of a complex system. > The main thrust of my argument about motivational systems that are > stable, is that we do have precise control over the > motivations. > > Which shfts the question to this: given that we can control the > motivations that lie behind the nannying, can we design them in such a > way that they satisfy our requirements and are not overbearing? > > I think we can easily do that. > > First, consider that most of the things that drive us to do things that > are for our childrens' own good do not apply. There is no urgency to > get things right before we grow up. There are no hangups (like me > wanting to play a musical instrument when I was young). There is no need > to worry about other kids competing with us and making us feel bad as we > grow up. > > The more you think about it, the more that all the drivers that would > cause a system (human or AGI) to impose on its protectees would simply > not be there. > > Second, the AGI would be driven by general concerns about empathy with > the needs of the human race, so it would be motivated by something far > more subtle and flexible than "Make sure these kids grow up right." It > would get its kicks from the general satisfaction of needs and wants on > a case by case basis. If someone says "Let me live free and take my own > risks, even if that means I might accidentally kill myself", the AGI > would not be crudely and stupidly programmed to override that and say to > itself "Stupid human: I'll save it when it is in mortal peril, and it > will thank me afterwards", it will say "These creatures are grown ups: > if that is what this person wants, so be it." Actually, when you think this through you find that any attempt to optimize a particular outcome ultimately incurs an increased cost in terms of local entropy . The best one can do (with anything less than infinite computing power) is act to implement best-known principles, not preferred outcomes. What you seem to fail to realize, is that the greater understanding of principles, under conditions of significantly asymmetrical intelligence, is bound to displease the lesser entit(y|ies), who will in all sincerity see its actions as "bad" (against their perceived interests.) This is why I emphasize the merits of (lower-case) friendly machine intelligence assisting humans only at the upcoming phase of our development. > if we assume that the AGI would contradict the wishes of a human and do > something that it is in some abstract sense "good" for it, even though > "good" is painful, then we are implicitly assuming a stupid AGI that > cannot understand the most general, subtle sense of empathy. This is a blatant contradiction which, I'm afraid, may be effectively unresolvable over this low-bandwidth medium. Best to you, - Jef ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=48552188-082061
