Jef Allbright wrote:
On 9/30/07, Richard Loosemore <[EMAIL PROTECTED]> wrote:
The motivational system of some types of AI (the types you would
classify as tainted by complexity) can be made so reliable that
the likelihood of them becoming unfriendly would be similar to
the likelihood of the molecules of an Ideal Gas suddenly deciding
to split into two groups and head for opposite ends of their
container.
Richard, in the context of the foregoing, I'd like to know your
thoughts on the effective differences between a powerful entity being
"nice" like a friend versus doing the "right" thing in the bigger
picture, much like a parent doing what they perceive best despite
screams of pain and protest from their children.
I need to begin by making a general comment. When trying to answer
questions like this, it is very important that we don't accidentally
assume that the AGI would be dumb. I think that misunderstandings often
arise because of that.
For example, in the scenario/situation you suggest above, there is a
hint of the idea that an AGI might feel the need to "do the right thing
in the bigger picture". But when we conceive of that "bigger picture"
in our minds, we have to be careful that we don't simply assume that the
AGI would have a *particular* bigger picture in mind, when in fact we
can see that an intelligent system would have to be rather stupid or
narrow minded if it did not see the flaws in that bigger picture.
So, to address your question directly: would it be nice like a friend,
or would it do what was "best" for us whether it hurt us or not?
let's talk about the human case.
When human parents try to raise their children, they have anxieties
about things that will happen to their children when they grow up, if
the kids make certain mistakes.
We need a concrete example. I have a son who is learning to play cello,
and enjoys it very much, but is reluctant to pick it up spontaneously
and practice. He wants to have the pleasure and sense of achievement of
playing well, but would rather that this fun come without expense
(i.e. without cutting into valuable play time). So, of course, Dad
tries to make sure that he gets enough practice every day, even
though this leads to grumpiness, go-slows, big arguments, etc etc.
Now you have to ask very carefully, why do I consider it to be in his
big-picture interest to push him, rather than just let him practice
whenever he wants (i.e. for a few minutes a week, just before lesson)?
Well, the answer to that is that I have a whole bunch of agenda items
that drive me to do it, including wanting to make sure that he is going
to get along in life as well as any other kid, and wanting him to have
something that I missed. But these are all generated by specifics of
the way I, as a human being and as a particular individual, am built.
So here is the first conclusion: when we imagine an AGI-Nanny, I think
we tend to assume that it WILL have the motivations that drive us when
we act as parents. No intrinsic reason to believe that assumption.
There are two things we are doing wrong when we imagine this situation.
One is to think that the AGI will experience the same motivations as us,
and (a quite separate assumption) we think that the motivations it
brings to the table could be completely out of our control, or become
completely alien to us.
I am sure you will agree, on reflection, that we should not make mistake
number 1: we should not automatically assume the same motivations that
would drive us to be nannies.
But the second mistake is more subtle. There is no good reason to
suppose that it will come to its own decisions about whether to nanny,
or what kind of nannying to do. WE get to choose, when we design it,
what kind of motivations lie behind any nannying behavior.
The main thrust of my argument about motivational systems that are
stable, is that we do have precise control over the
motivations.
Which shfts the question to this: given that we can control the
motivations that lie behind the nannying, can we design them in such a
way that they satisfy our requirements and are not overbearing?
I think we can easily do that.
First, consider that most of the things that drive us to do things that
are for our childrens' own good do not apply. There is no urgency to
get things right before we grow up. There are no hangups (like me
wanting to play a musical instrument when I was young). There is no need
to worry about other kids competing with us and making us feel bad as we
grow up.
The more you think about it, the more that all the drivers that would
cause a system (human or AGI) to impose on its protectees would simply
not be there.
Second, the AGI would be driven by general concerns about empathy with
the needs of the human race, so it would be motivated by something far
more subtle and flexible than "Make sure these kids grow up right." It
would get its kicks from the general satisfaction of needs and wants on
a case by case basis. If someone says "Let me live free and take my own
risks, even if that means I might accidentally kill myself", the AGI
would not be crudely and stupidly programmed to override that and say to
itself "Stupid human: I'll save it when it is in mortal peril, and it
will thank me afterwards", it will say "These creatures are grown ups:
if that is what this person wants, so be it."
if we assume that the AGI would contradict the wishes of a human and do
something that it is in some abstract sense "good" for it, even though
"good" is painful, then we are implicitly assuming a stupid AGI that
cannot understand the most general, subtle sense of empathy.
In conclusion:
1) It is wrong to assume that an AGI would have the same (slightly
overbearing) motivations that we do, and
2) It is wrong to assume that the design of the AGI's motivational
system is not for use to choose.
3) The system would actually be driven by a very smart, flexible, subtle
sense of 'empathy' and would not force us to do painful things that were
"good" for us, for the simple reason that this kind of nannying would be
the antithesis of really intelligent empathy.
If you want, give specific cases and we will try to see how it would behave.
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=8660244&id_secret=48512123-c8ccdd
