Hi all,
Pleasure to meet all of you and looking forward to interesting
discussions. I have been reading Shane's blog post first and will try
to get to Ben's PDF later. Below you will find my take on a response.
Looking forward to your feedback.
Stefan
On 9/10/06, Ben Goertzel <[EMAIL PROTECTED]> wrote:
for just a little later, because my friend Shane Legg posted an
interesting and controversial blog entry
http://www.vetta.org
entitled "Friendly AI is Bunk" which seems to me worthy of discussion.
Shane wrote it after a conversation we (together with my wife
Izabela) had in Genova last week. (As a bit of background, Shane is
no AGI slacker: he is currently a PhD student of Marcus Hutter working
on the theory of near-infinitely-powerful AGI, and in past he worked
with me on the Webmind AI project in the late 1990's, and with Peter
Voss on the A2I2 project.)
Argument 1) Tough love or killing us with kindness
Premise 1.1) Super AGI will be obedient to our self harming wishes.
Premise 1.2) Super AGI will do good without us realizing it.
Argument 2) Deadly butterfly wings.
Premise 2.1) Inability to foresee all effects will lead to
unforeseeable 'bad' acts.
Argument 3) Beautiful tool, terrible owners
Premise 3.1) In the wrong hands a super AGI will be harmful
Argument 4) Evil in disguise.
Premise 4.1) A super AGI will pretend to do good to harm humanity
Argument 5) The provably unprovable
Premise 5.1) Since even a super AGI will not be able to proof its
actions will be friendly it is bound to do something bad eventually.
1.1 response)
I agree that an appropriate mechanism to counteract this scenario lies
in properly defining friendliness. When thinking about this aspect I
am reminded of Maslow's hierarchy of needs with self-transcendence at
its apex. (also see Yudkowsky's view on the apotheotic potential of
the singularity). Personally I would define a friendly AGI as a guide
leading humanity up and beyond Maslow's hierarchy.
1.2 response)
A super AGI will likely have to take measures that proof controversial
even if it can proof that those measures are beneficial. A friendly
AGI could then resort to manipulation and deception to achieve greater
good.
2.1 response)
Yes - and a super AGI should know that and - to stick with the
butterfly effect - should spend some of its resources to optimize
flood responses in case its actions cause a storm due to incomputable
chaotic mechanisms.
3.1 response)
Yes. The only way out I see here is to have an even stronger
'friendly' AGI to counter this threat.
4.1 response)
I see this as a special scenario of 1.1.
5.1 response)
I see this as a special scenario of 2.1.
--
Stefan Pernar
App. 1-6-I, Piao Home
No. 19 Jiang Tai Xi Lu
100016 Beijing
China
Mobil: +86 1391 009 1931
Skype: Stefan.Pernar
-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
