Thanks to Peter for starting this discussion and to Ben for following up. This seems
to me a more constructive way to talk about Friendly AI.
Now it's my turn to comment on the 8 Guidelines, according to my NARS design (for
people who have no idea what I'm talking about, see
http://www.cogsci.indiana.edu/farg/peiwang/papers.html).
1. Friendliness-topped goal system
Though in NARS initial goals can be specified by the designer, they are not the
supergoals described. Due to insufficient knowledge and resources, there is no way
to guarantee the consistency (or even the relevance) of the subgoals and initial
goals. Therefore, to have friendliness as the only initial goal does not make the
system friendly.
2. Cleanly causal goal system
Since all causal knowledge is empirical, and therefore maybe biased, the above problem
is also here. Though NARS indeed has a “cleanly causal goal system”, where all
goals are derived from the initial goals according to the current causal knowledge,
this mechanism cannot guarantee friendliness.
3. Probabilistic supergoal content
In NARS all empirical knowledge has uncertainty, so this is not an issue for me.
4. Acquisition of Friendliness sources
As an adaptive system, NARS will learn various types of knowledge from human beings. I
don’t see anything special about ethical knowledge on this issue.
5. Causal validity semantics.
Again, this is just causal inference, which can be applied to moral issues, but is not
limited there.
6. Injunctions.
With insufficient knowledge and resources, it is impossible “to prevent possible
negative outcomes”. Of course, the system will try to avoid them as much as
possible.
7. Self-modeling of fallibility
as above.
8. Controlled ascent
I don’t think it is possible to implement an improvements counter. An adaptive
system is changing all the time. I cannot tell when NARS is changing faster than
usual, and if it is, whether this is an alarm signal. Furthermore, whether a change is
an improvement or not is usually a subjective judgment based on feedback over a
certain period. Something may look like an improvement now, but lead to a big trouble
in the future.
In summary, I agree with some of the “Guidelines”, though I think they are
actually about AI, and are neutral to the friendliness issue. Of course, high
intelligence can make a system friendlier, but it can also make the system more
dangeours. To select proper initial goals and to provide proper feedback will reduce
the danger, but cannot eliminate it.
I post this reply just to show that I fully agree that AI can be dangerous, and we
should be ready to deal with the problems when designing an AI system. However, at the
current time, moral discussions contribute little to the real work. I haven’t seen
how the design of NARS should be changed to make it friendlier.
Pei
---Original Message---
From: Ben Goertzel [EMAIL PROTECTED]
Sent: 02/20/03 12:01 PM
To: [EMAIL PROTECTED]
Subject: RE: [agi] Building a safe AI
Peter,
At the end of the page you reference, you list
The Guidelines' eight design recommendations in the light of my theory of
mind/ intelligence:
1. ...
2. ...
...
All of your comments in that section apply to Novamente without
significant
modification. Although in detail your design is of course quite different
from Novamente.
-- Ben
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On
Behalf Of Peter Voss
Sent: Thursday, February 20, 2003 11:48 AM
To: [EMAIL PROTECTED]
Subject: [agi] Building a safe AI
http://www.optimal.org/peter/siai_guidelines.htm
Peter
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On
Behalf Of Ben Goertzel
I would recommend Eliezer's excellent writings on this topic if you
don't
know them, chiefly www.singinst.org/CFAI.html . Also, I have a brief
informal essay on the topic,
www.goertzel.org/dynapsyc/2002/AIMorality.htm ,
although my thoughts on the topic have progressed a fair bit since I
wrote
that. Note that I don't fully agree with Eliezer on this stuff, but I
do
think he's thought about it more thoroughly than anyone else
(including me).
It's a matter of creating an initial condition so that the
trajectory of the
evolving AI system (with a potentially evolving goal system) will have a
very high probability of staying in a favorable region of state space
;-)
---
To unsubscribe, change your address, or temporarily deactivate
your subscription,
please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
---
To unsubscribe, change your address, or temporarily deactivate your
subscription,
please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
---
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
