Benjamin Goertzel wrote:
On 10/26/07, Stefan Pernar < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
>
> My one sentence summary of CEV is: "What would a better
me/humanity want?"
> Is that in line with your understanding?
No...
I'm not sure I fully grok Eliezer's intentions/ideas, but I will
summarize here the
current idea I have of CEV.. which is quite different than yours, and not as
oversimplified.
My understanding is that it's more like this (taking some liberties)
X0 = me
Y0 = what X0 thinks is good for the world
X1 = what X0 wants to be
Y1 = what X1 would think is good for the world
X2 = what X1 would want to be
Y2 = what X2 would think is good for the world.
...
The only circularity here is in the sense of convergence-to-a-fixed-point,
i.e. the series may tend gradually (or quickly) toward
X = what X wants to be
Y = what X thinks is good for the world
In fact some people may already be exactly at this fixed point, i.e.
they may be exactly what they want to be already... For them,
we'd have X1=X0 and Y1=Y0
You can sensibly argue that CEV is poorly-defined, or that it's
not likely to converge to anything given a normal human as an initial
condition, or that it is likely to converge to totally different things
for different sorts of people, giving no clear message...
But I don't think you can argue it's circular...
Stefan can correct me if I am wrong here, but I think that both yourself
and Aleksei have misunderstood the sense in which he is pointing to a
circularity.
If you build an AGI, and it sets out to discover the convergent desires
(the CEV) of all humanity, it will be doing this because it has the goal
of using this CEV as the basis for the "friendly" motivations that will
henceforth guide it.
But WHY would it be collecting the CEV of humanity in the first phase of
the operation? What would motivate it to do such a thing? What exactly
is it in the AGI's design that makes it feel compelled to be friendly
enough toward humanity that it would set out to assess the CEV of humanity?
The answer is: its initial feelings of friendliness toward humanity
would have to be the motivation that drove it to find out the CEV.
The goal state of its motivation system is assumed in the initial state
of its motivation system. Hence: circular.
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=4007604&id_secret=57834496-9ed753
