Benjamin Goertzel wrote:
Loosemore wrote:
But WHY would it be collecting the CEV of humanity in the first
phase of
the operation? What would motivate it to do such a thing? What exactly
is it in the AGI's design that makes it feel compelled to be friendly
enough toward humanity that it would set out to assess the CEV of
humanity?
The answer is: its initial feelings of friendliness toward humanity
would have to be the motivation that drove it to find out the CEV.
The goal state of its motivation system is assumed in the initial state
of its motivation system. Hence: circular.
No. Eliezer specifically indicated that the CEV-evaluator machine is not
supposed to be a sentient being with a human-like motivational system.
It's supposed to be a Very Powerful Optimization Process which is programmed
and configured specifically to perform a certain kind of extrapolation.
This
extrapolation process is a goal in itself, independently of whatever
results it
may or may not produce...
The feasibility of creating such a VPOP is a whole other question, but you
seem to be misunderstanding the idea, not just questioning the feasibility.
No misunderstanding, methinks.
The thing that does the assessment of the CEV is an AGI. Nobody would
be so dumb as to think that the collective wishes of humanity could be
evaluated, reconciled and then built into the motivation system of
another AGI by something that was less than fully intelligent.
Or could they...? Ah, yes: Yudkowsky thinks that he can rename that
initial machine, calling it a "VPOP" instead of an "AGI", with the
built-in assumption that a VPOP is both smart enough to carry out a task
that requires incredible intelligence, and yet it also has a safe and
trustworthy motivation system.
So a VPOP is defined to be a safe AGI. And its purpose is to solve the
problem of building the first safe AGI...
And that would not be circular, that would just be silly old me
misunderstanding the greatness of the Wise One.
[Damn, busted back down to Chapter One of Teach Yourself Diplomacy again
>-( ].
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=4007604&id_secret=57871461-3d7060
