Loosemore wrote: But WHY would it be collecting the CEV of humanity in the first phase of > the operation? What would motivate it to do such a thing? What exactly > is it in the AGI's design that makes it feel compelled to be friendly > enough toward humanity that it would set out to assess the CEV of > humanity? > > The answer is: its initial feelings of friendliness toward humanity > would have to be the motivation that drove it to find out the CEV. > > The goal state of its motivation system is assumed in the initial state > of its motivation system. Hence: circular.
No. Eliezer specifically indicated that the CEV-evaluator machine is not supposed to be a sentient being with a human-like motivational system. It's supposed to be a Very Powerful Optimization Process which is programmed and configured specifically to perform a certain kind of extrapolation. This extrapolation process is a goal in itself, independently of whatever results it may or may not produce... The feasibility of creating such a VPOP is a whole other question, but you seem to be misunderstanding the idea, not just questioning the feasibility. -- Ben G ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=4007604&id_secret=57844722-27f52c
