[agi] Screwing up Friendliness

[Eliezer S. Yudkowsky]

Roger "localroger" Williams has published online at kuro5hin his novel "The Metamorphosis of Prime Intellect", the first member of the new "Seed AI Programmer Screws Up Friendliness" genre of science fiction. I would like to recommend it to, at the very least, Ben Goertzel, Bill Hibbard, and Kevin Copple, since the novel not only illustrates vividly some of the problems with "hard-wiring", but also some of the problems that "experiential learning" as an answer to hard-wiring does *not* solve. One of the Big Lessons in AI is that just because you've solved one piece of the problem, doesn't mean you can stop there.

http://www.kuro5hin.org/prime-intellect

Roger Williams has emphasized for the record that this story is meant to emphasize the *importance* of thinking through the Singularity, not as a prediction of dystopia; also that the story was written in 1994, and set in 1988; hence, if the story fails to make any mention of more recent thinking on the Singularity, it may be excused.

--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence

-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]