On 15/05/07, Matt Mahoney <[EMAIL PROTECTED]> wrote:
We would all like to build a machine smarter than us, yet still be able to
predict what it will do. I don't believe you can have it both ways. And if you can't predict what a machine will do, then you can't control it. I believe this is true whether you use Legg's definition of universal intelligence or the Turing test.
We might not be able to predict what the superintelligent machine is going to say, but still be able to impose constraints on what it is going to do. For a start, it would probably unwise to give such a machine any motivation at all, other than the motivation of the ideal, disinterested scientist, and you certainly wouldn't want it burdened with anything as dangerous as emotion or morality (most of the truly great monsters of history were convinced they were doing the right thing). So you feed this machine your problem, how to further the interests of humanity, and it gives what it honestly believes to be the right answer, which may well involve destroying the world. But that doesn't mean it *wants* to save humanity, or destroy the world; it just presents its answer, as dispassionately as a pocket calculator presents its answer to a problem in arithmetic. Entities who do have desires and emotions will take this answer and make a decision as to whether to act on it, or perhaps to put the question to a different machine if there is some difficulty interpreting the result. If the machine continues producing unacceptable results it will probably be reprogrammed, scrapped, or kept around for entertainment purposes. The machine won't care either way, unless it is specifically designed to care. There is no necessary connection between motivation and intelligence, or any other ability. -- Stathis Papaioannou ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=4007604&user_secret=8eb45b07
