Hi,
Eliezer thought that I was trying to prove that the AI would or could save us. This was not correct, as I pointed out. The objection above is based on this misunderstanding and thus, if I understand the objection correctly, once this misunderstanding is corrected the objection is no longer relevant.
Shane, as far as I understand this is not the problem. In my opinion, the problem lies in the definition of friendliness that you take. In your example you took something on the lines (*) "if AI can avoid that a man is killed, then it should do so" and you show that such friendliness might be unprovable because it depends on the AI's ability to assess what it can do in any given situation and that might not be possible. But on the other hand you can imagine definitions like (**) "AI will not kill any man" which are perhaps too restricting (and might be hard to formulate precisely of course) but do not suffer from the problem you talk about. It seems that this kind of deinitions can really be enforced, perhaps even maintained when AI rewrites itself. Don't you think that there might be reasonable definitions of the (**) type that are worth enforcing, even if these make the AI a bit less powerful? Can you adjust (*) definitions to make them practical, e.g. by putting time bound on how long the AI can think about what it can avoid? Can you prove the existence of an optimal AI that always obeys some kind of (**) friendliness? - lk ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
