--- On Sat, 8/30/08, Vladimir Nesov <[EMAIL PROTECTED]> wrote: > You start with "what is right?" and end with > Friendly AI, you don't > start with "Friendly AI" and close the circular > argument. This doesn't > answer the question, but it defines Friendly AI and thus > "Friendly AI" > (in terms of "right").
In your view, then, the AI never answers the question "What is right?". The question has already been answered in terms of the algorithmic process that determines its subgoals in terms of Friendliness. Therefore it comes down to whether an algorithmic process can provide the basis of action in a way that "what is right?" could always be answered correctly in advance. You say "yes it can, but we don't know how, yet". I say "no it can't, because Friendliness can't be formalized". > All things change through time, which doesn't make them > cease to exist. Friendliness changes through time in the same sense that our morals change through time. I didn't say it ceases to exist. The implication here is that Friendliness is impossible to specify in any static sense - it must be updated as it changes. > Maybe it > complicates the > procedure a little, making the decision procedure > conditional, "if(A) > press 1, else press 1", or maybe it complicates it > much more, but it > doesn't make the challenge ill-defined. Ah, the expert-systems approach to morality. If I have a rule-book large enough, I can always act rightly in the world, is that the idea? If the challenge is not ill-defined, that should be possible, correct? But the challenge *is* ill-defined, because the kinds of contextual distinctions that make a difference in moral evaluation can be so minor or seemingly arbitrary. For example, it's easy to contrive a scenario in which the situation's moral evaluation changes based on what kind of shoes someone is wearing: an endorsed athlete willingly wears a competitor's brand in public. To make matters more difficult, moral valuations depend as often as not on intention. If the athlete in the above example knowingly wears a competitor's brand, it suggests a different moral valuation than if he mistakenly wears it. That suggests that an AGI will require a theory of mind that can make judgments about the intentions of fellow agents/humans, and that these judgments of intentionality are fed in to the Friendliness algorithm. On top of all that, novel situations with novel combinations of moral concerns occur constantly - witness humanity's struggle to understand the moral implications of cloning - an algorithm is going to answer that one for us? This is not an argument from ignorance. I leave open the possibility that I am too limited to see how one could resolve those objections. But it doesn't matter whether I can see it or not, it's still true that whoever hopes to design a Friendliness algorithm must deal with these objections head-on. And you haven't dealt with them, yet, you've merely dismissed them. Which is especially worrying coming from someone who essentially tethers the fate of humanity to our ability to solve this problem. Let me put it this way: I would think anyone in a position to offer funding for this kind of work would require good answers to the above. Terren ------------------------------------------- agi Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: https://www.listbox.com/member/?member_id=8660244&id_secret=111637683-c8fa51 Powered by Listbox: http://www.listbox.com