On Thu, Aug 28, 2008 at 9:08 PM, Terren Suydam <[EMAIL PROTECTED]> wrote: > --- On Wed, 8/27/08, Vladimir Nesov <[EMAIL PROTECTED]> wrote: >> One of the main motivations for the fast development of >> Friendly AI is >> that it can be allowed to develop superintelligence to >> police the >> human space from global catastrophes like Unfriendly AI, >> which >> includes as a special case a hacked design of Friendly AI >> made >> Unfriendly. > > That is certainly the most compelling reason to do this kind of research. > And I wish I had something more than "disallow self-modifying approaches", > as if that would be enforcible. But I just don't see Friendliness as > attainable, > in principle, so I think we treat this like nuclear weaponry - we do our best > to > prevent it.
Won't work, Moore's law is ticking, and one day a morally arbitrary self-improving optimization will go FOOM. We have to try. >> If we can understand it and know that it does what we want, >> we don't >> need to limit its power, because it becomes our power. > > Whose power? Who is referred to by "our"? More importantly, whose > agenda is served by this power? Power corrupts. One culture's good is > another's evil. What we call Friendly, our political enemies might call > Unfriendly. If you think no agenda would be served, you're naive. And if > you think the AGI would somehow know to not serve its masters in service > to Friendliness to humanity, then you believe in an objective morality... > in a universally compelling argument. Given the psychological unity of humankind, giving the focus of "right" to George W. Bush personally will be enormously better for everyone than going in any direction assumed by AI without the part of Friendliness structure that makes it absorb the goals from humanity. CEV is an attempt to describe how to focus AI on humanity as a whole, rather than on a specific human. >> With simulated >> intelligence, understanding might prove as difficult as in >> neuroscience, studying resulting design that is unstable >> and thus in >> long term Unfriendly. Hacking it to a point of Friendliness >> would be >> equivalent to solving the original question of >> Friendliness, >> understanding what you want, and would in fact involve >> something close >> to hands-on design, so it's unclear how much help >> experiments can >> provide in this regard relative to default approach. > > Agreed, although I would not advocate hacking Friendliness. I'd advocate > limiting the simulated environment in which the agent exists. The point of > this line of reasoning is to avoid the Singularity, period. Perhaps that's > every > bit as unrealistic as I believe Friendliness to be. And you are assembling the H-bomb (err, evolved intelligence) in the garage just out of curiosity, and occasionally to use it as a tea table, all the while advocating global disarmament. >> It's self-improvement, not self-retardation. If >> modification is >> expected to make you unstable and crazy, don't do that >> modification, >> add some redundancy instead and think again. > > The question is whether its possible to know in advance that an modification > won't be unstable, within the finite computational resources available to an > AGI. If you write something redundantly 10^6 times, it won't all just spontaneously *change*, in the lifetime of the universe. In the worst case, it'll all just be destroyed by some catastrophe or another, but it won't change in any interesting way. > With the kind of recursive scenarios we're talking about, simulation is the > only > way to guarantee that a modification is an improvement, and an AGI simulating > its own modified operation requires exponentially increasing resources, > particularly > as it simulates itself simulating itself simulating itself, and so on for N > future > modifications. Again, you are imagining an impossible or faulty strategy, pointing to this image, and saying "don't do that!". Doesn't mean there is no good strategy. -- Vladimir Nesov [EMAIL PROTECTED] http://causalityrelay.wordpress.com/ ------------------------------------------- agi Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: https://www.listbox.com/member/?member_id=8660244&id_secret=111637683-c8fa51 Powered by Listbox: http://www.listbox.com
