>From: Derek Zahn <[EMAIL PROTECTED]> >Date: Sun, 30 Sep 2007 08:57:53 -0600 >... >One thing that could improve safety is to reject the notion that AGI >projects should be focused on, or even capable of, recursive self >improvement in the sense of reprogramming its core implementation. >... >Let's take Novamente as an example. ... It cannot improve itself >until the following things happen: > >1) It acquires the knowledge and skills to become a competent > programmer, a task that takes a human many years of directed > training and practical experience. > >2) It is given access to its own implementation and permission to alter it. > >3) It understands its own implementation well enough to make a helpful change. >...
I agree that resource #1, competent programming, is essential for any interesting takeoff scenario. I don't think the other two matter, though. Consider these two scenarios: Self-improvement: The AI builds a new, improved version of itself, turns on the new one, and turns off the old one. Engineering: The AI builds a better AI and turns it on. These are essentially the same scenario, except in the latter case the AI doesn't turn itself off. If we assume that the new, improved AI really is improved, we don't care much whether the old one turns itself off because the new one is now the dominant player and the old one therefore doesn't matter much. So giving an AI access to its own source code and permission to alter it (resource #2 above) is irrelevant. Giving it understanding of its own source code is irrelevant too; it's just as good for it to understand some other actual or potential AI implementation, and that's subsumed by the competent programming requirement. No value is added by introducing considerations about self-reference into conversations about the consequences of AI engineering. Junior geeks do find it impressive, though. -- Tim Freeman http://www.fungible.com [EMAIL PROTECTED] ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=52968233-2b8b16
