On Feb 3, 2008 10:22 PM, Richard Loosemore <[EMAIL PROTECTED]> wrote: > My argument was (at the beginning of the debate with Matt, I believe) > that, for a variety of reasons, the first AGI will be built with > peaceful motivations. Seems hard to believe, but for various technical > reasons I think we can make a very powerful case that this is exactly > what will happen. After that, every other AGI will be the same way > (again, there is an argument behind that). Furthermore, there will not > be any "evolutionary" pressures going on, so we will not find that (say) > the first few million AGIs are built with perfect motivations, and then > some rogue ones start to develop.
In the context of a distributed AGI, like the one I propose at http://www.mattmahoney.net/agi.html this scenario would require the first AGI to take the form of a worm. It may indeed be peaceful if it depends on human cooperation to survive and spread, as opposed to exploiting a security flaw. So it seems a positive outcome depends on solving the security problem. If a worm is smart enough to debug software and discover vulnerabilities faster than humans can (with millions of copies working in parallel), the problem becomes more difficult. (And this *is* an evolutionary process). I guess I don't share Richard's optimism. I suppose a safer approach would be centralized, like most of the projects of people on this list. But I don't see how these systems could compete with the vastly greater resources (human and computer) already available on the internet. A distributed system with, say, Novamente and Google as two of its millions of peers is certainly going to be more intelligent than either system alone. You may wonder why I would design a dangerous system. First, I am not building it. (I am busy with other projects). But I believe that for practical reasons something like this will eventually be built anyway, and we need to study the design to make it safer. -- Matt Mahoney, [EMAIL PROTECTED] ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
