Yes, I see your point now.
If an AI has a percentage p chance of going feral, then in the case of a society of AI's, only p percent of them will go feral, and the odds are that other AI's will be able to stop it from doing anything bad. But in the case of only one AI, then there's just a p% chance of it going feral, without much to do about it...
Unknowns are the odds of AI's going feral and supersmart at the same time, and the effects of society-size on the probability of ferality...
But you do have a reasonable point, I'll admit, and I'll think about it more...
This does not follow. If an AI has a P chance of going feral, then a society of AIs may have P chance of all simultaneously going feral - it depends on how much of the probability is independent among different AIs. Actually, for the most worrisome factors, such as theoretical flaws in the theory of AI morality, I would expect the risk factor to be almost completely shared among all AIs. Furthermore, risk factors stemming from divergent rates of self-enhancement or critical thresholds in self-enhancement may not be at all improved by multiply copying or multiply diverging AIs if AI improvement is less than perfectly synchronized.
Remember that humans are evolved and trained to pay attention to variances among humans - the things we all have in common are not noticed. Humans exist in an evolved state where everyone has certain variances and societies are built around the interaction of those variances. That evolution emergently produced this kind of society following social selection pressures on individual fitness does not show that it is the best method given intelligent design, or even a good method at all. I am very suspicious of proposals to construct AI societies. What we regard as beneficial social properties are very contingent on our evolved individual designs.
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
