--- Linas Vepstas <[EMAIL PROTECTED]> wrote: > On 07/03/2008, Matt Mahoney <[EMAIL PROTECTED]> wrote: > > > > --- Mark Waser <[EMAIL PROTECTED]> wrote: > > > > > >> Attractor Theory of Friendliness > > > >> > > > >> There exists a describable, reachable, stable attractor in state > space > > > >> that > > > >> is sufficiently Friendly to reduce the risks of AGI to acceptable > levels > > > > > > > > Proof: something will happen resulting in zero or more intelligent > agents. > > > > Those agents will be Friendly to each other and themselves, because > the > > > > action > > > > of killing agents without replacement is an irreversible dynamic, and > > > > therefore cannot be part of an attractor. > > > > > > Huh? Why can't an irreversible dynamic be part of an attractor? (Not > that > > > I need it to be) > > > > An attractor is a set of states that are repeated given enough time. If > > agents are killed and not replaced, you can't return to the current > state. > > False. There are certainly attractors that disappear, first > seen by Ruelle, Takens, 1971 its called a "blue sky catastrophe" > > http://www.scholarpedia.org/article/Blue-sky_catastrophe
Also the simple point attractor x = 0 in the dynamical system dx/dt = -x with solution exp(-t) never repeats. But dynamical systems with real-valued states are just approximations of a discrete universe, and a discrete system must repeat. But I have a different objection to Mark's proposal: the only attractor in an evolutionary system is a dead planet. Evolution is not a stable system. It is on the boundary between stability and chaos. Evolution is punctuated by mass extinctions as well as smaller disasters, plagues, and population explosions. Right now I believe we are in the midst of a mass extinction larger than the Permian extinction. There are two reasons why I think we are still alive today: the anthropic principle, and a range of environments wide enough that no species can inhabit all of them (until now). Omohundro's goals are stable in an evolutionary system (as long as that system persists) because they improve fitness. In Mark's proposal, Friendliness is a subgoal to fitness because (if I understand correctly) agents that cooperate with each other are fitter as a group than agents that fight among themselves. So an outcome where the Earth is turned into a Dyson sphere of gray goo would be Friendly in the sense that the biggest army of nanobots kills off all their unFriendly competition (including all DNA based life) and they cooperate with each other. This is not the risk that concerns me. The real risk is that a single, fully cooperating system has no evolutionary drive for self improvement. Having one world government with perfect harmony among its population is a bad idea because there is no recourse from it making a bad collective decision. In particular, there is no evolutionary pressure to maintain a goal of self preservation. You need competition between countries, but unfortunately this means endless war. -- Matt Mahoney, [EMAIL PROTECTED] ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
