On Wed, Aug 27, 2008 at 7:44 AM, Terren Suydam <[EMAIL PROTECTED]> wrote: > > --- On Tue, 8/26/08, Vladimir Nesov <[EMAIL PROTECTED]> wrote: >> But what is safe, and how to improve safety? This is a >> complex goal >> for complex environment, and naturally any solution to this >> goal is >> going to be very intelligent. Arbitrary intelligence is not >> safe >> (fatal, really), but what is safe is also intelligent. > > Look, the bottom line is that even if you could somehow build a > self-modifying AI that was provably Friendly, some evil hacker could > come along and modify the code. One way or another, we have to > treat all smarter-than-us intelligences as inherently risky.
One of the main motivations for the fast development of Friendly AI is that it can be allowed to develop superintelligence to police the human space from global catastrophes like Unfriendly AI, which includes as a special case a hacked design of Friendly AI made Unfriendly. > So safe, for me, refers instead to the process of creating the intelligence. > Can we stop it? Can we understand it? Can we limit its scope, its power? > With simulated intelligences, the answer to all of the above is yes. Pinning > your hopes of safe AI on the Friendliness of the AI is the mother of all > gambles, one that in a well-informed democratic process would surely not > be undertaken. If we can understand it and know that it does what we want, we don't need to limit its power, because it becomes our power. With simulated intelligence, understanding might prove as difficult as in neuroscience, studying resulting design that is unstable and thus in long term Unfriendly. Hacking it to a point of Friendliness would be equivalent to solving the original question of Friendliness, understanding what you want, and would in fact involve something close to hands-on design, so it's unclear how much help experiments can provide in this regard relative to default approach. >> There is no law that makes large computations less lawful >> than small >> computations, if it is in the nature of computation to >> preserve >> certain invariants. A computation that multiplies two huge >> numbers >> isn't inherently more unpredictable than computation >> that multiplies >> two small numbers. > > I'm not talking about straight-forward, linear computation. Since > we're talking about self-modification, the computation is necessarily > recursive and iterative. Recursive computation can easily lead to > chaos (as in chaos theory, not disorder). It's self-improvement, not self-retardation. If modification is expected to make you unstable and crazy, don't do that modification, add some redundancy instead and think again. > I'm making a rather broad analogy here by comparing the above > example to a self-modifying AGI, but the principle holds. An AGI with > present goal system G computes the Friendliness of a modification M, > based on G. It decides to go ahead with the modification. This next > iteration results in goal system G'. And so on, performing Friendliness > computations against the resulting goal systems. In what sense could > one guarantee that this process would not lead to chaos? I'm not sure > you could even guarantee it would continue self-modifying. It's kinda the point of superintelligence: it should be able to think ahead, 2 steps ahead, and N steps ahead. So, if it expects that applying modification M will lead to transition to goal system G' that derails it from Friendliness, it won't do that. >> You have intuitive >> expectation that making Z will make AI uncontrollable, >> which will lead >> to a bad outcome, and so you point out that this design >> that suggests >> doing Z will turn out bad. But the answer is that AI itself >> will check >> whether Z is expected to lead to a good outcome before >> making a >> decision to implement Z. > > As has been pointed out before, by others, the goal system can > drift as the modifications are applied. The question once again is, > in what *objective sense* can the AI validate that its Friendliness > algorithm corresponds to what humans actually consider to be Friendly? > What does it compare *against*? Originally, it "compares" against humans, later on it improves on the information about the initial conditions, renormalizing the concept against itself. >> Causal models are not perfect, you say. But perfection is >> causal, >> physical laws are the most causal phenomenon. All the >> causal rules >> that we employ in our approximate models of environment are >> not >> strictly causal, they have exceptions. Evolution has the >> advantage of >> optimizing with the whole flow of environment, but >> evolution doesn't >> have any model of this environment, the counterpart of >> human models in >> evolution is absent. What it has is a simple regularity in >> the >> environment, natural selection. Will all the imperfections, >> human >> models of environment are immensely more precise than this >> regularity >> that relies on natural repetition of context. Evolution >> doesn't have a >> perfect model, it has an exceedingly simplistic model, so >> simple in >> fact that it managed to *emerge* by chance. Humans with >> their >> admittedly limited intelligence, on the other hand, already >> manage to >> create models far surpassing their own intelligence in >> ability to >> model the environment (computer simulations and >> mathematical models). > > What I'm trying to get across here is that evolution gives us access > to areas of the solution space that our intellect can't go. Because it's > possible that the solution to AGI lies in that space, then I'm advocating > putting evolution to work and exploring that space. It's slow, it's time > and resource intensive, but those are actually advantages when you look > at the risks involved with hard-takeoff. There are other advantages as well, > as I discussed in my linked article. Admittedly, these are disadvantages > when it comes to funding (slow != sexy), but with computational power > being so cheap, it's not too expensive to set something interesting up. Agreed. -- Vladimir Nesov [EMAIL PROTECTED] http://causalityrelay.wordpress.com/ ------------------------------------------- agi Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: https://www.listbox.com/member/?member_id=8660244&id_secret=111637683-c8fa51 Powered by Listbox: http://www.listbox.com
