It is my understanding that the basic problem in Friendly AI is that it is possible for the AI to interpret the command "help humanity" etc wrong, and then destroy humanity (what we don't want it to do). The whole problem is to find some way to make it more probable to not destroy us all. It is correct that a simple sentence can be interpreted to mean something that we don't really mean, even though the interpretation is logical for the AI.
2007/7/12, Mike Tintner <[EMAIL PROTECTED]>:
Kaj, If I look closely at what you write, you are somewhat close to me - but you are in fact not saying explicitly and precisely what I am. I am saying upfront: language and symbol systems are general and abstract and open to infinite, particular, concrete interpretations, including the opposite of any interpretation you might prefer. It is therefore impossible when programming an agent in general language or concepts, to control its particular interpretations of those concepts - whether those concepts are "help humanity" or "make humans happy" etc. You don't say this upfront, and you do seem to imply that it might be possible to control the agent sometimes, if not at others. If you basically agree with my statement, then both your exposition and, I'm sure, mine can be improved. > On 7/12/07, Mike Tintner <[EMAIL PROTECTED]> wrote: >> Comment: You and others seem to be missing the point, which obviously >> needs >> spelling out. There is no way of endowing any agent with conceptual goals >> that cannot be interpreted in ways opposite to the designer's >> intentions - >> that is in the general, abstract nature of language & symbolic systems. >> >> For example, the general, abstract goal of "helping humanity" can >> legitimately in particular, concrete situations be interpreted as wiping >> out >> the entire human race (bar, say, two) - for the sake of future >> generations. >> >> And there is no reasonable way to circumvent this. You couldn't, say, >> instruct an agent... "help humanity but don't kill any human beings..." >> because what if some humans (like, say Bush) are threatening to kill >> vastly >> greater numbers of other humans...wouldn't you want the agent to >> intervene? >> And if you decided that even so, you would instruct the agent not to >> kill, >> it could still as good as kill by rendering humans vegetables while still >> alive. >> >> So many people here and everywhere are unaware of the general and >> deliberately imprecise nature of language - much stressed by Derrida and >> exemplified in the practice of law. > > I am confused, now. The sentence from my essay that you quoted was > from a section of it that was *talking about the same very thing as > you are talking about now*. In fact, had you not cut out the rest of > the sentence, it would've been apparent that it was talking exactly > *about* how "helping humanity" is too vague and ill-defined to be > useful: > > "" An AI programmed only to help humanity will only help humanity, but > in what way? Were it programmed only to make all humans happy, it > might wirehead us - place us into constant states of pure, > artificially-induced states of orgasmic joy that preclude all other > thought and feeling. While that would be a happy state, many humans > would prefer not to end up in one - but even humans can easily argue > that pure happiness is more important than the satisfaction of desires > (in fact, I have, though I'm unsure of my argument's soundness), so > "forcibly wireheading is a bad thing" is not an obvious conclusion for > a mind." > > There are many, many things that we hold valuable, most of which feel > so obvious that we never think about them. An AI would have to be > built to preserve many of them - but it shouldn't preserve them > absolutely, since our values might change over time. Defining the > values in question might also be difficult: producing an exact > definition for any complex, even slightly vague concept often tends to > be next to impossible. We might need to give the AI a somewhat vague > definition and demonstrate by examples what we mean - just as we > humans have learnt them - and then try to make sure that the engine > the AI uses to draw inferences works the same way as ours, so that it > understands the concepts the same way as we do. "" > > Isn't this just what you're saying? The /entire section/ was talking > about this very issue. > > > > > -- > http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ > > Organizations worth your time: > http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ > > ----- > This list is sponsored by AGIRI: http://www.agiri.org/email > To unsubscribe or change your options, please go to: > http://v2.listbox.com/member/?&; > > > > -- > No virus found in this incoming message. > Checked by AVG Free Edition. Version: 7.5.476 / Virus Database: > 269.10.4/897 - Release Date: 11/07/2007 21:57 > > ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?&;
----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=4007604&id_secret=20436939-126b2d
