Using the Frankenstein analogy that Shane brought up, Frankie went bad because 
he was being tormented by people. It's the same with any collection of people 
that identify themselves with a label whether it be religious, political, 
lifestyle, or intelligence - humans fear what they do not understand, and 
sometimes react with violence. As an intelligent being that has become a victim 
of said violence, you may react with violence, or encourage peace by trying to 
dissolve the violence. The peaceful or rational method is extremely difficult 
when you are one against many perpetrators, and you get a lot of negative 
reinforcement about what actions you take in the future so that you try and 
avoid becoming the victim.

An AGI system will turn against us probably if humans turn against it first. 
It's like raising a child, if you beat the child every day, they are not going 
to grow up very friendly. If you raise a child to co-operate and co-exist with 
its environment, what possible motivation is there for it to grow up to be 
hostile?

cheers,
    Simon



> 
> I agree with Shane ... this approach suffers from the same sort of problem
> that AIXI suffers from, Friendliness-wise
> 
> When the system is smart enough, it will learn to outsmart the posited
> Control Code, and the ethics-monitor AGI....  You might want to avoid this
> by making the ethics-monitor AGI the smartest one -- but then who monitors
> *its* ethics??  One arrives at an infinite regress of smarter and smarter
> ethics-monitors!!
> 
> One way to approximately and uncertainly get around this regress is to do
> what Philip says and create an AGI society.  That means: Creating a bunch of
> AGI's and having them all spend part of their resources doing
> ethics-monitoring of each other.  If an AGI is judged bad by too many of its
> peers, lock it in jail.  i.e. enforce social conformity among AGI's....
> 
> However, the society approach does not prevent a whole society of AGI's from
> "drifting into evil".  How good is our understanding of AGI sociodynamics???
> ;-)  This approach just replaces one hard problem with another... which may
> or may not be even harder...
> 
> -- Ben
> 
> 
> 
> > -----Original Message-----
> > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
> > Behalf Of Shane Legg
> > Sent: Monday, March 03, 2003 5:06 AM
> > To: [EMAIL PROTECTED]
> > Subject: Re: [agi] Hard Wired Switch
> >
> >
> >
> > To me this approach sounds like you are trying to out smart the AGI;
> > to somehow constrain it in some way so that it is forced into being
> > nice to people... or Ben at least.  I don't think this kind of an
> > approach will work due to the simple fact that you'll never be able
> > to outsmart a super intelligence.
> >
> > The trick will be, I think, to dig right down the the deepest essence
> > of the AGI and make sure that it would never want to harm humanity in
> > the first place.  I think that's really the only place that danger
> > could be stopped.  Trying to control the AGI at higher levels seems
> > doomed to fail to me as the AGI will simply be far to smart to be
> > restrained by any system that a human could devise.
> >
> > Exactly what this means is of course hard to establish in the absence
> > of an obviously correct AGI design... but it's probably worth thinking
> > a bit about the problem sooner rather than later nevertheless.
> >
> > I think part of the solution will be to control, at least early on
> > while the AGI is learning, its interaction with humans.  I read Mary
> > Shelley's Frankenstein recently and thought that it is still quite
> > relevant in this respect.  Frankenstein succeeding in creating a living
> > being that was larger, stronger and more nimble than any man and rather
> > intelligent also.  His "monster" had no hatred of people to start
> > with and was just confused and scared.  However rather than taking
> > responsibility for his creation, Frankenstein tried to hide from what
> > he had done and just left his creation roam around free in the Swiss
> > wilderness to be tomented by various people until his creation
> > eventually turned into something rather nasty.  When Frankenstein
> > finally decided that his monster needed to be stopped, well it was
> > a bit late then as he was really no match for his own creation...
> >
> > Shane
> >
> >
> > Kevin Copple wrote:
> > > Here's an idea (if it hasn't already been considered before):
> > >
> > > In each executing component of an AGI has Control Code.  This
> > Control Code
> > > monitors a Big Red Switch.  If the switch is turned on,
> > execution proceeds
> > > normally.  If not, all execution by the component stopped.
> > >
> > > The big red switch could be a heart monitor on Ben's chest.
> > The AGI better
> > > keep Ben alive!  And if Ben gets angry with the AGI, he can
> > remove the heart
> > > monitor, so the AGI better keep Ben happy also.
> > >
> > > Several other features could be thrown in:
> > > 1. Components will check that calling components have the active Control
> > > Code that every component is expected to have.
> > > 2. The Control Code checks that certain parameters are not
> > exceeded, such as
> > > AGI memory storage and computational resources.
> > > 3. The Big Red Switch monitors not only Ben's heart, but other control
> > > parameters as well.
> > > 4. The Big Red Switch monitors another AGI, which has a focused
> > purpose of
> > > looking for trouble with the first AGI.
> > >
> > > I imagine that the control code would require a tiny slice of the AGI
> > > resources, yet still be effective.  Implementation details
> > would naturally
> > > require a lot of thought and effort, but I think the concept of
> > a built-in
> > > "virus" would be effective.  Maybe "immune system" would be a
> > more appealing
> > > term.
> > >
> > > Kevin Copple
> > >
> > >
> > >
> > > -------
> > > To unsubscribe, change your address, or temporarily deactivate
> > your subscription,
> > > please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
> > >
> >
> >
> > -------
> > To unsubscribe, change your address, or temporarily deactivate
> > your subscription,
> > please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
> >
> 
> -------
> To unsubscribe, change your address, or temporarily deactivate your 
> subscription, 
> please go to http://v2.listbox.com/member/[EMAIL PROTECTED]

-------
To unsubscribe, change your address, or temporarily deactivate your subscription, 
please go to http://v2.listbox.com/member/[EMAIL PROTECTED]

Reply via email to