[agi] Some thoughts of an AGI designer

[Richard Loosemore]

I find myself totally bemused by the recent discussion of AGI friendliness.

I am in sympathy with some aspects of Mark's position, but I also see a 
serious problem running through the whole debate:  everyone is making 
statements based on unstated assumptions about the motivations of AGI 
systems.  EVERYTHING depends on what assumptions you make, and yet each 
voice in this debate is talking as if their own assumption can be taken 
for granted.


The three most common of these assumptions are:

  1) That it will have the same motivations as humans, but with a 
tendency toward the worst that we show.


  2) That it will have some kind of "Gotta Optimize My Utility 
Function" motivation.


  3) That it will have an intrinsic urge to increase the power of its 
own computational machinery.


There are other assumptions, but these seem to be the big three.

So what I hear is a series of statements that are analogous to:

   "Well, since the AGI will be bright yellow, it will clearly
do this and this and this.."

   "Well, since the AGI will be a dull sort of Cambridge blue,
it will clearly do this and this and this.."

   "Well, since the AGI will be orange, it will clearly do this
and this and this.."

(Except, of course, that nobody is actually coming right out and saying 
what color of AGI they assume.)


In the past I have argued strenuously that (a) you cannot divorce a 
discussion of friendliness from a discussion of what design of AGI you 
are talking about, and (b) some assumptions about AGI motivation are 
extremely incoherent.


And yet in spite of all my efforts that I have made, there seems to be 
no acknowledgement of the importance of these two points.






Richard Loosemore

---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Ben Goertzel]

>  The three most common of these assumptions are:
>
>1) That it will have the same motivations as humans, but with a
>  tendency toward the worst that we show.
>
>2) That it will have some kind of "Gotta Optimize My Utility
>  Function" motivation.
>
>3) That it will have an intrinsic urge to increase the power of its
>  own computational machinery.
>
>  There are other assumptions, but these seem to be the big three.

And IMO, the truth is likely to be more complex...

For instance,  a Novamente-based AGI will have an explicit utility
function, but only a percentage of the system's activity will be directly
oriented toward fulfilling this utility function

Some of the system's activity will be "spontaneous" ... i.e. only
implicitly goal-oriented .. and as such may involve some imitation
of human motivation, and plenty of radically non-human stuff...

ben g

---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Vladimir Nesov]

On Mon, Mar 10, 2008 at 5:47 PM, Richard Loosemore <[EMAIL PROTECTED]> wrote:
>
>  In the past I have argued strenuously that (a) you cannot divorce a
>  discussion of friendliness from a discussion of what design of AGI you
>  are talking about, and (b) some assumptions about AGI motivation are
>  extremely incoherent.
>
>  And yet in spite of all my efforts that I have made, there seems to be
>  no acknowledgement of the importance of these two points.
>

Would it be so easy to reliably transfer semitechnical understanding,
and better yet to check its rationality...

-- 
Vladimir Nesov
[EMAIL PROTECTED]

---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

I am in sympathy with some aspects of Mark's position, but I also see a 
serious problem running through the whole debate:  everyone is making 
statements based on unstated assumptions about the motivations of AGI 
systems.


Bummer.  I thought that I had been clearer about my assumptions.  Let me try 
to concisely point them out again and see if you can show me where I have 
additional assumptions that I'm not aware that I'm making (which I would 
appreciate very much).


Assumption - The AGI will be a goal-seeking entity.

And I think that is it.:-)

EVERYTHING depends on what assumptions you make, and yet each voice in 
this debate is talking as if their own assumption can be taken for 
granted.


I agree with you and am really trying to avoid this.  I will address your 
specific examples below and would appreciate any others that you can point 
out.



The three most common of these assumptions are:
  1) That it will have the same motivations as humans, but with a tendency 
toward the worst that we show.


I don't believe that I'm doing this.  I believe that all goal-seeking 
generally tends to be optimized by certain behaviors (the Omohundro drives). 
I believe that humans show many of these behaviors because these behaviors 
are relatively optimal in relation to the alternatives (and because humans 
are relatively optimal).  But I also believe that the AGI will also have 
dramatically different motivations from humans where the human motivations 
were evolved stepping stones that were on the necessary and optimal path for 
one environment but haven't been eliminated now that they are unnecessary 
and sub-optimal in the current environment/society (Richard's "the worst 
that we show").


  2) That it will have some kind of "Gotta Optimize My Utility Function" 
motivation.


I agree with the statement but I believe that it is a logical follow-on to 
my assumption that the AGI is a goal-seeking entity (i.e. it's an Omohundro 
drive).  Would you agree, Richard?


  3) That it will have an intrinsic urge to increase the power of its own 
computational machinery.


Again, I agree with the statement but I believe that it is a logical 
follow-on to my single initial assumption (i.e. it's another Omohundro 
drive).  Wouldn't you agree?



There are other assumptions, but these seem to be the big three.


And I would love to go through all of them, actually (or debate one of my 
answers above).


So what I hear is a series of statements  (Except, of course, that 
nobody is actually coming right out and saying what color of AGI they 
assume.)


I thought that I pretty explicitly was . . . . :-(

In the past I have argued strenuously that (a) you cannot divorce a 
discussion of friendliness from a discussion of what design of AGI you are 
talking about,


And I have reached the conclusion that you are somewhat incorrect.  I 
believe that goal-seeking entities OF ANY DESIGN of sufficient intelligence 
(goal-achieving ability) will see an attractor in my particular vision of 
Friendliness (which I'm deriving by *assuming* the attractor and working 
backwards from there -- which I guess you could call a second assumption if 
you *really* had to  ;-).



and (b) some assumptions about AGI motivation are extremely incoherent.


If you perceive me as incoherent, please point out where.  My primary AGI 
motivation is "self-interest" (defined as achievement of *MY* goals -- which 
directly derives from my assumption that "the AGI will be a goal-seeking 
entity").  All other motivations are clearly logically derived from that 
primary motivation.  If you see an example where this doesn't appear to be 
the case, *please* flag it for me (since I need to fix it  :-).


And yet in spite of all my efforts that I have made, there seems to be no 
acknowledgement of the importance of these two points.


I think that I've acknowledged both in the past and will continue to do so 
(despite the fact that I am now somewhat debating the first point -- more 
the letter than the spirit  :-). 



---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

For instance,  a Novamente-based AGI will have an explicit utility
function, but only a percentage of the system's activity will be directly
oriented toward fulfilling this utility function

Some of the system's activity will be "spontaneous" ... i.e. only
implicitly goal-oriented .. and as such may involve some imitation
of human motivation, and plenty of radically non-human stuff...


Which, as Eliezer has pointed out, sounds dangerous as all hell unless you 
have some reason to assume that it wouldn't be (like being sure that the AGI 
sees and believes that Friendliness is in it's own self-interest). 



---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Richard Loosemore]

Mark Waser wrote:
I am in sympathy with some aspects of Mark's position, but I also see 
a serious problem running through the whole debate:  everyone is 
making statements based on unstated assumptions about the motivations 
of AGI systems.


Bummer.  I thought that I had been clearer about my assumptions.  Let me 
try to concisely point them out again and see if you can show me where I 
have additional assumptions that I'm not aware that I'm making (which I 
would appreciate very much).


Assumption - The AGI will be a goal-seeking entity.

And I think that is it.:-)


Okay, I can use that as an illustration of what I am getting at.

There are two main things.

One is that the statement "The AGI will be a goal-seeking entity" has 
many different interpretations, ad I am arguing that these different 
interpretations have a massive impact on what kind of behavior you can 
expect to see.


It is almost impossible to list all the different interpretations, but 
two of the more extreme variants are the two that I have described 
before:  a "Goal-Stack" system in which the goals are represented in the 
same form as the knowledge that the system stores, and a "Motivational 
Emotional System" which biasses the functioning of the system and is 
intimately connected with the development of its knowledge.  The GS 
system has the dangerous feature that any old fool could go in and 
rewrite the top level goal so it reads "make as much computronium as 
possible" or "cultivate dandelions" or "learn how to do crochet".  The 
MES system, on the other hand, can be set up to have values such as ours 
and to feel empathy with human beings, and once set up that way you 
would have to re-grow the system before you could get it to have some 
other set of values.


Clearly, these two interpretations of "The AGI will be a goal-seeking 
entity" have such different properties that, unless there is detailed 
clarification of what the meaning is, we cannot continue to discuss what 
they would do.


My second point is that some possible choices of the meaning of "The AGI 
will be a goal-seeking entity" will actually not cash out into a 
coherent machine design, so we would be wasting our time if we 
considered how that kind of AGI would behave.


In particular, there are severe doubts about whether the Goal-Stack type 
of system can ever make it up to the level of a full intelligence.  I'll 
go one further on that:  I think that one of the main reasons we have 
trouble getting AI systems to be AGI is precisely because we have not 
yet realised that they need to be driven by something more than a Goal 
Stack.  It is not the only reason, but its a big one.


So the message is:  we need to know exactly details of the AGI's 
motivation system ("The AGI will be a goal-seeking entity" is not 
specific enough), and we need to then be sure that the details we give 
are going to lead to a type of AGI that can actually be an AGI.


These questions, I think, are the real battleground.

BTW, this is not a direct attack on what you were saying, because I 
believe that there is a version of what you are saying (about an 
intrinsic tendency toward a Friendliness attractor) that I agree with. 
My problem is that so much of the current discussion is tangled up with 
hidden assumptions that I think that the interesting part of your 
message is getting lost.





EVERYTHING depends on what assumptions you make, and yet each voice in 
this debate is talking as if their own assumption can be taken for 
granted.


I agree with you and am really trying to avoid this.  I will address 
your specific examples below and would appreciate any others that you 
can point out.



The three most common of these assumptions are:
  1) That it will have the same motivations as humans, but with a 
tendency toward the worst that we show.


I don't believe that I'm doing this.  I believe that all goal-seeking 
generally tends to be optimized by certain behaviors (the Omohundro 
drives). I believe that humans show many of these behaviors because 
these behaviors are relatively optimal in relation to the alternatives 
(and because humans are relatively optimal).  But I also believe that 
the AGI will also have dramatically different motivations from humans 
where the human motivations were evolved stepping stones that were on 
the necessary and optimal path for one environment but haven't been 
eliminated now that they are unnecessary and sub-optimal in the current 
environment/society (Richard's "the worst that we show").


I am in complete disagreement with Omuhundro's idea that there are a 
canonical set of drives.


This is like saying that there is a canonical set of colors that AGIs 
will come in:  Cambridge Blue, Lemon Yellow and True Black.


What color the thing is will be what color you decide to paint it!

Ditto for its goals and motivations:  what you decide to put into it is 
what it does, so I cannot make any sense of statements like "I also 
believe that

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

First off -- yours was a really helpful post.  Thank you!

I think that I need to add a word to my initial assumption . . . .
Assumption - The AGI will be an optimizing goal-seeking entity.

> There are two main things.
> One is that the statement "The AGI will be a goal-seeking entity" has 
> many different interpretations, ad I am arguing that these different 
> interpretations have a massive impact on what kind of behavior you can 
> expect to see.

I disagree that it has many interpretations.  I am willing to agree that my 
original assumption phrase didn't sufficiently circumscribe the available space 
of entities to justify some of my further reasoning (most particularly because 
Omohundro drives *ASSUME* an optimizing entity -- my bad for not picking that 
up before  :-).

> The 
> MES system, on the other hand, can be set up to have values such as ours 
> and to feel empathy with human beings, and once set up that way you 
> would have to re-grow the system before you could get it to have some 
> other set of values.

As a system that (arguably) finds itself less able to massively (and possibly 
dangerously) optimize itself, the MES system is indeed less subject to my 
reasoning to the extent that it is not able to optimize itself (or, to the 
extent that it is constrained in optimizing itself).  On the other hand, to the 
extent that the MES system *IS* able to optimize itself, I would contend that 
my Omohundro-drive-based reasoning is valid and correct.

> Clearly, these two interpretations of "The AGI will be a goal-seeking 
> entity" have such different properties that, unless there is detailed 
> clarification of what the meaning is, we cannot continue to discuss what 
> they would do.

Hopefully my statement just above will convince you that we can continue since 
we really aren't arguing different properties -- merely the degree to which a 
system can self-optimize.  That should not prevent a useful discussion.

> My second point is that some possible choices of the meaning of "The AGI 
> will be a goal-seeking entity" will actually not cash out into a 
> coherent machine design, so we would be wasting our time if we 
> considered how that kind of AGI would behave.

I disagree.  Even if 50% of the possible choices can't be implemented, then I 
still don't believe that we shouldn't investigate the class as a whole.  It has 
interesting characteristics that lead me to believe that the remaining 50% of 
implementable choices may hit the jackpot.

> In particular, there are severe doubts about whether the Goal-Stack type 
> of system can ever make it up to the level of a full intelligence.  

Ah.  But this is an intelligence argument rather than a Friendliness argument 
and doubly irrelevant because I am not proposing or nor assuming a goal-stack.  
I prefer your system of a large, diffuse set of (often but not always simple) 
goals and constraints and don't believe it to be at all contrary to what I am 
envisioning.  I particularly like it because *I BELIEVE* that such an approach 
is much more likely to produce a safe, orderly/smooth transition into my 
Friendliness attractor that a relatively easily breakable Goal-Stack system.

> I'll go one further on that:  I think that one of the main reasons we have 
> trouble getting AI systems to be AGI is precisely because we have not 
> yet realised that they need to be driven by something more than a Goal 
> Stack.  It is not the only reason, but its a big one.

I agree with you (but it's still not relevant to my argument:-).

> So the message is:  we need to know exactly details of the AGI's 
> motivation system ("The AGI will be a goal-seeking entity" is not 
> specific enough), and we need to then be sure that the details we give 
> are going to lead to a type of AGI that can actually be an AGI.

No, we don't need to know the details.  I'm contending that my vision/theory 
applies regardless of the details.  If you don't believe so, please supply 
contrary details and I'll do whatever necessary to handle them.:-)

> These questions, I think, are the real battleground.

We'll see . . . . :-)

> BTW, this is not a direct attack on what you were saying, 

Actually, I prefer a direct attack:-).  I should have declared Crocker's 
rules with the "Waste of my time" exception (i.e. I reserve the right to be 
rude to anyone who both is rude *and* wastes my time  :-).

> My problem is that so much of the current discussion is tangled up with 
> hidden assumptions that I think that the interesting part of your 
> message is getting lost.

So let's drag those puppies into the light!  This is not an easy message.  It 
touches on (and, I believe, revises) one helluva lot.  That's why I laugh when 
someone just wants a link to the "completed" paper.  Trust me -- the wording on 
the "completed" paper changes virtually every time there is an e-mail on the 
subject.  And I *don't* want people skipping ahead to the punch line if I'm not 
explai

Re: [agi] Some thoughts of an AGI designer

[Charles D Hixson]

Mark Waser wrote:

...
The motivation that is in the system is "I want to achieve *my* goals".
 
The goals that are in the system I deem to be entirely irrelevant 
UNLESS they are deliberately and directly contrary to Friendliness.  I 
am contending that, unless the initial goals are deliberately and 
directly contrary to Friendliness, an optimizing system's motivation 
of achieve *my* goals (over a large enough set of goals) will 
eventually cause it to finally converge on the goal of Friendliness 
since Friendliness is the universal super-meta-subgoal of all it's 
other goals (and it's optimizing will also drive it up to the 
necessary intelligence to understand Friendliness).  Of course, it may 
take a while since we humans are still in the middle of it . . . . but 
hopefully we're almost there.;-)

...
 
Mark
I think here we need to consider A. Maslow's hierarchy of needs.  That 
an AGI won't have the same needs as a human is, I suppose, obvious, but 
I think it's still true that it will have a "hierarchy"  (which isn't 
strictly a hierarchy).  I.e., it will have a large set of motives, and 
which it is seeking to satisfy at any moment will alter as the 
satisfaction of the previous most urgent motive changes.


It it were a human we could say that breathing was the most urgent 
need...but usually it's so well satisfied that we don't even think about 
it.  Motives, then, will have satisficing  as their aim.  Only aberrant 
mental functions will attempt to increase the satisfying of some 
particular goal without limit.  (Note that some drives in humans seem to 
occasionally go into that "satisfy increasingly without limit" mode, 
like quest for wealth or power, but in most sane people these are reined 
in.  This seems to indicate that there is a real danger here...and also 
that it can be avoided.)


---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

I think here we need to consider A. Maslow's hierarchy of needs.  That an 
AGI won't have the same needs as a human is, I suppose, obvious, but I 
think it's still true that it will have a "hierarchy"  (which isn't 
strictly a hierarchy).  I.e., it will have a large set of motives, and 
which it is seeking to satisfy at any moment will alter as the 
satisfaction of the previous most urgent motive changes.


I agree with all of this.

It it were a human we could say that breathing was the most urgent 
need...but usually it's so well satisfied that we don't even think about 
it.  Motives, then, will have satisficing  as their aim.  Only aberrant 
mental functions will attempt to increase the satisfying of some 
particular goal without limit.  (Note that some drives in humans seem to 
occasionally go into that "satisfy increasingly without limit" mode, like 
quest for wealth or power, but in most sane people these are reined in. 
This seems to indicate that there is a real danger here...and also that it 
can be avoided.)


I agree this except that I believe that humans *frequently* aim to optimize 
rather than satisfy (frequently to their detriment -- in terms of happiness 
as well as in the real costs of performing the search past a simple 
satisfaction point).


Also, quest for pleasure (a.k.a. addiction) is also distressingly frequent 
in humans.


Do you think that any of this contradicts what I've written thus far?  I 
don't immediately see any contradictions. 



---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Linas Vepstas]

On 10/03/2008, Mark Waser <[EMAIL PROTECTED]> wrote:
>
>  Do you think that any of this contradicts what I've written thus far?  I
>  don't immediately see any contradictions.

The discussions seem to entirely ignore the role of socialization
in human and animal friendliness. We are a large collection of
autonomous agents that are well-matched in skills and abilities.
If we were unfriendly to one another, we might survive as a species,
but we would not live in cities and posses hi-tech.

We also know from the animal kingdom, as well as from the
political/economic sphere, what happens when abilities are
mis-matched. Lions eat gazelles, and business tycoons eat
the working class.  We've evolved political systems to curb
the worst abuses of feudalism and serfdom, but have not yet
achieved nirvana.

As parents, we apply social pressure to our children, to make
them friendly. Even then, some grow up unfriendly, and for them,
we have the police. Unless they achieve positions of power first
(Hitler, Stalin, Mao).

I don't see how a single AGI could be bound by the social
pressures that we are bound by. There won't be a collection
of roughly-equal AGI's keeping each other in check, not if they
are self-improving. Self-preservation is rational, and so is
paranoia; its reasonable to assume that agi will race to
self-improve merely for the benefit of self-preservation, so
that they've enough power so that others can't hurt them.

Our hope is that AGI will conclude that humans are harmless
and worthy of study and preservation; this is what will make
them friendly to *us*.. until one day we look like mosquitoes
or microbes to them.

--linas

---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

The discussions seem to entirely ignore the role of socialization
in human and animal friendliness. We are a large collection of
autonomous agents that are well-matched in skills and abilities.
If we were unfriendly to one another, we might survive as a species,
but we would not live in cities and posses hi-tech.


You are correct.  The discussions are ignoring the role of socialization.


We also know from the animal kingdom, as well as from the
political/economic sphere, what happens when abilities are
mis-matched. Lions eat gazelles, and business tycoons eat
the working class.  We've evolved political systems to curb
the worst abuses of feudalism and serfdom, but have not yet
achieved nirvana.


Because we do *not* have a common definition of goals and socially 
acceptable behavior.  Political systems have not acheived nirvana because 
they do not agree on what nirvana looks like.  *THAT* is the purpose of this 
entire thread.



As parents, we apply social pressure to our children, to make
them friendly. Even then, some grow up unfriendly, and for them,
we have the police. Unless they achieve positions of power first
(Hitler, Stalin, Mao).


OK.


I don't see how a single AGI could be bound by the social
pressures that we are bound by. There won't be a collection
of roughly-equal AGI's keeping each other in check, not if they
are self-improving. Self-preservation is rational, and so is
paranoia; its reasonable to assume that agi will race to
self-improve merely for the benefit of self-preservation, so
that they've enough power so that others can't hurt them.

Our hope is that AGI will conclude that humans are harmless
and worthy of study and preservation; this is what will make
them friendly to *us*.. until one day we look like mosquitoes
or microbes to them.

--linas

---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?&;

Powered by Listbox: http://www.listbox.com




---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

Pesky premature e-mail problem . . .


The discussions seem to entirely ignore the role of socialization
in human and animal friendliness. We are a large collection of
autonomous agents that are well-matched in skills and abilities.
If we were unfriendly to one another, we might survive as a species,
but we would not live in cities and posses hi-tech.


You are correct.  The discussions are ignoring the role of socialization.


We also know from the animal kingdom, as well as from the
political/economic sphere, what happens when abilities are
mis-matched. Lions eat gazelles, and business tycoons eat
the working class.  We've evolved political systems to curb
the worst abuses of feudalism and serfdom, but have not yet
achieved nirvana.


Because we do *not* have a common definition of goals and socially
acceptable behavior.  Political systems have not acheived nirvana because
they do not agree on what nirvana looks like.  *THAT* is the purpose of 
this

entire thread.


As parents, we apply social pressure to our children, to make
them friendly. Even then, some grow up unfriendly, and for them,
we have the police. Unless they achieve positions of power first
(Hitler, Stalin, Mao).


OK.  But I'm actually not attempting to use social pressure (or use it 
solely).  I seem to have gotten somewhat shunted down that track by Vladmir 
since a Friendly society is intelligent enough to use social pressure when 
applicable but it is not the primary (or necessary) thrust of my argument.



I don't see how a single AGI could be bound by the social
pressures that we are bound by. There won't be a collection
of roughly-equal AGI's keeping each other in check, not if they
are self-improving. Self-preservation is rational, and so is
paranoia; its reasonable to assume that agi will race to
self-improve merely for the benefit of self-preservation, so
that they've enough power so that others can't hurt them.


Again, social pressure is not my primary argument.  It just made an easy 
convenient correct-but-not-complete argument for Vladimir (and now I'm 
regretting it  :-).



Our hope is that AGI will conclude that humans are harmless
and worthy of study and preservation; this is what will make
them friendly to *us*.. until one day we look like mosquitoes
or microbes to them.


No, our hope is that the AGI will conclude that anything with enough 
intelligence/goal-success is more an asset than a liability and that wiping 
us out without good cause has negative utility. 



---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Charles D Hixson]

Mark Waser wrote:
I think here we need to consider A. Maslow's hierarchy of needs.  
That an AGI won't have the same needs as a human is, I suppose, 
obvious, but I think it's still true that it will have a "hierarchy"  
(which isn't strictly a hierarchy).  I.e., it will have a large set 
of motives, and which it is seeking to satisfy at any moment will 
alter as the satisfaction of the previous most urgent motive changes.


I agree with all of this.

It it were a human we could say that breathing was the most urgent 
need...but usually it's so well satisfied that we don't even think 
about it.  Motives, then, will have satisficing  as their aim.  Only 
aberrant mental functions will attempt to increase the satisfying of 
some particular goal without limit.  (Note that some drives in humans 
seem to occasionally go into that "satisfy increasingly without 
limit" mode, like quest for wealth or power, but in most sane people 
these are reined in. This seems to indicate that there is a real 
danger here...and also that it can be avoided.)


I agree this except that I believe that humans *frequently* aim to 
optimize rather than satisfy (frequently to their detriment -- in 
terms of happiness as well as in the real costs of performing the 
search past a simple satisfaction point).


Also, quest for pleasure (a.k.a. addiction) is also distressingly 
frequent in humans.


Do you think that any of this contradicts what I've written thus far?  
I don't immediately see any contradictions.


If the motives depend on "satisficing", and the questing for unlimited 
fulfillment is avoided, then this limits the danger.   The universe 
won't be converted into toothpicks, if a part of setting the goal for 
"toothpicks!" is limiting the quantity of toothpicks.  (Limiting it 
reasonably might almost be a definition of friendliness ... or at least 
neutral behavior.)


And, though I'm not clear on how this should be set up, this 
"limitation" should be a built-in primitive, i.e. not something subject 
to removal, but only to strengthening or weakening via learning.  It 
should ante-date the recognition of visual images.  But it needs to have 
a slightly stronger residual limitation that it does with people.  Or 
perhaps it's initial appearance needs to be during the formation of the 
statement of the problem.  I.e., a solution to a problem can't be sought 
without knowing limits.  People seem to just manage that via a dynamic 
sensing approach, and that sometimes suffers from inadequate feedback 
mechanisms (saying "Enough!").


(It's not clear to me that it differs from what you are saying, but it 
does seem to address a part of what you were addressing,  and I wasn't 
really clear about how you intended the satisfaction of to be limited.)


---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

If the motives depend on "satisficing", and the questing for unlimited 
fulfillment is avoided, then this limits the danger.   The universe won't 
be converted into toothpicks, if a part of setting the goal for 
"toothpicks!" is limiting the quantity of toothpicks.  (Limiting it 
reasonably might almost be a definition of friendliness ... or at least 
neutral behavior.)


You have a good point.  Goals should be fulfilled after satisficing except 
when the goals are of the form "as  as possible" (hereafter referred 
to as "unbounded" goals).  Unbounded-goal-entities *are* particularly 
dangerous (although being aware of the danger should mmitigate it to some 
degree).


My Friendliness basically works by limiting the amount of interference with 
other's goals (under the theory that doing so will prevent other's from 
interfering with your goals).  Stupid entities that can't see the 
self-interest in the parenthetical point are not inclined to be Friendly. 
Stupid unbounded-goal-entities are Eliezer's paperclip-producing nightmare.


And, though I'm not clear on how this should be set up, this "limitation" 
should be a built-in primitive, i.e. not something subject to removal, but 
only to strengthening or weakening via learning.  It should ante-date the 
recognition of visual images.  But it needs to have a slightly stronger 
residual limitation that it does with people.  Or perhaps it's initial 
appearance needs to be during the formation of the statement of the 
problem.  I.e., a solution to a problem can't be sought without knowing 
limits.  People seem to just manage that via a dynamic sensing approach, 
and that sometimes suffers from inadequate feedback mechanisms (saying 
"Enough!").


The limitation is "Don't stomp on other people's goals unless it is truly 
necessary" *and* "It is very rarely truly necessary".


(It's not clear to me that it differs from what you are saying, but it 
does seem to address a part of what you were addressing,  and I wasn't 
really clear about how you intended the satisfaction of to be limited.)


As far as my theory/vision goes, I was pretty much counting on the fact that 
we are multi-goal systems and that our other goals will generally limit any 
single goal from getting out of hand.  Further, if that doesn't do it, the 
proclamation of not stepping on other's goals unless absolutely necessary 
should help handle the problem . . . . but . . . . actually you do have a 
very good point.  My theory/vision *does* have a vulnerability toward 
single-unbounded-goal entities in that my Friendly attractor has no benefit 
for such a system (unless, of course it's goal is Friendliness or it is 
forced to have a secondary goal of Friendliness).



---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Charles D Hixson]

Mark Waser wrote:
If the motives depend on "satisficing", and the questing for 
unlimited fulfillment is avoided, then this limits the danger.   The 
universe won't be converted into toothpicks, if a part of setting the 
goal for "toothpicks!" is limiting the quantity of toothpicks.  
(Limiting it reasonably might almost be a definition of friendliness 
... or at least neutral behavior.)


You have a good point.  Goals should be fulfilled after satisficing 
except when the goals are of the form "as  as possible" 
(hereafter referred to as "unbounded" goals).  Unbounded-goal-entities 
*are* particularly dangerous (although being aware of the danger 
should mmitigate it to some degree).


My Friendliness basically works by limiting the amount of interference 
with other's goals (under the theory that doing so will prevent 
other's from interfering with your goals).  Stupid entities that can't 
see the self-interest in the parenthetical point are not inclined to 
be Friendly. Stupid unbounded-goal-entities are Eliezer's 
paperclip-producing nightmare.


And, though I'm not clear on how this should be set up, this 
"limitation" should be a built-in primitive, i.e. not something 
subject to removal, but only to strengthening or weakening via 
learning.  It should ante-date the recognition of visual images.  But 
it needs to have a slightly stronger residual limitation that it does 
with people.  Or perhaps it's initial appearance needs to be during 
the formation of the statement of the problem.  I.e., a solution to a 
problem can't be sought without knowing limits.  People seem to just 
manage that via a dynamic sensing approach, and that sometimes 
suffers from inadequate feedback mechanisms (saying "Enough!").


The limitation is "Don't stomp on other people's goals unless it is 
truly necessary" *and* "It is very rarely truly necessary".


(It's not clear to me that it differs from what you are saying, but 
it does seem to address a part of what you were addressing,  and I 
wasn't really clear about how you intended the satisfaction of to be 
limited.)


As far as my theory/vision goes, I was pretty much counting on the 
fact that we are multi-goal systems and that our other goals will 
generally limit any single goal from getting out of hand.  Further, if 
that doesn't do it, the proclamation of not stepping on other's goals 
unless absolutely necessary should help handle the problem . . . . but 
. . . . actually you do have a very good point.  My theory/vision 
*does* have a vulnerability toward single-unbounded-goal entities in 
that my Friendly attractor has no benefit for such a system (unless, 
of course it's goal is Friendliness or it is forced to have a 
secondary goal of Friendliness).


The trouble with "not stepping on other's goals unless absolutely 
necessary" is that it relies on mind-reading.  The goals of others are 
often opaque and not easily verbalizable even if they think to.  Then 
there's the question of "unless absolutely necessary".  How and why 
should I decide that their goals are more important than mine?  So one 
needs to know not only how important their goals are to them, but also 
how important my conflicting goals are to me.  And, of course, whether 
there's a means for mutual satisfaction that it's too expensive.   (And 
just try to define that "too".)


For some reason I'm reminded of the story about the peasant, his son, 
and the donkey carrying a load of sponges.  I'd just as soon nobody ends 
up in the creek.  ("Please all, please none.")


---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

> The trouble with "not stepping on other's goals unless absolutely 
> necessary" is that it relies on mind-reading.  The goals of others are 
> often opaque and not easily verbalizable even if they think to.  

The trouble with the optimal implementation of "not stepping on other's goals 
unless absolutely necessary" is that it relies on mind-reading. 

Your honest, best attempt at not doing so is all that is required of/in 
Friendliness.  The rest is an intelligence problem.

> Then 
> there's the question of "unless absolutely necessary".  

Again, your honest, best attempt is all that is required of/in Friendliness. 

> How and why 
> should I decide that their goals are more important than mine?  

You should *never* decide that (this is, after all, the ultimate in 
self-interest -- remember?).  You should frequently decide that their goals are 
sufficiently compatible enough with your super-goals (i.e. Friendliness) that 
it is worth going a bit out of your way to avoid conflict (particularly if they 
feel strongly enough about their goals that any effort that you make in 
conflict will be wasted -- see the next paragraph).

> So one 
> needs to know not only how important their goals are to them, but also 
> how important my conflicting goals are to me.  

Sort of but the true situation would be clearer if I restate it.  Knowing how 
important their immediate goals are to them will give you some idea as to how 
hard they will strive to fulfill them.  Knowing how important your immediate 
goals are to you will give you some idea how hard you should strive to fulfill 
them.  If you both could redirect an equal amount of directly competing 
striving into other efforts, you both would come out ahead by that amount so 
you both should reduce the importance of the conflicting goals by that amount 
(the alternative is to just waste the effort striving against each other).  If 
the party with the less important goal gives up without a fight (striving), 
both parties gain.  Further, if the losing party gets the agreement of the 
winning party for a reasonable favor in return -- they both end up way ahead.

The only requirement that Friendliness insists upon for this to work is that 
you have to be as honest as you can about how important something is to you 
(otherwise a lot of effort is wasting upon truth verification, hiding 
information, etc.).

> And, of course, whether 
> there's a means for mutual satisfaction that it's too expensive.   (And 
> just try to define that "too".)

I think that I just handled this in the paragraph above -- keeping in mind that 
all that Friendliness requires is your honest, best attempt.

> For some reason I'm reminded of the story about the peasant, his son, 
> and the donkey carrying a load of sponges.  I'd just as soon nobody ends 
> up in the creek.  ("Please all, please none.")

Friendliness is supposed to appeal to geniuses as beng in their self-interest.  
It can't do that and be stupid at the same time.  If it's not possible to 
please everyone then Friendliness isn't going to attempt to do so.  The entire 
point to Friendliness is to REDUCE UNNECESSARY CONFLICT AS MUCH AS POSSIBLE 
because it is in *everyone's* best interest to do so.  Look at Friendliness as 
the ultimate social lubricant that gets the gears of society moving as 
efficiently as possible -- which is only to the benefit of everyone in the 
society.

Mark

Vision/Slogan -- Friendliness:  The Ice-9 of Ethics and Ultimate in 
Self-Interest

---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Charles D Hixson]

Mark Waser wrote:

> The trouble with "not stepping on other's goals unless absolutely
> necessary" is that it relies on mind-reading.  The goals of others are
> often opaque and not easily verbalizable even if they think to. 
 
The trouble with */ the optimal implementation of /* "not stepping on 
other's goals unless absolutely necessary" is that it relies on 
mind-reading. 
 
Your honest, best attempt at not doing so is all that is required 
of/in Friendliness.  The rest is an intelligence problem.
 
> Then
> there's the question of "unless absolutely necessary". 
 
Again, your honest, best attempt is all that is required of/in 
Friendliness. 
 
> How and why
> should I decide that their goals are more important than mine? 
 
You should *never* decide that (this is, after all, the ultimate in 
self-interest -- remember?).  You should frequently decide that their 
goals are sufficiently compatible enough with your super-goals (i.e. 
Friendliness) that it is worth going a bit out of your way to avoid 
conflict (particularly if they feel strongly enough about their goals 
that any effort that you make in conflict will be wasted -- see the 
next paragraph).
 
> So one

> needs to know not only how important their goals are to them, but also
> how important my conflicting goals are to me. 
 
Sort of but the true situation would be clearer if I restate it.  
Knowing how important their immediate goals are to them will give you 
some idea as to how hard they will strive to fulfill them.  Knowing 
how important your immediate goals are to you will give you some idea 
how hard you should strive to fulfill them.  If you both could 
redirect an equal amount of directly competing striving into other 
efforts, you both would come out ahead by that amount so you both 
should reduce the importance of the conflicting goals by that amount 
(the alternative is to just waste the effort striving against each 
other).  If the party with the less important goal gives up without a 
fight (striving), both parties gain.  Further, if the losing party 
gets the agreement of the winning party for a reasonable favor in 
return -- they both end up way ahead.
 
The only requirement that Friendliness insists upon for this to work 
is that you have to be as honest as you can about how important 
something is to you (otherwise a lot of effort is wasting upon truth 
verification, hiding information, etc.).
 
> And, of course, whether

> there's a means for mutual satisfaction that it's too expensive.   (And
> just try to define that "too".)
 
I think that I just handled this in the paragraph above -- keeping in 
mind that all that Friendliness requires is your honest, best attempt.
 
> For some reason I'm reminded of the story about the peasant, his son,
> and the donkey carrying a load of sponges.  I'd just as soon nobody 
ends

> up in the creek.  ("Please all, please none.")
 
Friendliness is supposed to appeal to geniuses as beng in their 
self-interest.  It can't do that and be stupid at the same time.  If 
it's not possible to please everyone then Friendliness isn't going to 
attempt to do so.  The entire point to Friendliness is to */REDUCE 
UNNECESSARY CONFLICT AS MUCH AS POSSIBLE/* because it is in 
*everyone's* best interest to do so.  Look at Friendliness as the 
ultimate social lubricant that gets the gears of society moving as 
efficiently as possible -- which is only to the benefit of everyone in 
the society.
 
Mark
 
I *think* you are assuming that both sides are friendly.  If one side is 
a person, or group of people, then this is definitely not guaranteed.  
I'll grant all your points if both sides are friendly, and each knows 
the other to be friendly.  Otherwise I think things get messier.  So 
objective measures and tests are desireable.


---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

> I *think* you are assuming that both sides are friendly.  If one side is 
> a person, or group of people, then this is definitely not guaranteed.  
> I'll grant all your points if both sides are friendly, and each knows 
> the other to be friendly.  Otherwise I think things get messier.  So 
> objective measures and tests are desireable.

Most of the stuff in the post that you were replying to *did* assume that both 
sides are *declared* Friendly.

I'm trying to set it up so it is OBVIOUS to any UnFriendly entity that 
*declaring* Friendliness without actually being Friendly and then being 
declared UnFriendly has enough automatic penalties to offset any advantages 
such that it will almost *NEVER* be worthwhile (i.e. Friendliness has a 
*serious* vested interest in ensuring that declarations of Friendliness are 
genuine.  It will seriously encourage and attempt the (hopefully restrained but 
not constrained) pummeling and minor abuse of any offenders).

If a Friendly is dealing with an UnFriendly, then it will constantly attempt to 
convert the UnFriendly by 
  a.. calling attention to the fact that it *has* to raise the prices of it's 
services/interactions to cover protective measures (against the UnFriendly 
itself), the extra negotiating and verification costs necessitated by dealing 
with an untrustworthy UnFriendly, and any/all other similar additional expenses 
incurred by dealing with an UnFriendly; 
  b.. by threatening to go (or actually going) somewhere else because the cost 
of business is cheaper elsewhere after UnFriendly covereage costs are included 
and because Friendlies are *preferred customers/partners*; 
  c.. by constantly pestering the Friendly about how much better the 
UnFriendly's life would be if he were Friendly; etc., etc.
The two things which a Friendly would *NOT* do is allow an UnFriendly to take 
advantage of him *OR* be UnFriendly when it isn't necessary -- but if 
UnFriendliness *IS* necessary, the Friendly is *VERY* likely to have *A LOT* of 
assistance because all the other Friendlies know that it could be them in that 
situation and it is (all together now) in their long-term best interest to help.

= = = = = = = = = = 

"Play the game" by *assuming* that you are a Friendly and asking yourself what 
you would do to protect yourself without breaking your declaration of 
Friendliness.  It's fun and addictive and hopefully will lead you to declaring 
Friendliness yourself. 
(Yes, I really *am* serious about spreading Friendliness.  It's my own little, 
but hopefully growing, cult and I'm sticking to it.)

---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Charles D Hixson]

Mark Waser wrote:

...
= = = = = = = = = =
 
"Play the game" by *assuming* that you are a Friendly and asking 
yourself what you would do to protect yourself without breaking your 
declaration of Friendliness.  It's fun and addictive and hopefully 
will lead you to declaring Friendliness yourself.
(Yes, I really *am* serious about spreading Friendliness.  It's my own 
little, but hopefully growing, cult and I'm sticking to it.)
I think that you need to look into the simulations that have been run 
involving Evolutionarily Stable Strategies.  Friendly covers many 
strategies, including (I think) Dove and Retaliator.  Retaliator is 
almost an ESS, and becomes one if the rest of the population is either 
Hawk or Dove.  In a population of Doves, Probers have a high success 
rate, better than either Hawks or Doves.  If the population is largely 
Doves with an admixture of Hawks, Retaliators do well.  Etc.  (Note that 
each of these Strategies is successful depending on a model with certain 
costs of success an other costs for failure specific to the strategy.)  
Attempts to find a pure strategy that is uniformly successful have so 
far failed.  Mixed strategies, however, can be quite successful, and 
different environments yield different values for the optimal mix.  (The 
model that you are proposing looks almost like Retaliator, and that's a 
pretty good Strategy, but can be shown to be suboptimal against a 
variety of different mixed strategies.  Often even against 
Prober-Retaliator, if the environment contains sufficient Doves, though 
it's inferior if most of the population is simple Retaliators.)



---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mike Dougherty]

On Wed, Mar 12, 2008 at 8:54 PM, Charles D Hixson <
[EMAIL PROTECTED]> wrote:

> I think that you need to look into the simulations that have been run
> involving Evolutionarily Stable Strategies.  Friendly covers many
> strategies, including (I think) Dove and Retaliator.  Retaliator is
> almost an ESS, and becomes one if the rest of the population is either
> Hawk or Dove.  In a population of Doves, Probers have a high success
> rate, better than either Hawks or Doves.  If the population is largely
> Doves with an admixture of Hawks, Retaliators do well.  Etc.  (Note that
> each of these Strategies is successful depending on a model with certain
> costs of success an other costs for failure specific to the strategy.)
> Attempts to find a pure strategy that is uniformly successful have so
> far failed.  Mixed strategies, however, can be quite successful, and
> different environments yield different values for the optimal mix.  (The
> model that you are proposing looks almost like Retaliator, and that's a
> pretty good Strategy, but can be shown to be suboptimal against a
> variety of different mixed strategies.  Often even against
> Prober-Retaliator, if the environment contains sufficient Doves, though
> it's inferior if most of the population is simple Retaliators.)
>

I believe Mark's point is that the honest commitment to Friendly as an
explicit goal is an attempt to minimize wasted effort achieving all other
goals.  Exchanging information about goals with other Friendly agents helps
all parties invest optimally in achieving the goals in order of priority
acceptable to the consortium of Friendly.  I think one (of many) problems is
that our candidate AGI must not only be capable of self-reflection when
modeling its goals, but also capable of modeling the goals of other Friendly
agents (with respect to each other and to the goal-model of the collective)
as well as be able to decide when an UnFriendly behavior is worth declaring
(modeling the consequences and impact to the group of which it is a member)
That seems to be much more difficult than a selfish or ignorant Goal Stack
implementation (which we would typically attempt to control via an
imperative Friendly Goal)

---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Charles D Hixson]

Mike Dougherty wrote:
On Wed, Mar 12, 2008 at 8:54 PM, Charles D Hixson 
<[EMAIL PROTECTED] > wrote:


I think that you need to look into the simulations that have been run
involving Evolutionarily Stable Strategies.  Friendly covers many
strategies, including (I think) Dove and Retaliator.  Retaliator is
almost an ESS, and becomes one if the rest of the population is either
Hawk or Dove.  In a population of Doves, Probers have a high success
rate, better than either Hawks or Doves.  If the population is largely
Doves with an admixture of Hawks, Retaliators do well.  Etc.
 (Note that
each of these Strategies is successful depending on a model with
certain
costs of success an other costs for failure specific to the strategy.)
Attempts to find a pure strategy that is uniformly successful have so
far failed.  Mixed strategies, however, can be quite successful, and
different environments yield different values for the optimal mix.
 (The
model that you are proposing looks almost like Retaliator, and
that's a
pretty good Strategy, but can be shown to be suboptimal against a
variety of different mixed strategies.  Often even against
Prober-Retaliator, if the environment contains sufficient Doves,
though
it's inferior if most of the population is simple Retaliators.)


I believe Mark's point is that the honest commitment to Friendly as an 
explicit goal is an attempt to minimize wasted effort achieving all 
other goals.  Exchanging information about goals with other Friendly 
agents helps all parties invest optimally in achieving the goals in 
order of priority acceptable to the consortium of Friendly.  I think 
one (of many) problems is that our candidate AGI must not only be 
capable of self-reflection when modeling its goals, but also capable 
of modeling the goals of other Friendly agents (with respect to each 
other and to the goal-model of the collective) as well as be able to 
decide when an UnFriendly behavior is worth declaring (modeling the 
consequences and impact to the group of which it is a member)  That 
seems to be much more difficult than a selfish or ignorant Goal Stack 
implementation (which we would typically attempt to control via an 
imperative Friendly Goal)


And it's a very *good* strategy.  But it's not optimal except in certain 
constrained situations.   Note that all the strategies that I listed 
were VERY simple strategies.   Tit-for-tat was better than any of them, 
but it requires more memory and the remembered recognition of 
individuals.  As such it's more expensive to implement, so in some 
situations it looses out to Retaliator.  (Anything sophisticated enough 
to be even a narrow AI should be able to implement tit-for-tat, however, 
if it could handle the recognition of individuals.)  (Retaliator doesn't 
retain memory of individuals between encounters.  It's SIMPLE.)


Now admittedly the research on ESSs via simulations has focused on 
strategies that don't require any reasonable degree of intelligence.  
The simulator is needing to run large populations over large numbers of 
generations multiple times with slightly different assumptions.  As 
such, it doesn't speak directly to "What is a good strategy for an 
advanced AI with lots of resources?", but it provides indications.  
E.g., a population of Hawks does very poorly.  A population of Doves 
does well, but if it's infiltrated by a few Hawks, the Hawks soon come 
to dominate.  Etc.  And "Kill them All!!" is a very poor strategy unless 
there it is adopted by a single individual that is vastly stronger than 
any opposition that it might encounter.  (Even then it's not clearly a 
good strategy...except with certain specialized model conditions.  
Generally it will have a maximal size, and two "Kill them All!!"s would 
attempt to kill each other.  So the payoff for a win is much less than 
the payoff would be for a population even of Hawks.  [Hawks only 
initiate an attack if there are resources present that they have a use 
for.])


---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

And it's a very *good* strategy.  But it's not optimal except in certain 
constrained situations.   Note that all the strategies that I listed were 
VERY simple strategies.   Tit-for-tat was better than any of them, but it 
requires more memory and the remembered recognition of individuals.  As 
such it's more expensive to implement, so in some situations it looses out 
to Retaliator.  (Anything sophisticated enough to be even a narrow AI 
should be able to implement tit-for-tat, however, if it could handle the 
recognition of individuals.)  (Retaliator doesn't retain memory of 
individuals between encounters.  It's SIMPLE.)


So, in other words, even if Friendliness is *only* just intelligent 
tit-for-tat . . . . (which it is -- don't bug me and I won't bug you  :-)


Now admittedly the research on ESSs via simulations has focused on 
strategies that don't require any reasonable degree of intelligence.


Which pretty much invalidates it for AGI purposes as you say with  . . . .

The simulator is needing to run large populations over large numbers of 
generations multiple times with slightly different assumptions.  As such, 
it doesn't speak directly to "What is a good strategy for an advanced AI 
with lots of resources?", but it provides indications.


And I would argue that I've got a far better, more analogous study with 
several large populations over large numbers of generations.  It's called 
the study of human religion.:-)


And "Kill them All!!" is a very poor strategy unless there it is adopted 
by a single individual that is vastly stronger than any opposition that it 
might encounter.  (Even then it's not clearly a good strategy...except 
with certain specialized model conditions.


Thank you!  So why am I having to argue this with Vladimir?



---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Charles D Hixson]

Mark Waser wrote:

...
The simulator is needing to run large populations over large numbers 
of generations multiple times with slightly different assumptions.  
As such, it doesn't speak directly to "What is a good strategy for an 
advanced AI with lots of resources?", but it provides indications.


And I would argue that I've got a far better, more analogous study 
with several large populations over large numbers of generations.  
It's called the study of human religion.:-)


...

It's better in the sense of "more clearly analogous", but it's worse 
because 1) it's harder to analyze and 2) the results are *MUCH* more 
equivocal.  I'd argue that religion has caused more general suffering 
than it has ameliorated.  Probably by several orders of magnitude.  But 
the "results" are so messy and hard to separate from other simultaneous 
causes that this can't be conclusively proven.  (And, also, with 
sufficient desire to disbelieve, the law of gravity itself could be 
thrown into doubt.  [That's a paraphrase of somebody else talking about 
commercial interests, but the more general statement is correct.])


---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=98558129-0bdb63
Powered by Listbox: http://www.listbox.com

Re: [agi] Some thoughts of an AGI designer

[Mark Waser]

It's better in the sense of "more clearly analogous", but it's worse 
because 1) it's harder to analyze and 2) the results are *MUCH* more 
equivocal.  I'd argue that religion has caused more general suffering than 
it has ameliorated.  Probably by several orders of magnitude.


I agree.  Why do you think that my "religion" has a "you can't argue by 
using what you think God would want you to do" clause?


I've been doing some work on an article tentatively titled "The Scientific 
Basis and Logical Correctness of Belief and Religion".  I believe that I 
have a pretty good argument for why Belief In The Face of Rational 
Uncertainty is a logical, rational behavior and why the God Meme is the 
simplest meme that enforces ethics (aka Friendliness).  The problem is that 
the God meme does *not* allow for human error in the anticipation of what 
God wants -- and fighting over this point continues to plague us massively 
to this day.


But the "results" are so messy and hard to separate from other 
simultaneous causes that this can't be conclusively proven.  (And, also, 
with sufficient desire to disbelieve, the law of gravity itself could be 
thrown into doubt.  [That's a paraphrase of somebody else talking about 
commercial interests, but the more general statement is correct.])


Obviously.  The biggest problem with the current implementation of the God 
meme is that it is carried over boundary of Rational Uncertainty.  When your 
religion requires you to ignore truths in the face of compelling evidence, 
then believing in it is no longer rational.  Up until that point, however, 
belief can be and most frequently *IS* a logical, rational behavior.



---
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
http://www.listbox.com/member/?member_id=8660244&id_secret=98558129-0bdb63
Powered by Listbox: http://www.listbox.com