[agi] Reward function vs utility

[Joshua Fox]

This has probably been discussed at length, so I will appreciate a reference
on this:

Why does Legg's definition of intelligence (following on Hutters' AIXI and
related work) involve a reward function rather than a utility function? For
this purpose, reward is a function of the word state/history which is
unknown to the agent while  a utility function is known to the agent.

Even if  we replace the former with the latter, we can still have a
definition of intelligence that integrates optimization capacity over
possible all utility functions.

What is the real  significance of the difference between the two types of
functions here?

Joshua



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Reward function vs utility

[Matt Mahoney]

The definition of universal intelligence being over all utility functions 
implies that the utility function is unknown. Otherwise there is a fixed 
solution.

 -- Matt Mahoney, matmaho...@yahoo.com





From: Joshua Fox 
To: agi 
Sent: Sun, June 27, 2010 4:22:19 PM
Subject: [agi] Reward function vs utility


This has probably been discussed at length, so I will appreciate a reference on 
this:

Why does Legg's definition of intelligence (following on Hutters' AIXI and 
related work) involve a reward function rather than a utility function? For 
this purpose, reward is a function of the word state/history which is unknown 
to the agent while  a utility function is known to the agent. 

Even if  we replace the former with the latter, we can still have a definition 
of intelligence that integrates optimization capacity over possible all utility 
functions. 

What is the real  significance of the difference between the two types of 
functions here?

Joshua
agi | Archives  | Modify Your Subscription  


---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Reward function vs utility

[Ben Goertzel]

You can always build the utility function into the assumed universal Turing
machine underlying the definition of algorithmic information...

I guess this will improve learning rate by some additive constant, in the
long run ;)

ben

On Sun, Jun 27, 2010 at 4:22 PM, Joshua Fox  wrote:

> This has probably been discussed at length, so I will appreciate a
> reference on this:
>
> Why does Legg's definition of intelligence (following on Hutters' AIXI and
> related work) involve a reward function rather than a utility function? For
> this purpose, reward is a function of the word state/history which is
> unknown to the agent while  a utility function is known to the agent.
>
> Even if  we replace the former with the latter, we can still have a
> definition of intelligence that integrates optimization capacity over
> possible all utility functions.
>
> What is the real  significance of the difference between the two types of
> functions here?
>
> Joshua
>*agi* | Archives 
>  | 
> ModifyYour Subscription
> 
>



-- 
Ben Goertzel, PhD
CEO, Novamente LLC and Biomind LLC
CTO, Genescient Corp
Vice Chairman, Humanity+
Advisor, Singularity University and Singularity Institute
External Research Professor, Xiamen University, China
b...@goertzel.org

"
“When nothing seems to help, I go look at a stonecutter hammering away at
his rock, perhaps a hundred times without as much as a crack showing in it.
Yet at the hundred and first blow it will split in two, and I know it was
not that blow that did it, but all that had gone before.”



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Reward function vs utility

[Joshua Fox]

I found the answer as given by Legg, *Machine Superintelligence*, p. 72,
copied below. A reward function is used to bypass potential difficulty in
communicating a utility function to the agent.

Joshua

The existence of a goal raises the problem of how the agent knows what the
goal is. One possibility would be for the goal to be known in advance and
for this knowledge to be built into the agent. The problem with this is that
it limits each agent to just one goal. We need to allow agents that are more
flexible, specifically, we need to be able to inform the agent of what the
goal
is. For humans this is easily done using language. In general however, the
possession of a suffciently high level of language is too strong an
assumption
to make about the agent. Indeed, even for something as intelligent as a dog
or a cat, direct explanation is not very effective.

Fortunately there is another possibility which is, in some sense, a blend of
the above two. We define an additional communication channel with the sim-
plest possible semantics: a signal that indicates how good the agent’s
current
situation is. We will call this signal the reward. The agent simply has to
maximise the amount of reward it receives, which is a function of the goal.
In
a complex setting the agent might be rewarded for winning a game or solving
a puzzle. If the agent is to succeed in its environment, that is, receive a
lot of
reward, it must learn about the structure of the environment and in
particular
what it needs to do in order to get reward.




On Mon, Jun 28, 2010 at 1:32 AM, Ben Goertzel  wrote:

> You can always build the utility function into the assumed universal Turing
> machine underlying the definition of algorithmic information...
>
> I guess this will improve learning rate by some additive constant, in the
> long run ;)
>
> ben
>
> On Sun, Jun 27, 2010 at 4:22 PM, Joshua Fox  wrote:
>
>> This has probably been discussed at length, so I will appreciate a
>> reference on this:
>>
>> Why does Legg's definition of intelligence (following on Hutters' AIXI and
>> related work) involve a reward function rather than a utility function? For
>> this purpose, reward is a function of the word state/history which is
>> unknown to the agent while  a utility function is known to the agent.
>>
>> Even if  we replace the former with the latter, we can still have a
>> definition of intelligence that integrates optimization capacity over
>> possible all utility functions.
>>
>> What is the real  significance of the difference between the two types of
>> functions here?
>>
>> Joshua
>>*agi* | Archives 
>>  | 
>> ModifyYour Subscription
>> 
>>
>
>
>
> --
> Ben Goertzel, PhD
> CEO, Novamente LLC and Biomind LLC
> CTO, Genescient Corp
> Vice Chairman, Humanity+
> Advisor, Singularity University and Singularity Institute
> External Research Professor, Xiamen University, China
> b...@goertzel.org
>
> "
> “When nothing seems to help, I go look at a stonecutter hammering away at
> his rock, perhaps a hundred times without as much as a crack showing in it.
> Yet at the hundred and first blow it will split in two, and I know it was
> not that blow that did it, but all that had gone before.”
>
>*agi* | Archives 
>  | 
> ModifyYour Subscription
> 
>



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Reward function vs utility

[Steve Richfield]

To all,

There may be a fundamental misdirection here on this thread, for your
consideration...

There have been some very rare cases where people have lost the use of one
hemisphere of their brains, and then subsequently recovered, usually with
the help of recently-developed clot-removal surgery. What they report seems
to be completely at odds with the present discussion. I will summarize and
probably overgeneralize, because there aren't many such survivors. One was a
brain researcher who subsequently wrote a book, about which I heard a review
on the radio, but I don't remember the details like title or name.
Hopefully, one of you has found and read this book.

It appears that one hemisphere is a *completely* passive observer, that does
*not* even bother to distinguish you and not-you, other than noting a
probable boundary. The other hemisphere concerns itself with manipulating
the world, regardless of whether particular pieces of it are you or not-you.
It seems unlikely that reward could have any effect at all on the passive
observer hemisphere.

In the case of the author of the book, apparently the manipulating
hemisphere was knocked out of commission for a while, and then slowly
recovered. This allowed her to see the passively observed world, without the
overlay of the manipulating hemisphere. Obviously, this involved severe
physical impairment until she recovered.

Note that AFAIK all of the AGI efforts are egocentric, while half of our
brains are concerned with passively filtering/understanding the world enough
to apply egocentric "logic". Note further that since the two hemispheres are
built from the same types of neurons, that the computations needed to do
these two very different tasks are performed by the same wet-stuff. There is
apparently some sort of advanced "Turing machine" sort of concept going on
in wetware.

This sounds to me like a must-read for any AGIer, and I certainly would have
read it, had I been one.

Hence, I see goal direction, reward, etc., as potentially useful only in
some tiny part of our brains.

Any thoughts?

Steve



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Reward function vs utility

[Joshua Fox]

Another point. I'm probably repeating the obvious, but perhaps this will be
useful to some.

On the one hand,  an agent could not game a Legg-like intelligence metric
by altering the utility function, even an internal one,, since the metric is
based on the function before any such change.

On the other hand, since an  internally-calculated utility function would
necessarily be a function of observations, rather than of actual world
state, it could be successfully gamed by altering observations.

This latter objection does not apply to functions which are externally
calculated, whether known or unknown.

Joshua



On Fri, Jul 2, 2010 at 7:23 PM, Joshua Fox  wrote:

> I found the answer as given by Legg, *Machine Superintelligence*, p. 72,
> copied below. A reward function is used to bypass potential difficulty in
> communicating a utility function to the agent.
>
> Joshua
>
> The existence of a goal raises the problem of how the agent knows what the
> goal is. One possibility would be for the goal to be known in advance and
> for this knowledge to be built into the agent. The problem with this is
> that
> it limits each agent to just one goal. We need to allow agents that are
> more
> flexible, specifically, we need to be able to inform the agent of what the
> goal
> is. For humans this is easily done using language. In general however, the
> possession of a suffciently high level of language is too strong an
> assumption
> to make about the agent. Indeed, even for something as intelligent as a dog
> or a cat, direct explanation is not very effective.
>
> Fortunately there is another possibility which is, in some sense, a blend
> of
> the above two. We define an additional communication channel with the sim-
> plest possible semantics: a signal that indicates how good the agent’s
> current
> situation is. We will call this signal the reward. The agent simply has to
> maximise the amount of reward it receives, which is a function of the goal.
> In
> a complex setting the agent might be rewarded for winning a game or solving
> a puzzle. If the agent is to succeed in its environment, that is, receive a
> lot of
> reward, it must learn about the structure of the environment and in
> particular
> what it needs to do in order to get reward.
>
>
>
>
> On Mon, Jun 28, 2010 at 1:32 AM, Ben Goertzel  wrote:
>
>> You can always build the utility function into the assumed universal
>> Turing machine underlying the definition of algorithmic information...
>>
>> I guess this will improve learning rate by some additive constant, in the
>> long run ;)
>>
>> ben
>>
>> On Sun, Jun 27, 2010 at 4:22 PM, Joshua Fox  wrote:
>>
>>> This has probably been discussed at length, so I will appreciate a
>>> reference on this:
>>>
>>> Why does Legg's definition of intelligence (following on Hutters' AIXI
>>> and related work) involve a reward function rather than a utility function?
>>> For this purpose, reward is a function of the word state/history which is
>>> unknown to the agent while  a utility function is known to the agent.
>>>
>>> Even if  we replace the former with the latter, we can still have a
>>> definition of intelligence that integrates optimization capacity over
>>> possible all utility functions.
>>>
>>> What is the real  significance of the difference between the two types of
>>> functions here?
>>>
>>> Joshua
>>>*agi* | Archives 
>>>  | 
>>> ModifyYour Subscription
>>> 
>>>
>>
>>
>>
>> --
>> Ben Goertzel, PhD
>> CEO, Novamente LLC and Biomind LLC
>> CTO, Genescient Corp
>> Vice Chairman, Humanity+
>> Advisor, Singularity University and Singularity Institute
>> External Research Professor, Xiamen University, China
>> b...@goertzel.org
>>
>> "
>> “When nothing seems to help, I go look at a stonecutter hammering away at
>> his rock, perhaps a hundred times without as much as a crack showing in it.
>> Yet at the hundred and first blow it will split in two, and I know it was
>> not that blow that did it, but all that had gone before.”
>>
>>*agi* | Archives 
>>  | 
>> ModifyYour Subscription
>> 
>>
>
>



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Reward function vs utility

[Abram Demski]

Joshua,

But couldn't it game the external utility function by taking actions which
modify it? For example, if the suggestion is taken literally and you have a
person deciding the reward at each moment, an AI would want to focus on
making that person *think* the reward should be high, rather than focusing
on actually doing well at whatever task it's set...and the two would tend to
diverge greatly for more and more complex/difficult tasks, since these tend
to be harder to judge. Furthermore, the AI would be very pleased to knock
the human out of the loop and push its own buttons. Similar comments would
apply to automated reward calculations.

--Abram

On Sun, Jul 4, 2010 at 4:40 AM, Joshua Fox  wrote:

> Another point. I'm probably repeating the obvious, but perhaps this will be
> useful to some.
>
> On the one hand,  an agent could not game a Legg-like intelligence metric
> by altering the utility function, even an internal one,, since the metric is
> based on the function before any such change.
>
> On the other hand, since an  internally-calculated utility function would
> necessarily be a function of observations, rather than of actual world
> state, it could be successfully gamed by altering observations.
>
> This latter objection does not apply to functions which are externally
> calculated, whether known or unknown.
>
> Joshua
>
>
>
> On Fri, Jul 2, 2010 at 7:23 PM, Joshua Fox  wrote:
>
>> I found the answer as given by Legg, *Machine Superintelligence*, p. 72,
>> copied below. A reward function is used to bypass potential difficulty in
>> communicating a utility function to the agent.
>>
>> Joshua
>>
>> The existence of a goal raises the problem of how the agent knows what the
>> goal is. One possibility would be for the goal to be known in advance and
>> for this knowledge to be built into the agent. The problem with this is
>> that
>> it limits each agent to just one goal. We need to allow agents that are
>> more
>> flexible, specifically, we need to be able to inform the agent of what the
>> goal
>> is. For humans this is easily done using language. In general however, the
>> possession of a suffciently high level of language is too strong an
>> assumption
>> to make about the agent. Indeed, even for something as intelligent as a
>> dog
>> or a cat, direct explanation is not very effective.
>>
>> Fortunately there is another possibility which is, in some sense, a blend
>> of
>> the above two. We define an additional communication channel with the sim-
>> plest possible semantics: a signal that indicates how good the agent’s
>> current
>> situation is. We will call this signal the reward. The agent simply has to
>> maximise the amount of reward it receives, which is a function of the
>> goal. In
>> a complex setting the agent might be rewarded for winning a game or
>> solving
>> a puzzle. If the agent is to succeed in its environment, that is, receive
>> a lot of
>> reward, it must learn about the structure of the environment and in
>> particular
>> what it needs to do in order to get reward.
>>
>>
>>
>>
>> On Mon, Jun 28, 2010 at 1:32 AM, Ben Goertzel  wrote:
>>
>>> You can always build the utility function into the assumed universal
>>> Turing machine underlying the definition of algorithmic information...
>>>
>>> I guess this will improve learning rate by some additive constant, in the
>>> long run ;)
>>>
>>> ben
>>>
>>> On Sun, Jun 27, 2010 at 4:22 PM, Joshua Fox wrote:
>>>
 This has probably been discussed at length, so I will appreciate a
 reference on this:

 Why does Legg's definition of intelligence (following on Hutters' AIXI
 and related work) involve a reward function rather than a utility function?
 For this purpose, reward is a function of the word state/history which is
 unknown to the agent while  a utility function is known to the agent.

 Even if  we replace the former with the latter, we can still have a
 definition of intelligence that integrates optimization capacity over
 possible all utility functions.

 What is the real  significance of the difference between the two types
 of functions here?

 Joshua
*agi* | Archives 
  | 
 ModifyYour Subscription
 

>>>
>>>
>>>
>>> --
>>> Ben Goertzel, PhD
>>> CEO, Novamente LLC and Biomind LLC
>>> CTO, Genescient Corp
>>> Vice Chairman, Humanity+
>>> Advisor, Singularity University and Singularity Institute
>>> External Research Professor, Xiamen University, China
>>> b...@goertzel.org
>>>
>>> "
>>> “When nothing seems to help, I go look at a stonecutter hammering away at
>>> his rock, perhaps a hundred times without as much as a crack showing in it.
>>> Yet at the hundred and first blow it will split in two, and I know it was
>>> not that blow that did it, but all that had gone before.”
>>>
>>> 

Re: [agi] Reward function vs utility

[Matt Mahoney]

Perhaps we now have a better understanding of the risks of uploading to a form 
where we could modify our own software. We already do this to some extent using 
drugs. Evolution will eliminate such failures.

 -- Matt Mahoney, matmaho...@yahoo.com





From: Abram Demski 
To: agi 
Sent: Sun, July 4, 2010 11:43:46 AM
Subject: Re: [agi] Reward function vs utility

Joshua,

But couldn't it game the external utility function by taking actions which 
modify it? For example, if the suggestion is taken literally and you have a 
person deciding the reward at each moment, an AI would want to focus on making 
that person *think* the reward should be high, rather than focusing on actually 
doing well at whatever task it's set...and the two would tend to diverge 
greatly for more and more complex/difficult tasks, since these tend to be 
harder to judge. Furthermore, the AI would be very pleased to knock the human 
out of the loop and push its own buttons. Similar comments would apply to 
automated reward calculations.

--Abram


On Sun, Jul 4, 2010 at 4:40 AM, Joshua Fox  wrote:

Another point. I'm probably repeating the obvious, but perhaps this will be 
useful to some.
>
>
>On the one hand,  an agent could not game a Legg-like intelligence metric by 
>altering the utility function, even an internal one,, since the metric is 
>based on the function before any such change.
>
>
>On the other hand, since an  internally-calculated utility function would 
>necessarily be a function of observations, rather than of actual world state, 
>it could be successfully gamed by altering observations.  
>
>
>This latter objection does not apply to functions which are externally 
>calculated, whether known or unknown.
>
>Joshua
>
>
>
>
>
>
>>
>On Fri, Jul 2, 2010 at 7:23 PM, Joshua Fox  wrote:
>
>>>
>>
>>I found the answer as given by Legg, Machine Superintelligence, p. 72, copied 
>>below. A reward function is used to bypass potential difficulty in 
>>communicating a utility function to the agent.
>>
>>
>>Joshua
>>
>>
>>
>>The existence of a goal raises the problem of how the agent knows what the
>>goal is. One possibility would be for the goal to be known in advance and
>>for this knowledge to be built into the agent. The problem with this is that
>>it limits each agent to just one goal. We need to allow agents that are more
>>flexible, specifically, we need to be able to inform the agent of what the 
>>goal
>>is. For humans this is easily done using language. In general however, the
>>possession of a suffciently high level of language is too strong an assumption
>>to make about the agent. Indeed, even for something as intelligent as a dog
>>or a cat, direct explanation is not very effective.
>>
>>
>>Fortunately there is another possibility which is, in some sense, a blend of
>>the above two. We define an additional communication channel with the sim-
>>plest possible semantics: a signal that indicates how good the agent’s current
>>situation is. We will call this signal the reward. The agent simply has to
>>maximise the amount of reward it receives, which is a function of the goal. In
>>a complex setting the agent might be rewarded for winning a game or solving
>>a puzzle. If the agent is to succeed in its environment, that is, receive a 
>>lot of
>>reward, it must learn about the structure of the environment and in particular
>>what it needs to do in order to get reward.
>>
>>
>>
>>
>>
>>
>>
>>On Mon, Jun 28, 2010 at 1:32 AM, Ben Goertzel  wrote:
>>
>>>>>You can always build the utility function into the assumed universal 
>>>>>Turing machine underlying the definition of algorithmic information...
>>>
>>>I guess this will improve learning rate by some additive constant, in the 
>>>long run ;)
>>>
>>>ben
>>>
>>>
>>>On Sun, Jun 27, 2010 at 4:22 PM, Joshua Fox  wrote:
>>>
>>>This has probably been discussed at length, so I will appreciate a reference 
>>>on this:
>>>>
>>>>
>>>>Why does Legg's definition of intelligence (following on Hutters' AIXI and 
>>>>related work) involve a reward function rather than a utility function? For 
>>>>this purpose, reward is a function of the word state/history which is 
>>>>unknown to the agent while  a utility function is known to the agent. 
>>>>
>>>>
>>>>Even if  we replace the former with the latter, we can still have a 
>>>>definition of intelligence that integra

Re: [agi] Reward function vs utility

[Ian Parker]

No it would not. AI willk "press its own buttons" only if those buttons are
defined. In one sense you can say that Goedel's theorem is a proof of
friendliness as it means that there must always be one button that AI cannot
press.


  - Ian Parker

On 4 July 2010 16:43, Abram Demski  wrote:

> Joshua,
>
> But couldn't it game the external utility function by taking actions which
> modify it? For example, if the suggestion is taken literally and you have a
> person deciding the reward at each moment, an AI would want to focus on
> making that person *think* the reward should be high, rather than focusing
> on actually doing well at whatever task it's set...and the two would tend to
> diverge greatly for more and more complex/difficult tasks, since these tend
> to be harder to judge. Furthermore, the AI would be very pleased to knock
> the human out of the loop and push its own buttons. Similar comments would
> apply to automated reward calculations.
>
> --Abram
>
>
>
>   *agi* | Archives 
>  | 
> ModifyYour Subscription
> 
>



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Reward function vs utility

[Jim Bromer]

On Fri, Jul 2, 2010 at 2:35 PM, Steve Richfield
wrote:

> It appears that one hemisphere is a *completely* passive observer, that
> does *not* even bother to distinguish you and not-you, other than noting a
> probable boundary. The other hemisphere concerns itself with manipulating
> the world, regardless of whether particular pieces of it are you or not-you.
> It seems unlikely that reward could have any effect at all on the passive
> observer hemisphere.
>
> In the case of the author of the book, apparently the manipulating
> hemisphere was knocked out of commission for a while, and then slowly
> recovered. This allowed her to see the passively observed world, without the
> overlay of the manipulating hemisphere. Obviously, this involved severe
> physical impairment until she recovered.
>
> Note that AFAIK all of the AGI efforts are egocentric, while half of our
> brains are concerned with passively filtering/understanding the world enough
> to apply egocentric "logic". Note further that since the two hemispheres are
> built from the same types of neurons, that the computations needed to do
> these two very different tasks are performed by the same wet-stuff. There is
> apparently some sort of advanced "Turing machine" sort of concept going on
> in wetware.
> Hence, I see goal direction, reward, etc., as potentially useful only in
> some tiny part of our brains.
>
> Any thoughts?


I don't buy the hemisphere disconnect, but I do feel that it makes sense to
say that some parts are (like) passive observers and other parts are more
concerned with the interactive aspects of reasoning.  The idea that
reinforcement might operate on the interactive aspects but not the passive
observers is really interesting.  My only criticism is that there is
evidence that human beings will often interpret events according to the
projections of their primary concerns onto their observations.
Jim Bromer



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Reward function vs utility

[Joshua Fox]

Abram,

Good point. But I am ignoring the implementation of the  utility/reward
function , and treating it as a Platonic  mathematical function of
world-state or observations which cannot be changed without reducing the
total utility/reward. You are quite right that when we do bring
implementation into account, as one must in the real world,
the implementation (e.g., the person you mentioned) can be gamed.

Even the pure mathematical function, however, can be gamed if you can alter
its inputs "unfairly", as in the example I gave of altering observations to
optimize a function of the observations.

Regards,

Joshua

On Sun, Jul 4, 2010 at 6:43 PM, Abram Demski  wrote:

> Joshua,
>
> But couldn't it game the external utility function by taking actions which
> modify it? For example, if the suggestion is taken literally and you have a
> person deciding the reward at each moment, an AI would want to focus on
> making that person *think* the reward should be high, rather than focusing
> on actually doing well at whatever task it's set...and the two would tend to
> diverge greatly for more and more complex/difficult tasks, since these tend
> to be harder to judge. Furthermore, the AI would be very pleased to knock
> the human out of the loop and push its own buttons. Similar comments would
> apply to automated reward calculations.
>
> --Abram
>
>
> On Sun, Jul 4, 2010 at 4:40 AM, Joshua Fox  wrote:
>
>> Another point. I'm probably repeating the obvious, but perhaps this will
>> be useful to some.
>>
>> On the one hand,  an agent could not game a Legg-like intelligence metric
>> by altering the utility function, even an internal one,, since the metric is
>> based on the function before any such change.
>>
>> On the other hand, since an  internally-calculated utility function would
>> necessarily be a function of observations, rather than of actual world
>> state, it could be successfully gamed by altering observations.
>>
>> This latter objection does not apply to functions which are externally
>> calculated, whether known or unknown.
>>
>> Joshua
>>
>>
>>
>> On Fri, Jul 2, 2010 at 7:23 PM, Joshua Fox  wrote:
>>
>>> I found the answer as given by Legg, *Machine Superintelligence*, p. 72,
>>> copied below. A reward function is used to bypass potential difficulty in
>>> communicating a utility function to the agent.
>>>
>>> Joshua
>>>
>>> The existence of a goal raises the problem of how the agent knows what
>>> the
>>> goal is. One possibility would be for the goal to be known in advance and
>>> for this knowledge to be built into the agent. The problem with this is
>>> that
>>> it limits each agent to just one goal. We need to allow agents that are
>>> more
>>> flexible, specifically, we need to be able to inform the agent of what
>>> the goal
>>> is. For humans this is easily done using language. In general however,
>>> the
>>> possession of a suffciently high level of language is too strong an
>>> assumption
>>> to make about the agent. Indeed, even for something as intelligent as a
>>> dog
>>> or a cat, direct explanation is not very effective.
>>>
>>> Fortunately there is another possibility which is, in some sense, a blend
>>> of
>>> the above two. We define an additional communication channel with the
>>> sim-
>>> plest possible semantics: a signal that indicates how good the agent’s
>>> current
>>> situation is. We will call this signal the reward. The agent simply has
>>> to
>>> maximise the amount of reward it receives, which is a function of the
>>> goal. In
>>> a complex setting the agent might be rewarded for winning a game or
>>> solving
>>> a puzzle. If the agent is to succeed in its environment, that is, receive
>>> a lot of
>>> reward, it must learn about the structure of the environment and in
>>> particular
>>> what it needs to do in order to get reward.
>>>
>>>
>>>
>>>
>>> On Mon, Jun 28, 2010 at 1:32 AM, Ben Goertzel  wrote:
>>>
 You can always build the utility function into the assumed universal
 Turing machine underlying the definition of algorithmic information...

 I guess this will improve learning rate by some additive constant, in
 the long run ;)

 ben

 On Sun, Jun 27, 2010 at 4:22 PM, Joshua Fox wrote:

> This has probably been discussed at length, so I will appreciate a
> reference on this:
>
> Why does Legg's definition of intelligence (following on Hutters' AIXI
> and related work) involve a reward function rather than a utility 
> function?
> For this purpose, reward is a function of the word state/history which is
> unknown to the agent while  a utility function is known to the agent.
>
> Even if  we replace the former with the latter, we can still have a
> definition of intelligence that integrates optimization capacity over
> possible all utility functions.
>
> What is the real  significance of the difference between the two types
> of functions here?
>
> Joshua
>  

Re: [agi] Reward function vs utility

[Abram Demski]

Joshua,

Fortunately, this is not that hard to fix by abandoning the idea of a reward
function and going back to a normal utility function... I am working on a
paper on how to do that.

--Abram

On Mon, Jul 5, 2010 at 9:43 AM, Joshua Fox  wrote:

> Abram,
>
> Good point. But I am ignoring the implementation of the  utility/reward
> function , and treating it as a Platonic  mathematical function of
> world-state or observations which cannot be changed without reducing the
> total utility/reward. You are quite right that when we do bring
> implementation into account, as one must in the real world,
> the implementation (e.g., the person you mentioned) can be gamed.
>
> Even the pure mathematical function, however, can be gamed if you can alter
> its inputs "unfairly", as in the example I gave of altering observations to
> optimize a function of the observations.
>
> Regards,
>
> Joshua
>
> On Sun, Jul 4, 2010 at 6:43 PM, Abram Demski wrote:
>
>> Joshua,
>>
>> But couldn't it game the external utility function by taking actions which
>> modify it? For example, if the suggestion is taken literally and you have a
>> person deciding the reward at each moment, an AI would want to focus on
>> making that person *think* the reward should be high, rather than focusing
>> on actually doing well at whatever task it's set...and the two would tend to
>> diverge greatly for more and more complex/difficult tasks, since these tend
>> to be harder to judge. Furthermore, the AI would be very pleased to knock
>> the human out of the loop and push its own buttons. Similar comments would
>> apply to automated reward calculations.
>>
>> --Abram
>>
>>
>> On Sun, Jul 4, 2010 at 4:40 AM, Joshua Fox  wrote:
>>
>>> Another point. I'm probably repeating the obvious, but perhaps this will
>>> be useful to some.
>>>
>>> On the one hand,  an agent could not game a Legg-like intelligence metric
>>> by altering the utility function, even an internal one,, since the metric is
>>> based on the function before any such change.
>>>
>>> On the other hand, since an  internally-calculated utility function would
>>> necessarily be a function of observations, rather than of actual world
>>> state, it could be successfully gamed by altering observations.
>>>
>>> This latter objection does not apply to functions which are externally
>>> calculated, whether known or unknown.
>>>
>>> Joshua
>>>
>>>
>>>
>>> On Fri, Jul 2, 2010 at 7:23 PM, Joshua Fox  wrote:
>>>
 I found the answer as given by Legg, *Machine Superintelligence*, p.
 72, copied below. A reward function is used to bypass potential difficulty
 in communicating a utility function to the agent.

 Joshua

 The existence of a goal raises the problem of how the agent knows what
 the
 goal is. One possibility would be for the goal to be known in advance
 and
 for this knowledge to be built into the agent. The problem with this is
 that
 it limits each agent to just one goal. We need to allow agents that are
 more
 flexible, specifically, we need to be able to inform the agent of what
 the goal
 is. For humans this is easily done using language. In general however,
 the
 possession of a suffciently high level of language is too strong an
 assumption
 to make about the agent. Indeed, even for something as intelligent as a
 dog
 or a cat, direct explanation is not very effective.

 Fortunately there is another possibility which is, in some sense, a
 blend of
 the above two. We define an additional communication channel with the
 sim-
 plest possible semantics: a signal that indicates how good the agent’s
 current
 situation is. We will call this signal the reward. The agent simply has
 to
 maximise the amount of reward it receives, which is a function of the
 goal. In
 a complex setting the agent might be rewarded for winning a game or
 solving
 a puzzle. If the agent is to succeed in its environment, that is,
 receive a lot of
 reward, it must learn about the structure of the environment and in
 particular
 what it needs to do in order to get reward.




 On Mon, Jun 28, 2010 at 1:32 AM, Ben Goertzel  wrote:

> You can always build the utility function into the assumed universal
> Turing machine underlying the definition of algorithmic information...
>
> I guess this will improve learning rate by some additive constant, in
> the long run ;)
>
> ben
>
> On Sun, Jun 27, 2010 at 4:22 PM, Joshua Fox wrote:
>
>> This has probably been discussed at length, so I will appreciate a
>> reference on this:
>>
>> Why does Legg's definition of intelligence (following on Hutters' AIXI
>> and related work) involve a reward function rather than a utility 
>> function?
>> For this purpose, reward is a function of the word state/history which is
>> unknown to the

Re: [agi] Reward function vs utility

[Abram Demski]

Ian,

The reward button *would* be amoung the well-defined ones, though... sounds
to me like you are just abusing Goedel's theorem. Can you give a more
detailed argument?

--Abram

On Sun, Jul 4, 2010 at 4:47 PM, Ian Parker  wrote:

>
>
> No it would not. AI willk "press its own buttons" only if those buttons are
> defined. In one sense you can say that Goedel's theorem is a proof of
> friendliness as it means that there must always be one button that AI cannot
> press.
>
>
>   - Ian Parker
>
> On 4 July 2010 16:43, Abram Demski  wrote:
>
>> Joshua,
>>
>> But couldn't it game the external utility function by taking actions which
>> modify it? For example, if the suggestion is taken literally and you have a
>> person deciding the reward at each moment, an AI would want to focus on
>> making that person *think* the reward should be high, rather than focusing
>> on actually doing well at whatever task it's set...and the two would tend to
>> diverge greatly for more and more complex/difficult tasks, since these tend
>> to be harder to judge. Furthermore, the AI would be very pleased to knock
>> the human out of the loop and push its own buttons. Similar comments would
>> apply to automated reward calculations.
>>
>> --Abram
>>
>>
>>
>>   *agi* | Archives 
>>  | 
>> ModifyYour Subscription
>> 
>>
>
>*agi* | Archives 
>  | 
> ModifyYour Subscription
> 
>



-- 
Abram Demski
http://lo-tho.blogspot.com/
http://groups.google.com/group/one-logic



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com