Kaj Sotala wrote:
On 1/29/08, Richard Loosemore <[EMAIL PROTECTED]> wrote:
Summary of the difference:
1) I am not even convinced that an AI driven by a GS will ever actually
become generally intelligent, because of the self-contrdictions built
into the idea of a goal stack. I am fairly sure that whenever anyone
tries to scale one of those things up to a real AGI (something that has
never been done, not by a long way) the AGI will become so unstable that
it will be an idiot.
2) A motivation-system AGI would have a completely different set of
properties, and among those properties would be extreme stability. It
would be possible to ensure that the thing stayed locked on to a goal
set that was human-empathic, and which would stay that way.
Omohundros's analysis is all predicated on the Goal Stack approach, so
my response is that nothing he says has any relevance to the type of AGI
that I talk about (which, as I say, is probably going to be the only
type ever created).
Hmm. I'm not sure of exact definition that you're using of the term
"motivational AGI", so let me wager a guess based on what I remember
reading from you before - do you mean something along the lines of a
system built out of several subsystems, each with partially
conflicting desires, that are constantly competing for control and
exerting various kinds of "pull" to the behavior of the system as a
whole? And you contrast this with a goal stack AGI, which would only
have one or a couple of such systems?
While this is certainly a major difference on the architectural level,
I'm not entirely convinced how large of a difference it makes in
behavioral terms, at least in this context. In order to accomplish
anything, the motivational AGI would still have to formulate goals and
long-term plans. Once it managed to hammer out acceptable goals that
the majority of its subsystems agreed on, it would set out on
developing ways to fulfill those goals as effectively as possible,
making it subject to the pressures outlined in Omohundro's paper.
The utility function that it would model for itself would be
considerably more complex than for an AGI with less subsystems, as it
would have to be a compromise between the desires of each subsystem
"in power", and if the balance of power would be upset too radically,
the modeled utility function may even be changed entirely (like the
way different moods in humans give control to different networks,
altering the current desires and effective utility functions).
However, AGI designers likely wouldn't make the balance of power
between the different subsystems /too/ unstable, as an agent that
constantly changed its mind about what it wanted would just go around
in circles. So it sounds plausible that the utility function it
generated would remain relatively stable, and the motivational AGI's
behavior optimized just as Omohundro analysis suggests.
I obviously need to get more detail of this idea down in a published
paper, but in the mean time let me try to give a quick feel for what I mean.
AGIs need *two* types of control system. One of these certainly does
look like a conventional goal stack: this is the "planner" that handles
everyday priorities and missions. There would probably be some strong
differences from a conventional GS, but these we can talk about another
time. For brevity, I'll just call this the "GS".
The second component would be the Motivational-Emotional System (MES).
To understand the difference between this and the GS, try to imagine
something like a Boltzmann Machine or a backpropagation neural net that
has been designed in such a way as to implement a kind-of regular
symbolic AI system. This would be a horrible, ugly hybrid (as you can
imagine), but you could probably see how it could be done in principle,
and it serves as a good way for me to get my point across.
Now imagine that the neural net level of this system was built with some
"modulating" parameters that biassed the functioning of the neurons, but
where these parameters are not global, but rather are local to the
neurons (so we could talk about a vector field of these parameters
across the net).
The purpose of these parameters is to bias the behavior of the neurons,
so that if one parameter goes high in one area of the net, the firing of
those neurons is elevated, and the functioning of the system is somehow
enhanced in that area.
What *exactly* is the effect of the vector field on the behavior of the
system? Well, that is not easy to say, because the field has a diffuse
effect at the symbol level - there is no one-to-one correspondence
between particular symbols and he field values. Instead what you get is
a soft change in the functioning of the system. Without getting into
details, I am sure you can see how, in general, such a thing could be
possible.
Now one more idea: the field itself is locally connected, so it is
almost as if there is a complete parallel universe of neurons lying
underneath the real neurons, not controlling the "activation level" of
the neurons, but instead controlling the changes in the vector field.
Once again, let's leave the details out and just say that the field is
dynamically changing in both fast-time (the way that regular neural
activation changes in with a fast time constant), and also in slow-time
(the way that connection strengths generally change with a slow time
constant).
How exactly is the vector-field system changing? Well, not in the same
way as the neural system, for sure. Two things: the fast-time
component is changing according to the dictates of the things that lie
at the root of the Motivational Emotional System: a set of modules that
represent the system's basic drives (which is what I think you meant
when you talked of my "modules" above). Simple examples: if the
system's [hunger] drive is becoming strongly activated, this causes one
set of (fast-time) changes in the vector field. Similar for an
elevation in the system's sex drive, or curiosity, or feelings of
empathy. Each causes rapid changes in the field, which immediately
propagate across the system. (Do they propagate *uniformly* across the
field? No! But the reason why comes in the next paragraph...).
The other type of change is slower, and this has to do with the way that
concepts develop in the system. Remember, this is supposed to be a real
AGI, so we do not fill it with all of its adult knowledge by hand, we
get it to develop its own knowledge from childhood. The system starts
off with some very primitive ideas about the world, each of which is
rooted in one of those motivation modules. As time goes on more
sophisticated concepts are built on top of those primitive ones, but
what happens during this process is that the connections in the vector
field system (this "parallel universe" of neurons that govern the vector
field) are strongly connected as if they were each parts of trees rooted
in each of the primitives. As a result, when one of the MES modules
fires strongly in the adult system, this causes a systematic bias in a
particular set of concepts that all can be traced back, developmentally,
to the original primitives. The complicated structure of this tree of
connections (different trees for each of the MES modules) determines
what effect the field has on the overall system. This is very much a
complex system: you will not be able to write down a closed-form
specification for how the MES affects the working system.
I have left out a good deal from this quick sketch. Most notably, the
fact that the "concepts" or "symbols" are not just passive things that
encode things in the world. The way i have sketched it, you might think
that the MES simply biasses the system to think certain thoughts , so it
just sits there and tends to daydream different sets of ideas depending
on which MES module is dominant. In fact, the "symbols" most affected
by the MES are different, and closely allied with the previously
mentioned GS, so the effect of the MES is not just to bias the daydreams
but also to bias the goal priorities and "values". In fact, the state
of the MES field is what the deeper part of the "value system" actually is.
Okay, sorry to hit you with incomprehensible technical detail, but maybe
there is a chance that my garbled version of the real picture will
strike a chord.
The message to take home from all of this is that:
1) There are *huge* differences between the way that a system would
behave if it had a single GS, or even a group of conflicting GS modules
(which is the way you interpreted my proposal, above) and the kind of
MES system I just described: the difference would come from the type of
influence exerted, because the vector field is operating on a completely
different level than the symbl processing.
2) The effect of the MES is to bias the system, but this "bias" amounts
to the following system imperative: [Make your goals consistent with
this *massive* set of constraints] .... where the "massive set of
constraints" is a set of ideas built up throughout the entire
development of the system. Rephrasing that in terms of an example: if
the system gets an idea that it should take a certain course of action
because it seems to satisfy an immediate goal, the implications of that
action will be quickly checked against a vast range o constraints, and
if there is any hint of an inconsistency with teh value system, this
will "pull" the thoughts of the AGI toward that issue, whereupon it will
start to elaborate the issue in more detail and try to impose an even
wider net of constraits, finally making a decision based on the broadest
possible set of considerations. This takes care of all the dumb
examples where people suggest that an AGI could start with the goal
"Increase global happiness" and then finally decide that this would be
accomplished by tiling the universe with smiley faces. Another way to
say this: there is no such thing as a single "utility function" in this
type of system, nor is there a small set of utility functions .... there
is a massive-dimensional set of utility functions (as many as there are
concepts or connections in the system), and this "diffuse" utility
function is what gives the system its stability.
So that, in the end, is why Omohundro's paper initially struck me as
invalid: any talk of a "utility function" simply does not work in the
type of system I have in mind.
Richard Loosemore.
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=8660244&id_secret=91103577-9ab385
