Chuck,
Regarding "AGI tests", this is something I've thought about a bit because
some people from the nonprofit world have told me they felt it would be
relatively easy to raise money for some kind of AGI Prize, similar to the
Hutter Prize or the Methuselah Mouse Prize.
However, I thought about it a bit and failed to achieve my usual level
of brilliant insight ;-)
But I will summarize a couple of my non-brilliant, hopefully non-completely-
stupid ideas on the theme, in this email.
A preliminary note: One issue is
that all the tests one naturally uses in the course of AGI engineering, are
simply too easy to "game." For instance, we spent months getting Novamente
to learn to play fetch, but of course there are way easier ways to do that
than building a Novamente system.
As a good example of tests that are useful if one is sincerely pursuing
AGI, but not useful if one is willing to "game" the tests, see the Piagetan
psychology based tests in this book
http://www.amazon.com/Computational-Developmental-Psychology-Bradford-Books/dp/026219483X/ref=sr_1_1/102-9099924-7398536?ie=UTF8&s=books&qid=1173875017&sr=8-1
"Computational Developmental Psychology" by Shultz.
He describes a bunch of nicely computer-ized versions of Piagetan
cognitive tests used to assess childrens' developmental level. Then,
he proceeds to show how machine learning algorithms can solve these
problems in an interesting but narrow-AI-ish way. He shows that his
cascade-correlational neural nets do a better job of this than decision
trees and some other algorithms.... But, clearly, his algorithms are not
solving the tasks in any way related to how a general intelligence
system would do so.
Now: A related test that I thought of is as follows. Suppose one made
a really nice user interface for a 3D simulation world, so that both
-- human children
-- AIs
could easily control the same humanoid "child agent" in the sim world.
This isn't so hard to envision, since many human children are pretty
good with video games.
There would also be a humanoid "teacher agent" in the sim world.
The teacher agent would then give the child agent a number of tests,
modeled on standard Piagetan (and more modern) tests aimed at
understanding the system's cognitive level.
If we want to partly work around the language issue, we could have a variant
experimental scenario in which the teacher agent talks and the child
agent doesn't talk, but only carries out physical actions.
The instructions from the teacher agent could be given in text, which
could then be speech-synthesized for human children to see.
We could then have real child psychologists play the role of the
teacher agent....
The goal would not be a Turing test goal of seeing if the AI can
imitate a human, but rather to see if the child psychologists think the
AI has mastered a certain cognitive level or not.
Child psychologists are used to dealing with kids with autism and
other cognitive peculiarities, so dealing with a baby AI might not
be so far out-there.
Another, different sort of idea is the Textual Entailment
Challenge, see
http://www.pascal-network.org/Challenges/RTE2
I have thought of a Situational Entailment Challenge, in which
you interact with an AI in a certain manner in a sim world, and
then ask it binary questions about the situation in which it's
been interacting. This tests whether the AI understands the
situations it's involved in. Of course, the set of situations and
questions would need to be carefully chosen.
Anyway those are my thoughts on the topic so far. Better
ideas would be appreciated...
-- Ben
Chuck Esterbrook wrote:
There's a good chance this topic has been discussed before, so feel
free to point the way if that's the case. It's certainly been touched
on since I joined the list, but I wanted to break it out for its own
sake of discussion.
Background:
There is a contest that implements the Turing Test for AI called the
Loebner Prize:
http://www.loebner.net/Prizef/loebner-prize.html
http://en.wikipedia.org/wiki/Loebner_prize
It has plenty of controversy and seems to be shunned by most of the AI
community. See:
http://dir.salon.com/story/tech/feature/2003/02/26/loebner_part_one/index.html
http://dir.salon.com/story/tech/feature/2003/02/27/loebner_part_2/index.html
And briefly discussed on this list here:
http://www.mail-archive.com/agi@v2.listbox.com/msg01155.html
On the one hand, I respect that someone set up a contest modeled after
the Turing Test. On the other hand, there are legitimate criticisms of
it. In particular, an AI may be an AGI but not be capable of passing
the Turing Test due to having subhuman processing power and/or a lack
of human experience and/or primitive language skills. Not recognizing
such an AGI would be unfortunate since having a "baby" AGI would be
pretty damn exciting!
Also, there are aspects of AI that would be interesting to test such
as "getting along" in a spatial+temporal environment. The Turing Test
can only cover these things indirectly.
Still the Turing Test *is* a neat invention and it will certainly be
interesting to see what year it is passed.
But the real topic is this:
*** Can we create an analogous test for identifying Baby AGI? ***
In an email today, Josh mentioned "demonstrate adaptiveness,
robustness, learning, and reflective control". Ben speaks of
"achieving complex goals in complex environments." I like these
definitions and I think the test (or series of tests) would be looking
for things like that.
So more specifically:
Can we set up a series of tests that when passed would yield a high
probability that an AI is a baby AGI?
Is someone already working on this?
Is this worth working on now or are we so far off that it would be
premature?
For those that are implementing AGI, to what extent have you already
set up such tests for your system? And what form do those tests take?
-Chuck
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?list_id=303
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?list_id=303
