***
And anyway why would your pure
experience-based learning approach be any less likely to lead to subtly but
dangerously warped ethical systems? The trainers could make errors and a
Novamente's self-learning could be skewed by the limits of its experience and
the modelling it observes.
***
I don't have a good argument on this point, just an
intuition, based on the fact that generally speaking in narrow AI, inductive
learning based rules based on a very broad range of experience, are much more
robust than expert-encoded rules. The key is a broad range of experience,
otherwise inductive learning can indeed lead to rules that are "overfit" to
their training situations and don't generalize well to fundamentally novel
situations.
A key form of ethical testing will be to give the
system situations very different from those it has ever seen in the past, and
see how well it generalizes its ethical knowledge to these
domains.
***
For example, why couldn't someone outside
your group get a copy of a Novamente and just strip out the learned data and
then retrain the new copy of Novamente themselves?
***
That danger exists no matter what
method you use. If you use expert rules for ethics, someone could steal a
copy and fill it up with expert rules for evil
instead...
***
Getting back to your basic preferred
concept of using expience-based learning to build a Novamente's ethical beliefs
- this means that every Novamente has to start as a tabula rasa and in effect
learn all the lessons of evolution all by itself.
***
If there need to be multiple Novamentes (not clear --
one might be enough), they could be produced through "cloning" rather than
raising each one from scratch.
The engineering/teaching of ethics in an AI system is
pretty different from its evolution in natural
systems...
-- Ben G
