Ben, 

OK - so Novamente has a system for handling 'importance' already and there is an importance updating function that feeds back to other aspects of Attention Value.  That's good in terms of Novamente having an internal architecture capable of supporting and ethical system. 

> You're asking the AGI to solve the inverse problem: Find the concept
> that is consistent with these descriptions and associations, and then
> embody that concept in your own behavior.  I think this is a very hard
> learning problem

.... which presumably means that it will be put off until the AGI has the capacity to undertake the leaning process.  So why is this a problem?  

> and the AGI might will come up with something subtly but dangerously
> twisted.... I don't trust this 1/1000 at much as experience-based
> learning.

But it's not either/or - under the approach that I've suggested, Novamente would have an itch to learn about certain ethical concepts AND it would gain experience-based learning - so if experience-based learning is so good why wouldn't it help Novamente to handle it internal itch-driven learning without the subtle but dangerous twisting that you fear?  

And anyway why would your pure experience-based learning approach be any less likely to lead to subtly but dangerously warped ethical systems? The trainers could make errors and a Novamente's self-learning could be skewed by the limits of its experience and the modelling it observes. 

> Hmmmm.....  I am not certain, but I don't have a good feeling about it.
>  I think it's fine to stimulate things related to compassion and
> inhibit things opposed to it, but, I think this will be useful as a
> *guide* to an experience-based learning process, not as a primary means
> of stimulating the development of a sense of compassion.

Your approach to ethics seems to be based almost 100% on learning and you seem to thing that your own team will be training all the Novamentes before they leave the sandbox. How can you guarantee that your team will always be the trainers and the quality standards will always be maintained? 

For example, why couldn't someone outside your group get a copy of a Novamente and just strip out the learned data and then retrain the new copy of Novamente themselves? 

--- 

Getting back to your basic preferred concept of using expience-based learning to build a Novamente's ethical beliefs - this means that every Novamente has to start as a tabula rasa and in effect learn all the lessons of evolution all by itself. 

With anything less intelligent than a human-equivalent Novamente this would be a highly inefficient approach.  But with something as intelligent as human-equivalent Novamente this a hugely dangerous strategy. 

Given that ethics were not hard wired into early animals - you have to ask why this hard wiring eventually emerged.  My guess is that as animals became more powerful and potentially dangerous to their own kind it was only the ones with inbuilt ethics that could be activated soon after birth that were safe enough to survive and pass on their genes. 

In other word the lesson of evolution was that evolutionary recapitulation could not be relied on to get each animal to a point where it was safe for its fellows. 

---- 

Just as an aside, it seems that autism is a condition caused by problems with a human's pre-wired empathy system.  According to your preferred approach to GI training it should only be a matter of training human GI in ethics and empathy.  Why then does autism exist as a problem since 99% of autistic kids are put through a major training program by parents and others to get them to relate socially?  I simply can't see why a Novamente that is  without a modicum of ethical hardwiring will not end up being autisitic - no matter how good the training program you might give it. 

Why will your Novamentes not be autistic - despite the training regime that you intend? 

---- 

At this stage in the discussion on the AGI list I haven't heard anything to convinve me that a certain amount of ethical pre-wiring is certain to cause problems that are any greater than the problems that could be caused by NOT having a modicum of carefully designed ethical hardwiring. 

---- 

You have said many times that we need to suck it an see through experiment - that the theory of AGI psychological development is too underdeveloped because we don't know what we are dealing with. 

So why not proceed to develop Novamente's down two different paths simultaneously - the path you have already designed - where experience- based learning is virtually the only strategy, and a variant where some Novamentes have a modicum of carefully designed pre-wiring for ethics? 

Then you've got some experiential basis for comparing the two proposed strategies - and quick corrective action will be easier if one or other strategy shows signs of running into problems. 

And less negatively, there might be useful things learned from both strategies than can be cross fertilised even if major catastrophic problems don't appear to be emerging. 

Would you be prepared to give the dual (simultaneous) development path approach a go? 

Cheers, Philip

Reply via email to