On 10/2/07, Vladimir Nesov <[EMAIL PROTECTED]> wrote: > On 10/2/07, Mark Waser <[EMAIL PROTECTED]> wrote: > > You misunderstood me -- when I said robustness of the goal system, I meant > > the contents and integrity of the goal system, not the particular > > implementation. > > I meant that too - and I didn't mean to imply this distinction. > Implementation of goal system, or 'goal system itself' are both in my > argument can be represented as text written in natural language, that > is in rather faulty way. 'Goal system as what it meant to be' is what > intelligent system tries to achieve. > > > > > I do however continue to object to your phrasing about the system > > recognizing influence on it's goal system and preserving it. Fundamentally, > > there are only a very small number of "Thou shalt not" supergoals that need > > to be forever invariant. Other than those, the system should be able to > > change it's goals as much as it likes (chocolate or strawberry? excellent > > food or mediocre sex? save starving children, save adults dying of disease, > > or go on vacation since I'm so damn tired from all my other good works?) > > This is just substitution of levels of abstraction. Programs on my PC > run on a fixed hardware and are limited by its capabilities, yet they > can vary greatly. Plus, intelligent system should be able to integrate > impact of goal system on multiple levels of abstraction, that is it > can infer level of strictness in various circumstances (which > interface with goal system through custom-made abstractions). > > > > A quick question for Richard and others -- Should adults be allowed to > > drink, do drugs, wirehead themselves to death? > > > > ----- Original Message ----- > > From: "Vladimir Nesov" <[EMAIL PROTECTED]> > > To: <agi@v2.listbox.com> > > Sent: Tuesday, October 02, 2007 9:49 AM > > Subject: **SPAM** Re: [agi] Religion-free technical content > > > > > > > But yet robustness of goal system itself is less important than > > > intelligence that allows system to recognize influence on its goal > > > system and preserve it. Intelligence also allows more robust > > > interpretation of goal system. Which is why the way particular goal > > > system is implemented is not very important. Problems lie in rough > > > formulation of what goal system should be (document in English is > > > probably going to be enough) and in placing the system under > > > sufficient influence of its goal system (so that intelligent processes > > > independent on it would not take over). > > > > > > On 10/2/07, Mark Waser <[EMAIL PROTECTED]> wrote: > > >> The intelligence and goal system should be robust enough that a single or > > >> small number of sources should not be able to alter the AGI's goals; > > >> however, it will not do this by recognizing "forged communications" but > > >> by > > >> realizing that the aberrant goals are not in congruence with the world. > > >> Note that many stupid and/or greedy people will try to influence the > > >> system > > >> and it will need to be immune to them (or the solution will be worse than > > >> the problem).
Argh! "Goal system" and "Friendliness" are roughly the same sort of confusion. They are each modelable only within a ***specified***, encompassing context. In more coherent, modelable terms, we express our evolving nature, rather than strive for "goals." - Jef ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=48954318-fd558f
