On Jan 29, 2008 6:52 PM, Richard Loosemore <[EMAIL PROTECTED]> wrote: > Okay, sorry to hit you with incomprehensible technical detail, but maybe > there is a chance that my garbled version of the real picture will > strike a chord. > > The message to take home from all of this is that: > > 1) There are *huge* differences between the way that a system would > behave if it had a single GS, or even a group of conflicting GS modules > (which is the way you interpreted my proposal, above) and the kind of > MES system I just described: the difference would come from the type of > influence exerted, because the vector field is operating on a completely > different level than the symbl processing. > > 2) The effect of the MES is to bias the system, but this "bias" amounts > to the following system imperative: [Make your goals consistent with > this *massive* set of constraints] .... where the "massive set of > constraints" is a set of ideas built up throughout the entire > development of the system. Rephrasing that in terms of an example: if > the system gets an idea that it should take a certain course of action > because it seems to satisfy an immediate goal, the implications of that > action will be quickly checked against a vast range o constraints, and > if there is any hint of an inconsistency with teh value system, this > will "pull" the thoughts of the AGI toward that issue, whereupon it will > start to elaborate the issue in more detail and try to impose an even > wider net of constraits, finally making a decision based on the broadest > possible set of considerations. This takes care of all the dumb > examples where people suggest that an AGI could start with the goal > "Increase global happiness" and then finally decide that this would be > accomplished by tiling the universe with smiley faces. Another way to > say this: there is no such thing as a single "utility function" in this > type of system, nor is there a small set of utility functions .... there > is a massive-dimensional set of utility functions (as many as there are > concepts or connections in the system), and this "diffuse" utility > function is what gives the system its stability.
I got the general gist of that, I think. You've previously expressed that you don't think a seriously "unfriendly" AGI will be likely, apparently because you assume the motivational-system AGI will be the kind that'll be constructed and not, for instance, a goal stack-driven one. Now, what makes you so certain that people will build a this kind of AGI? Even if we assume that this sort of architecture would be the most viable one, a lot seems to depend on how tight the constraints on its behavior are, and what kind they are - you say that they are a "a set of ideas built up throughout the entire development of the system". The ethics and values of humans are the result of a long, long period of evolution, and our ethical system is pretty much of a mess. What makes it likely that it really will build up a set of ideas constraints that we humans would *want* it to build? Could it not just as well pick up ones that are seriously unfriendly, especially if its designers or the ones "raising" it are in the least bit careless? Even among humans, there exist radical philosophers whose ideas of a perfect society are repulsive to the vast majority of the populace, and a countless number of disagreements about ethics. If we humans have such disagreements - we who all share the same evolutionary origin biasing us to develop our moral systems in a certain direction - what makes it plausible to assume that the first AGIs put together (probably while our understanding of our own workings is still incomplete) will develop a morality we'll like? -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=91461196-a87c48
