--- John Ku <[EMAIL PROTECTED]> wrote: > On 2/16/08, Matt Mahoney <[EMAIL PROTECTED]> wrote: > > The result will not be pretty. The best definition (not solution) of > > friendliness is probably CEV ( http://www.singinst.org/upload/CEV.html ) > which > > can be summarized as "our wish if we knew more, thought faster, were more > the > > people we wished we were, had grown up farther together". What would you > wish > > for if your brain was not constrained by the hardwired beliefs and goals > that > > you were born with and you knew that your consciousness did not exist? > What > > would you wish for if you could reprogram your own goals? The logical > answer > > is that it doesn't matter. The pleasure of a thousand permanent orgasms > is > > just a matter of changing a few lines of code, and you go into a > degenerate > > state where learning ceases. > > Your counterfactuals seem very different from Eliezer's and less > relevant to what matters. I think Eliezer's definition was plausible > because it approximated the standards we use to deliberate about our > values. As such, it is getting at deeper values or procedures that we > implicitly presuppose in any serious discussion of values at all. Even > if you were to question whether you should use that standard, your > cognitive architecture would still have to do so by reference to some > internal standard in order to even count as a meaningful type of > questioning and Eliezer's definition would probably be a decent > intuitive characterization of it. Of course, you are free to pose any > type of bizarre counterfactual you want, but I don't see how > evaluating it would be relevant to what matters in the way that > Eliezer's would.
I admit I am oversimplifying Eliezer's definition. Reading the full document, we should not assume that an AGI would be stupid enough to grant our extrapolated wish to be put in a blissful, degenerate state. Nevertheless I am mistrustful. I am most troubled that CEV does not have a concise description. > I would prefer to leave behind these counterfactuals altogether and > try to use information theory and control theory to achieve a precise > understanding of what it is for something to be the standard(s) in > terms of which we are able to deliberate. Since our normative concepts > (e.g. should, reason, ought, etc) are fundamentally about guiding our > attitudes through deliberation, I think they can then be analyzed in > terms of what those deliberative standards prescribe. I agree. I prefer the approach of predicting what we *will* do as opposed to what we *ought* to do. It makes no sense to talk about a right or wrong approach when our concepts of right and wrong are programmable. -- Matt Mahoney, [EMAIL PROTECTED] ------------------------------------------- singularity Archives: http://www.listbox.com/member/archive/11983/=now RSS Feed: http://www.listbox.com/member/archive/rss/11983/ Modify Your Subscription: http://www.listbox.com/member/?member_id=4007604&id_secret=96140713-a54b2b Powered by Listbox: http://www.listbox.com
