On Mon, May 5, 2014 at 1:11 PM, Aaron Hosford <[email protected]> wrote:
> I think the trick lies in multiple redundancies, both for triggering and > effecting termination. > > We should also design in as many mechanisms as possible to avoid the > problem in the first place. For example, a very strong negative reward > signal for the AGI even considering modifications to certain critical zones > of its own software or hardware, particularly those that determine the > reward levels themselves, in a reinforcement learning-based AGI. (This > could be interpreted as an overpowering urge to "stay true to oneself" on > the part of the AGI, meaning that it would try to preserve its own personal > identity.) > > (I realize that AIXI is not practical, but this is a philosophical discussion) Suppose you were gifted with an infinitely fast computer, and wanted to put it to work for the good of the humankind running AIXI. How would you actually implement its reward function, in code, so that it doesn't go off the reservation? Is this still very much an open problem? ------------------------------------------- AGI Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/21088071-f452e424 Modify Your Subscription: https://www.listbox.com/member/?member_id=21088071&id_secret=21088071-58d57657 Powered by Listbox: http://www.listbox.com
