Hi Ben,
I have written two essays on the nature of future AGI's that are smart
enough to use theorem-proving to guide their every action:
...
Yes, we are a long way from there now! But it's always fun to speculate....
The two essays are a series; by the end of the second one things get fairly
interesting...
Fun stuff :->.
Maybe I'm missing something, but let's compare the behavior of two "rational" AIs with the goal of "be nice to puppies", one with only this goal (AI0) and one this same goal, along with the fancied up ITSSIM, T0 (AI1).. Lets further assume that "be nice to puppies" is encoded, for example, as a few million weighted examples of puppy-niceness and non-puppy-niceness, and that the AI knows how to generalized correctly from these cases and won't tile the universe with identical copies of puppy-niceness; the ability to create such an AGI seems to be presupposed, otherwise ITSSIM can't take off, as best I understand. Right?
Now, lets say that AI0 and AI1 are both evaluating an improved system, AI'. If AI0 accepts AI', that implies that, to the best of its knowledge, AI' will not only be nice to puppies, but lead to puppy-niceness going forward in time as well (possible including any future AI''s). Now, a subgoal of this should be the ITSSIM property "I will only take an action if I can prove that it is expected not to decrease safety.", because decreased safety (in the ITSSIM sense) should clearly lead to decreased puppy-niceness over the future course of the universe. So it seems like AI1 would also want to accept AI' as well. Similarly, AI0 should accept any AI' that AI1 accepts..
As I see it, the only difference in behavior between AI0 and AI1 is a matter of weighting and priorities between base goal and the subgoals of safety and stability; AI0 might conceivably be swayed that AI' was safe, whereas AI1 would require a "formal proof". Note that the converse is also true; AI1 might accept AI' based on some formal proof that AI0 would reject, e.g., due to an insufficiently conservative axiomatization. But since all of the nice properties of ITSSIM seem to be subgoals of AI0, why must they be explicit? Wouldn't a sufficiently advanced AI discover them on its own and apply them properly (in a way more elegant and rational than "spend r% of my time thinking about X"), in pursuit of its base goal?
Moshe
-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
