Monday, March 3, 2003, 1:18:18 AM, Kevin Copple wrote: KC> Here's an idea (if it hasn't already been considered before):
KC> In each executing component of an AGI has Control Code. This Control Code KC> monitors a Big Red Switch. If the switch is turned on, execution proceeds KC> normally. If not, all execution by the component stopped. KC> The big red switch could be a heart monitor on Ben's chest. The AGI better KC> keep Ben alive! And if Ben gets angry with the AGI, he can remove the heart KC> monitor, so the AGI better keep Ben happy also. Bypass the monitor signal. Create a system or subsystem that will make the Ben heart monitor always give out correct signals. Worst case, kill Ben but take his heart out and hook it up to an electrical stimulus. I should really point out that from a certain angle, the "fixed reward problem" looks exactly like the "computer security" problem. Some relevant resources are: http://www.counterpane.com/crypto-gram.html (archives of the newsletter "crypto-gram", by Bruce Schneier, one of the foremost cryptography writers) http://www.cl.cam.ac.uk/ftp/users/rja14/satan.pdf (programming Satan's computer) http://catless.ncl.ac.uk/Risks (archives of the moderated comp.risks group) In general I'd recommend adopting the super-paranoid hypotheses of the cryptography community when considering such issues: - always assume fantastically powerful entities want to subvert or negate your goals - always assume they have just about all information about you and your work (algorithms, ciphertext) - design your work so that none of that makes a difference: design codes so that even posession of ciphertext and some plaintexts doesn't let your opponent crack important messages; design your AI so that even posession of the AI code doesn't help to create unfriendly technologies. For further indoctrination into full cypherparanoia, consult the cypherpunks mailing list archive, read "cryptoanarchy" by Tim May, read about medusa/tentacles, etc. The summary is, really, assume the worst: plan and design for it. -- Cliff ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
