I'll try and avoid a repeat of the lenghtly, fairly futile and extremely disruptive discussion of Loosemore's assertions that occurred on the SL4 mailing list. I am willing to address the implicit questions/assumptions about my own position.
Richard Loosemore wrote: > The contribution of complex systems science is not to send across a > whole body of plug-and-play theoretical work: they only need to send > across one idea (an empirical fact), and that is enough. This empirical > idea is the notion of the disconnectedness of global from local behavior > - what I have called the 'Global-Local Disconnect' and what, roughly > speaking, Wolfram calls 'Computational Irreducibility'. This is only an issue if you're using open-ended selective dynamics on or in a substrate with softly-constrained, implicitly-constrained or unconstrained side effects. Nailing that statement down precisely would take a few more paragraphs of definition, but I'll skip that for now. The point is that plenty of complex engineered systems, including almost all existing software systems, don't have this property. The assertion that it is possible (for humans) to design an AGI with fully explicit and rigorous side effect control is contraversial and unproven; I'm optimistic about it, but I'm not sure and I certainly wouldn't call it a fact. What you failed to do was show that it is impossible, and indeed below you seem to acknowledge that it may in fact be possible. The question of whether is more desirable to build an AGI with strong structural constraints is more complicated. Eliezer Yudkowsky has spent hundreds of thousands of words arguing fairly convincingly for this, including a fairly good essay on the subject that was forwarded to this list earlier by Ben and I'm not going to rehash it here. >> It is entirely possible to build an AI in such a way that the general >> course of its behavior is as reliable as the behavior of an Ideal >> Gas: can't predict the position and momentum of all its particles, >> but you sure can predict such overall characteristics as temperature, >> pressure and volume. A highly transhuman intelligence could probably do this, though I suspect it would be very inefficient, partially I expect you'd need strong passive constraints on the power of local mechanisms (the kind the brain has in abundance), which will always sacrifice performance on many tasks compared to unconstrained or intelligently-verified mechanisms. The chances of humans being able to do this are pretty remote, much worse than the already not-promising chances for doing constraint/logic-based FAI. Part of that is due to the fact that while there are people making theoretical progress on constraint-based analysis of AGI, all the suggestions for developing the essential theory for this kind of FAI seem to involve running experiments on highly dangerous proto-AGI or AGI systems (necessarily built before any such theory can be developed and verified). Another problem is the fact that people advocating this kind of approach usually don't appreciate the difficult of designing a good set of FAI goals in the first place, nor the difficulty of verifying that an AGI has a precisely human-like motivational structure if they're going with the dubious plan of hoping an enhanced-human-equivalent can steer humanity through the Singularity successfully. Finally the most serious problem is that an AGI of this type isn't capable of doing safe full-scale self modification until it has full competence in applying all of this as yet undeveloped emergent-FAI theory; unlike constraint-based FAI you don't get any help from the basic substrate and the self-modification competence doesn't grow with the main AI. Until both the abstract knowledge of the reliable-emergent-goal-system-design and the Friendly goal system to use it properly are fully in place (i.e. in all of your prototypes) you're relying on adversarial methods to prevent arbitary self-modification, hard takeoff and general bad news. In short this approach is ridiculously risky and unlikely to work, orders of magnitude more so than actively verified FAI on a rational AGI substrate, which is already extremely difficult and pretty damn risky to develop. Hybrid approaches (e.g. what Ben's probably envisioning) are almost certainly better than emergence-based theories (and I use the word theories loosely there), and I accept that if fully formal FAI turns out to be impossibly difficult we might have to downgrade to some form of probabilistic verification. I'd add that I have yet to see any evidence that you or anyone else are actually making progress on 'emergent' FAI design, or any evidence or even detailed arguments for an AGI design capable of this. > The motivational system of some types of AI (the types you would > classify as tainted by complexity) can be made so reliable that the > likelihood of them becoming unfriendly would be similar to the > likelihood of the molecules of an Ideal Gas suddenly deciding to split > into two groups and head for opposite ends of their container. I second Ben here; let's see the design specs for one of these systems, along with some evidence that it's scalable to AGI, otherwise this is a completely unsupported personal hunch. > And by contrast, the type of system that the Rational/Normative AI > community want to build (with logically provable friendliness) is either > never going to arrive, or will be as brittle as a house of cards: it > will not degrade gracefully. In many cases it deliberately doesn't degrade gracefully; sensible designs are laced with redundant checks that shut down the system completely at the first sign of trouble. Any other assertions of brittleness are probably a layer confusion and misgeneralisation from classic symbolic AI (possibly even non-AI software systems). > For that reason, I believe that if/when you do get impatient and decide > to forgo a definitive proof of friendliness, Which I won't (I might grudgingly permitt probabilistic analysis if progress on provable FAI completely fails, but I will never accept superficial generalisation from in-simulation experiments or just 'hell, let's go for it'). Someone else might of course. > and push the START button on your AI, you will create something > incredibly dangerous. You don't appear to have any understanding of the functional mechanisms involved in a 'rational/normative' AI system that could actually scale to AGI, and you have yet to produce any general arguments that tell us something about the behaviour of such a system (acknowledging that they are designed to avoid the kind of unexpected global dynamics you are so fond of, and presumably accepting that this is possible at an unknown potential cost in flexibility). Can you articulate a specific failure scenario that you think a system like this would undergo and that your ideas would avoid? Michael Wilson Director of Research and Development Bitphase AI Ltd - http://www.bitphase.com ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
