My apologies for the duplication of my previous post; I thought my mail client failed to send the original, but actually it just dropped the echo from the server.
Matt Mahoney wrote: > Michael Wilson wrote: >> Hybrid approaches (e.g. what Ben's probably envisioning) are almost certainly >> better than emergence-based theories... if fully formal FAI turns out to be >> impossibly difficult we might have to downgrade to some form of >> probabilistic verification. > > I think a hybrid approach is still risky. Alas, Seed AI is inherently risky. All we can do is compare levels of risk. > By hybrid, do you mean AGI augmented with conventional computing > capability? All AGIs implemented on general purpose computers will have access to 'conventional computing capability' unless (sucessfully) kept in a sandbox - and even then anything with a Turing-complete substrate has the potential to develop such capability internally. 'Access to' isn't the same thing as 'augmented with' of course, but I'm not sure exactly what you mean by this (and I'd rather wait for you to explain than guess). Certainly there is the potential for combing formal and informal control mechanisms (as opposed to just local inference and learning machanisms, where informality is much easier to render safe) in an FAI system. Given a good understanding of what's going on I would expect this to be a big improvement on purely informal/implicit methods, though it is possible to imagine someone throwing together the two approaches in such a way that the result is even worse than an informal approach on its own (largely because the kind of reflective analysis and global transforms a constraint-based system can support override what little protection the passive causality constraints in a typical localised-connectionist system give you). My statement above was referencing the structure of the theory used to design/verify the FAI though, not the structure of the FAI itself. I'd characterise a hybrid FAI theory as one that uses some directly provable constraints to narrow down the range of possible behaviours, and then some probabilistic calculation (possibly incorporating experimental evidence) to show that the probability of the AGI staying Friendly is high. The biggest issues with probabilistic calculations are the difficultly of generalising them across self-modification, the fact that any nontrivial uncertainty that compounds across self-modification steps will quickly render the theory useless when applied to an AGI undergoing takeoff, and the fact that humans are just so prone to making serious mistakes when trying to reason probabilistically (even when formal probability theory is used, though that's still /much/ better than intuition/guessing for a problem this complex). As I've mentioned previously, I am optimistic about using narrow AI to help develop AGI designs and FAI theories, and have had some modest success in this area already. I'm not sure if this counts as 'augmenting with conventional computing capability'. > Suppose we develop an AGI using a neural model, with all the strengths > and weaknesses of humans, such as limited short term memory, inefficient > and error prone symbolic reasoning and arithmetic skills, slow long term > learning rate, inability to explicitly delete data, etc. Then we > observe: > > A human with pencil and paper can solve many more problems than one > without. > A human with a calculator is even more powerful. > A human with a computer and programming skills is even more powerful. > A human with control over a network of millions of computers is even more > powerful. > > Substitute AGI for human and you have all the ingredients to launch a > singularity. Absolutely. Plus the AGI has the equivalent of these things directly interfaced into a human's brain, not manipulated through slow and unreliable physical interfaces, and even a brain-like AGI may well be running at a much higher effective clock rate than a human in the first place. This is essentially why AGI is so dangerous even if you don't accept hard and/or early takeoff in an AGI system on its own. > If your goal is friendly AI, then not only must you get it right, but so > must the AGI when it programs the network to build a more powerful AGI, > and so must that AGI, and so on. You cannot make a mistake anywhere > along the chain. Thus probabilistic methods have a serious problem remaining effective under recursive self-modification; any flaws in the original theory that don't get quickly and correctly fixed by the AGI (which requires an accurate meta-description of what your FAI theory is supposed to do...) are likely to deviate the effective goal system out of the human-desirable space. If you /have/ to use probabilistic methods, they are all kinds of mitigating strategies you can take; Eliezer actually covered quite a few of them back in Creating A Friendly AI. But provable Friendliness (implemented with many layers of redundancy just to be sure) is better if it's available. Michael Wilson Director of Research and Development Bitphase AI Ltd - http://www.bitphase.com ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
