Ben,
I guess the issue I have with your critique is that you say that I have given no details, no rigorous argument, just handwaving, etc.
But you are being contradictory: on the one hand you say that the proposal is vague/underspecified/does not give any arguments .... but then having said that, you go on to make specific criticisms and say that it is wrong on this or that point.
I don't think you can have it both ways. Either you don't see an argument, and rest your case, or you do see an argument and want to critique it. You are trying to do both: you repeatedly make broad accusations about the quality of the proposal ("some verrrrry hand-wavy, intuitive suggestions", "you have not given any sort of rigorous argument", "... your intuitive suggestions...", "you did not give any details as to why you think your proposal will 'work'", etc. etc.), but then go on to make specific points about what is wrong with it.
Now, if the specific points you make were valid criticisms, I could perhaps overlook the inconsistency and just address the criticisms. But that is exactly what I just did, and your specific criticisms, as I explained in the last message, were mostly about issues that had nothing to do with the general class of architectures I proposed, but only with weird cases or weird issues that had no bearing on my case.
Since you just dropped most of those issues (except one, which I will address in a moment), I must assume that you accept that I have given a good reply to each of them. But instead of conceding that the argument I gave must therefore have some merit, you repeat -- even more insistently than before -- that there is nothing in the argument, that it is all just vague handwaving etc.
No fair! This kind of response: - Your argument is either too vague or I don't understand it. Would be fine, and I would just try to clarify it in the future. But this response: - This is all just handwaving, with no details and no argument. - It is also a wrong argument, for these reasons: - [Reasons that are mostly just handwaving or irrelevant]. Is not so good. *************************I will say something about the specific point you make about my claim that as time goes on the system will check new ideas against previous ones to make sure that new ones are consistent with ALL the old ones, so therefore it will become more and more stable.
What you have raised is a minor technical issue, together with some confusion about what exactly I meant:
The "ideas" being checked against "all previous ideas" are *not* the incoming general learned concepts (cup, salt, cricket, democracy, sneezes..... etc.) but the concepts related to planned actions and the system's base of moral/ethical/motivational concerns. Broadly speaking, it is when there is a new "perhaps I should do this ..." idea that the comparison starts. I did actually say this, but it was a little obscurely worded.
Now, when I said "checked for consistency against all previous ideas" I was speaking rather loosely (my bad). Obviously I would not do this by an exhaustive comparison [please: I don't need to have it explained to me that this is O(n^^2)! :-) ]. The mechanism would work something like a parallel terraced scan: issues are represented at different levels of granularity, and if any kind of inconsistency is detected at one of the high (low-granularity) levels, it provokes a focussing on the problem and an elaboration of everything involved in the idea, which then can bring in lots more consideration, potentially resulting in a complete comparison on that one issue. In addition, but the system would use various other (monte-carlo-esque) techniques for taking random looks at the implications of some issue, to catch problems that might not get past the top level scan.
Specific example. The system thinks that maybe selling its mother into the white slave trade is a good way to make money. But this very idea causes simple associations with [white slave trade] to kick in (for example [misery], [brutality], [betrayal], and so on). These simple associations get connected with [mother] and in a moment the system finds that the concept [unhappy mother] sends a big fat negative signal back to the motivational system, waking up the module that is responsible for the [social group attachment] motivation. Pretty soon this kicks in a full-scale reexamination of the entire idea, and when examined in detail it is found to be inconsistent with the system's prime motivations.
So although you made a reasonable point, this is not a technical difficulty that cannot be handled easily.
I note that you did anticipate this reply, when you said "Some heuristic shortcuts must be used to decrease the number of comparisons, and such heuristics introduce the possibility of error...", and then also "The kind of distributed system you are describing seems NOT to solve the computational problem of verifying the consistency of each new knowledge item with each other knowledge item."
But these two statements are actually very hard to defend. Heuristics that decrease the number of comparisons IN A CONVENTIONAL AI SYSTEM are unreliable, precisely because of the "fragile, mechanistic" nature of such AI designs (see my reply to Hank Conn) ... but the whole force of my argument is to do the job without such conventional AI techniques, so that one won't fly unless you can say why. As for the type of distributed system I propose being unable to solve this kind of problem, the very reverse is true: parallel terraced scans are among the very best methods known for dealing with this kind of problem! I couldn't have chosen a better architecture. Your statement is mystifying.
***********What I feel I have done now is to address every one of the specific criticisms that you have put on the table to date.
I am certainly willing to accept that, beyond those specific points, you may have a gut feeling that it doesn't work, or that you prefer not to address it in more detail at this stage. I'd be happy to postpone further debate until I can get a more detailed version in print.
What I would find extremely unfair would be more accusations that it is "just vague handwaving" without specific questions designed to show that the argument falls apart under probing. I don't see the argument falling apart, so making that accusation again would be unjustified.
Richard Loosemore Ben Goertzel wrote:
Hi,There is something about the gist of your response that seemed strange to me, but I think I have put my finger on it: I am proposing a general *class* of architectures for an AI-with-motivational-system. I am not saying that this is a specific instance (with all the details nailed down) of that architecture, but an entire class..... an approach. However, as I explain in detail below, most of your criticisms are that there MIGHT be instances of that architecture that do not work.No. I don't see why there will be any instances of your architecture that do "work" (in the sense of providing guaranteeable Friendliness under conditions of radical, intelligence-increasing self-modification). And you have not given any sort of rigorous argument that such instances will exist.... Just some verrrrry hand-wavy, intuitive suggestions, centering on the notion that (to paraphrase) "because there are a lot of constraints, a miracle happens" ;-) I don't find your intuitive suggestions foolish or anything, just highly sketchy and unconvincing. I would say the same about Eliezer's attempt to make a Friendly AI architecture in his old, now-repudiated-by-him essay Creating a Friendly AI. A lot in CFAI seemed plausible to me , and the intuitive arguments were more fully fleshed out than your in your email (naturally, because it was an article, not an email) ... but in the end I felt unconvinced, and Eliezer eventually came to agree with me (though not on the best approach to fixing the problems)...> In a radically self-improving AGI built according to your > architecture, the set of constraints would constantly be increasing in > number and complexity ... in a pattern based on stimuli from the > environment as well as internal stimuli ... and it seems to me you > have no way to guarantee based on the smaller **initial** set of > constraints, that the eventual larger set of constraints is going to > preserve "Friendliness" or any other criterion. On the contrary, this is a system that grows by adding new ideas whose motivatonal status must be consistent with ALL of the previous ones, and the longer the system is allowed to develop, the deeper the new ideas are constrained by the sum total of what has gone before.This does not sound realistic. Within realistic computational constraints, I don't see how an AI system is going to verify that each of its new ideas is consistent with all of its previous ideas. This is a specific issue that has required attention within the Novamente system. In Novamente, each new idea is specifically NOT required to be verified for consistency against all previous ideas existing in the system, because this would make the process of knowledge acquisition computationally intractable. Rather, it is checked for consistency against those other pieces of knowledge with which it directly interacts. If an inconsistency is noticed, in real-time, during the course of thought, then it is resolved (sometimes by a biased random decision, if there is not enough evidence to choose between two inconsistent alternatives; or sometimes, if the matter is important enough, by explicitly maintaining two inconsistent perspectives in the system, with separate labels, and an instruction to pay attention to resolving the inconsistency as more evidence comes in.) The kind of distributed system you are describing seems NOT to solve the computational problem of verifying the consistency of each new knowledge item with each other knowledge item.Thus: if the system has grown up and acquired a huge number of examples and ideas about what constitutes good behavior according to its internal system of values, then any new ideas about new values must, because of the way the system is designed, prove themselves by being compared against all of the old ones.If each idea must be compared against all other ideas, then cognition has order n^2 where n is the number of ideas. This is not workable. Some heuristic shortcuts must be used to decrease the number of comparisons, and such heuristics introduce the possibility of error...And I said "ridiculously small chance" advisedly: if 10,000 previous constraints apply to each new motivational idea, and if 9,900 of them say 'Hey, this is inconsistent with what I think is a good thing to do', then it doesn't have a snowball's chance in hell of getting accepted. THIS is the deep potential well I keep referring to.The problem, as I said, is posing a set of constraints that is both loose enough to allow innovative new behaviors, and tight enough to prevent the wrong behaviors...I maintain that we can, during early experimental work, understand the structure of the motivational system well enough to get it up to a threshold of acceptably friendly behavior, and that beyond that point its stability will be self-reinforcing, for the above reasons.Well, I hope so ;-) I don't rule out the possibility, but I don't feel you've argued for it convincingly, either...Overall, the goal would be, not to give it an intrinsic definition of "friendliness", but actually something closer to a "desire to discover and empathize with the normative aspirations of the human species". In other words, make it want to be part of the community of humankind, the way that you and I, as compassionate individual humans, want to be part of that community. Make it one of us, but without certain irrascible motivational mechanisms that evolution stuck us with.This does sound reasonable to me. BTW, "the normative aspirations of the human species" sounds a fair bit like Eliezer's Coherent Extrapolated Volition, though ;-) ... the question with both your formulation and his is how do you really define this, either formally or even pragmatically... Human aspirations are quite diverse on the individual and also cultural level; is there really such a thing as our "species-wide aspirations"? -- Ben ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
