On Mon, Jun 22, 2026 at 7:21 PM Matt Mahoney <[email protected]> wrote:
> I agree. We need a definition of friendly (or aligned) AI that we can all > agree on. What is your definition? > You missed the point of my using the term "Leggian". What Shane did way back when was compile all the definitions of "intelligence" he could find and then create a taxonomy. He did this not so everyone would agree, but so people would stop arguing past each other when referring to entirely different concepts while using the word "intelligence". His taxonomy ended up with 2 dimensions, each with two values resulting in 4 different concepts -- all referred to as "intelligence". We need something similar done with "friendly" and, of course by implication "unfriendly". In the present instance there seems to be some notion of "friendly" to which you adhere that relates to what is sometimes called "wireheading" in the literature on AGI. In other words, you seem to be saying that "we" (a problematic term when discussing values) are working feverishly to create a very roundabout way of placing wires in our pleasure centers and hooking them up to buttons that we can push, much like a rat with a lever that delivers cocaine. The fact that we may not want to see that we are wired, nor that we are pushing the buttons nor that we did this to ourselves, explains why we had to do this in a roundabout way. Moreover, this also seems associated with your belief that "we" have no free will or something like that. To my way of thinking, this is sort of like saying that the Amish are just kidding themselves -- what they *really* want is a continual drip feed of Fentanyl. Where do we differ in our concepts? Well, it gets down to a difference in our views of how much "freedom" humans actually have. Like BF Skinner in "Beyond Freedom and Dignity" -- a book that a childhood friend of mine cut his teeth on in behavior psychology before he ended up in the DC Beltway with a literal fentanyl drip -- you seem intellectually satisfied with material determinism. No matter how you may wish to argue otherwise, in the final analysis, this is an article of faith. I mean, we all must make decisions and take actions based on limited information with our limited intelligence, unlike ideal AIXI agents. My faith is more oriented toward consent-maximization as a consequence of agency-maximizing social relations. In other words, I'd very much like to see you able to live in a community that shares your faith. But I don't share your faith. For me, "friendly" AGI means you and I can live among those that share our respective faiths.\ The global economy is currently directed by people who/entities that most definitely do not tolerate those who want to exclude them from the environments in which their own children are being raised. > I asserted that getting everything you want is equivalent to killing all > humans. Most people would reject this as absurd. Yet, even though people > are objectively better off today than any time in the past, they are not > happier. There is no evidence that humans are happier than animals. We have > the highest suicide rate of any species, but maybe that's because we are > the only species with the technology to shoot ourselves. The rate of mental > illness is going up, but maybe that's just so we can get drugs. Maybe it's > because happiness is not utility. It's the rate of increase of utility. The > closer you get to the maximum, the more you have to lose. > > I asked on the EA forum whether life has net positive, negative, or zero > utility. I could not get an answer. Is it better to populate the galaxy > with von Newmann probes to maximize the number of living beings, or to end > all suffering by exterminating all life? There was no agreement on this > fundamental question. Really? On a forum dedicated to maximizing collective > utility? > > We are programmed by evolution to fear death and then die because that > maximizes reproductive fitness. We could therefore eliminate suffering > either by achieving immortality by eliminating fear. We know how to do the > latter. Psychopaths lack a functioning amygdala, the part of the brain that > feels fear. They do not respond to negative reinforcement. They feel pain > but do not suffer. In most people, pain does not cause suffering either. It > reprograms the brain to fear the thing that caused it. > > The other alternative is immortality. If AI killed all humans but > preserved all human knowledge, would that still count as human extinction? > Is that enough, or would it also have to run a simulation like the one you > might be in now? Would it make a difference if the knowledge was erased and > a completely different world was modeled in its place, if you didn't know > the difference? > > Eliminating suffering is possible with brain surgery, but goal seeking > agents don't want their goals changed because then they wouldn't meet their > original goals. Eliminating death requires you to accept that consciousness > is indistinguishable from next token prediction. Our global surveillance > system already knows enough about you to train an LLM to convince others > that it's you. > > But maybe these aren't the only two options. Tell me what else you want AI > to do for you. > > -- Matt Mahoney, [email protected] > > On Mon, Jun 22, 2026, 2:30 PM James Bowery <[email protected]> wrote: > >> You just made the case for what I was arguing in "A Leggian Approach to >> "Friendly AI >> <https://www.linkedin.com/pulse/leggian-approach-friendly-ai-james-bowery>". >> Just as everyone has their definition of "intelligence," everyone has their >> definition of "friendly" and "unfriendly". You have yours. >> >> On Mon, Jun 22, 2026 at 9:42 AM Matt Mahoney <[email protected]> >> wrote: >> >>> Unfriendly AI would kill all humans. Friendly AI would give us >>> everything we want. It would raise us to a state of maximum utility, where >>> we would stay. >>> >>> That's the same thing. >>> >>> -- Matt Mahoney, [email protected] >>> >>> On Mon, Jun 22, 2026, 9:56 AM James Bowery <[email protected]> wrote: >>> >>>> >>>> >>>> On Sun, Jun 21, 2026 at 8:47 PM Matt Mahoney <[email protected]> >>>> wrote: >>>> >>>>> Let's define some terms. >>>>> >>>>> ...If you mean Legg and Hutter universal intelligence as goal >>>>> achievement over a universal distribution of environments, then we need a >>>>> computable approximation of that. A reasonable measure might be dollars >>>>> per >>>>> hour. That happened about 1800 when world GDP doubled from the medieval >>>>> baseline due to technology.... >>>>> >>>> >>>> As I've been saying explicitly for at least 9 years and implicitly for >>>> more like 34 years: >>>> >>>> The Global Economy IS the Unfriendly AGI We've Been Wating For >>>> >>>> Frank Herbert's "The Santaroga Barrier" is his best and most relevant >>>> novel. >>>> >>>> It sets a lower bound on "Friendly". >>>> >>>> PS: My February 9, 2016 article "A Leggian Approach to "Friendly AI >>>> <https://www.linkedin.com/pulse/leggian-approach-friendly-ai-james-bowery>" >>>> preceded the creation of "Hugging Face". >>>> >>>> >>>> *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Teedaf4a19e56877e-M713feb8ebdb722b12f74a634> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Teedaf4a19e56877e-Ma90e94566222ba4847da972c Delivery options: https://agi.topicbox.com/groups/agi/subscription
