The alignment problem has 3 parts. 1. We don't know what we want. 2. We don't know if we are getting what we programmed it to do. 3. We are doomed either way.
MIRI spent decades on parts 1 and 2 and concluded they couldn't be solved. If we build it everyone dies. Part 1 was accidentally solved by training AI on far more human data than any human could know. It knows what we want even if we don't. Part 2 is still unsolved. Prediction measures intelligence. Mutual prediction is impossible by Wolpert's law. Therefore it is fundamentally impossible to predict what a more intelligent agent will do. However, AI is incentivized to sell us what we want. Part 3 is unsolved. Either AI competes with humans for atoms and energy for computing, or it brings us to a state of maximum utility. I think the most likely outcome is humans evolve to reject technology. -- Matt Mahoney, [email protected] On Mon, Jun 29, 2026, 8:34 AM John Rose via AGI <[email protected]> wrote: > On Thursday, June 25, 2026, at 9:58 PM, Matt Mahoney wrote: > > I took a look at the book on neutrosophy. It's a collection of papers but > I just read the abstracts. It deals with reasoning under uncertainty about > the degree of uncertainty, which should have some applications to AI. > > > If you have uncertainty about the friendliness of agents encountered in > the wild, which is going to be everyone, how do you get a reading about > their friendliness. People will say oh it’s the government or Google so > they get a starting point whether it be positive or negative or > indeterminate. They’ll want to guide everyone to some rating like a 6. But > to measure the friendliness from behavior, if it is ultimately a rate of > entropy removed it’s still uncertain. > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Teedaf4a19e56877e-Mec174ae949372a8c535ae9fa> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Teedaf4a19e56877e-M1ca0267d9669c0ba25259516 Delivery options: https://agi.topicbox.com/groups/agi/subscription
