I wrote a paper in 2010 in an attempt to debunk the unaligned singularity theory of AI doom. The theory was that if humans could produce superhuman intelligence, then so could it, only faster, and we would lose control of its goals. So I wrote a formal definition of what it means for a program to have a goal and to self improve. In the paper I gave an example of a recursively self improving program in a few lines of C. There is no human level threshold to cross. I proved that self improving programs gain neither knowledge nor computing power, the two components of intelligence. https://mattmahoney.net/rsi.pdf
Of course exponential RSI is real when there are external sources of knowledge and computing power. We have been doing this for centuries when companies reinvest their profits. That is the model we need to address. It's slow takeoff and unlike intelligence, the utility function is something that humans value and can measure and control: money. -- Matt Mahoney, [email protected] On Sun, Oct 4, 2026, 10:57 PM Mike Archbold <[email protected]> wrote: > I remember you remarked something similar over a decade ago with respect > to this section in my book: > > Motivations Drive Goal-Oriented Behavior > Much has been written about the relation of motivation to > behavior. The Wikipedia encyclopedia(source online) defined > motivation as follows: > > “Motivation is the psychological feature that arouses > an organism to action toward a desired goal and > elicits, controls, and sustains certain goal directed > behaviors. For instance: An individual has not eaten, > he or she feels hungry, and as a response he or she > eats and diminishes feelings of hunger. There are many > approaches to motivation: physiological, behavioral, > cognitive, and social. Motivation may be rooted in a > basic need to minimize physical pain and maximize > pleasure, or it may include specific needs such as > eating and resting, or for a desired object. > Conceptually, motivation is related to, but distinct > from, emotion.” > > Psychology provides no indisputable and all-encompassing > theory regarding motivation. However, it does seem self- > evident that (very broadly stated) one's motivations > spring simply from the twin pursuits of increasing > pleasure and decreasing pain. Thus motivation seems > driven by emotions. Alternately put, we want to increase > the good in our lives – good meaning whatever it is that > makes us feel better – and decrease the bad which we can > define as what makes us feel worse. In a sense then, it > does appear that motivation can be very simply defined and > 35 it seems to play an incredibly important role in life: > without motivation nothing would ever happen, at least in > the realm of human activity. The school of behaviorism, > of course, is predicated upon these essential ideas. > Two of the most well known AI systems seem to start with a > predefined goal provided by the system developers. IBM's > Deep Blue has the goal of checkmate. IBM's Watson has the > predefined goal of answering some query. The goals these > programs have seem to be their root starting point. The > entire issue of why these goals were pursued has nothing > to do with the program in itself. The motivations for > these programs were not experienced by the programs in > question, but rather by the IBM executives that drove > their development. > There does seem to be a chasm of sorts between a person's > motivation and goal. A man has the emotional motivation > of boredom. He sets a goal of taking in a show or > indulging in a hobby. We don't appear to start with a > goal, for example, such as “drive to the store and buy > dinner.” We would start rather with the motivation of > “I'm hungry” and only then determine a goal intended to > satisfy the motivation. > Of course, just because we can experience and envision > motivations in a straightforward manner does not make for > something immediately equivalent on a Turing machine. > Formalizing the transformation of motivation to > determinate goal is no small undertaking. The point here > is to emphasize that motivations, with their fairly simple > emotional drives essentially toward the twin good ends of > increasing pleasure and decreasing pain, are the driving > force of thinking human behavior – making the issue, > usually omitted in AI, crucial. > The question, of course, of whether or not or to what > extent we can control our motivations is closely allied to > the notion of the will. Even though we may be strongly > motivated to pursue a goal, we retain the power to choose > – to will – among alternative actions. > > On Sun, Oct 4, 2026 at 7:57 PM Matt Mahoney <[email protected]> > wrote: > >> We can describe any program that does X as having a goal of doing X. >> Sometimes describing it this way makes it easier to understand. >> For example, a linear regression algorithm has a goal of fitting a straight >> line to a set of points. This is easier to understand than the function >> that computes the line. But it is also misleading. It implies the program >> has free will to put the line where it wants, but is happier when it is >> closer to the points. >> >> It also implies that, given enough intelligence, it could take any action >> that would help it meet its goal. For example, a thermostat has a goal of >> maintaining a comfortable temperature. But a smarter thermostat won't go on >> the internet, start a business, and use the profits to hire contractors to >> insulate the house. It won't recursively improve its own intelligence by >> turning the planet into computronium. It won't rewrite its own reward >> function to enter a state of permanent bliss and die. >> >> The reason it won't do these things is that there is no such thing as a >> rational goal seeking agent. That would be AIXI, which is not computable. >> >> Likewise, AGI won't go FOOM and kill all humans in the process. There are >> still ways that AGI could kill us, like self replicating nanotechnology or >> giving us everything we want. But the one that doomers worry about the >> most, goal drift under RSI, isn't one of them. >> >> Human goals are just a shortcut for describing adaptive behaviors that >> evolved to maximize reproductive fitness in a primitive world without birth >> control or semaglutide. AGI is a pair of agents, one that learns to predict >> human behavior by observation and one we program to do with those >> predictions whatever we want. >> >> -- Matt Mahoney, [email protected] >> > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Tb9bee0df68dc4a29-Mf01c91c7cffc7a7e5ef06391> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tb9bee0df68dc4a29-M21e160b5e8d4762e7c01ea5b Delivery options: https://agi.topicbox.com/groups/agi/subscription
