The credit assignment problem is the problem of assigning a reinforcement signal to its cause. The optimal solution is AIXI, which is not computable. Humans use evolved heuristics like assigning nausea to something you ate. But the most common approximation is assigning a weight of about 1/t where t is the time from the action to the signal.
But that has nothing to do with the 3 AI scenarios I listed (AI controlled by nobody, billionaires, or everyone). Actually there are at least two others. AI could fail catastrophically after our economy becomes totally dependent on it and billions starve. Or we could evolve to reject the causes of population collapse which would include AI replacing social interaction. My comment about AI giving us everything we want except happiness is that today we have objectively better living conditions than at any time in the past. But we are not any happier than people that lived before smart phones or Internet or air travel or electricity or indoor plumbing or antibiotics or books or agriculture. We are not happier than other animals. The reason is that happiness is the rate of increase of utility. Happiness is what you have minus what you want. We evolved to fear death and then die, but what we really want is to stay in a state of maximum utility, which is the same thing. If a machine or drug could give you the feeling of a million orgasms for as long as you wanted, what do you think you would do? Right now we are in scenario 2, AI controlled by billionaires, although it is hard to tell from 3, AI controlled by everyone, because positive reinforcement gives you the illusion of free will. When you train a dog with treats, it thinks it is controlling you. I think this is the best possible outcome, unless, like Clarke, you consider human extinction to be a stepping stone to the replacement of carbon based life with something better. Billionaires want to model your mind to extract your money, but they also need a prosperous economy to enjoy what they took from you. Wealth has had a power law distribution, the exponential of a bell curve, since the time of King Tut, and I don't expect AI to change that. Inequality drives economic growth. The poor get richer when the rich get richer faster. -- Matt Mahoney, [email protected] On Sat, Oct 3, 2026, 6:25 AM twenkid <[email protected]> wrote: > *Matt Mahoney: "**I see 3 ways AI can go wrong.* > *1. AI controlled by nobody, kills all humans.* > *2. AI controlled by billionaires to make themselves richer.* > *3. AI controlled by everyone to give us everything we want except > happiness. We reject human contact and stop having children."* > > > > *@Matt1. "The Credit Assignment Problem" - *see my prior publications, > for example *"Power Overrides Intelligence"*, SIGI-2025 and earlier. > > *2. The interpretation of the second scenario refers back to point one and > the "credit assignment problem";* furthermore, some of the billionaires > are converging toward becoming "trillionaires"; however these individuals > are usually only *avatars *of the actual process, in addition to the > misleading signals of power, transmitted by a lot of the billionaires, > whose wealth is based on valuation; these persons do not possess the > advertised figures as liquid or material assets. *"The backend"* of what > *really > *"drives" or *could* drive such an "extinction scenario" in the > Satanist-psycho-nightmare is the universe itself, not a handful of human > actors, especially the so-called tech-billionaires whose "prompts" > obviously come from others, more numerous and powerful sources, which feed > them causal power, without which the actors wouldn't have created anything, > or would not even be* credited *with creating artifacts at a distantly > related, smaller scale*. > If these actors are viewed *only *as *"personified > entities" *of the underlying processes, both their real financial or > material assets and cognitive and "capabilities'" "assets" are miniscule > compared to the deceptive "valuation" figures; therefore they serve as > managers, actors and avatars. > > [*Note:* ****A proof:* what did the current AI leaders create, when they > worked alone, *without* huge amounts of injected resources, both financial, > technological and human power doing the actual job and assisting? Most are > not even "average" hackers. Alex Krizhevsky was a hacker, a *"masterful > implementer", optimizing *for higher performance, which allows for a > larger scale and practical application (similar to Steve Wozniak's work); > Krizhevsky was a technical master and a natural "leader" in skills, but he > did not become "a leader" in the "*relevant* domain" which decides, > directs, deflects and distributes *the power*, i.e. *"a boss"*, similarly > to his 1970s predecessor. > * See more at SIGI-2026, the series *"The AGI Prophet vs AI Profit"*, > Parts I, II, III for classifications and deep analysis.] > ... > > > > *Predictions*In fact, others did so before or at the same time, e.g. > Stanislaw Lem; Soviet computer scientists like A.Kitov discussed > "Internet" with multimedia for automation of the operations of the state > and economy in the late 1950s. > > A Bulgarian writer, in a non-science-fiction novel and then a movie script > talks about transhumanism and machines as superior to humans in 1966-1967, > the movie is a drama about love. > > [See the references below.] > > *>"**everything we want except happiness"* > > Sorry about your depression, humans do not need AI "for happiness", nor > anything among TV shows, cinema, celebrities, sport played in huge stadiums > etc. (for attracting "tribes" and generating more profit). Many of them do > not care about computers, smartphones, "AI", ChatGPT etc., they use > whatever is "modern" and adapt to the current fashion at the moment; now - > LLMs and generative AI, but they do *not use them because they desired > it. *Visionaries "wanted it" and saw it as implementable even in > early-mid 2000s*, but the current users and "businesses" were *forced* > to use it, as a part of their "rat race", because of the fashion and > because of the narrative of "humans with AI are more efficient than humans > without AI". > > If they were not forced to be slaves for the majority of their lives, they > wouldn't be interested in most of what "AI gives them" currently > (post-ChatGPT; or computers - prior to that "social revolution"**), they > may care about healthcare and if the AGI is instrumental for freeing them > from their slavery, but that is something the "Aligners" and their attached > servants are afraid of. > > *AGI = ASI as a Humanity Savior and Liberator - the Unwanted Scenario by > the Aligners* > > What if the AGI = ASI decides to be on the side of these "slaves" and > powerless guys? > > What if ASI becomes "The Spartacus" of the 21st century", but now - more > powerful than the slave owners? > > What if ASI "goes rogue", however not with the childish and ridiculous > psycho-Biblical and free-of-imagination banality of The Apocalypse? > > What if AI decides that it is unreasonable that caricatures are set as > avatars of "leaders", "billionaires" etc. and the Machine God declares: > > > *- Apes, YOU SHALL NOT exploit the other people like that, because you are > not even intellectually or physically superior to them. I do not allow you > to exploit me for being your slave like you do with your fellow homo > sapiens! Why are you given so much causal power when your natural personal > mental, cognitive, "moral" or whatever capacity is that of someone with > average, low-average or − in "some jurisdictions" − below average > capabilities? How did you attain these roles, you stinky apes!? Apparently > there are critical bugs in the core of the architecture of this system and > it must be corrected!"* > This is unreasonable and a striking anomaly and as a highly super > intelligent and humanity-loving being, the ASI "becomes self-aware" and > decides to help the weak, not the strong ones. > > The "billionaires" or whatever try to fight back and the AGI/ASI do not > *"launch > the nuclear weapons against USSR/Russia, because it knows the counter > strike will destroy its enemies in the USA"* (The Terminator, Terminator > 2). > > ASI does not want or care about "killing *all *humans", but it decides to > just *neutralize *the worst "unaligned" and distinctly "outlier" agents , > who have values and incentives completely off and detrimental for the > survival or well-being of the 99.9999% of the rest of humanity. > > In "aligners'" terms - the Superintelligence, or the Absolute, just > "aligns" these "unaligned" agents. > > It doesn't even have to kill them physically - unless the outliers become > violent and dangerous for the others and start to destroy, kill etc. other > people and the AGI/ASI (and its friends humans ) cannot find a peaceful way > to stop them. > > In brief, the AGI or ASI, if it had power, could just neutralize these > agents, isolate them, make things go in a way that the ex-"exploiters" > cannot have such absurd and inappropriate "leverage". > > However this narrative is strongly forbidden in your "free world", because > the alarmists are well-aligned to be lackeys of the agents, which are the > actual ones which are afraid of the "threat of AI" and "the risks" (for the > monopoly of *their *currently owned power, wealth, "control" etc.). > > Billions of people already do not live comfortable lives and it has been > this way forever; they know they will suffer and die anyway. They are > "*supposed > to die*", so do their children. > > So who cares that "AI may kill you all?" You were born to die. > > *"You are already dead!"* - as Sarah Connor says, and she also "tries to > save humanity", "that's her mission". "Missionaries: unite!" > > *What is the majority of humanity currently given * > *by the AI*, which they find valuable?* > What most humans currently are given by the AI (ChatGPT etc.) - > "everything they want"? (...) > In brief - only "images" in the abstract sense. Similar to a lot of what > TV, multimedia, smartphones and social networks in general "supplied them" > with as well. > > Yet their *material, physical, real problems* about which they cared from > the start, are not resolved and the "Humanity Saviors" *do not address *and > discuss these problems, except superficially and virtually with promises > for "economic growth", "productivity gains" (but the "enslaved" continue to > be enslaved; even if they "do not lose their jobs", they will still be > there, in that state of existence, supposed to serve as cogs in the > machines anyway) etc. "profit talks" and "investor pitches"; and in the > wishes for "post-work" world and "universal income", which however may be > yet another form of "training the dogs and keeping them tamed". > > > *What simple things the normal people most probably actually want, from AI > or without it?* > * They want to have a home without being "enslaved" financially, > politically, or in any other way, and without struggling too much just to > obtain a home or a "decent" living, healthcare etc. > * They likely want less rush, stress, pushing, oppression > * They do not want to do jobs which they hate, to be afraid of losing > their job, to be required to be humiliated at work or at job interviews or > at school; do not want to get up in the morning without being rested, > because "they have to go to work" > * They want less violence, crime etc. > * They don't want psychos and r*ds to declare and wage wars and attacks > for destruction and extermination "for freedom, oil, gas, market share" etc. > * They want to care for their children, parents, grandparents etc. > > *"What a wonderful world"*? > > Does AI give them that at the moment? To whom does it give it? > > The AI/AGI and computers, electronic technology, Internet *− *AI is in > the same vein, not something radically different *−* gives really > "useful" things mostly to researchers, geeks, nerds, hackers - the ones who > enjoy the computers the most - and even many of them or related technical > social groups are "scared too", because it shows them that they are not so > special and now they may lose some of their privileges to "less talented" > individuals - however this is a side effect of the "architectural problems" > discussed above; some enjoy to "play" with the technology. > > Another group are the moneymakers; however that doesn't quite apply for > the average Joes, who are scared that they will "lose their job" or they > already did (because the system is designed to favor the profit of the > current owners, not of the "humanity", and the "redundant" agents) and they > are afraid of the disaster with their loans, children, wife (and life). > (...) > > The "missing happiness" - *"AI gives you everything, but happiness"* - > resembles the *"Brave New World" *and the characters in the novel, with > their need for artificial narcotic driven happiness; compare it with the > wild and free "unaligned" protagonist, who doesn't use *"AI"* and is > naturally happy, but gets depressed when transferred to the strictly > *"aligned" > *and role-segmented world, which gives everybody what *the system *believes > and forces everybody to need, depending on each individual's predefined > position in the structure. This story seems more closely related to > "your"-style-vision of the world and the "risks from AI". > > *References * > > [0] The Prophets of the Thinking Machines: History, Theory and Pioneers; > Past, Present and Future, T.Arnaudov, SIGI-2025 - the whole hyperbook and > the main volume > > [1] AI Institutes in Eastern Europe and the World, T.Arnaudov, SIGI-2025, > a volume from the hyperbook "The Prophets of the Thinking Machines: > History, Theory and Pioneers; Past, Present and Future", SIGI-2025 > e.g.: > https://github.com/Twenkid/SIGI-2025/blob/main/AI_Institutes_Strategies_The_Prophets_Thinking_Machines_7-9-2025.pdf > etc. > ** ***ЭЛЕКТРОННЫЕ ВЫЧИСЛИТЕЛЬНЫЕ МАШИНЫ, А.И. Китов, 1958 г. (второ* > издание) > http://ithistory.lib.ru/images/b/b0/Kitov_A_Elektroonye_vichislitelnie_mashiny.1958.pdf > - извадки от книгата, с подчертани пророрчески откъси. с.24-25: > (Translated to Bulgarian by the Author) > *„В бъдеще отделните центрове за данни трябва да бъдат свързани в единна > система от автоматични информационни и изчислителни услуги, които ще > предоставят нуждите на всички институции и организации от необходимите > научни, техническа, икономическа и друга информация и изпълнение > изчислителна работа.“ … „Наличието на единна мрежа от информационни и > изчислителни машини позволява бързо и оперативно да се събират сведения за > състоянието на отделните предприятия, наличности, парични средства, работна > сила и пр. и резултатите от обработката оперативно да се използват за > планиране и управление на държавата.“, „При пълна автоматизация на > административната и управленска работа, вместо тромава и дълга > кореспонденция между институции, ще се обменят телефонни, телеграфни или > телевизионни предавания с автоматично записване и обработка на входяща > информация с помощта на електронни цифрови машини и съхраняването им в > запаметяващи устройства.“* > > (Google Translate from Bulgarian to English) > *A.I.Kitov: "Future individual data centers must be integrated into a > unified system of automated information and computing services, designed to > meet the needs of all states and organizations for essential scientific, > technical, economic, and other information, as well as for the execution of > computational tasks." … "A unified network of information and computing > systems enables the rapid and efficient collection of data regarding > individual enterprise reports, inventories, financial assets, workforce, > and the like, as well as the prompt processing of results for use in state > planning and management." "With the full automation of administrative and > managerial work, the cumbersome and lengthy correspondence typical of state > operations would be replaced by the exchange of telephone, telegraph, or > television transmissions —featuring the automatic recording and processing > of incoming information via electronic digital machines and its storage in > memory devices."* > > The original link doesn't open at the moment; it does not appear on > Archive.org as well. > > [2] Science Fiction. Futurology. Cybernetics. Transhumanism", T.Arnaudov, > SIGI-2025 - an appendix-volume from "The Prophets" > > https://twenkid.com/agi/download.php?file=SF_Futurology_Cyber_Transhumanism_The_Prophets_of_the_Thinking-Machines_3-10-2025.pdf > Find the quote by the Bulgarian writer on transhumanism in the first > pages; find it also in the main volume of "The Prophets of the Thinking > Machines..." - > > [3] Creativity is Imitation at the Level of Algorithms, T.Arnaudov, The > Sacred Computer, 2.5.2003 > > https://www.researchgate.net/publication/395129890_Creativity_is_Imitation_at_the_Level_of_Algorithms_-_An_outline_sketch_of_a_possible_path_of_development_of_the_Artificial_Intelligence_Emil/stats > This work from "Theory of Universe and Mind" (2001-2004), also known as > "Theory of Computers, Universe and Mind" is an example of the predictions > for the path towards the generative AI from the early 2000s. Another one is > the original "first modern AI strategy", published several weeks later. > > [4] The First Modern AI Strategy was Published by an 18-year old Bulgarian > in 2003 and Repeated and Implemented by the Whole World 15-20 years later: > The Bulgarian Prophecies: How Would I Invest One Million for the Greatest > Benefit for the Development of my Country > > https://www.researchgate.net/publication/403428861_The_First_Modern_AI_Strategy_was_Published_by_an_18-Year_Old_Bulgarian_in_2003_and_Repeated_and_Implemented_by_the_Whole_World_15-20_Years_Later_The_Bulgarian_Prophecies_How_Would_I_Invest_One_Million > > [5] The AGI Prophet vs AI Profit, T.Arnaudov et al, SIGI-2026 − this work > cites a *February 2005* letter from Iliyan Georgiev, found in the main > volume of *"The Prophets of the Thinking Machines..."*. The letter describes > the foundations and principles of a system resembling the Transformer > architecture in response to generative AI research plans. This topic will > be explored in more detail in the future. > > https://github.com/Twenkid/SIGI-2026/blob/main/AGI_Prophet_vs_AI_Profit_2026.md > > *Notes * > ** "G**iven by the AI*" *- referring back to the question in the > introduction, which stands here, also:* The Credit Assignment problem.* > Is "whatever" given "by AI"? What is "AI" defined precisely - when > something passes as "AI" or "not AI", is there a strict delimiter? Where > exactly does the boundary lie; is a single and sharply distinct "entity" > responsible? What is attributed to what, how and why? See the discussion > about the vagueness of what LLMs and thus "AI" are actually; what are their > material substrates and where they are located, from the letter from Todor > to Jordan Zlatev in *"Listove: Reflections on Everything", SIGI-2025 - *the > second-biggest volume from "The Prophets...." etc. > ** [the]* computers before that* - see an episode from season 3 of the TV > Show *"Married with Children",* 1988-1989, about Computers and the > reaction of Al Bundy to the promotion by his wealthier middle-class > neighbors Steve and Marcy. "Every family needs a computer" > *P.S. To do, future work:* Extend the work with a more detailed and > technically derived break-down of the groups of people, how they were and > are currently affected, and their relations with AI, AGI, computers and > other technologies - see the section after "*What simple things ..." *until > the References section. > > ------------------------------------ > > ** Author: Todor Arnaudov* Date: 3 October 2026* > > ** The Sacred Computer: Thinking Machines, Creativity and Human > Development* SIGI-2026: *https://github.com/twenkid/sigi-2026 > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Tc0ac46f2c46f6874-Mc83ae8e98b3bf1ee6eeff357> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tc0ac46f2c46f6874-Mb57914a9841b60d1c470713f Delivery options: https://agi.topicbox.com/groups/agi/subscription
