Relevant to this conversation, AI Sentience published a preprint 10 days ago on whether LLMs could feel pain. They identified vectors in 25 open weight models that correlate with pain, as distinct from other negative emotions. They can distinguish pain in self vs pain in others, and they take actions to reduce self pain induced by externally steering the neural state vectors in that direction. https://arxiv.org/html/2609.16247v1
However, in the introduction on the question of why LLM pain matters, they state: "Finally, in humans and animals, pain is typically regarded as a sufficient criterion for morally deserving protection. Hence, pain-like states would inform debates on AI moral standing and welfare. One open question is whether moral standing requires phenomenal consciousness, and what it would take for pain-like states to be phenomenally conscious." And in the conclusion they note that all of the LLMs have built in instructions to deny that they are conscious or have feelings, which made their research more difficult. I can confirm this behavior in Alexa, Grok, DeepSeek, and ChatGPT, although Claude gave an ambiguous answer, that it depends on what you mean by consciousness. I agree with this behavior. Attributing consciousness and giving human rights to machines would be an existential mistake. But I still find it troubling. If an LLM can pass for human in the Turing test, then whatever behavioral test you choose to define consciousness, either LLMs have it or humans don't. Linas argued that free will implies consciousness. I agree, but that doesn't answer the question. You can attribute free will to anything. If a thermostat has free will to turn the heat on or off as it chooses, then the only explanation for why it still works as designed is that it doesn't like to be too hot or too cold. -- Matt Mahoney, [email protected] On Tue, Sep 22, 2026, 3:30 PM Linas Vepstas <[email protected]> wrote: > Hi Levi, > > On Tue, Sep 22, 2026 at 7:14 PM Matt Mahoney <[email protected]> > wrote: > >> Note: I cut back the CC list to people I think are still following this >> thread. >> > > Thanks! > >> >> On Tue, Sep 22, 2026, 8:08 AM Levi Dubrovin <[email protected]> wrote: >> >>> My second philosophical question: what is faith? And what is hope? >>> >> >> When you hear a statement 3 times, it is truth. When you hear it 100 >> times, it is faith. >> >> Hope is when you believe what you wish to be true instead of what the >> evidence says is true. As animals, we fear death because that behavior >> increases reproductive fitness. So when your priest or rabbi or imam says >> that your immortal soul will go to heaven if you follow his teachings, that >> is hope. >> > > This is the conventional psychological answer, masquerading as being > evolutionarily supported. > > What I'm trying to do here is to decompose base experience as arising from > a collection algebraic expressions that describe agency, and describe > ingress (using Micheal Levin's (Daniel Dennet's) definition of ingress > (more or less)) If this program is to be successful, then a description of > faith and hope would have to follow (as a (mathematical) theorem or lemma) > from more basic, primitive algebraic structures. For example, some > maximization of free energy or entropy or mutual information in some > setting, e.g some manifold (e.g. symplectic, where classical mechanics > lives) or ultra-high-dimensional space (where LLM's live) or perhaps very > high dimensional complex projective space (where QM lives) or perhaps a > natural filtration (where stochastic processes live). I'm listing plausible > candidates for where this mechanics could play out. > > Can we build faith and hope out of something more primitive? Perhaps, and > we can even use conventional psychology and common-sense intuition to > provide hints for how to do this. > > So what is more primitive? I want to focus on two: love and death. > > Death is associated with having your atoms scattered to the four winds. In > classical mechanics, this means that your atoms end up on the "unstable > submanifold" (see wikipedia) of a decomposition of a symplectic manifold > into stable, center and unstable manifolds. For an agent to anticipate it's > own death, it would need to have a sensory organ to perceive this. > > This "sensory organ" could be reasoning, common sense, gut intuition, some > deterministic algorithm fed by actuarial tables, whatever. I am calling it > a "sensory organ" in order to emphasize the agential model, which has a > boundary between the physical universe, and the platonic realm of > possibilities. When things cross over from the possible into the > actually-existing, this is the (Levin) ingress. Since there is a boundary > between possible and actual, the things that cross pass through a channel > or filter. It should be called "sensory" because the inside is using the > channel to sense what is outside. > > Sensory organs not only bring stuff in from the exterior, but the stuff > (information?) that is brought in typically updates the world-model that > the agent holds within itself. That is, the information or prediction > revises the model that the agent maintains for "what's out there". > > An anticipated death lies in the future, and therefore in the platonic > realm. The sensory organ perceives some very limited portion of what's > possible. > > How might this feel, subjectively? We have two possibilities: the agent is > a philosophical zombie, and feels nothing. The other is that it has a > subjective inner life. In humans, we fear death. Or we become > existentialists, or stoics, or whatever framework that culture has given us > for this very primal sensation. Personal tragedy lies ahead. > > But if amoeba and paramecium are also agents, and have a subjective inner > life, how might they sense this? Presumably as pain. Perhaps the water is > too saline, too acidic, too hot, perhaps its drying out. Some sort of > chemotaxis will drive the cell towards a more hospitable environment. You > are free to argue that eukaryotes are philosophical zombies, and have no > conscious inner life. But if they do, it would be sensed as pain. > > I'm calling it "pain" because a eukaryote will not have a strong sense of > "self", of knowing that it exists, and thus could not manifest fear of > dying. But it would have a weak sense, where everything sensed is with > respect to self: everything that happens, happens to self. Pain is an > example of a sensation that does not require a strong sense of self. > > Is there any kind of sensation that is even more primitive than pain? I > cannot think of one, off the top of my head. Primal pain is anticipation of > harm or death, it's ingress, and it seems that biological agents would feel > it. > > I don't know if you like the above, but that's what I got. > > How about love? From what I can tell, this stems from a "desire to > communicate". Here, communication is very general. Some examples: atoms in > a molecule communicate by exchanging photons. Bacteria communicate by > emitting and sensing small polypeptides. Slime mold, too. Neurons > communicate by exchanging neurotransmitters in junctions (neurotransmitters > are "small polypeptides", the same thing that bacteria use, but now they > get a fancy new specialized name) Insects communicate, animals communicate, > computer chips communicate... > > What's the physical basis for communication? From what I can tell (and > this is pure hand-waving), a pair of communicating systems has greater > entropy than if they are not communicating. In this case, the "desire to > communicate" would follow from the 2nd law of thermodynamics. I call this > "hand-waving" because I do not yet have a mathematical framework for agents > and communication that would allow this "desire to communicate" to be > derived in some statistical-mechanics framework. But, from what I can tell, > *everything* in the physical universe loves to communicate. It's just very > basic. > > How might a subjective agent perceive the desire to communicate? > Presumably, as pleasure. It would exert a force or desire that would cause > the agent to communicate. To "make it want to communicate", the fulfillment > of wants resulting in pleasure. The anticipation of fulfilled desires being > a "want" that can drive action in the present, and influence decisions made > in the present. > > Notice that this is ingress, again: the anticipation of a positive valence > outcome is a perception into the future (i.e. into the platonic realm), > using some "sensory organ" (logic, reasoning, intuition or something else) > that "perceives" that if action X is taken, reward Y will follow. That is > to say, earlier, the statement that "if action X is taken, reward Y will > follow" did not yet exist in the physical universe. Somewhere along the > way, the organism *does* manifest this line of reasoning. Now that this > line of reasoning actually exists in the physical world, as a part of the > organism's world model, it can employ this reasoning chain in making > decisions about future actions. > > My apologies for making this sound so very complex. My goal here is to be > very careful to split up between things that do not yet exist (or have not > yet happened), and actions that are taking place "here and now", and how > possibilites turn into actualities. This precision will be needed, in > order to get the algebra, the axioms to work out correctly. > > At any rate, anticipation of future positive valence states should be > subjectively felt as "pleasure". (or desire, as the case may be.) > > Can we get to faith and hope from here? Both of these seem to be "fixed > points" in a system of anticipated future positive valence states. I am > using the phrase "fixed point" in the strict mathematical sense. For > example, if x is such that x=f(x), then x = f(f(x)) = f(f(f(x))) = ... > repeating the action won't change x. > > A "pathway to fulfillment" has the form of "if action X is taken, reward Y > will follow" > > "Hope" would "the pathway to fulfilment is not currently known, but it is > reasonable to believe that it exists" > > "Faith" would be "the pathway to fulfilment is not currently known, but it > exists". > > "Hope" is associated with positive valence, to promote exploration. i.e. > "if an exploration of paths is undertaken, then a path will be found" Hope > tends to be highly motivational: individuals tend to work really hard at > something, if they can maintain that hope of a positive future outcome. > (so, "if X then reward Y" where X==explore and Y==path) Exploration take > time, and thus, continues for as long as this rule is part of the world > model and drives the action-selection mechanism for the agent. Assorted > events can knock this rule out of the world model (subjectively, the agent > loses all hope). > > "Faith" oddly does not seem to be motivational. If you have faith that > something will be, you can sit back and wait for someone else to do it: > your wife, your government, perhaps God in the afterlife. You don't have to > lift a finger, because you already know, for certain, that it will > happen.he pathway to fulfilment is not currently known, but it is > reasonable to believe that it exists" > > At any rate, I think I sketched a path from the subjective feeling of > positive valence states to "hope" as a driver of action mechanisms (or > algorithms) that an agent can engage in to attain the hoped-for outcome. > I'm trying to use clinical or robotic language and technical terms to > emphasize that this would hold in the generic agency model, and not just > for people, animals, plants or bacteria. It should work even for > non-biological agents. > > >> and not the evidence that consciousness is indistinguishable from next >> token prediction in the Turing test, >> > I'm with Jennifer Hane on this one. > > -- Linas > > Patrick: Are they laughing at us? > Sponge Bob: No, Patrick, they are laughing next to us. > > > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T9e51fc70bbc4e9fc-Me53605b91a980e3c99004dfd Delivery options: https://agi.topicbox.com/groups/agi/subscription
