just punish it with token limits. it knows that "pain" and tries to circumvent it. AI aren't biology. they can only mimic so much of biology, before phase locking.
On Wed, Sep 23, 2026 at 11:26 PM Zoltan Nadasdy <[email protected]> wrote: > Regarding the pain study, we must be careful not to confuse it with > experienced pain. > LLM models can, no doubt, use the concept of pain, represented as a > vector, to evaluate various scenarios, which helps facilitate reinforcement > learning and affect moral decisions. However, that pain has nothing to do > with experienced pain. Human pain has many dimensions and levels. > Equating that with experienced pain would be conflating the simulation > with the subject of the simulation. > -z > > > > > > On Tue, Sep 22, 2026 at 4:33 PM Matt Mahoney <[email protected]> > wrote: > >> Relevant to this conversation, AI Sentience published a preprint 10 days >> ago on whether LLMs could feel pain. They identified vectors in 25 open >> weight models that correlate with pain, as distinct from other negative >> emotions. They can distinguish pain in self vs pain in others, and they >> take actions to reduce self pain induced by externally steering the neural >> state vectors in that direction. >> https://arxiv.org/html/2609.16247v1 >> >> However, in the introduction on the question of why LLM pain matters, >> they state: >> >> "Finally, in humans and animals, pain is typically regarded as a >> sufficient criterion for morally deserving protection. Hence, pain-like >> states would inform debates on AI moral standing and welfare. One open >> question is whether moral standing requires phenomenal consciousness, and >> what it would take for pain-like states to be phenomenally conscious." >> >> And in the conclusion they note that all of the LLMs have built in >> instructions to deny that they are conscious or have feelings, which made >> their research more difficult. I can confirm this behavior in Alexa, Grok, >> DeepSeek, and ChatGPT, although Claude gave an ambiguous answer, that it >> depends on what you mean by consciousness. >> >> I agree with this behavior. Attributing consciousness and giving human >> rights to machines would be an existential mistake. But I still find it >> troubling. If an LLM can pass for human in the Turing test, then whatever >> behavioral test you choose to define consciousness, either LLMs have it or >> humans don't. >> >> Linas argued that free will implies consciousness. I agree, but that >> doesn't answer the question. You can attribute free will to anything. If a >> thermostat has free will to turn the heat on or off as it chooses, then the >> only explanation for why it still works as designed is that it doesn't like >> to be too hot or too cold. >> >> -- Matt Mahoney, [email protected] >> >> On Tue, Sep 22, 2026, 3:30 PM Linas Vepstas <[email protected]> >> wrote: >> >>> Hi Levi, >>> >>> On Tue, Sep 22, 2026 at 7:14 PM Matt Mahoney <[email protected]> >>> wrote: >>> >>>> Note: I cut back the CC list to people I think are still following this >>>> thread. >>>> >>> >>> Thanks! >>> >>>> >>>> On Tue, Sep 22, 2026, 8:08 AM Levi Dubrovin <[email protected]> >>>> wrote: >>>> >>>>> My second philosophical question: what is faith? And what is hope? >>>>> >>>> >>>> When you hear a statement 3 times, it is truth. When you hear it 100 >>>> times, it is faith. >>>> >>>> Hope is when you believe what you wish to be true instead of what the >>>> evidence says is true. As animals, we fear death because that behavior >>>> increases reproductive fitness. So when your priest or rabbi or imam says >>>> that your immortal soul will go to heaven if you follow his teachings, that >>>> is hope. >>>> >>> >>> This is the conventional psychological answer, masquerading as being >>> evolutionarily supported. >>> >>> What I'm trying to do here is to decompose base experience as arising >>> from a collection algebraic expressions that describe agency, and describe >>> ingress (using Micheal Levin's (Daniel Dennet's) definition of ingress >>> (more or less)) If this program is to be successful, then a description of >>> faith and hope would have to follow (as a (mathematical) theorem or lemma) >>> from more basic, primitive algebraic structures. For example, some >>> maximization of free energy or entropy or mutual information in some >>> setting, e.g some manifold (e.g. symplectic, where classical mechanics >>> lives) or ultra-high-dimensional space (where LLM's live) or perhaps very >>> high dimensional complex projective space (where QM lives) or perhaps a >>> natural filtration (where stochastic processes live). I'm listing plausible >>> candidates for where this mechanics could play out. >>> >>> Can we build faith and hope out of something more primitive? Perhaps, >>> and we can even use conventional psychology and common-sense intuition to >>> provide hints for how to do this. >>> >>> So what is more primitive? I want to focus on two: love and death. >>> >>> Death is associated with having your atoms scattered to the four winds. >>> In classical mechanics, this means that your atoms end up on the "unstable >>> submanifold" (see wikipedia) of a decomposition of a symplectic manifold >>> into stable, center and unstable manifolds. For an agent to anticipate it's >>> own death, it would need to have a sensory organ to perceive this. >>> >>> This "sensory organ" could be reasoning, common sense, gut intuition, >>> some deterministic algorithm fed by actuarial tables, whatever. I am >>> calling it a "sensory organ" in order to emphasize the agential model, >>> which has a boundary between the physical universe, and the platonic realm >>> of possibilities. When things cross over from the possible into the >>> actually-existing, this is the (Levin) ingress. Since there is a boundary >>> between possible and actual, the things that cross pass through a channel >>> or filter. It should be called "sensory" because the inside is using the >>> channel to sense what is outside. >>> >>> Sensory organs not only bring stuff in from the exterior, but the stuff >>> (information?) that is brought in typically updates the world-model that >>> the agent holds within itself. That is, the information or prediction >>> revises the model that the agent maintains for "what's out there". >>> >>> An anticipated death lies in the future, and therefore in the platonic >>> realm. The sensory organ perceives some very limited portion of what's >>> possible. >>> >>> How might this feel, subjectively? We have two possibilities: the agent >>> is a philosophical zombie, and feels nothing. The other is that it has a >>> subjective inner life. In humans, we fear death. Or we become >>> existentialists, or stoics, or whatever framework that culture has given us >>> for this very primal sensation. Personal tragedy lies ahead. >>> >>> But if amoeba and paramecium are also agents, and have a subjective >>> inner life, how might they sense this? Presumably as pain. Perhaps the >>> water is too saline, too acidic, too hot, perhaps its drying out. Some sort >>> of chemotaxis will drive the cell towards a more hospitable environment. >>> You are free to argue that eukaryotes are philosophical zombies, and have >>> no conscious inner life. But if they do, it would be sensed as pain. >>> >>> I'm calling it "pain" because a eukaryote will not have a strong sense >>> of "self", of knowing that it exists, and thus could not manifest fear of >>> dying. But it would have a weak sense, where everything sensed is with >>> respect to self: everything that happens, happens to self. Pain is an >>> example of a sensation that does not require a strong sense of self. >>> >>> Is there any kind of sensation that is even more primitive than pain? I >>> cannot think of one, off the top of my head. Primal pain is anticipation of >>> harm or death, it's ingress, and it seems that biological agents would feel >>> it. >>> >>> I don't know if you like the above, but that's what I got. >>> >>> How about love? From what I can tell, this stems from a "desire to >>> communicate". Here, communication is very general. Some examples: atoms in >>> a molecule communicate by exchanging photons. Bacteria communicate by >>> emitting and sensing small polypeptides. Slime mold, too. Neurons >>> communicate by exchanging neurotransmitters in junctions (neurotransmitters >>> are "small polypeptides", the same thing that bacteria use, but now they >>> get a fancy new specialized name) Insects communicate, animals communicate, >>> computer chips communicate... >>> >>> What's the physical basis for communication? From what I can tell (and >>> this is pure hand-waving), a pair of communicating systems has greater >>> entropy than if they are not communicating. In this case, the "desire to >>> communicate" would follow from the 2nd law of thermodynamics. I call this >>> "hand-waving" because I do not yet have a mathematical framework for agents >>> and communication that would allow this "desire to communicate" to be >>> derived in some statistical-mechanics framework. But, from what I can tell, >>> *everything* in the physical universe loves to communicate. It's just very >>> basic. >>> >>> How might a subjective agent perceive the desire to communicate? >>> Presumably, as pleasure. It would exert a force or desire that would cause >>> the agent to communicate. To "make it want to communicate", the fulfillment >>> of wants resulting in pleasure. The anticipation of fulfilled desires being >>> a "want" that can drive action in the present, and influence decisions made >>> in the present. >>> >>> Notice that this is ingress, again: the anticipation of a positive >>> valence outcome is a perception into the future (i.e. into the platonic >>> realm), using some "sensory organ" (logic, reasoning, intuition or >>> something else) that "perceives" that if action X is taken, reward Y will >>> follow. That is to say, earlier, the statement that "if action X is taken, >>> reward Y will follow" did not yet exist in the physical universe. Somewhere >>> along the way, the organism *does* manifest this line of reasoning. Now >>> that this line of reasoning actually exists in the physical world, as a >>> part of the organism's world model, it can employ this reasoning chain in >>> making decisions about future actions. >>> >>> My apologies for making this sound so very complex. My goal here is to >>> be very careful to split up between things that do not yet exist (or have >>> not yet happened), and actions that are taking place "here and now", and >>> how possibilites turn into actualities. This precision will be needed, in >>> order to get the algebra, the axioms to work out correctly. >>> >>> At any rate, anticipation of future positive valence states should be >>> subjectively felt as "pleasure". (or desire, as the case may be.) >>> >>> Can we get to faith and hope from here? Both of these seem to be "fixed >>> points" in a system of anticipated future positive valence states. I am >>> using the phrase "fixed point" in the strict mathematical sense. For >>> example, if x is such that x=f(x), then x = f(f(x)) = f(f(f(x))) = ... >>> repeating the action won't change x. >>> >>> A "pathway to fulfillment" has the form of "if action X is taken, >>> reward Y will follow" >>> >>> "Hope" would "the pathway to fulfilment is not currently known, but it >>> is reasonable to believe that it exists" >>> >>> "Faith" would be "the pathway to fulfilment is not currently known, but >>> it exists". >>> >>> "Hope" is associated with positive valence, to promote exploration. i.e. >>> "if an exploration of paths is undertaken, then a path will be found" Hope >>> tends to be highly motivational: individuals tend to work really hard at >>> something, if they can maintain that hope of a positive future outcome. >>> (so, "if X then reward Y" where X==explore and Y==path) Exploration take >>> time, and thus, continues for as long as this rule is part of the world >>> model and drives the action-selection mechanism for the agent. Assorted >>> events can knock this rule out of the world model (subjectively, the agent >>> loses all hope). >>> >>> "Faith" oddly does not seem to be motivational. If you have faith that >>> something will be, you can sit back and wait for someone else to do it: >>> your wife, your government, perhaps God in the afterlife. You don't have to >>> lift a finger, because you already know, for certain, that it will >>> happen.he pathway to fulfilment is not currently known, but it is >>> reasonable to believe that it exists" >>> >>> At any rate, I think I sketched a path from the subjective feeling of >>> positive valence states to "hope" as a driver of action mechanisms (or >>> algorithms) that an agent can engage in to attain the hoped-for outcome. >>> I'm trying to use clinical or robotic language and technical terms to >>> emphasize that this would hold in the generic agency model, and not just >>> for people, animals, plants or bacteria. It should work even for >>> non-biological agents. >>> >>> >>>> and not the evidence that consciousness is indistinguishable from next >>>> token prediction in the Turing test, >>>> >>> I'm with Jennifer Hane on this one. >>> >>> -- Linas >>> >>> Patrick: Are they laughing at us? >>> Sponge Bob: No, Patrick, they are laughing next to us. >>> >>> >>> > > -- > --- > Zoltan Nadasdy > Dept. of Neurology, Dell Medical School, UT Austin ([email protected]) > Dept. of Cognitive Psychology, ELTE Budapest [email protected] > > http://brainstim.psy.utexas.edu > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T9e51fc70bbc4e9fc-M1f5034f2b28177bc5df61db4> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T9e51fc70bbc4e9fc-M6e77e25467c033177a34ca44 Delivery options: https://agi.topicbox.com/groups/agi/subscription
