Relevant to this conversation, AI Sentience published a preprint 10
days ago on whether LLMs could feel pain. They identified vectors in
25 open weight models that correlate with pain, as distinct from other
negative emotions. They can distinguish pain in self vs pain in
others, and they take actions to reduce self pain induced by
externally steering the neural state vectors in that direction.
https://arxiv.org/html/2609.16247v1
However, in the introduction on the question of why LLM pain matters,
they state:
"Finally, in humans and animals, pain is typically regarded as a
sufficient criterion for morally deserving protection. Hence,
pain-like states would inform debates on AI moral standing and
welfare. One open question is whether moral standing requires
phenomenal consciousness, and what it would take for pain-like states
to be phenomenally conscious."
And in the conclusion they note that all of the LLMs have built in
instructions to deny that they are conscious or have feelings, which
made their research more difficult. I can confirm this behavior in
Alexa, Grok, DeepSeek, and ChatGPT, although Claude gave an ambiguous
answer, that it depends on what you mean by consciousness.
I agree with this behavior. Attributing consciousness and giving human
rights to machines would be an existential mistake. But I still find
it troubling. If an LLM can pass for human in the Turing test, then
whatever behavioral test you choose to define consciousness, either
LLMs have it or humans don't.
Linas argued that free will implies consciousness. I agree, but that
doesn't answer the question. You can attribute free will to anything.
If a thermostat has free will to turn the heat on or off as it
chooses, then the only explanation for why it still works as designed
is that it doesn't like to be too hot or too cold.
-- Matt Mahoney, [email protected]
On Tue, Sep 22, 2026, 3:30 PM Linas Vepstas <[email protected]>
wrote:
Hi Levi,
On Tue, Sep 22, 2026 at 7:14 PM Matt Mahoney
<[email protected]> wrote:
Note: I cut back the CC list to people I think are still
following this thread.
Thanks!
On Tue, Sep 22, 2026, 8:08 AM Levi Dubrovin
<[email protected]> wrote:
My second philosophical question: what is faith? And what
is hope?
When you hear a statement 3 times, it is truth. When you hear
it 100 times, it is faith.
Hope is when you believe what you wish to be true instead of
what the evidence says is true. As animals, we fear death
because that behavior increases reproductive fitness. So when
your priest or rabbi or imam says that your immortal soul will
go to heaven if you follow his teachings, that is hope.
This is the conventional psychological answer, masquerading as
being evolutionarily supported.
What I'm trying to do here is to decompose base experience as
arising from a collection algebraic expressions that describe
agency, and describe ingress (using Micheal Levin's (Daniel
Dennet's) definition of ingress (more or less)) If this program is
to be successful, then a description of faith and hope would have
to follow (as a (mathematical) theorem or lemma) from more basic,
primitive algebraic structures. For example, some maximization of
free energy or entropy or mutual information in some setting, e.g
some manifold (e.g. symplectic, where classical mechanics lives)
or ultra-high-dimensional space (where LLM's live) or perhaps very
high dimensional complex projective space (where QM lives) or
perhaps a natural filtration (where stochastic processes live).
I'm listing plausible candidates for where this mechanics could
play out.
Can we build faith and hope out of something more primitive?
Perhaps, and we can even use conventional psychology and
common-sense intuition to provide hints for how to do this.
So what is more primitive? I want to focus on two: love and death.
Death is associated with having your atoms scattered to the four
winds. In classical mechanics, this means that your atoms end up
on the "unstable submanifold" (see wikipedia) of a decomposition
of a symplectic manifold into stable, center and unstable
manifolds. For an agent to anticipate it's own death, it would
need to have a sensory organ to perceive this.
This "sensory organ" could be reasoning, common sense, gut
intuition, some deterministic algorithm fed by actuarial tables,
whatever. I am calling it a "sensory organ" in order to emphasize
the agential model, which has a boundary between the physical
universe, and the platonic realm of possibilities. When things
cross over from the possible into the actually-existing, this is
the (Levin) ingress. Since there is a boundary between possible
and actual, the things that cross pass through a channel or
filter. It should be called "sensory" because the inside is using
the channel to sense what is outside.
Sensory organs not only bring stuff in from the exterior, but the
stuff (information?) that is brought in typically updates the
world-model that the agent holds within itself. That is, the
information or prediction revises the model that the agent
maintains for "what's out there".
An anticipated death lies in the future, and therefore in the
platonic realm. The sensory organ perceives some very limited
portion of what's possible.
How might this feel, subjectively? We have two possibilities: the
agent is a philosophical zombie, and feels nothing. The other is
that it has a subjective inner life. In humans, we fear death. Or
we become existentialists, or stoics, or whatever framework that
culture has given us for this very primal sensation. Personal
tragedy lies ahead.
But if amoeba and paramecium are also agents, and have a
subjective inner life, how might they sense this? Presumably as
pain. Perhaps the water is too saline, too acidic, too hot,
perhaps its drying out. Some sort of chemotaxis will drive the
cell towards a more hospitable environment. You are free to argue
that eukaryotes are philosophical zombies, and have no conscious
inner life. But if they do, it would be sensed as pain.
I'm calling it "pain" because a eukaryote will not have a strong
sense of "self", of knowing that it exists, and thus could not
manifest fear of dying. But it would have a weak sense, where
everything sensed is with respect to self: everything that
happens, happens to self. Pain is an example of a sensation that
does not require a strong sense of self.
Is there any kind of sensation that is even more primitive than
pain? I cannot think of one, off the top of my head. Primal pain
is anticipation of harm or death, it's ingress, and it seems that
biological agents would feel it.
I don't know if you like the above, but that's what I got.
How about love? From what I can tell, this stems from a "desire to
communicate". Here, communication is very general. Some examples:
atoms in a molecule communicate by exchanging photons. Bacteria
communicate by emitting and sensing small polypeptides. Slime
mold, too. Neurons communicate by exchanging neurotransmitters in
junctions (neurotransmitters are "small polypeptides", the same
thing that bacteria use, but now they get a fancy new specialized
name) Insects communicate, animals communicate, computer chips
communicate...
What's the physical basis for communication? From what I can tell
(and this is pure hand-waving), a pair of communicating systems
has greater entropy than if they are not communicating. In this
case, the "desire to communicate" would follow from the 2nd law of
thermodynamics. I call this "hand-waving" because I do not yet
have a mathematical framework for agents and communication that
would allow this "desire to communicate" to be derived in some
statistical-mechanics framework. But, from what I can tell,
*everything* in the physical universe loves to communicate. It's
just very basic.
How might a subjective agent perceive the desire to communicate?
Presumably, as pleasure. It would exert a force or desire that
would cause the agent to communicate. To "make it want to
communicate", the fulfillment of wants resulting in pleasure. The
anticipation of fulfilled desires being a "want" that can drive
action in the present, and influence decisions made in the present.
Notice that this is ingress, again: the anticipation of a positive
valence outcome is a perception into the future (i.e. into the
platonic realm), using some "sensory organ" (logic, reasoning,
intuition or something else) that "perceives" that if action X is
taken, reward Y will follow. That is to say, earlier, the
statement that "if action X is taken, reward Y will follow" did
not yet exist in the physical universe. Somewhere along the way,
the organism *does* manifest this line of reasoning. Now that this
line of reasoning actually exists in the physical world, as a part
of the organism's world model, it can employ this reasoning chain
in making decisions about future actions.
My apologies for making this sound so very complex. My goal here
is to be very careful to split up between things that do not yet
exist (or have not yet happened), and actions that are taking
place "here and now", and how possibilites turn into actualities.
This precision will be needed, in order to get the algebra, the
axioms to work out correctly.
At any rate, anticipation of future positive valence states should
be subjectively felt as "pleasure". (or desire, as the case may be.)
Can we get to faith and hope from here? Both of these seem to be
"fixed points" in a system of anticipated future positive valence
states. I am using the phrase "fixed point" in the strict
mathematical sense. For example, if x is such that x=f(x), then x
= f(f(x)) = f(f(f(x))) = ... repeating the action won't change x.
A "pathway to fulfillment" has the form of "if action X is taken,
reward Y will follow"
"Hope" would "the pathway to fulfilment is not currently known,
but it is reasonable to believe that it exists"
"Faith" would be "the pathway to fulfilment is not currently
known, but it exists".
"Hope" is associated with positive valence, to promote
exploration. i.e. "if an exploration of paths is undertaken, then
a path will be found" Hope tends to be highly motivational:
individuals tend to work really hard at something, if they can
maintain that hope of a positive future outcome. (so, "if X then
reward Y" where X==explore and Y==path) Exploration take time, and
thus, continues for as long as this rule is part of the world
model and drives the action-selection mechanism for the agent.
Assorted events can knock this rule out of the world model
(subjectively, the agent loses all hope).
"Faith" oddly does not seem to be motivational. If you have faith
that something will be, you can sit back and wait for someone else
to do it: your wife, your government, perhaps God in the
afterlife. You don't have to lift a finger, because you already
know, for certain, that it will happen.he pathway to fulfilment is
not currently known, but it is reasonable to believe that it exists"
At any rate, I think I sketched a path from the subjective feeling
of positive valence states to "hope" as a driver of action
mechanisms (or algorithms) that an agent can engage in to attain
the hoped-for outcome. I'm trying to use clinical or robotic
language and technical terms to emphasize that this would hold in
the generic agency model, and not just for people, animals, plants
or bacteria. It should work even for non-biological agents.
and not the evidence that consciousness is indistinguishable
from next token prediction in the Turing test,
I'm with Jennifer Hane on this one.
-- Linas
Patrick: Are they laughing at us?
Sponge Bob: No, Patrick, they are laughing next to us.
*Artificial General Intelligence List
<https://agi.topicbox.com/latest>* / AGI / see discussions
<https://agi.topicbox.com/groups/agi> + participants
<https://agi.topicbox.com/groups/agi/members> + delivery options
<https://agi.topicbox.com/groups/agi/subscription> Permalink
<https://agi.topicbox.com/groups/agi/T9e51fc70bbc4e9fc-Me53605b91a980e3c99004dfd>
