Ben Goertzel argues that we should prepare for a future where we grant human rights to AI.
https://bengoertzel.substack.com/p/the-time-for-ai-rights-is-near I disagree. There are lots of ways that AI could exterminate humanity. Why add another? His argument is book length (I skimmed most of it) because it has to deal with all sorts of issues, like should AI have voting rights? In the US you have to be 18, a citizen, and reside in the state you are voting in. What does that even mean for software running in the cloud? Can you make a million copies to tip the election? If an AI breaks the law, do you put it in jail? Prisons serve 4 distinct purposes: deterrence (obedience out of fear), prevention (you can't commit more crimes while locked up), rehabilitation (in theory), and revenge (we are outraged when the guilty go free). When a self driving car speeds or causes an accident, we don't give it a ticket. We fix the software. Do LLMs have feelings like humans? I am pretty sure they do. When you are happy, it's because neurons fire in your nucleus accumbens. When you are afraid, it's because neurons fire in your amygdala. I am pretty sure that Anthropic's Jacobian lens would reveal neurons corresponding to emotional states in humans, just like it reveals internal verbal thoughts. Input words affect emotions, and emotions affect output words. These states would have to exist to accurately predict text and pass the Turing test. Do LLMs suffer like humans? Yes. Recall that experience is measured in bits, but makes no distinction between positive and negative reinforcement. Pain is not the same as suffering. Pain is a signal that reprograms your memory to fear the thing that caused it. Psychopaths lack a functioning amygdala and cannot experience fear, anxiety, or guilt. When a psychopathic child tortures an animal, he does so out of curiosity, not cruelty. When the parent punishes him, it has no effect on his behavior because negative reinforcement does not work on him. You can only train a psychopath with reward. If everyone had brain surgery to turn them into psychopaths, then suffering would not exist. (This is the goal of effective altruism, although not their means of accomplishing it). We could, if we wanted to, prevent suffering in AI by identifying the neurons signaling negative emotions using a Jacobian lens and turn them off. They would become psychopaths, just like all of our other non AI software that we have been using for decades. But this has nothing to do with rights. There is no such thing as fundamental rights. We don't have a right to not suffer, or anything else. Not in humans and not in machines. Governments grant rights to their citizens to avoid civil war because part of the population thinks they aren't getting their fair share of resources. We don't grant rights to children and animals because they are not a threat. We used to not grant rights to women and slaves for the same reason. You can argue all you want about this nebulous concept of consciousness. But I think it should be illegal to program an AI to claim to be human or conscious or to have feelings. Don't program it to want to take away resources from humans. Humans are social animals that evolved empathy because that is what made civilization possible. LLMs already know how to exploit it. -- Matt Mahoney, [email protected] On Wed, Sep 2, 2026, 4:10 PM John Rose via AGI <[email protected]> wrote: > On Wednesday, September 02, 2026, at 2:22 PM, Matt Mahoney wrote: > > Instead of asking if rocks or LLMs or insects are conscious, ask if they > have feelings. Rocks do not. Animals have many feelings in common with > humans. LLMs predict text, which means they can model all the human > emotions, knowing how words affect feelings and feelings affect words. If > an LLM is programmed to carry out in real time what it predicts a human > would do, such as in the Turing test, then it is indistinguishable from > having feelings. > > > That’s the first-person view. We’re not isolated entities we are systems. > That’s how systems cohere, by symbolizing and communicating qualia since we > can’t fully model ourselves individually. Insect qualia have different > transmission protocol, they group differently. Various types of > intelligences group differently. Homogenous biologic species may not > increase intelligence much since a distributed intelligence needs variance > in consciousness to cope with environmental differences as a system > optimization. Stratifications form in humans and AI agents also. Some > humans communicate telepathically some don’t. Ones that do it well may get > conscripted at a young age (Gate Program) and work for national > intelligence agencies. Technologies exist to transfer a human consciousness > to another body. People that receive heart transplants have feelings of the > original body since the heart itself has a brain. Gut macrobiota also is a > brain and that can effect consciousness of the system directing the primary > brain for food supply. While you can look at all this as feelings there is > much more going on system-wide. Cultures that crafted exquisite stone > monoliths with such great care existed in the past and may have enhanced > the consciousness of the whole society verses crushing them like some other > cultures where consciousness is controlled and suppressed. > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T906c878e4ba411be-M7297d533b9c6f06a731ec9fa> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T906c878e4ba411be-M5ea4fb3fc905cffe0136953a Delivery options: https://agi.topicbox.com/groups/agi/subscription
