On Fri, Apr 17, 2026, 9:13 PM James Bowery <[email protected]> wrote:
> On Sat, Apr 11, 2026 at 5:24 PM Matt Mahoney <[email protected]> > wrote: > >> I'm not concerned about AGI turning us into clones in a hive mind. I'm >> concerned about AGI making humans irrelevant. Why should humans exist when >> machines can do everything better? >> >> I don't mean just that humans would be irrelevant to AGI. >> > > That depends on the utility function, or telos with which they've been > parameterized. > > > >> I mean that humans would be irrelevant to other humans too. >> > > This returns to our terminology problem and the profound division > occurring between humans and de facto "can't make a pencil > <https://en.wikipedia.org/wiki/I,_Pencil>" cyborgs. I understand and > agree with you when you say "humans" aren't built for space. Nor are > humans built for "civilization" (as we know it). > Then what should our utility function be? All living things optimize for reproductive fitness. Should we do that? Or should we optimize for our evolved secondary goals, the things we think we want like food and sex and not dying? The problem with all utility functions over a finite number of states is that they have a maximum where no further state changes are desired. This is exactly the state we evolved to fear. We don't have an infinite number of states. The most states that can exist in the observable universe is exp(Mc^2/kT) ≈ 2^10^92, where M ≈ 10^53 Kg is the mass of the universe, c is the speed of light, k is Boltzmann's constant, and T ≈ 3 K is the cosmic microwave background temperature. That's a big number that should last us until the heat death of the universe, but it's not forever. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tc2616dc0a8577813-M6fd1859ea507397818ddfe79 Delivery options: https://agi.topicbox.com/groups/agi/subscription
