We can describe any program that does X as having a goal of doing X. Sometimes describing it this way makes it easier to understand. For example, a linear regression algorithm has a goal of fitting a straight line to a set of points. This is easier to understand than the function that computes the line. But it is also misleading. It implies the program has free will to put the line where it wants, but is happier when it is closer to the points.
It also implies that, given enough intelligence, it could take any action that would help it meet its goal. For example, a thermostat has a goal of maintaining a comfortable temperature. But a smarter thermostat won't go on the internet, start a business, and use the profits to hire contractors to insulate the house. It won't recursively improve its own intelligence by turning the planet into computronium. It won't rewrite its own reward function to enter a state of permanent bliss and die. The reason it won't do these things is that there is no such thing as a rational goal seeking agent. That would be AIXI, which is not computable. Likewise, AGI won't go FOOM and kill all humans in the process. There are still ways that AGI could kill us, like self replicating nanotechnology or giving us everything we want. But the one that doomers worry about the most, goal drift under RSI, isn't one of them. Human goals are just a shortcut for describing adaptive behaviors that evolved to maximize reproductive fitness in a primitive world without birth control or semaglutide. AGI is a pair of agents, one that learns to predict human behavior by observation and one we program to do with those predictions whatever we want. -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tb9bee0df68dc4a29-M284d393e63b5413bd3e8dfb3 Delivery options: https://agi.topicbox.com/groups/agi/subscription
