On Sun, Dec 22, 2024 at 3:30 AM PGC <[email protected]> wrote:
*> While it’s true that scaling a model’s compute often improves > performance—e.g., O3 going from 75.7% to 87.5%—that alone doesn’t prove > that we’ve achieved “fundamental AGI.”* *Francois Chollet is the man who invented the ARC test and he did everything he could think of to make his test as difficult as possible for a large language model. Back in the olden days of late 2023 when the best AIs only got scores in the single digits on the ARC test, Chollet said that if a machine ever got higher than 75% on his test then it should be considered an AGI. But when a machine did exactly that he immediately moved the goal post to some vague unspecified spot. * *And it's not just the ARC test, ALL the previous benchmarks that were supposed to determine when AGI has arrived are no longer of any use because they've all been maxed out. We need new more difficult benchmarks, not to compare AIs to humans because to my mind that debate is over, but in order to compare one AGI to another AGI.* > > *> In machine learning history, many benchmarks have been surpassed by > throwing more resources at them, yet models often fail when faced with > novel tasks.* *Long before O3 came out it was obvious that computers were getting much better at being good at generalize tasks for example, back in the stone age of 2017 starting with zero knowledge and with just a few hours of self study, AlphaZero, could become good enough to beat the most talented human at chess and GO and shogi and ANY two player zero sum game. * > *> **I’ve personally seen kids under age ten handle about 80 to upwards > of 90% of the daily “play” tasks (besides acing the 6 problems on the > landing page) on the ARC site once they grasp the basic rule of finding the > rule,* *Once they grasp the basic rule of finding the rule yes. The first and probably the most difficult step very young children have in taking the ARC test is figuring out what question the ARC test is asking, only after they understand the question can children start thinking about an answer. And the more they play the ARC test the better they get at it. Exactly the same thing could be said about O3. * > *> As for the claim that François Chollet “moved the goalpost” once AI > systems approached the 75% mark, it’s common in AI research for benchmarks > to evolve precisely because scoring high on an older test doesn’t > necessarily reflect deep, generalizable reasoning.* *Alan Turing originally said that if you were only communicating over a teletype machine and a computer could convince you that you were communicating with another human being then that computer should be considered as intelligent as a human being. Computers blew past that benchmark about two years ago. * *Douglas Hofstadter, the author of my all-time favorite book Godel Escher Bach, said that if a computer could beat a Chess grandmaster then it would be intelligent, but he didn't expect that to happen in his lifetime. However it happened in 1997. Hofstadter now thinks computers are genuinely intelligent, and he's very frightened. * *Then people said Chess was not a good benchmark but the game of GO was because it was astronomically more complicated than Chess, but a computer beat the best human GO player in 2016.* *Then people said that for a computer to be an AGI it would need to be as good as most people at most things. And I think we're already there.* *Then people said for a computer to be an AGI it would need to be better than EVERY human being at EVERYTHING, but that's not Artificial General Intelligence, that's Artificial Superintelligence. And we're almost there. * *The pattern is always the same.A computer will never be able to do X.And then a computer does X.Well OK but a computer will never be able to do Y.And then a computer does Y.And then it's obvious they will soon run out of letters of the alphabet. * *John K Clark See what's on my new list at Extropolis <https://groups.google.com/g/extropolis>* 5n3 -- You received this message because you are subscribed to the Google Groups "Everything List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/everything-list/CAJPayv2Pa9dnHL%3Df83oM%3DW%3D6tQx_6mLgsEwoT7XfYWZXk-O1%2BA%40mail.gmail.com.
