Thank you Matt for doing this. Danko
Dr. Danko Nikolić CEO, Robots Go Mental www.robotsgomental.com www.danko-nikolic.com https://www.linkedin.com/in/danko-nikolic/ -- I wonder, how is the brain able to generate insight? -- On Sun, Nov 15, 2026 at 1:09 AM Matt Mahoney <[email protected]> wrote: > My wife and I just returned from 18 days of exploring ancient sites in > Rome and Tunisia, hiking the canyon walls of Camino del Rey in Spain, and > running the Porto marathon in Portugal. We were fortunate that our return > flight from Madrid to Orlando was the day after a nationwide strike halted > travel in Spain and the day before the FAA grounded all air traffic in the > US because of a software error. > > I didn't bring my laptop so I spent most of the next day updating the > Large Text Compression Benchmark on my website with a flurry of Hutter > prize submissions. The rate of progress since July has been astounding. > From 2006 when I created the benchmark until earlier this year, the record > improved from 127 MB by durilca (a PPM compressor) to 107 MB set by nncp in > 2019 using a transformer, an average of 2% gain per year. Now another > transformer, cmix-neif 2.0 holds the record (pending testing) at 89 MB. > That is 20 years of progress in 4 months, a 60 fold speedup. Now nncp sits > at 16'th place and durilca at 31'st among 238 compressors. All of this > progress is due to AI generated code and (high quality) documentation. > > I do not believe in a fast takeoff singularity, but using AI to write AI > sure looks like one. I just read that Anthropic used 1000 agents on an > unreleased model to write a complete operating system in 8 days, 1000 times > faster than a team of humans. I remind you that the cost of human > intellectual work is $10K per million input tokens and $50K per million > output tokens. The most advanced models are 1000 times cheaper. > > In 1999 I wrote a proposal to NSF for a text compression prize to promote > AI development. It was rejected, as were 90% of proposals. So I created > https://mattmahoney.net/dc/text.html in 2006 without prize money until it > was funded by the Hutter prize. Now I am 71, retired 12 years, and > supporting this research voluntary because I think that AI is important > > One early result was that compression increases logarithmically with > computing power, as shown in the 2 graphs I made in 2008. That relationship > still holds. Now the new result is two lineages of transformer based models > derived from cmix and nncp. Both use offline training by multiple passes of > back propagation (8-10 GPU days) quantizing 6-8M weights to 4 bits, > compressing and appending to the compressed file. The cmix lineage uses a > 12 layer network and the nncp lineage, led by DNA4, uses 5 layers. DNA4 > only compresses to 100 MB but is much faster, 2.5 GPU days to train and 2.5 > GPU hours to model using 14 bit tokens vs 8 bits and 2 CPU days for cmix. > > I still believe there are better algorithms than transformers. The brain > learns by Hebb's rule, which is purely local, not back propagation. Human > learning is single pass, not offline in dozens or hundreds of passes. An > online neural network would eliminate the 3-4 MB penalty for storing the > weights in the decompressor (or double penalty in the Hutter prize for > storing another copy in the compressor). Once that happens, the number and > precision of the weights could be increased to improve compression further. > I believe that 75 MB is possible if hardware limits are removed. > > -- Matt Mahoney, [email protected] > > > > > > > > > > > > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-Mbd1bc33f7aad1a4d7573352c> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-M5303034b9a7b5ab2f5617392 Delivery options: https://agi.topicbox.com/groups/agi/subscription
