On Tue, Nov 17, 2026, 9:30 PM Nano Great <[email protected]> wrote: > Matt, thanks for sharing. Very interesting progress. Any idea on how soon > a processor-compressor would run as software on an 8 GB notebook computer? >
It almost does now. The current Hutter prize entries are required to run in 50 hours on my laptop on one core and no GPU using 10 GB memory. That is already 10,000 times faster than the human brain. 1 GB of text is a lifetime of learning. But I think the trend will be faster hardware and more capabilities as it always has been, once the temporary hardware price spike is over. But long term, Moore's law is over, at least for silicon. You can't make transistors smaller than atoms. There was some discussion on the Hutter prize list about whether the transformer architecture models how humans process language. I argued that the brain doesn't train using thousands of epochs of back propagation, then freeze the weights and use a 1 MB context window to predict words, forgetting and replaying the whole conversation before every question. LLMs do this so they don't leak information between users. I argued that humans learn continuously and that a winning decompressor should be updating the weights continuously in a single pass on the partially decoded output. Otherwise the architecture is pretty similar. Real neural networks are recurrent, with lateral (sideways) inhibition producing winner take all states (attention) and backwards inhibition to scale inputs logarithmically. In a transformer, the recurrence is unrolled to dozens or hundreds of feed forward layers, and softmax layers are inserted to implement scaling and attention or short term memory. Current entries train a 6-8M parameter, 5-12 layer transformer for 1-2 GPU weeks, quantize the weights to 4 bits, compress them by making the compressed size part of the objective function, and then append the 3-4 MB of compressed weights to both the compressor and archive before inference with a 2-4K context window of 8-14 bit tokens. I argued that online training could remove this penalty and allow larger and higher precision modes for better inference, but requiring a GPU. Since July, enwik9 went from 110 MB to 89 MB, 20 years of progress in 4 months. I believe that 75 MB is possible on LTCB, which does not have any time or hardware limits. That is the lower bound of Shannon's 1950 estimate of the entropy of written English. The Hutter prize requires 50 hours, 10 GB, 1 core, and no GPU. So far, all of the progress is within these limits. But how to train online? Marcus Wölk, author of the leading entry, has studied childhood language development extensively and argued that humans don't learn in a single pass either. In my own research I saw that children learn about 5000 words per year after having used the word 20 times. Assuming a Zipf distribution where the n'th most frequent word has frequentcy 0.1/n and 6 hours per day of reading or listening to speech, then that child also heard or read the word 2000 times before learning it. I argued that the transformer weights could be updated after each article. I argued that this models the brains of marsupial and placental mammals including humans which store about 100K tokens in the hippocampus during the day and then replay those chopped up memories during REM sleep to train the cerebral cortex, which is otherwise read only. The cortex has a capacity of 10^9 bits, so 90% is ultimately forgotten. But an LLM should have no such limitation, even one running on a high end PC. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-Mb435b02e8f28b413d4947ded Delivery options: https://agi.topicbox.com/groups/agi/subscription
