On Tue, Nov 17, 2026, 9:30 PM Nano Great <[email protected]> wrote:

> Matt, thanks for sharing. Very interesting progress. Any idea on how soon
> a processor-compressor would run as software on an 8 GB notebook computer?
>

It almost does now. The current Hutter prize entries are required to run in
50 hours on my laptop on one core and no GPU using 10 GB memory. That is
already 10,000 times faster than the human brain. 1 GB of text is a
lifetime of learning.

But I think the trend will be faster hardware and more capabilities as it
always has been, once the temporary hardware price spike is over. But long
term, Moore's law is over, at least for silicon. You can't make transistors
smaller than atoms.

There was some discussion on the Hutter prize list about whether the
transformer architecture models how humans process language. I argued that
the brain doesn't train using thousands of epochs of back propagation, then
freeze the weights and use a 1 MB context window to predict words,
forgetting and replaying the whole conversation before every question. LLMs
do this so they don't leak information between users. I argued that humans
learn continuously and that a winning decompressor should be updating the
weights continuously in a single pass on the partially decoded output.

Otherwise the architecture is pretty similar. Real neural networks are
recurrent, with lateral (sideways) inhibition producing winner take all
states (attention) and backwards inhibition to scale inputs
logarithmically. In a transformer, the recurrence is unrolled to dozens or
hundreds of feed forward layers, and softmax layers are inserted to
implement scaling and attention or short term memory.

Current entries train a 6-8M parameter, 5-12 layer transformer for 1-2 GPU
weeks, quantize the weights to 4 bits, compress them by making the
compressed size part of the objective function, and then append the 3-4 MB
of compressed weights to both the compressor and archive before inference
with a 2-4K context window of 8-14 bit tokens. I argued that online
training could remove this penalty and allow larger and higher precision
modes for better inference, but requiring a GPU. Since July, enwik9 went
from 110 MB to 89 MB, 20 years of progress in 4 months. I believe that 75
MB is possible on LTCB, which does not have any time or hardware limits.
That is the lower bound of Shannon's 1950 estimate of the entropy of
written English. The Hutter prize requires 50 hours, 10 GB, 1 core, and no
GPU. So far, all of the progress is within these limits.

But how to train online? Marcus Wölk, author of the leading entry, has
studied childhood language development extensively and argued that humans
don't learn in a single pass either. In my own research I saw that children
learn about 5000 words per year after having used the word 20 times.
Assuming a Zipf distribution where the n'th most frequent word has
frequentcy 0.1/n and 6 hours per day of reading or listening to speech,
then that child also heard or read the word 2000 times before learning it.

I argued that the transformer weights could be updated after each article.
I argued that this models the brains of marsupial and placental mammals
including humans which store about 100K tokens in the hippocampus during
the day and then replay those chopped up memories during REM sleep to train
the cerebral cortex, which is otherwise read only. The cortex has a
capacity of 10^9 bits, so 90% is ultimately forgotten. But an LLM should
have no such limitation, even one running on a high end PC.

------------------------------------------
Artificial General Intelligence List: AGI
Permalink: 
https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-Mb435b02e8f28b413d4947ded
Delivery options: https://agi.topicbox.com/groups/agi/subscription

Reply via email to