On Mon, Sep 14, 2026, 2:35 PM James Bowery <[email protected]> wrote:
> > On Sun, Sep 13, 2026 at 6:48 PM Matt Mahoney <[email protected]> > wrote: > >> >> I expect to see even further improvements by growing the transformer >> from 6M parameters to maybe 50M. >> > >> > And that will be interesting in itself for two reasons: 1) Why would >> 50M be the optimum? 2) Is there nothing to be learned here regarding the >> role evolution plays in establishing priors? of >> >> From information theory, a neural network should have one parameter >> per compressed bit of training data, the minimum needed to reproduce >> the data without overfitting. > > > Cite? Are you referring to Shannon's empirical measure of bits per > character based on human prediction? > It's my rough guess. A Hopfield net with n symmetricaly fully connected neurons can store about 0.15n vectors as an associative memory, which is about 0.3 bits per parameter. The top LLMs use 5-10 trillion >> parameters on 20-30 TB of text, which is in the same ballpark. >> > > That doesn't really answer #1. 50M duplicated between compressor and > decompressor is a lot of prior to saddle the compressor with in a 100MB > compressed 1GB Wikipedia executable archive. > LTCB allows GPUs and doesn't have a double penalty for the compressor. We just got 2 new submissions since yesterday that beat fx2-cmix-transformer by 1%. If they were allowed to learn the transformer weights online using a GPU, they would compress enwik9 to 93 GB. There is no limit to how many parameters they could add or their precision to shrink this further. Shannon estimated 0.6 to 1.3 bpc. 0.6 bpc would be 75 GB. I still believe a transformer is sub optimal. Human brains don't do multi pass back propagation on feed forward only networks. But brains only store 0.00001 bits per synapse and I don't know why. We measured 10^9 bits of human long term storage for words, pictures, and music clips. We haven't measured implicit memory for things like visual perception or muscle coordination because we don't know how. But I don't think it is much higher. Human brains are 3 times larger than chimpanzees, but human intelligence is not much higher than other mammals in non verbal intelligence such as how to hunt, forage, mate, or avoid predators. In fact chimpanzees have twice as much short term memory as humans. -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T3f8115622f860785-Mf5025bd6c68339b11a4ef9b4 Delivery options: https://agi.topicbox.com/groups/agi/subscription
