On Fri, Sep 11, 2026, 2:29 PM James Bowery <[email protected]> wrote:
> On Fri, Sep 11, 2026 at 12:27 PM Matt Mahoney <[email protected]> > wrote:... > >> ...Inference takes a lot less compute than training. We know this works, >> because that's how the Hutter prize leader does it. >> > > I view Vladimir's winning entry the same way I view Kolmogorov's failure > to include directed *cyclic* graphs of universal gates in his measure of > information complexity > <https://claude.ai/share/f5179150-9eab-470d-9cdc-6a9754bdc38e>: > He did? I am pretty sure that Kolmogorov complexity applies to any Turing complete language. Aren't cyclic graphs of universal gates Turing complete? Anyway my statement was about the efficiency of offline training of transformer weights. The top 4 compressors in the large text benchmark are transformers, but 3 of them use online training, which takes a week on a GPU to compress enwik9 to 105-107 MB. Vladimir's winning entry trained 6M transformer weights offline for 26 hours on 8 GPUs, then used the fixed weights to compress to under 97 GB in 2 days with just a single CPU. https://mattmahoney.net/dc/text.html The reason is that training a transformer or any deep neural network requires multiple passes. Inference is single pass. OpenAI and Anthropic have not disclosed the number of parameters in their best models, but others estimate around 10 trillion, with 10% active in a mixture of experts. The top open weight models are Chinese: Kimi K3, Alibaba Qwen, and DeepSeek with around 2-3 trillion. To run the top one, K3, you need 1.7 TB memory and at least 18 enterprise grade GPUs, which I estimate would cost around $300K. I find it stunning that state actors wouldn't spend this much to protect their most important military secrets. -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T3f8115622f860785-M82927dcc3a769c14d1771100 Delivery options: https://agi.topicbox.com/groups/agi/subscription
