On Fri, Sep 11, 2026 at 3:27 PM Matt Mahoney <[email protected]> wrote:
> On Fri, Sep 11, 2026, 2:29 PM James Bowery <[email protected]> wrote: > >> On Fri, Sep 11, 2026 at 12:27 PM Matt Mahoney <[email protected]> >> wrote:... >> >>> ...Inference takes a lot less compute than training. We know this >>> works, because that's how the Hutter prize leader does it. >>> >> >> I view Vladimir's winning entry the same way I view Kolmogorov's failure >> to include directed *cyclic* graphs of universal gates in his measure of >> information complexity >> <https://claude.ai/share/f5179150-9eab-470d-9cdc-6a9754bdc38e>: >> > > He did? I am pretty sure that Kolmogorov complexity applies to any Turing > complete language. Aren't cyclic graphs of universal gates Turing complete? > > Finite state machines are not Turing complete. My NiNOR complexity argument <https://jimbowery.blogspot.com/2023/10/ninor-complexity.html> is based on the notion that one can maintain a finite fiction of Turing completeness without being Turing complete (ie: infinite tape). People like to pretend that it’s no big deal that no one outside of Cambridge had access to Turing’s 1948 paper “Intelligent Machinery”, based on finite states <https://dn790006.ca.archive.org/0/items/turing1948/turing1948_text.pdf>, until *after* Solomonoff’s papers in the 1960s. Indeed, a colleague of Solomonoff’s with whom I’m working just told me he’d never even heard of it until I sent it to him. My late colleague, Robert Johnson (the Seymour Cray of Burroughs who is better known for having invented magnetic ink found on your checks), when in 2006 I told him about the Hutter Prize, said he thought the British still classified Turing-algorithms pertaining to prediction. > Anyway my statement was about the efficiency of offline training of > transformer weights. > You're speaking of computational efficiency as opposed to descriptive efficiency. That's true but the Hutter Prize includes the size of the compressor along with the size of the description of the sample of human knowledge. Vladimir has to pay for those pretrained weights *twice* The top 4 compressors in the large text benchmark are transformers, but 3 > of them use online training, which takes a week on a GPU to compress enwik9 > to 105-107 MB. Vladimir's winning entry trained 6M transformer weights > offline for 26 hours on 8 GPUs, then used the fixed weights to compress to > under 97 GB in 2 days with just a single CPU. > https://mattmahoney.net/dc/text.html > > The reason is that training a transformer or any deep neural network > requires multiple passes. Inference is single pass. > The strongest argument I think you can make, correct me if I'm wrong, is that this games the intent of the Hutter Prize by achieving a higher *effective* computational complexity (time and memory) algorithm within the single cpu Geekbench5 10GB RAM + ~6GB cache resource limits, without delivering additional descriptive efficiency. For the latter to be true, there would have to be a *potential* leader in a resource-unconstrained version of the Hutter Prize (ie: LTCB that scored the compressor size + compressed size) that used essentially the same compression algorithm. Correct? OpenAI and Anthropic have not disclosed the number of parameters in their > best models, but others estimate around 10 trillion, with 10% active in a > mixture of experts. The top open weight models are Chinese: Kimi K3, > Alibaba Qwen, and DeepSeek with around 2-3 trillion. To run the top one, > K3, you need 1.7 TB memory and at least 18 enterprise grade GPUs, which I > estimate would cost around $300K. > > I find it stunning that state actors wouldn't spend this much to protect > their most important military secrets. > > -- Matt Mahoney, [email protected] > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T3f8115622f860785-M82927dcc3a769c14d1771100> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T3f8115622f860785-M5317f3b9d9c3a7107b4829ae Delivery options: https://agi.topicbox.com/groups/agi/subscription
