On Fri, Sep 11, 2026, 5:17 PM James Bowery <[email protected]> wrote:
> On Fri, Sep 11, 2026 at 3:27 PM Matt Mahoney <[email protected]> > wrote: > >> On Fri, Sep 11, 2026, 2:29 PM James Bowery <[email protected]> wrote: >> >>> On Fri, Sep 11, 2026 at 12:27 PM Matt Mahoney <[email protected]> >>> wrote:... >>> >>>> ...Inference takes a lot less compute than training. We know this >>>> works, because that's how the Hutter prize leader does it. >>>> >>> >>> I view Vladimir's winning entry the same way I view Kolmogorov's failure >>> to include directed *cyclic* graphs of universal gates in his measure >>> of information complexity >>> <https://claude.ai/share/f5179150-9eab-470d-9cdc-6a9754bdc38e>: >>> >> >> He did? I am pretty sure that Kolmogorov complexity applies to any Turing >> complete language. Aren't cyclic graphs of universal gates Turing complete? >> >> > Finite state machines are not Turing complete. My NiNOR complexity > argument <https://jimbowery.blogspot.com/2023/10/ninor-complexity.html> > is based on the notion that one can maintain a finite fiction of Turing > completeness without being Turing complete (ie: infinite tape). > It's true that the shortest Turing machine that outputs a given finite string must halt after finite time and write to finite memory, so there is an equivalent finite state machine. However Kolmogorov's proof of language independence no longer works. For Turing machines, any language can be translated into any other language by a fixed length program, but that program might use an arbitrary amount of memory. For example there is no finite state machine that can translate unary to binary or vice versa because the input and output can be arbitrarily long.. > Anyway my statement was about the efficiency of offline training of >> transformer weights. >> > > You're speaking of computational efficiency as opposed to descriptive > efficiency. > > That's true but the Hutter Prize includes the size of the compressor along > with the size of the description of the sample of human knowledge. > Vladimir has to pay for those pretrained weights *twice* > I disagree with adding the size of the compressor because it has nothing to do with Kolmogorov complexity. But Marcus Hutter is funding it so he makes the rules. Anyway, it should not be hard to modify fx2-cmix-transformer to train online to remove the 2.9 MB penalty and compress enwik9 to 94.1 GB in 8-10 days with one GPU. I expect to see even further improvements by growing the transformer from 6M parameters to maybe 50M. But what I would really like to see is a faster, one pass training algorithm that we know must exist because that's how our brains work. Our brains don't do back propagation. They are recurrent and use Hebb's rule. What are we missing? ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T3f8115622f860785-M621f9cbb7422634a5edf7619 Delivery options: https://agi.topicbox.com/groups/agi/subscription
