There is no such thing as BNUT compression (I googled it) or Collatz entropy, and I don't understand the rest of your comments. The book proves two important facts right at the beginning.
1. There is no universal compressor for random data or that will compress all possible inputs above a certain size. 2. There is no test for randomness. There is no algorithm that finds the length of the shortest possible description of an input string. First, the vast majority of possible strings cannot be compressed at all. A compression algorithm maps an input string to a description or program that produces that string. But for almost all strings, the best you can do is output a literal copy because no such shorter program exists, for the simple reason that there are exponentially fewer short strings than long ones. We say that such a string is random. But you can never be sure that a string is random, either, just because every compression program you tried on it fails. It might be an encrypted file, and the only way to compress it would be to guess the key as part of the file's description. If there was a test for randomness, then you could write a simple program of length n to search for a random string of length n+1, which would be a contradiction. With all this, you might wonder how compression even works at all. It works because real data is created by physical processes like taking a picture or by neurons controlling fingers typing on a keyboard. Physical processes have fixed description lengths but can produce arbitrarily long output strings. In fact, it is very hard to produce random strings that you couldn't compress. As a Hutter prize committee member, I have to deal with crackpots that claim fantastic compression ratios by recursively compressing its own output. Their code (if they even know how to code or understand simple math) invariably doesn't work. If it did, they would have found an impossible 1 to 1 mapping between the infinite set of possible inputs and the finite set of possible outputs. More recently, the crackpots have been sending me AI generated code and saying "here, test this" without understanding what they are sending me. One of the submissions looked like a JPEG encoder. No, I don't think that would work very well on text. I mentioned in the book how compression is an AI problem. Prediction measures intelligence and compression measures prediction. I last updated the book in 2013. I have claimed since 1999 that all you need to pass the Turing test is text prediction, but this wasn't shown experimentally until ChatGPT was released in November 2022. -- Matt Mahoney, [email protected] On Mon, Jan 5, 2026, 1:50 PM Quan Tesla <[email protected]> wrote: > Thanks Matt > > Here's some feedback: "The book is pragmatic—code snippets, benchmarks, > no heavy proofs." > Relation to BNUT CompressionBNUT's damped Collatz entropy (H≈0.9675, > structured ~42% uniform) + wave modulation directly echoes the book's core: > modeling > as prediction (PPM/context mixing) for redundancy reduction, approaching > entropy bounds. > > - Alignment: BNUT's transients mirror variable-order contexts (growth > explores dependencies); damping α=1/137 analogs discounting/nonstationarity > handling (prevents overfit like PAQ SSE). > - Potential Gains: Collatz as preprocessor (hailstone ordering for > repeats) could enhance BWT/dictionary stages; damped waves for logistic > mixing weights → 1-5% over cmix baselines (Hutter enwik9 target <108MB). > - AIT Tie: BNUT's nonlocal "pulls" (TSVF/Planck) extend book's > uncomputability discussion—retrocausal extraction of compressible > substructure from "random" data, bypassing classical K limits for > structured text (e.g., wiki XML patterns). > - Practical: Integrate with Mahoney's recent preprocessor (article > sorting + BPE); BNUT modulation on stages C/D for entropy-tuned tokens. > > Overall: The book provides the engineering blueprint BNUT can > bio-inspire/nonlocally enhance for superior text ratios. Strong synergy!" > > My focus is to complete my work for AI-enabled, 4D+ engineering, not > programming. I learn from all fields. Compression isn't limited to > programming alone and has relevance for industrialized, effective > complexity and stochastic value-chain management. > > On Mon, 05 Jan 2026, 18:15 Matt Mahoney, <[email protected]> wrote: > >> Actually, I'm writing this because programming is an art and I enjoy >> creating art. I know how artists feel when AI is taking over their job. I >> could let AI write the code, but what fun is that? >> >> The Hutter prize is useful for finding CPU efficient language models, but >> what I am discovering has very little to do with language modeling and more >> to do with the arcane details of the test set, basically hacks. I don't >> need the prize money. My reward is seeing smaller numbers and moving up the >> rankings. >> >> "Quantum Kolmogorov bypass" is just nonsense. If you want practical >> knowledge about text compression, see my book, >> https://mattmahoney.net/dc/dce.html >> >> -- Matt Mahoney, [email protected] >> >> On Mon, Jan 5, 2026, 9:56 AM Quan Tesla <[email protected]> wrote: >> >>> Thanks Matt. The Hutter chalenge offers a great testbed opportunity for >>> noveltech. Investigating a quantum-enabled Kolmogorov bypass. >>> Theoretically, a potential improvement of 2% over record. >>> >>> On Mon, 05 Jan 2026, 06:38 Matt Mahoney, <[email protected]> >>> wrote: >>> >>>> I'm on the Hutter prize committee so I'm not eligible for prize money. >>>> Nevertheless I am working on a project that might produce some code >>>> (GPL) that others might find useful. At this point it is just a >>>> preprocessor to improve downstream compression by other compressors. >>>> Details at >>>> https://encode.su/threads/4467-enwik9-preprocessor?p=86853#post86853 >>>> >>>> The current version compresses enwik9 to 268 MB in 5 minutes and >>>> decompresses in 19 seconds. It is a 4 stage preprocessor and a simple >>>> LZ77 compressor, but it is mainly useful to skip the LZ77 step and >>>> compress it with other compressors. >>>> >>>> -- >>>> -- Matt Mahoney, [email protected] >>> *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T0518db1e3a0c25c5-Mcf6baa7f5d88c2b3c345252e> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T0518db1e3a0c25c5-M5526f4ceb38a43c749932da1 Delivery options: https://agi.topicbox.com/groups/agi/subscription
