I went back to the beginning of the thread on encode that you posted in 2021 so it is a little more clear. I think you are doing byte level context mixing. PPM also computes a probability distribution over the next byte but it only uses one variable length context, whatever is the longest context match for each possible next byte. PAQ predicts one bit at a time using statistics stored in hash tables, but I think your program stores them in a context tree like PPM or CTW.
CTW was one of the first bit level context mixing algorithms, developed in 1995, but it was never widely used because it was protected by patents at the time (now expired). http://mattmahoney.net/dc/dce.html#Section_423 But maybe you are on the right track. LLMs predict a token at a time, which is fast on a GPU because each component of the output vector can be computed in parallel. The future of data compression research is language modeling, not reducing storage or bandwidth, which is cheap. Zip (deflate) has one of the worst compression ratios but everyone still uses it because it is free, fast, standardized, and well supported. -- Matt Mahoney, [email protected] On Mon, Dec 8, 2025, 11:39 PM <[email protected]> wrote: > Wait 1 more thing: No, my mixer does not use a context. The mixer only > checks the counts and how many symbols are in an order's prediction set and > determines how confident it is, next it sends it to an exp() and next it > uses weights that are hardcoded. > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Tf0bedfcd44454678-M5f874255c7be6ea96f5979f1> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tf0bedfcd44454678-Me9959d7df3046a5f021522d3 Delivery options: https://agi.topicbox.com/groups/agi/subscription
