Hutter prize announcement by James Bowery. As you know, the purpose of the Hutter prize is to encourage research in small language models that can be run on home computers or phones. Text prediction is all you need to pass the Turing test, and compression measures prediction. https://spasim.org/docs/HutterPrize/TwentyYearBenchmarkShattered.html
After 2 years of no progress, 3 submissions since June were awarded a total of almost 50,000 euros for an almost 10% improvement in compression ratio of a 1 GB text file of Wikipedia articles used in a benchmark I started in 2006. Another 4 submissions are pending with a possible 2-3% further improvement. One has been running on my laptop now for 3 days. http://prize.hutter1.net/ The 3 winning programs are also listed on my large text compression benchmark (LTCB) in 13'th, 10'th, and 4'th place, along with descriptions of the algorithms. Ten other entries are listed in the top 20 that are either pending or submitted too late. Those are shown with a decompressor size of 0 because they are self extracting archives. https://mattmahoney.net/dc/text.html Both benchmarks test the same data under different rules. The Hutter sizes shown include the size of the compressor, which adds about 3 MB for the most recent entries. The Hutter prize also has hardware (one CPU, no GPU), memory (10 GB), and time limits (50 hours) but LTCB does not. The newest winning entry, fx2-cmix-transformer by Vladimir Ivanov, improved by 7.5% over the other two winners by replacing the LSTM model with a 6M 4-bit parameter transformer trained for 26 hours on 8 GPUs. The weights are compressed and appended to both the compressor and decompressor, which adds a 3% penalty to LTCB and 6% to Hutter, costing €5000 per 1%. The top 3 pending entries plus the unreleased entry currently running on my laptop are all derived from this program by retraining the weights with an objective function to reduce the compressed size of both the text and the weights themselves. Many of the entries were partially written by AI, which explains the flood of entries. I haven't examined the code yet (all open source) to know all the details of how it works. But generally most are context mixing models like PAQ that include a PPM model and a transformer neural network in addition to hundreds of indirect context models with specialized contexts. Each of these components independently predicts the next bit, and those predictions are combined by single layer neural networks by weighted averaging in the logistic domain, x=ln(p(1)/p(0)). The output is converted back to a prediction p(1)=1/(1+e^-x) and arithmetic coded. Then the weights are updated to reduce the prediction errors. Indirect context models map a context or hash to a bit history (8 but state) and then to a probability by a table. Then the table is updated to reduce prediction errors. Contexts can be the last n bytes or words or something more complex like table column, token grammatical category, or token sequence with gaps. PPM predicts byte distributions following the longest matching contexts using statistics stored in a context tree. These are converted to bit predictions and mixed. The Hutter entries map this tree to a 14 GB temporary file to avoid going over the 10 GB limit, but this usually pushes the program over the 50 hour time limit because I am running Linux in a WSL2 window and Windows reserves 4 of 16 GB, leaving only 2 GB for buffering memory mapped files. The 1 GB Wikipedia file is preprocessed by sorting 243K articles by topic, decoding XML, HTML, and Wiki formatting, encoding capitalization, stemming, and tokenizing using a dictionary of about 80K words organized to group related words like "brother" with "sister". The article sort order and dictionary order are CPU intensive, so is done offline and supplied to both the compressor and decompressor along with the transformer weights for a size penalty in LTCB and double penalty in the Hutter prize. I say "most" are derived from fx2-cmix-transformer, which is derived from the PAQ- cmix line and WRT dictionaries. However open source tufazip at #6 is derived from the closed source nncp at #8. nncp held the top spot on LTCB as the only transformer for 2 years until July. Neither are Hutter prize entries. Tufazip takes 7.5 days to compress or decompress with 2 GPUs and 73 GB memory. The entry I am testing now is a self extracting archive of size 94,128,758 bytes. I think without hardware limits we could see 75 MB, which would be at the lower bound of Shannon's 1950 estimate of 0.6 to 1.3 bit per character for the entropy of written English. Two things we have not tested are whether humans could predict as well as these models or whether the models could pass the Turing test with appropriate training data. -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T9c671768539c096c-M0d94ce8958f5027e6f01597f Delivery options: https://agi.topicbox.com/groups/agi/subscription
