That sounds like you're saying benchmarks for language modeling algorithms aka training algorithms are uninteresting because we've learned all we need to learn about them. Surely you don't mean to say that!
On Tue, Jul 23, 2024 at 5:42 PM Matt Mahoney <mattmahone...@gmail.com> wrote: > The Large Text Benchmark and Hutter prize test language modeling > algorithms, not language models. An actual language model wouldn't be > trained on just 1 GB of Wikipedia from 2006. But what we learned from this > is that neural networks is the way to go, specifically transformers running > on GPUs. > > On Tue, Jul 23, 2024, 3:10 PM James Bowery <jabow...@gmail.com> wrote: > >> I directed the question at you because you are likely to understand how >> different training and inference are since you said you "pay my bills by >> training" -- so far from levelling a criticism at you I was hoping you had >> some insight into the failure of the industry to use training benchmarks as >> opposed to inference benchmarks. >> >> Are you saying you don't see the connection between training and >> compression? >> >> On Mon, Jul 22, 2024 at 8:08 PM Aaron Hosford <hosfor...@gmail.com> >> wrote: >> >>> Sorry, I'm not sure what you're saying. It's not clear to me if this is >>> intended as a criticism of me, or of someone else. Also, I lack the context >>> to draw the connection between what I've said and the topic of >>> compression/decompression, I think. >>> >>> On Mon, Jul 22, 2024 at 5:17 PM James Bowery <jabow...@gmail.com> wrote: >>> >>>> >>>> >>>> On Mon, Jul 22, 2024 at 4:12 PM Aaron Hosford <hosfor...@gmail.com> >>>> wrote: >>>> >>>>> ... >>>>> >>>>> I spend a lot of time with LLMs these days, since I pay my bills by >>>>> training them.... >>>>> >>>> >>>> Maybe you could explain why it is that people who get their hands dirty >>>> training LLMs, and are therefore acutely aware of the profound difference >>>> between training and inference (if for no other reason than that training >>>> takes orders of magnitude more resources), seem to think that these >>>> benchmark tests should be on the inference side of things whereas the >>>> Hutter Prize has, *since 2006*, been on the training *and* inference >>>> side of things, because a winner must both train (compress) and infer >>>> (decompress). >>>> >>>> Are the "AI experts" really as oblivious to the obvious as they appear >>>> and if so *why*? >>>> >>> *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T6510028eea311a76-Mb81011d0bfa13655b772ecae> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T6510028eea311a76-M6d84ad9194dadef221251f4c Delivery options: https://agi.topicbox.com/groups/agi/subscription
