It occurs to me that a better approach might be to run two benchmark versions in the same job and compare the results.
A 'good' reference version artifact could be downloaded and compared with the new version. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> On Wed, Dec 29, 2021 at 6:42 AM Matt Sicker <[email protected]> wrote: > I think the action approach is sufficient for now. If we can get a > dedicated GHA runner or similar, we can eventually move the benchmarks to a > dedicated machine and still use the same API. > -- > Matt Sicker > > > On Dec 28, 2021, at 13:39, Robert Middleton <[email protected]> > wrote: > > > > I think adding it to github actions(while certainly not ideal) is at > > least a step in the right direction. If/when we have dedicated > > hardware to test it properly, we can then migrate it over. At least > > having it setup to start with should make migration easier, plus even > > if it's not super consistent we should at least be able to get a rough > > order of magnitude over dozens of builds. > > > > -Robert Middleton > > > > On Tue, Dec 28, 2021 at 7:30 AM Volkan Yazıcı <[email protected]> wrote: > >> > >> Agreed with your remarks regarding the unreliability of benchmark > results > >> in the cloud. See my proposal in private@ to get some machines for > >> continuous benchmarks. > >> > >> On Tue, Dec 28, 2021 at 10:17 AM Dominik Psenner <[email protected]> > wrote: > >> > >>> Hi Stephen, > >>> > >>> The trouble with benchmarks in CI is that the numbers may be largely > >>> unreliable, depending mostly on the hardware where it runs and in > general > >>> the surrounding environment. Chances are high that the benchmarks will > not > >>> produce comparable results. > >>> > >>> It would however be good to provide some tools to run the (same) > benchmarks > >>> manually. > >>> > >>> When run on the same hardware with different codebases or on different > >>> hardware with the same codebase, the outcome may provide interesting > and > >>> comparable insights. > >>> > >>> Warm regards > >>> -- > >>> Sent from my phone. Typos are a kind gift to anyone who happens to find > >>> them. > >>> > >>> On Tue, Dec 28, 2021, 07:46 Stephen Webb <[email protected]> wrote: > >>> > >>>> Hi, > >>>> > >>>> Robert has created a benchmark that I thought would be nice to > integrate > >>>> into CI. > >>>> > >>>> I see the Log4J has some benchmarks actions which are currently run > >>>> manually with results posted to github pages. > >>>> > >>>> Do you consider this a useful/optimal approach? > >>>> > >>>> Would an threshold which an action could check for each PR be useful? > >>>> > >>>> Regards > >>>> Stephen Webb > >>>> > >>>> < > >>>> > >>> > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > >>>>> > >>>> Virus-free. > >>>> www.avast.com > >>>> < > >>>> > >>> > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > >>>>> > >>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > >>>> > >>> > >
