Re: [log4cxx] CI Benchmarking

It occurs to me that a better approach might be to run two benchmark versions in the same job and compare the results. A 'good' reference version artifact could be downloaded and compared with the new version.

Re: [log4cxx] CI Benchmarking

I think the action approach is sufficient for now. If we can get a dedicated GHA runner or similar, we can eventually move the benchmarks to a dedicated machine and still use the same API. -- Matt Sicker > On Dec 28, 2021, at 13:39, Robert Middleton wrote: > > I think adding it to github actio

Re: [log4cxx] CI Benchmarking

I think adding it to github actions(while certainly not ideal) is at least a step in the right direction. If/when we have dedicated hardware to test it properly, we can then migrate it over. At least having it setup to start with should make migration easier, plus even if it's not super consisten

Re: [log4cxx] CI Benchmarking

Agreed with your remarks regarding the unreliability of benchmark results in the cloud. See my proposal in private@ to get some machines for continuous benchmarks. On Tue, Dec 28, 2021 at 10:17 AM Dominik Psenner wrote: > Hi Stephen, > > The trouble with benchmarks in CI is that the numbers may

Re: [log4cxx] CI Benchmarking

Yes... Benchmark is only meaningful when on same machine, and totally same situation. Benchmark on the cloud seems weird and pointless... Besides, they are time-costing, thus will make our ci worse. Dominik Psenner 于2021年12月28日周二 17:17写道： > Hi Stephen, > > The trouble with benchmarks in CI is t

Re: [log4cxx] CI Benchmarking

Hi Stephen, The trouble with benchmarks in CI is that the numbers may be largely unreliable, depending mostly on the hardware where it runs and in general the surrounding environment. Chances are high that the benchmarks will not produce comparable results. It would however be good to provide som

[log4cxx] CI Benchmarking

Hi, Robert has created a benchmark that I thought would be nice to integrate into CI. I see the Log4J has some benchmarks actions which are currently run manually with results posted to github pages. Do you consider this a useful/optimal approach? Would an threshold which an action could check