On Fri, 2016-06-10 at 23:20 +1000, Steven D'Aprano wrote: > On Fri, Jun 10, 2016 at 01:13:10PM +0200, Victor Stinner wrote: > > Hi, > > > > Last weeks, I made researchs on how to get stable and reliable > > benchmarks, especially for the corner case of microbenchmarks. The > > first result is a serie of article, here are the first three: > > Thank you for this! I am very interested in benchmarking. > > > https://haypo.github.io/journey-to-stable-benchmark-system.html > > https://haypo.github.io/journey-to-stable-benchmark-deadcode.html > > https://haypo.github.io/journey-to-stable-benchmark-average.html > > I strongly question your statement in the third: > > [quote] > But how can we compare performances if results are random? > Take the minimum? > > No! You must never (ever again) use the minimum for > benchmarking! Compute the average and some statistics like > the standard deviation: > [end quote] > > > While I'm happy to see a real-world use for the statistics module, I > disagree with your logic. > > The problem is that random noise can only ever slow the code down, it > cannot speed it up.
Consider a workload being benchmarked running on one core, which has a particular pattern of cache hits and misses. Now consider another process running on a sibling core, sharing the same cache. Isn't it possible that under some circumstances the 2nd process could prefetch memory into the cache in such a way that the workload under test actually gets faster than if the 2nd process wasn't running? [...snip...] Hope this is constructive Dave _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com