On 9/20/12 2:37 PM, Jacob Carlborg wrote:
On 2012-09-20 14:36, Andrei Alexandrescu wrote:
Let's use the minimum. It is understood it's not what you'll see in
production, but it is an excellent proxy for indicative and reproducible
performance numbers.
Why not min, max and average?
For a very simple reason: unless the algorithm under benchmark is very
long-running, max is completely useless, and it ruins average as well.
For virtually all benchmarks I've run, the distribution of timings is a
half-Gaussian very concentrated around the minimum. Say you have a
minimum of e.g. 73 us. Then there would be a lot of results close to
that; the mode of the distribution would be very close, e.g. 75 us, and
the more measurements you take, the closer the mode is to the minimum.
Then you have a few timings up to e.g. 90 us. And finally you will
inevitably have a few outliers at some milliseconds. Those are orders of
magnitude larger than anything of interest and are caused by system
interrupts that happened to fall in the middle of the measurement.
Taking those into consideration and computing the average with those
outliers simply brings useless noise into the measurement process.
Andrei
