> > You're right that microbenchmarks often do not reflect real-world > performance, but in defense of using the sample mean for benchmarking, it's > a good estimator of the population mean (provided the distribution has > finite variance), and if you perform an operation n times, it will take > about nμ seconds. If you want your code to run as fast as possible that's > probably what matters. > I don't agree. In general, whole programs with I/O has approximate log-normal distribution (from my experience). In which case the geometric mean is a far better estimator of the location and the mode of the distribution.
If you want your code to run as fast as possible, you should profile it. If you want to profile code snippets, to get the best local performances, then, and it's my whole point, you should at least plot the distributions. Means can be deceptive. A quick example: <https://lh4.googleusercontent.com/-zRvU2_hznYc/VBM_pgRuSfI/AAAAAAAAAC8/csvUyP4hfdc/s1600/proba.png> Imagine the experimental data are obtained from function `f1` with a mean of 550 s (asymmetric distribution). Imagine the normal data are obtained from function `f2` with a mean of 490 s. You would choose `f2` without any doubt. Now, Stefan, who has the same computer as you but invested in an SSD found that you made a poor choice in choosing function `f2` because on his computer, function `f1` is normal and gives a mean of 460 s vs. 490 s for `f1`. Who's correct? (We could prove that the tail was produced by inefficient IO) > As long as you run an operation enough times, the mean will accurately > reflect garbage collection overhead. (That isn't just by chance.) > What was by chance is the almost linear trend which I proved wrong by sampling values a while longer.
