In the world of technical computing, we are mostly interested in running the same computationally intensive functions over and over, and we usually only care about how long that will take. The best way to approximate how long that will take is to run the function over and over and see how long it takes. Moreover, the central limit theorem guarantees that, given a function that takes a mean time μ to run, the amount of time it takes to run that function n times is approximately normally distributed with mean nμ for sufficiently large n regardless of the distribution of the individual function evaluation times as long as it has finite variance. Of course if something takes longer to run the more times you run it the central limit theorem will not apply, but then a histogram won't be much use then either, although plotting function eval time vs. clock time would be.
Your effect from running the function many times is curious but it's not reproducible for me. After running for 20 minutes I still just see two bands. The difference between the means of the 2020 samples at the beginning, middle, and end is <2%. Can you reproduce that? Simon On Friday, September 12, 2014 3:03:49 PM UTC-4, gael....@gmail.com wrote: > > You're right that microbenchmarks often do not reflect real-world >> performance, but in defense of using the sample mean for benchmarking, it's >> a good estimator of the population mean (provided the distribution has >> finite variance), and if you perform an operation n times, it will take >> about nμ seconds. If you want your code to run as fast as possible that's >> probably what matters. >> > I don't agree. In general, whole programs with I/O has approximate > log-normal distribution (from my experience). In which case the geometric > mean is a far better estimator of the location and the mode of the > distribution. > > If you want your code to run as fast as possible, you should profile it. > > If you want to profile code snippets, to get the best local performances, > then, and it's my whole point, you should at least plot the distributions. > > Means can be deceptive. A quick example: > > > <https://lh4.googleusercontent.com/-zRvU2_hznYc/VBM_pgRuSfI/AAAAAAAAAC8/csvUyP4hfdc/s1600/proba.png> > Imagine the experimental data are obtained from function `f1` with a mean > of 550 s (asymmetric distribution). > Imagine the normal data are obtained from function `f2` with a mean of 490 > s. > > You would choose `f2` without any doubt. Now, Stefan, who has the same > computer as you but invested in an SSD found that you made a poor choice in > choosing function `f2` because on his computer, function `f1` is normal and > gives a mean of 460 s vs. 490 s for `f1`. Who's correct? > > (We could prove that the tail was produced by inefficient IO) > > > >> As long as you run an operation enough times, the mean will accurately >> reflect garbage collection overhead. (That isn't just by chance.) >> > What was by chance is the almost linear trend which I proved wrong by > sampling values a while longer. > >
