On 4/8/12 11:59 AM, Denis Shelomovskij wrote:
Very good but minimum isn't a best guess. Personally I (and there will be a lot of such maniacs I suppose) will think that this (minimum) time can be significantly smaller than average time.
I've analyzed this quite a bit at work and the average and median are not very informative. You need the mode, but in most benchmarks the mode is very close to the minimum, so using the minimum is even better.
In speed measurements, all noise is additive (there's no noise that may make a benchmark appear to run faster). There are also quite a few outliers. Recording the average will include a fair amount of noise.
Clearly there is noise during normal use as well, but incorporating it in benchmarks as a matter of course reduces the usefulness of benchmarks as a mean to improve performance of the benchmarked code.
So a parameter (probably with a default value) should be added. Something like enum of flags telling what we want to know. At least these looks usable: minTime, <some mean time>, maxTime, standardDeviation, graph (yes, good old ASCII art).
Standard deviation is also not very useful because it includes all outliers (some of which are very far away from the mode).
Yes, graph is needed.
I am not sure about that. We may provide the raw measurement data for programs that want to plot things, but plotting is beyond the charter of std.benchmark.
Andrei
