08.04.2012 21:31, Andrei Alexandrescu пишет:
On 4/8/12 11:59 AM, Denis Shelomovskij wrote:
Very good but minimum isn't a best guess. Personally I (and there will
be a lot of such maniacs I suppose) will think that this (minimum) time
can be significantly smaller than average time.
I've analyzed this quite a bit at work and the average and median are
not very informative. You need the mode, but in most benchmarks the mode
is very close to the minimum, so using the minimum is even better.
In speed measurements, all noise is additive (there's no noise that may
make a benchmark appear to run faster). There are also quite a few
outliers. Recording the average will include a fair amount of noise.
Yes of course. I mean than an algorithm itself should follow some
restrictions for such measurement. It should run with the same speed
every time. Many algorithms follows this convention, but not all.
Why will recording the average produce so much noise? As I see, floating
point arithmetic is now used without a strong reason so it looks like a
time of this part isn't valuable. Or is it just a temporary solution?
Anyway it should be configurable using a CT parameter, so it will not
abuse one who doesn't need it.
Clearly there is noise during normal use as well, but incorporating it
in benchmarks as a matter of course reduces the usefulness of benchmarks
as a mean to improve performance of the benchmarked code.
A graph is needed exactly because of that. Without a graph it really
gives very little.
So a parameter (probably with a default value) should be added.
Something like enum of flags telling what we want to know. At least
these looks usable: minTime, <some mean time>, maxTime,
standardDeviation, graph (yes, good old ASCII art).
Standard deviation is also not very useful because it includes all
outliers (some of which are very far away from the mode).
So a graph is needed.
Yes, graph is needed.
I am not sure about that. We may provide the raw measurement data for
programs that want to plot things, but plotting is beyond the charter of
std.benchmark.
Sorry, meant a histogram, not a curve. A histogram can be shown in a
console very well. And it is needed because its the easiest way to show
benchmarked program behaviour (and noise behaviour). It also requires
only about 80 integers to store information and shouldn't produce much
noise.
IMHO, a histogram gives lots of information and will be a good addition.
--
Денис В. Шеломовский
Denis V. Shelomovskij
