On 9/20/12 10:05 AM, Manu wrote:
On 20 September 2012 15:36, Andrei Alexandrescu
<seewebsiteforem...@erdani.org <mailto:seewebsiteforem...@erdani.org>>
wrote:
Let's use the minimum. It is understood it's not what you'll see in
production, but it is an excellent proxy for indicative and
reproducible performance numbers.
If you do more than a single iteration, the minimum will virtually
always be influenced by ideal cache pre-population, which is
unrealistic.
To measure performance against cold cache, you could always clear the
cache using one of the available methods, see
http://stackoverflow.com/questions/1756825/cpu-cache-flush. Probably
std.benchmark could include a routine that does that. But performance on
cold would actually be most unrealistic and uninformative, as loading
the memory into cache will dominate the work that the algorithm is
doing, so essentially the benchmark would evaluate the memory bandwidth
against the working set of the algorithm. That may be occasionally
useful, but I'd argue that most often the interest in benchmarking is to
measure repeated application of a function, not occasional use of it.
Memory locality is often the biggest contributing
performance hazard in many algorithms, and usually the most
unpredictable. I want to know about that in my measurements.
Reproducibility is not important to me as accuracy. And I'd rather be
conservative(/pessimistic) with the error.
>
What guideline would you apply to estimate 'real-world' time spent when
always working with hyper-optimistic measurements?
The purpose of std.benchmark is not to estimate real-world time. (That
is the purpose of profiling.) Instead, benchmarking measures and
provides a good proxy of that time for purposes of optimizing the
algorithm. If work is done on improving the minimum time given by the
benchmark framework, it is reasonable to expect that performance in-situ
will also improve.
Andrei
