On 6/2/06, Terry Reedy <[EMAIL PROTECTED]> wrote:
> Hardly a setting in which to run comparison tests, seems to me.
The point though was to show that the time distribution is non-Gaussian,
so intuition based on that doesn't help.
> > Using the minimum looks like the way to go for calibration.
>
> Or possibly the median.
Why? I can't think of why that's more useful than the minimum time.
Given an large number of samples the difference between the
minimum and the median/average/whatever is mostly providing
information about the background noise, which is pretty irrelevent
to most benchmarks.
> But even better, the way to go to run comparison timings is to use a system
> with as little other stuff going on as possible. For Windows, this means
> rebooting in safe mode, waiting until the system is quiescent, and then run
> the timing test with *nothing* else active that can be avoided.
A reason I program in Python is because I want to get work done and not
deal with stoic purity. I'm not going to waste all that time (or money to buy
a new machine) just to run a benchmark.
Just how much more accurate would that be over the numbers we get
now. Have you tried it? What additional sensitivity did you get and was
the extra effort worthwhile?
> Even then, I would look at the distribution of times for a given test to
> check for anomalously high values that should be tossed. (This can be
> automated somewhat.)
I say it can be automated completely. Toss all but the lowest.
It's the one with the least noise overhead.
I think fitting the smaller data points to a gamma distribution might
yield better (more reproducible and useful) numbers but I know my
stats ability is woefully decayed so I'm not going to try. My observation
is that the shape factor is usually small so in a few dozen to a hundred
samples there's a decent chance of getting a time with minimal noise
overhead.
Andrew
[EMAIL PROTECTED]
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
