The standard deviation can give you an estimation of the minimal running time. (99.xxxx of the samples are within X standard deviations below the average. Pick a high enough xxxx relative to the number of times you'll run the software, and you'll get an estimation of the minimum running time you'll get).
The scalar representing the minimum running time have the obnoxious mis-feature that it doesn't tend to a certain value. So with a small probability you can sample your input today 1m times, and tomorrow yet another 1m times, but still get very different results because of a very rare event. The probability this would happen with the mean or the standard deviation is much much much lower. But I know almost nothing about statistical analysis, so don't take my word for it, and I'll be glad to hear any corrections. On Sun, Jul 18, 2010 at 1:35 PM, Shlomi Fish <shlo...@iglu.org.il> wrote: > On Sunday 18 Jul 2010 11:56:33 Elazar Leibovich wrote: > > While you're on to it. > > I expect to read in a benchmark report, the number of time the software > was > > executed, the mean running time, and the standard deviation. Running and > > timing it once can hide a pretty large error. > > > > Perhaps you're right about that for more complex software. But for Freecell > Solver, I noticed that running it on the virtual console without X running > and > while renicing it to the maximal possible priority, yields about the same > runtime every time. This maybe because it is almost purely CPU/memory bound > and does not use the disk much. > > I'm also more interested in the minimal running time than I am in the mean > one. > > Regards, > > Shlomi Fish > > -- > ----------------------------------------------------------------- > Shlomi Fish http://www.shlomifish.org/ > "Humanity" - Parody of Modern Life - http://shlom.in/humanity > > God considered inflicting XSLT as the tenth plague of Egypt, but then > decided against it because he thought it would be too evil. > > Please reply to list if it's a mailing list post - http://shlom.in/reply . >
_______________________________________________ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
