On Tue, 18 Sep 2012 18:02:10 -0400 Andrei Alexandrescu <seewebsiteforem...@erdani.org> wrote:
> On 9/18/12 5:07 PM, "Øivind" wrote: > > * For all tests, the best run is selected, but would it not be > > reasonable in some cases to get the average value? Maybe excluding > > the runs that are more than a couple std. deviations away from the > > mean value.. > > After extensive tests with a variety of aggregate functions, I can > say firmly that taking the minimum time is by far the best when it > comes to assessing the speed of a function. > *Ahem*: http://zedshaw.com/essays/programmer_stats.html Your claim that the minimum time is sufficient is...ummm...extremely unorthodox, to say the least. As such, you're going to need a far more convincing argument than "It worked well for me." I assume I don't need to preach that "Extraordinary claims require extraordinary evidence". But, condensing benchmarks and statistics down to "take the minimum" and saying that's sufficient is one heck of an extraordinary claim. If you feel that you can provide sufficiently extraordinary justification, then please do. Otherwise, I think we'll need richer results. At the very least there should be an easy way to get at the raw results programmatically so we can run whatever stats/plots/visualizations/output-formats we want. I didn't see anything like that browsing through the docs, but it's possible I may have missed it. That brings up another question too: I like the idea of a one-stop-benchmarking-shop, much like we have for unittests, but maybe reporting shouldn't be so tightly integrated and left more open for integration with a proper statistics lib and more generalized output abilities? But of course, that doesn't preclude having a nice built-in, but optional, default report. (Again though, maybe I'm overlooking something already in the module?) One other nitpick: My initial impression is that the "benchmark_relative_file read" stuff seems a bit kludgey (and confusing to visually parse). Is there maybe a better way to handle that? For example, inspired by getopt: printBenchmarks!( "file write", { std.file.write("/tmp/deleteme", "hello, world!"); }, BenchmarkOption.relative, "file read", { std.file.read("/tmp/deleteme"); }, "array creation", { new char[32]; }) ();