On Tue, Aug 03, 2010 at 21:58:56 -0600, Zooko O'Whielacronx wrote: > > Are you using the canned tahoe-lafs repository that one fetches with > > darcs-benchmark, or are you using a more recent copy. > > The canned one, to make the results more comparable to other people's > benchmarks.
Hmm, how interesting... performance regression in record?! > > I don't know what the best practices are, but it does sound like this is > > a more conservative and informative way to report results, at least in > > the graphs. If nobody complains, I may tweak darcs-benchmark so that > > its graphs use the mean+sd figure. > > Using mean + sd or mean + 2 * sd is a reasonable approximation of a > percentile requirement. > > [1] > http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf HEAD darcs-benchmark uses mean + sd everywhere. In the tables, you'll see a number, followed by the stdev 401 ms (32 ms) The first number is the mean + sd figure; to get the mean, just subtract the stddev. If somebody with better statistical fu says mean + 2 sd is better, it's an easy switch to flip > I was thinking maybe a command to run everything at least 20 times so > that no '?' or '~' would appear (and the results would be more > statistically reliable). Maybe. The reason I avoided that was to make it easy for people to be able to interrupt darcs-benchmark and still get some good results. HEAD darcs-benchmark has a --converge flag. The granularity is only at the (repo, benchmark) level, which is still better than nothing. If you run darcs-benchmark run --converge, for each repository, it will only run benchmarks with less that 20 data points collected. Perhaps a simple tweak would be to have a special exit code indicating that there are no more benchmarks to run; then folks can loop. > > - fast darcs get over networks - Alexey's Summer of Code project will > > be refined over time. One aim is to reduce the number of files that > > darcs has to download. > > Yay! :-) Alexey's recent blog post (see planet Darcs is quite promising). I guess it's too late for darcs 2.5, but maybe that's for the better. It will give us 6 more months (until darcs 2.8) to make sure we have it right. -- Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow> For a faster response, please try +44 (0)1273 64 2905.
signature.asc
Description: Digital signature
_______________________________________________ darcs-users mailing list [email protected] http://lists.osuosl.org/mailman/listinfo/darcs-users
