On Fri, Jul 30, 2010 at 12:05 PM, Eric Kow <[email protected]> wrote: > >> 3. Comparing *optimized* 2.4.4 to 2.5 with the "mean + sd" value shows >> that there is a significant performance regression for "darcs record". … > * http://wiki.darcs.net/Benchmarks/Dewdrop > * http://wiki.darcs.net/Benchmarks/GhLinux
I added http://wiki.darcs.net/Benchmarks/TahoeLAFSOrg > Are you using the canned tahoe-lafs repository that one fetches with > darcs-benchmark, or are you using a more recent copy. The canned one, to make the results more comparable to other people's benchmarks. > I don't know what the best practices are, but it does sound like this is > a more conservative and informative way to report results, at least in > the graphs. If nobody complains, I may tweak darcs-benchmark so that > its graphs use the mean+sd figure. I think this is a good way to go. See for example the Amazon Dynamo paper [1]. If I recall correctly, established practice within Amazon server engineering is to measure and optimize for the 99.9th percentile. So, if your service would respond within 100 ms on mean, but would, one in 1000 times, respond in 500 ms, then you would fail the "300 ms 99.9 percentile" requirement. Using mean + sd or mean + 2 * sd is a reasonable approximation of a percentile requirement. [1] http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf > By the way, I hope that a future version of darcs-benchmark will include > a feature to just run the benchmarks with '?' or '~'. I was thinking maybe a command to run everything at least 20 times so that no '?' or '~' would appear (and the results would be more statistically reliable). > More progress to go! We still don't have automated benchmarking down pat > yet (or reporting of automated benchmarks). I do think automating the execution of benchmarks and collection of reports. We already have this: http://buildbot.darcs.net/builders/6.12.1%20Debian/builds/77/steps/shell_2/logs/stdio http://buildbot.darcs.net/builders/6.12.1%20Debian/builds/78/steps/shell_2/logs/stdio http://buildbot.darcs.net/builders/6.12.1%20Debian/builds/79/steps/shell_2/logs/stdio Hm... These are interesting results here. The build 77 had these benchmarks: get (full) 6.2s d=0.1 get (lazy) 1.5s d=1.1 pull 100 3.2s d=0.2 > Next up on the performance hacking front: > > - fast darcs annotate - I hope Benedikt will be able to push for getting > this into Darcs HEAD as soon as the 2.5 release is out the door Yay! :-) > - fast darcs get over networks - Alexey's Summer of Code project will > be refined over time. One aim is to reduce the number of files that > darcs has to download. Yay! :-) This should have the accidental side-effect of making darcs work much better over Tahoe-LAFS. > - fixing Mac-related performance regressions? Maybe clues from Florian's > dtracing will lead us to some simple fishy thing that we can just poke Yay! :-) Regards, Zooko _______________________________________________ darcs-users mailing list [email protected] http://lists.osuosl.org/mailman/listinfo/darcs-users
