Hi Zooko, Many thanks for the report.
Reordering slightly. > 3. Comparing *optimized* 2.4.4 to 2.5 with the "mean + sd" value shows > that there is a significant performance regression for "darcs record". Now that's surprising, given the fix in http://bugs.darcs.net/issue1106 In our other benchmarking efforts, darcs record improves rather dramatically: * http://wiki.darcs.net/Benchmarks/Dewdrop * http://wiki.darcs.net/Benchmarks/GhLinux Are you using the canned tahoe-lafs repository that one fetches with darcs-benchmark, or are you using a more recent copy. If the latter, how many patches since the last tag, please? For what it's worth, here's the output in our canned tahoe-lafs % darcs changes --count --from-tag . 10 > 2. I like to look at mean + sd or mean + 2 * sd instead of just mean. > My intuition is that I'd rather have something that's sort of fast > almost every time than something that's really fast half the time and > slow half the time, even if the latter thing has a better mean. I don't know what the best practices are, but it does sound like this is a more conservative and informative way to report results, at least in the graphs. If nobody complains, I may tweak darcs-benchmark so that its graphs use the mean+sd figure. One disadvantage for tables is that we'd have to explain that we do this, and that people are unlikely to notice the explanation. > Folks, I benchmarked darcs-2.4.4 vs. the current 2.5.0 pre-release on > one of my linux servers with the Tahoe-LAFS repo. I reran the > benchmark over and over until it stopped having any "?" or "~" in the > output. Here is the final results: By the way, I hope that a future version of darcs-benchmark will include a feature to just run the benchmarks with '?' or '~'. > Overall: way to go on making useful benchmarks that are easy to > install and run! And way to go on making a new release of darcs that > is mostly better and only possibly a little bit worse than 2.4.4! More progress to go! We still don't have automated benchmarking down pat yet (or reporting of automated benchmarks). Next up on the performance hacking front: - fast darcs annotate - I hope Benedikt will be able to push for getting this into Darcs HEAD as soon as the 2.5 release is out the door - fast darcs get over networks - Alexey's Summer of Code project will be refined over time. One aim is to reduce the number of files that darcs has to download. - fixing Mac-related performance regressions? Maybe clues from Florian's dtracing will lead us to some simple fishy thing that we can just poke ... -- Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow> For a faster response, please try +44 (0)1273 64 2905.
pgp8jRiEZluCp.pgp
Description: PGP signature
_______________________________________________ darcs-users mailing list [email protected] http://lists.osuosl.org/mailman/listinfo/darcs-users
