Folks, I benchmarked darcs-2.4.4 vs. the current 2.5.0 pre-release on
one of my linux servers with the Tahoe-LAFS repo. I reran the
benchmark over and over until it stopped having any "?" or "~" in the
output. Here is the final results:
Timings
====================================================
Tahoe LAFS
----------
============== ======== ========== ======== ==========
============== ==========
2.4.4 sdev op 2.4.4 sdev op
2.4.98.1+62 sdev
============== ======== ========== ======== ==========
============== ==========
wh 36.7ms (6.5ms) 12.8ms (2.4ms)
13.3ms (1.6ms)
wh mod 145.9ms (54.8ms) 115.0ms (34.8ms)
108.9ms (9.2ms)
wh -l 82.2ms (24.4ms) 49.6ms (13.7ms)
49.0ms (24.7ms)
record mod 210.3ms (118.1ms) 160.3ms (42.4ms)
182.1ms (113.9ms)
revert mod 137.6ms (47.0ms) 112.9ms (24.1ms)
114.5ms (30.0ms)
(un)revert mod 326.7ms (119.4ms) 286.8ms (145.8ms)
268.9ms (73.4ms)
get (full) 7.8s (1.4s) 8.1s (2.0s)
8.5s (2.6s)
get (lazy) 1523.2ms (1236.6ms) 2542.4ms (3133.7ms)
1876.2ms (1646.7ms)
pull 100 4.3s (1.4s) 4.7s (1.3s)
4.2s (0.9s)
pull 1000 32.4s (6.3s) 34.4s (7.0s)
33.5s (7.6s)
check 21.2s (4.9s) 21.3s (5.0s)
21.3s (4.9s)
repair 21.5s (4.9s) 21.5s (5.0s)
23.9s (6.2s)
annotate 17.8s (5.8s) 18.0s (6.3s)
30.7s (20.9s)
============== ======== ========== ======== ==========
============== ==========
My thoughts:
1. The automatic optimization in 2.5 is a big win. I think it is fair
to assume that many users, even those who are struggling with
performance issues, won't know to run "darcs optimize --pristine". (I
didn't.) So for those users it is fair to compare the 2.4.4 column to
the optimized 2.5 column.
However, now that I do know about "darcs optimize --pristine", I might
choose to stick with darcs-2.4.4 and run "darcs optimize --pristine"
instead of upgrading to darcs-2.5, so it is also fair to compare the
"op 2.4.4" column to the 2.5 column.
2. I like to look at mean + sd or mean + 2 * sd instead of just mean.
My intuition is that I'd rather have something that's sort of fast
almost every time than something that's really fast half the time and
slow half the time, even if the latter thing has a better mean.
3. Comparing *optimized* 2.4.4 to 2.5 with the "mean + sd" value shows
that there is a significant performance regression for "darcs record".
I think "darcs record" isn't too critical for my server, where darcs
record is rarely executed, so I don't mind upgrading to 2.5 on my
server. For local use this might be an issue--I'll benchmark darcs on
my laptop and report back.
4. There seems to be a performance regression for get (full) and get
(lazy). Again this won't prevent me from upgrading on the server but I
might advise my users not to upgrade their local versions. I'll write
more about that after benchmarking on my laptop.
Overall: way to go on making useful benchmarks that are easy to
install and run! And way to go on making a new release of darcs that
is mostly better and only possibly a little bit worse than 2.4.4!
Regards,
Zooko
_______________________________________________
darcs-users mailing list
[email protected]
http://lists.osuosl.org/mailman/listinfo/darcs-users
