On [many an occasion] "Many people" wrote:
> user=> (time foo)
...
> "Elapsed time: 245.152 msecs"
I hate to be a wet blanket, but how accurate is this? The doc doesn't
even say whether it measures wall clock time or cpu time. Even if you
knew that, it's at best a foundation to build a real benchmarking
system on.
By "real benchmarking" I mean one that does things like run the
benchmark a few times to fill caches and let the JIT do it's thing,
etc., makes sure it runs the target code enough times to get a chunk
of time large enough to means something, and takes into account the
overhead of making multiple runs. Now do that a dozen or so times so
you can take some stats, so when you compare two timings you actually
have enough information to decide if one is really faster than the
other, or the difference is just fuzz in the measurement system.
Ok, I started building something like that, and immediately started
getting ludicrous results. Googling for Java benchmarks turned up
exactly *one* such tool (which presumably works on any runnable or
callable, but fails on Clojure functions) amongst the sets of
benchmark code (most of which didn't talk about how they measured the
time).
First thing: a timer that returns the time so I can work with it:
(defmacro nano-timer "Time body, returning nanoseconds and result"
[body] `(do
(System/runFinalization)
(System/gc)
(let [start# (System/nanoTime)
res# ~body]
[(- (System/nanoTime) start#) res#])))
Second thing: run the benchmark many times, subtracting out the loop
overhead:
(defmacro loop-timer
"Time code in a loop, returning nanoseconds taken minus loop overhead"
[code count] `(let [overhead# (first (nano-timer (dotimes [_# ~count])))
mytime# (first (nano-timer (dotimes [_# ~count] ~code)))]
(- mytime# overhead#)))
Which has the annoying habit of returning negative numbers :-(. I'd
really like to know how the loop that calculates overhead takes more
time than the loop that actually runs the code, no matter how trivial
the code is!
Is anyone using anything more sophisticated than clojure.core/time for
benchmarking clojure code?
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
