On Jul 2, 7:41 pm, Mike Meyer <mwm-keyword-googlegroups. 620...@mired.org> wrote: > On Thu, 1 Jul 2010 22:19:56 -0400 > It depends on what you're benchmarking. If the loop time ... is > on the order of the same size as the standard deviation, then it can > fool you into falsely concluding that there's no statistically > significant difference between two algorithms. Once it gets to be > around half the total benchmark time, your results are pretty much > worthless.
I'm not a JVM benchmarking expert by any means, but I have been working with Java for close to a decade, and to the best of my knowledge that level of micro-benchmarking precision is not possible, for all the reasons listed and linked to - dynamic HotSpot optimizations, unpredictable garbage collection, limited timing resolution etc. For loops that are part of the program, like iterating over all the pixels in an image to compute a transform, it's a non-issue. Those loops are part of the algorithm, and if transform A takes the same time as transform B because both of them are dominated by the time it takes to loop through every pixel, you can correctly conclude that the difference between A and B does not matter. But the benchmarking loop time and other administrative overhead (including the extra overhead on garbage collection) needs to be an insignificant fraction of the total, yes. If the benchmarking framework does more work than the code being measured, you will be benchmarking the framework rather than the code you want to measure. I normally deal with it by putting larger chunks of code inside the benchmarking loop, aiming for at least a second of runtime per invocation. Even then, the PDF from Azul systems linked to earlier says that "performance varies from run to run - stable numbers within a single JVM invocation, but could vary > 20% with new JVM invocation", which matches well with my experience. In practice, I have given up on timing very small blocks of code, and I have given up on trying for precision better than +/-10%. Your mileage may vary. jf -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
