On Sun, Jan 31, 2010 at 9:02 AM, Patrick Wright <[email protected]> wrote:
In addition, I think that having an open-source micro-benchmarking > library for this purpose would be very helpful. It might not be > perfect or all-encompassing, but it would allow researchers and > tinkerers a base on which to test their own code, and on which to base > arguments of "improvements" to the current libs. In essence, it would > be like saying that the OpenJDK is written as it is because it works > well in these standard benchmarks, and if someone thinks they have > improvements, they have to prove it against the same benchmarks that > Sun/Oracle has used for their own decision-making. > > There is/was a Sun project I mentioned on another JDK thread, Japex > (search online) which is a test harness for micro-benchmarking. > Perhaps something like this could provide a standard benchmarking, > with records of historical results, into which different tests could > be plugged. I don't have any reason to prefer Japex over Caliper or > another framework; in general I think that a framework that could take > care (at least) of isolating the warm-up phase, taking and recording > measurements, etc. would be better than people starting from scratch > in main() each time they write a benchmark. > FWIW, this is a very accurate depiction of our goals for Caliper and the reasons why we're making it. Development has been proceeding in fits and starts (presently: it's throwing a "fit"), but we intend to keep at it until it has become a very useful tool for you and others who do this kind of work. If you'd like to keep tabs on our development -- possibly even help? (dare I dream?) -- please drop me a line and I'll add you to our mailing list! -- Kevin Bourrillion @ Google internal: http://go/javalibraries external: guava-libraries.googlecode.com
