Hello Nathaniel, Am 25.08.2010 07:35, schrieb Stefan Behnel: > Nathaniel Smith, 24.08.2010 18:48: >> On Sun, Aug 22, 2010 at 11:57 PM, Kay Hayen wrote: >>> I currently only have "pystone" as a benchmark, which is a bit >>> difficult to look at, because it's not good at telling you why it has >>> that speed, aggregating everything into one. >> >> You might want to take advantage of the work the Unladen Swallow folks >> have put into benchmark tooling: >> http://code.google.com/p/unladen-swallow/wiki/Benchmarks > > Yes, pystone and pybench are particularly bad benchmarks for translation to > C code. Pybench even has a benchmark that simply assigns numbers to > variables. When compiled into optimised C code, this becomes useless code > that the C compiler discards. So the benchmark sometimes fails because it > can happen that it runs a tiny bit faster than even the calibration loop.
I have looked at these and I believe it may indeed be a realistic benchmark to see how fast "hg" starts, the kind of things that Google uses there make a lot of sense. I also like the statistically significant analysis, so you know if it's just noise or you really achieved something. And I think I will use it, or at least part of it, to get an overall feeling of how an application benefits from using a compiler. But keeping track of the amount of CPU cycles and memory like Valgrind does, seems to give a lot less noise from the start. plus I would use Valgrind anyway. The thing is, we want to see, when a certain function increases the number of cycles, because something is no longer discarded, or even worse, one more or slower call into CPython library is done. Most of the time its the implicit coercions that surprise you. The good thing about it is that - if done right - you need no timing loop at all. You can use very basic tests and need no loop to make these tests. The bad thing about that is that caches will not be measured. But to me that's a hard to measure secondary effect anyway, more important it is to avoid calls into Python and branches not needed. Yours, Kay _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
