M.-A. Lemburg, 28.04.2011 22:23:
Stefan Behnel wrote:
DasIch, 28.04.2011 20:55:
the CPython
benchmarks have an extensive set of microbenchmarks in the pybench
package
Try not to care too much about pybench. There is some value in it, but
some of its microbenchmarks are also tied to CPython's interpreter
behaviour. For example, the benchmarks for literals can easily be
considered dead code by other Python implementations so that they may
end up optimising the benchmarked code away completely, or at least
partially. That makes a comparison of the results somewhat pointless.
The point of the micro benchmarks in pybench is to be able to compare
them one-by-one, not by looking at the sum of the tests.
If one implementation optimizes away some parts, then the comparison
will show this fact very clearly - and that's the whole point.
Taking the sum of the micro benchmarks only has some meaning
as very rough indicator of improvement. That's why I wrote pybench:
to get a better, more details picture of what's happening,
rather than trying to find some way of measuring "average"
use.
This "average" is very different depending on where you look:
for some applications method calls may be very important,
for others, arithmetic operations, and yet others may have more
need for fast attribute lookup.
I wasn't talking about "averages" or "sums", and I also wasn't trying to
put down pybench in general. As it stands, it makes sense as a benchmark
for CPython.
However, I'm arguing that a substantial part of it does not make sense as a
benchmark for PyPy and others. With Cython, I couldn't get some of the
literal arithmetic benchmarks to run at all. The runner script simply bails
out with an error when the benchmarks accidentally run faster than the
initial empty loop. I imagine that PyPy would eventually even drop the loop
itself, thus leaving nothing to compare. Does that tell us that PyPy is
faster than Cython for arithmetic? I don't think it does.
When I see that a benchmark shows that one implementation runs in 100% less
time than another, I simply go *shrug* and look for a better benchmark to
compare the two.
Stefan
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
