I'm writing some tests to check for performance regressions (i.e. you
change a function, and it becomes much slower) and I was hoping for some
guidelines or hints.
This is what I have come up with so far:
* The disclaimers about timing code snippets that can be found in the
timeit module apply. If possible, use timeit rather than roll-you-own
timers.
* Put performance tests in a separate test suite, because they're
logically independent of regression tests and functional tests, and
therefore you might not want to run them all the time.
* Never compare the speed of a function to some fixed amount of time,
since that will depend on the hardware you are running on, but compare it
relative to some other function's running time. E.g.:
# Don't do this:
time_taken = Timer(my_func).timeit() # or similar
assert time_taken <= 10
# This is bad, since the test is hardware dependent, and a change
# in environment may cause this to fail even if the function
# hasn't changed.
# Instead do this:
time_taken = Timer(my_func).timeit()
baseline = Timer(simple_func).timeit()
assert time_taken <= 2*baseline
# my_func shouldn't be more than twice as expensive as simple_func
# no matter how fast or slow they are in absolute terms.
Any other lessons or hints I should know?
If it helps, my code will be targeting Python 3.1, and I'm using a
combination of doctest and unittest for the tests.
Thanks in advance,
--
Steven.
--
http://mail.python.org/mailman/listinfo/python-list
