STINNER Victor added the comment: Not only I'm too lazy to compute manually the number of loops and repeat, but also I don't trust myself. It's even worse when someone publishs results of a micro-benchmark. I don't trust how the benchmark was calibrated. In my experience, micro-benchmark are polluted by noise in timings, so results are not reliable.
benchmarks.py calibration is based on time, whereas timeit uses hardcoded constants (loops=1000000, repeat=3) which can be modified on the command line. benchmarks.py has 3 main parameters: - minimum duration of a single run (--min-time): 100 ms by default - maximum total duration of the benchmark: benchmark.py does its best to respect this duration, but it can be longer: 1 second by default - minimum repeat: 5 by default The minimum duration is increased if the clock resolution is bad (1 ms or more). It's the case on Windows for time.clock() on Python 2 for example. Extract of benchmark.py: min_time = max(self.config.min_time, timer_precision * 100) The estimation of the number of loops is not reliable, but it's written to be "fast". Since I run a micro-benchmark many times, I don't want to wait too long. It's not a power of 10, but an arbitrary integer number. Usually, when running benchmark.py multiple times, the number of loops is different each time. It's not really a big issue, but it probably makes results more difficult to compare. My constrain is max_time. The tested function may not have a linear duration (time = time_one_iteration * loops). https://bitbucket.org/haypo/misc/src/348bfd6108e9985b3c2298d2745eb5ddfe7042e6/python/benchmark.py?at=default#cl-416 Repeat a test at least 5 times is a compromise between the stability of the result and the total duration of the benchmark. Feel free to reuse my code to enhance time.py. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue23693> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com