[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: I updated the documentation in 3.7 and master branches. -- ___ Python tracker ___ ___

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: New changeset cebd4b009adca6611e92eb337747f59818e941a6 by Victor Stinner (Miss Islington (bot)) in branch '3.7': bpo-28240: timeit: Update repeat() doc (GH-7419) (GH-7457) https://github.com/python/cpython/commit/cebd4b009adca6611e92eb337747f59818e941a6

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Change by miss-islington : -- pull_requests: +7080 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: New changeset 3ef769fcd378a7f1cda19c0dfec2e79613d79e48 by Victor Stinner in branch 'master': bpo-28240: timeit: Update repeat() doc (GH-7419) https://github.com/python/cpython/commit/3ef769fcd378a7f1cda19c0dfec2e79613d79e48 --

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Change by STINNER Victor : -- pull_requests: +7044 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Serhiy Storchaka added the comment: Yes, it was my thought. But seems you are right, it is easier to use Python as a programming language. In the past I used the CLI because the programming interface didn't supported autoranging. Although I would change the

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: > Wait, I didn't notice the change to the format of raw timings. It looks as a > regression to me. Do you mean that some applications may run timeit as a CLI and parse stdout to get raw values? Why doing so? timeit is a Python module,

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Serhiy Storchaka added the comment: Wait, I didn't notice the change to the format of raw timings. It looks as a regression to me. -- status: closed -> open ___ Python tracker

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Changes by Donald Stufft : -- pull_requests: +945 ___ Python tracker ___ ___

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Roundup Robot added the comment: New changeset 4e4d4e9183f5 by Victor Stinner in branch 'default': Issue #28240: Fix formatting of the warning. https://hg.python.org/cpython/rev/4e4d4e9183f5 -- ___ Python tracker

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Serhiy Storchaka added the comment: >> Sorry, I don't understand how running 1 iteration instead of 10 makes the >> benchmark less reliable. IMO the reliability is more impacted by the number > Caches. Not high-level caching that can make the measurement

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Serhiy: "It may be worth to emit a warning in case of using timeit for too short code." I suggest to simply warn users that timeit is not reliable at all :-) By the way, I close this issue, so I suggest you to open new issues if you want further

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Serhiy Storchaka added the comment: It may be worth to emit a warning in case of using timeit for too short code. -- ___ Python tracker ___

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Serhiy Storchaka: > This is a senseless example. 0.0339 usec is not a time of executing "pass", > it is an overhead of the iteration. You can't use timeit for measuring the > performance of the code that takes such small time. You just can't get the >

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Serhiy Storchaka added the comment: > Sorry, I don't understand how running 1 iteration instead of 10 makes the > benchmark less reliable. IMO the reliability is more impacted by the number > of repeatitions (-r). I changed the default from 3 to 5 repetitions, so > timeit should be *more*

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: I'm disappointed by the discussion on minumum vs average. Using the perf module (python3 -m perf timeit), it's very easy to show that the average is more reliable than the minimum. The perf module runs 20 worker processes by default: with so many processes,

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Serhiy Storchaka: >> * autorange: start with 1 loop instead of 10 for slow benchmarks like >> time.sleep(1) > This is good if you run relatively slow benchmark, but it makes the result > less reliable. You always can specify -n1, but on your own risk. Sorry,

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Changes by Maciej Fijalkowski : -- nosy: -fijall ___ Python tracker ___ ___

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: > For now default timeit run takes from 0.8 to 8 sec. Adding yet 5 sec makes a > user more angry. See the issue #28469 "timeit: use powers of 2 in autorange(), instead of powers of 10" for a simple fix to reduce the total duration of the worst case (reduce

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Steven D'Aprano: >> * Display large number of loops as power of 10 for readability, ex: >> "10^6" instead of "100". Also accept "10^6" syntax for the --num >> parameter. > > Shouldn't we use 10**6 or 1e6 rather than bitwise XOR? :-) Hum, with "10**6"

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Steven D'Aprano: >> * Don't disable the garbage collector anymore! Disabling the GC is not >> fair: real applications use it. > But that's just adding noise: you're not timing code snippet, you're timing code snippet plus garbage collector. > > I disagree

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Roundup Robot added the comment: New changeset 4d611957732b by Victor Stinner in branch 'default': timeit: enhance format of raw timings (in verbose mode) https://hg.python.org/cpython/rev/4d611957732b New changeset c3a93069111d by Victor Stinner in branch 'default': timeit: add nsec

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Roundup Robot added the comment: New changeset 975df4c13db6 by Victor Stinner in branch 'default': timeit: remove --clock and --time options https://hg.python.org/cpython/rev/975df4c13db6 -- ___ Python tracker

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Roundup Robot added the comment: New changeset 3aba5552b976 by Victor Stinner in branch 'default': timeit: start autorange with 1 iteration, not 10 https://hg.python.org/cpython/rev/3aba5552b976 New changeset 2dafb2f3e7ff by Victor Stinner in branch 'default': timeit: change default repeat to

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Oh, cfbolz just modified timeit in PyPy to display average (mean) and standard deviation: https://bitbucket.org/pypy/pypy/commits/fb6bb835369e Moreover, PyPy timeit now displays the following warning: --- WARNING: timeit is a very unreliable tool. use perf or

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Ned Deily added the comment: Until we have a consensus on this change and a final, reviewed patch, it is premature to consider inclusion in 3.6. If there is such prior to 360b2, we can reconsider. -- versions: -Python 3.6 ___ Python tracker

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Raymond Hettinger added the comment: I concur with all of Mark-Andre's comments. FWIW, when I do timings with the existing timeit, I use a repeat of 7. It gives stable and consistent results. The whole idea of using the minimum of those runs is to pick the least noisy measurement.

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Marc-Andre Lemburg added the comment: > Marc-Andre: "Consensus then was to use the minimum as basis for benchmarking: > (...) There are arguments both pro and con using min or avg values." > > To be honest, I expect that most developer are already aware that minimum is > evil and so I

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Antoine Pitrou added the comment: Le 23/09/2016 à 10:21, STINNER Victor a écrit : > > Quick rationale: the purpose of displaying the average rather than > the minimum in timeit is to make timeit more *reliable*. My goal is that running timeit 5 times would give exactly the same result. Why

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Marc-Andre: "Consensus then was to use the minimum as basis for benchmarking: (...) There are arguments both pro and con using min or avg values." To be honest, I expect that most developer are already aware that minimum is evil and so I wouldn't have to

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Steven D'Aprano added the comment: > I'd suggest to display all values and base the findings on > all available values, rather than just one: > min, max, avg, median, stddev. If we're going to go down that path, I suggest using something like: https://en.wikipedia.org/wiki/Five-number_summary

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Marc-Andre Lemburg added the comment: We had a similar discussion a while back for pybench. Consensus then was to use the minimum as basis for benchmarking: https://mail.python.org/pipermail/python-dev/2006-June/065525.html I had used the average before this discussion in pybench 1.0:

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: Maciej Fijalkowski also sent me the following article a few months ago, it also explains indirectly why using the minimum for benchmarks is not reliable: "Virtual Machine Warmup Blows Hot and Cold" http://arxiv.org/pdf/1602.00602.pdf Even if the article is

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Antoine Pitrou added the comment: Another point: timeit is often used to compare performance between Python versions. By changing the behaviour of timeit in a given Python version, you'll make it more difficult to compare results. -- ___ Python

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Changes by Antoine Pitrou : -- nosy: +tim.peters ___ Python tracker ___ ___ Python-bugs-list

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Antoine Pitrou added the comment: > Display the average, rather than the minimum, of the timings *and* display > the standard deviation. It should help a little bit to get more reproductible > results That entirely depends on which benchmark you are running. > Disabling the GC is not fair:

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: > * Display the average, rather than the minimum, of the timings *and* display > the standard deviation. It should help a little bit to get more reproductible > results. Rationale: * http://blog.kevmod.com/2016/06/benchmarking-minimum-vs-average/ *

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

STINNER Victor added the comment: The perf module displays the median rather than the mean (arithmeric average). The difference between median and mean is probably too subtle for most users :-/ The term "median" is also probably unknown by most users... I chose to use average: well known,

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

Changes by STINNER Victor : -- title: Enhance the timeit module: diplay average +- std dev instead of minimum -> Enhance the timeit module: display average +- std dev instead of minimum ___ Python tracker