[issue28637] Python startup performance regression

Changes by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___

[issue28637] Python startup performance regression

Changes by Christian Heimes : -- nosy: +christian.heimes ___ Python tracker ___ ___

[issue28637] Python startup performance regression

STINNER Victor added the comment: The initial bug (performance regression in python_startup) is fixed. Thanks everyone for the nice discussion. I close the issue. Please open new issues if you want to continue efforts on making CPython faster ;-) -- resolution: -> fixed status: open

[issue28637] Python startup performance regression

STINNER Victor added the comment: Ethan Furman: "Does this mean we can put enum back in re?" I checked manually: adding "import enum" to Lib/re.py has no more impact on "python_startup" benchmark. Guido van Rossum: "Honestly now the basic startup time is under control I am fine with

[issue28637] Python startup performance regression

Roundup Robot added the comment: New changeset 5fd69d4a93e0 by Victor Stinner in branch '3.6': Issue #28637: Reapply changeset 223731925d06 https://hg.python.org/cpython/rev/5fd69d4a93e0 -- ___ Python tracker

[issue28637] Python startup performance regression

Serhiy Storchaka added the comment: Thanks to Wolfgang. This is his idea, I just faster wrote the patch. -- ___ Python tracker ___

[issue28637] Python startup performance regression

Raymond Hettinger added the comment: Serhiy, thanks for your work. By isolating the few easily rewritten lines in site.py, you saved us from endless contortions trying to avoid using our own batteries. -- nosy: +rhettinger ___ Python tracker

[issue28637] Python startup performance regression

Guido van Rossum added the comment: I think it's okay to add `enum` back into `re`. -- ___ Python tracker ___

[issue28637] Python startup performance regression

Ethan Furman added the comment: Leaving `re` out of site.py is fine; the current question is whether to add `enum` back to `re`. -- ___ Python tracker

[issue28637] Python startup performance regression

STINNER Victor added the comment: Oh nice, python_startup benchmark at Nov 10 (rev at 8ebaa546a033) is now faster than python_startup at Feb 29 (rev dfeadfb4c8cc): new 16.8 ms, old 19.5. And it's much before when I opened the issue: 26.6 ms. Thanks.

[issue28637] Python startup performance regression

Guido van Rossum added the comment: Honestly now the basic startup time is under control I am fine with restoring the re enum. We can optimize that later. --Guido (mobile) -- ___ Python tracker

[issue28637] Python startup performance regression

STINNER Victor added the comment: About the re module, there is still a regression on the performance of the import time: see my msg280271. We may also revert the changeset 88110cfbf4dc (issue #28193) if we want to keep "import re" as fast as Python 3.5. The question is if we consider that the

[issue28637] Python startup performance regression

Serhiy Storchaka added the comment: On my computer: Importing empty module: 160 us Creating empty class: 30 us Creating empty function: 0.16 us Creating empty Enum/IntEnum: 125/150 us Creating Enum/IntEnum member: 25/27 us Creating empty namedtuple: 600 us Creating namedtuple member: 50 us

[issue28637] Python startup performance regression

Gregory P. Smith added the comment: re: Ethan's question - I think the enum use should be restored in re. I realize issue28082 (yay palindrome number) is not an urgent change but we created IntEnum for the purpose of more identifiable integer constants. So a microbenchmark of "import re"

[issue28637] Python startup performance regression

Guido van Rossum added the comment: OK. I believe we also found that the extra slowdown is only in a virtualenv so that may explain that I saw something different. -- title: Python startup performance regression due to enum in re -> Python startup performance regression

[issue28637] Python startup performance regression due to enum in re

Ethan Furman added the comment: Does this mean we can put enum back in re? -- title: Python startup performance regression -> Python startup performance regression due to enum in re ___ Python tracker

[issue28637] Python startup performance regression

STINNER Victor added the comment: Guido: "So what does that benchmark measure? For me, python 3.6 startup is 44 ms and python 2.7 startup is 78 ms (real time; user time is proportionally less)." It measures something like "time python -c pass". The performance module creates a virtual

[issue28637] Python startup performance regression

STINNER Victor added the comment: Cool, thanks Serhiy for the site enhancement :-) -- ___ Python tracker ___

[issue28637] Python startup performance regression

Roundup Robot added the comment: New changeset a822818ec74e by Serhiy Storchaka in branch '3.6': Issue #28637: No longer use re in site.py. https://hg.python.org/cpython/rev/a822818ec74e New changeset a4ea837a7f84 by Serhiy Storchaka in branch 'default': Issue #28637: No longer use re in

[issue28637] Python startup performance regression

Terry J. Reedy added the comment: The effect is limited to venv only in a narrow sense. The startup time for running programs from an IDLE editor is also affected. Many other programs also import re either directly or indirectly on startup. For instance, importing argparse imports re.

[issue28637] Python startup performance regression

Changes by Wolfgang Maier : -- nosy: +vinay.sajip ___ Python tracker ___

[issue28637] Python startup performance regression

Serhiy Storchaka added the comment: It seems to me that current regular expression is not correct. It can keep trailing whitespaces it a value. I'm not experienced with venv and don't know what tests are needed. -- ___ Python tracker

[issue28637] Python startup performance regression

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file45391/site-not-use-re.patch ___ Python tracker ___

[issue28637] Python startup performance regression

Wolfgang Maier added the comment: @serhiy.storchaka you've beaten me by a few minutes (still waiting for the test suite to finish). Your patch is "contaminated" by an additional change in collections/__init__.py though so I'm still uploading my version (which also tries to stick as close as

[issue28637] Python startup performance regression

Serhiy Storchaka added the comment: Oh, please ignore it. This is a sample code from other issue (issue28638). -- ___ Python tracker ___

[issue28637] Python startup performance regression

STINNER Victor added the comment: site-not-use-re.patch: The change on site.py LGTM. I don't see why re was used in the first place for such simple text format ;-) I suggest to push the fix into Python 3.6 and 3.7. But what is the change in Lib/collections/__init__.py? --

[issue28637] Python startup performance regression

Serhiy Storchaka added the comment: Proposed patch removes the use of re from site.py. -- keywords: +patch nosy: +serhiy.storchaka Added file: http://bugs.python.org/file45389/site-not-use-re.patch ___ Python tracker

[issue28637] Python startup performance regression

Wolfgang Maier added the comment: STINNER Victor added the comment: >BUT when Python is started from a virtual environment (created by the >"venv" module), the re module is important by default. > >haypo@speed-python$ venv/bin/python3 -c 'import sys; print("re" in >sys.modules)' >True

[issue28637] Python startup performance regression

Guido van Rossum added the comment: So what does that benchmark measure? For me, python 3.6 startup is 44 ms and python 2.7 startup is 78 ms (real time; user time is proportionally less). On Mon, Nov 7, 2016 at 5:40 PM, STINNER Victor wrote: > > STINNER Victor added

[issue28637] Python startup performance regression

STINNER Victor added the comment: python_startup: * 2.7: 7.74 ms +- 0.28 ms * 3.4: 18.7 ms +- 0.4 ms * 3.5: 20.3 ms +- 0.7 ms * 3.6: 26.9 ms +- 0.6 ms (rev c4319c0d0131, before the revert) * 3.7: 26.6 ms +- 0.5 ms (rev 36af3566b67a, before the revert) * 3.7: 24.6 ms +- 0.0 ms (rev cf7711887b4a,

[issue28637] Python startup performance regression

Guido van Rossum added the comment: Agreed. --Guido (mobile) -- ___ Python tracker ___ ___ Python-bugs-list

[issue28637] Python startup performance regression

STINNER Victor added the comment: Hum strange, the changeset d903a243c281 (the revert) optimized regex_compile: 480 ms (rev 573bc1f9900e) => 403 ms (rev cf7711887b4a). But it didn't restore python_startup performance: 26.ms (rev 573bc1f9900e) => 24.6 ms (rev cf7711887b4a). Another change

[issue28637] Python startup performance regression

Ned Deily added the comment: Please leave this as a "release blocker" until we have had a chance to explore all the options for 3.6.0b4. We may still very well decide that reverting is the least bad option but I don't want to rule out other options without some further investigation. We have

[issue28637] Python startup performance regression

STINNER Victor added the comment: Possible options: > 3. avoid the use of re in site.py venv That would be a tiny but nice enhancement. re is a commonly used module. I would prefer to not make its import time slower. Python 3 startup is already something like 3x slower than Python 2. Please

[issue28637] Python startup performance regression

STINNER Victor added the comment: I remove the release blocker flag since the change was reverted. -- priority: release blocker -> ___ Python tracker

[issue28637] Python startup performance regression

Ned Deily added the comment: With being this close to release and with two weeks to go until the final beta, it seems a bit premature to immediately revert this without exploring all the alternatives especially since it seems the effects are limited to venvs. In any case, we do need to have

[issue28637] Python startup performance regression

Roundup Robot added the comment: New changeset d903a243c281 by Victor Stinner in branch '3.6': Issue #28637: Revert issue #28082, don't import enum in re https://hg.python.org/cpython/rev/d903a243c281 -- nosy: +python-dev ___ Python tracker

[issue28637] Python startup performance regression

STINNER Victor added the comment: Oh, I understood why I had issues to reproduce the startup performance slowdown. When Python is run from the source code using ./python, the re module is not imported. When Python is installed, the re is not imported by default neither: haypo@speed-python$

[issue28637] Python startup performance regression

STINNER Victor added the comment: Ethan Furman added the comment: > Victor, can you add the commands you are using for timing so I can work on > making enum imports faster? Results of the python_startup benchmark: https://speed.python.org/timeline/#/?exe=4=python_startup=1=50=off=on=on This

[issue28637] Python startup performance regression

Ethan Furman added the comment: Ouch, that's a lot slower! Victor, can you add the commands you are using for timing so I can work on making enum imports faster? -- ___ Python tracker

[issue28637] Python startup performance regression

Guido van Rossum added the comment: Reverting is reasonable, though it may break some code that started using this in 3.6b1. Such is the life of beta testers. The benefit of using Enum for re constants doesn't seem worth such a big increase in startup time. -- nosy: +gvanrossum

[issue28637] Python startup performance regression

Changes by STINNER Victor : -- nosy: +ned.deily priority: normal -> release blocker ___ Python tracker ___

[issue28637] Python startup performance regression

New submission from STINNER Victor: The changeset 223731925d06 of the issue issue #28082 "use IntFlag for re constants" made Python startup 34% slower: - rev 5637c9b4dd4c: Median +- std dev: 19.5 ms +- 0.0 ms - rev 223731925d06: Median +- std dev: 26.2 ms +- 0.0 ms The change adds "import