[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: Thanks all! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: New changeset 81108375d9b2ccd0add1572da745311d4dfac505 by Serhiy Storchaka in branch 'master': bpo-30152: Reduce the number of imports for argparse. (#1269) https://github.com/python/cpython/commit/81108375d9b2ccd0add1572da745311d4dfac505 --

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: functools.wraps() is used in more complex programs. The purpose of this PR is speeding up small scripts that just start, parse arguments, do its fast work (so fast as outputting a help or a version) and exit. Even collections is not used in a half of my

[issue30152] Reduce the number of imports for argparse

INADA Naoki added the comment: Oh, the pull request is far larger than I thought! I doubt that avoiding functools and collections is worth enough, because it is very common. For example: * functools is very commonly imported, especially for wraps(). * collections is imported by functools,

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: Got rid of importing errno as Victor suggested. In gettext its import is deferred. It is only used in one exceptional case. In os errno no longer used. Initially errno was imported in os inside a function. The import was moved at module level for fixing

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: On my new laptop 100 runs are sped up from 2.357 sec to 2.094 sec (by 13%). Got rid of importing 6 modules: _struct, collections.abc, copy, struct, textwrap, warnings. Getting rid of importing errno leads to rewriting of os._execvpe() which looks less

[issue30152] Reduce the number of imports for argparse

STINNER Victor added the comment: I reviewed your PR, see my questions. Would you mind to run a new benchmark again, please? -- ___ Python tracker ___

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: I have removed changes for which it was requested and addressed other comments. Can this PR be merged now? -- ___ Python tracker

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: Implemented Wolfgang Maier's idea. The copy module is now imported in argparse only when the default value for 'append' or 'append_const' is not a list (uncommon case). -- ___ Python tracker

[issue30152] Reduce the number of imports for argparse

Terry J. Reedy added the comment: I believe importing argparse in the __main__ clause (or a main or test function called therein) is common. I prefer it there as it is not germain to the code that is usually imported. I have a similar view on not immediately importing warnings when only

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: > Measuring Python startup performance is painful, there is a huge deviation. That is why I run Python 100 times and repeat that several times for testing that the result is stable. With perf I got roughly the same result -- the absolute difference is 18-19

[issue30152] Reduce the number of imports for argparse

STINNER Victor added the comment: > $ time for i in `seq 100`; do ./python -S -c 'import argparse; > argparse.ArgumentParser()'; done Measuring Python startup performance is painful, there is a huge deviation. You may try the new "command" command that I added to perf 1.1:

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: After reverting some changes (import heapq on Raymond's request and import gettext since it is needed for the ArgumentParser constructor) the effect of the patch is reducing the number of imports by 6 and the time by 0.017 sec (> 10%) on my computer. $

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: I just realized that is is not so easy to avoid gettext import. gettext is used in the ArgumentParser constructor for localizing names of default groups and the default help (besides they are used only for formatting help). This adds importing the locale

[issue30152] Reduce the number of imports for argparse

Antoine Pitrou added the comment: I'd like to vote for a lazy import system that would benefit everyone (many third-party packages are also affected by startup time issues), but I've seen enough handwaving about it along the years that I'm not really hoping any soon. My own limited attempts

[issue30152] Reduce the number of imports for argparse

Christian Heimes added the comment: Instead of messing with all modules, we should rather try to improve startup time with lazy imports first. Mercurial added an on-demand importer to speed up CLI performance. Many years ago I worked on PEP 369 for lazy import. It should be much easier to

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: In general I agree with Raymond's principle and I am not very happy doing these changes. But heavy startup is a sore spot of CPython. For comparison Python 2.7 and optparse: $ python -c 'import sys; s = set(sys.modules); import argparse; print(len(s),

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: collections is very popular module. It is almost impossible to avoid importing it since so much stdlib modules use namedtuple, OrderedDict, deque, or defaultdict. The reason of moving a heapq import inside a Counter method is that importing heapq adds two

[issue30152] Reduce the number of imports for argparse

STINNER Victor added the comment: Raymond Hettinger added the comment: > Some parts of the patch are harmless but other parts needlessly constipates > the code for very little benefit. In general, Python startup is currently the most severe performance regression in Python 3 compared to Python

[issue30152] Reduce the number of imports for argparse

Serhiy Storchaka added the comment: > I'd move this into its own PR. Done. Issue30166. -- ___ Python tracker ___

[issue30152] Reduce the number of imports for argparse

Wolfgang Maier added the comment: @rhettinger: I do not quite understand this harsh reaction. Making argparse more responsive could, in fact, be quite a nice improvement. This is particularly true, I guess, if you want argument completion with a package like

[issue30152] Reduce the number of imports for argparse

Raymond Hettinger added the comment: Please stop rearranging everything in the standard library. Every day I look at the tracker and there is a new patch from this dev that alters dozens of files. Some parts of the patch are harmless but other parts needlessly constipates the code for

[issue30152] Reduce the number of imports for argparse

Changes by paul j3 : -- nosy: +paul.j3 ___ Python tracker ___ ___ Python-bugs-list

[issue30152] Reduce the number of imports for argparse

Changes by Wolfgang Maier : -- nosy: +wolma ___ Python tracker ___

[issue30152] Reduce the number of imports for argparse

Berker Peksag added the comment: > The patch also makes argparse itself be imported only when the module > is used as a script, not just imported. +1. I'd move this into its own PR. -- nosy: +berker.peksag ___ Python tracker

[issue30152] Reduce the number of imports for argparse

Changes by STINNER Victor : -- nosy: +Chi Hsuan Yen ___ Python tracker ___ ___

[issue30152] Reduce the number of imports for argparse

Changes by Serhiy Storchaka : -- pull_requests: +1381 ___ Python tracker ___ ___

[issue30152] Reduce the number of imports for argparse

New submission from Serhiy Storchaka: Since argparse becomes more used as standard way of parsing command-line arguments, the number of imports involved when import argparse becomes more important. Proposed patch reduces that number by 10 modules. Unpatched: $ ./python -c 'import sys; s =