[issue31415] Add -X option to show import time

miss-islington added the comment: New changeset 73943ce7d31595d152dc01bf0008b37c802c0d3b by Miss Islington (bot) in branch '3.9': bpo-31415: importtime was made by Inada Naoki (GH-31875) https://github.com/python/cpython/commit/73943ce7d31595d152dc01bf0008b37c802c0d3b --

[issue31415] Add -X option to show import time

miss-islington added the comment: New changeset 9f1587e8d3f18729a0982774c6c4409472cbd9e3 by Miss Islington (bot) in branch '3.10': bpo-31415: importtime was made by Inada Naoki (GH-31875) https://github.com/python/cpython/commit/9f1587e8d3f18729a0982774c6c4409472cbd9e3 --

[issue31415] Add -X option to show import time

Change by miss-islington : -- pull_requests: +29982 pull_request: https://github.com/python/cpython/pull/31884 ___ Python tracker ___

[issue31415] Add -X option to show import time

Change by miss-islington : -- nosy: +miss-islington nosy_count: 12.0 -> 13.0 pull_requests: +29981 pull_request: https://github.com/python/cpython/pull/31883 ___ Python tracker

[issue31415] Add -X option to show import time

Guido van Rossum added the comment: New changeset 29624e769c5c3c1e59c6acc8b69383ead53e8a9f by Victor Stinner in branch 'main': bpo-31415: importtime was made by Inada Naoki (GH-31875) https://github.com/python/cpython/commit/29624e769c5c3c1e59c6acc8b69383ead53e8a9f --

[issue31415] Add -X option to show import time

Change by STINNER Victor : -- pull_requests: +29973 pull_request: https://github.com/python/cpython/pull/31875 ___ Python tracker ___

[issue31415] Add -X option to show import time

Nico Schlömer added the comment: I just updated tuna [1] to support import time profiles as well. Install with ``` pip install tuna ``` and use with ``` python -X importprofile yourfile.py 2> import.log tuna import.log ``` See screenshot for example output. Cheers, Nico [1]

[issue31415] Add -X option to show import time

STINNER Victor added the comment: FYI I'm working on the Py_Main() function in bpo-32030. My latest PR (PR 4412) moves the code to parse -X importtime and PYTHONPROFILEIMPORTTIME into Py_Main(), to group all functions parsing the command line arguments and

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: I think there is nothing left to do for this issue, so I'm closing it. -- status: open -> closed ___ Python tracker

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: New changeset 088929cf62fa22c06f6a44e25915abce9048a545 by Serhiy Storchaka in branch 'master': bpo-31415: Improve error handling and caching of the importtime option. (#4138)

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: On Nov 4, 2017, at 14:29, Terry J. Reedy wrote: > The importtime output is sent to stderr. Should this be documented? Yes. :) -- ___ Python tracker

[issue31415] Add -X option to show import time

Terry J. Reedy added the comment: > with environment variable you get more information. For timing package import time, such as for idlelib.pyshell, rather than bare python startup time, the extra information is just a bit more noise to ignore. The importtime output is sent

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: Okay, given the non-propagation and Windows issues. I'm -1 on removing -X, +1 on fixing the negative cache. -- ___ Python tracker

[issue31415] Add -X option to show import time

Antoine Pitrou added the comment: (and Terry is right that environment variables can be significantly more annoying on Windows) -- ___ Python tracker

[issue31415] Add -X option to show import time

Antoine Pitrou added the comment: One benefit (or drawback, depending on how you see it) of the -X option is that it wouldn't propagate to child processes spawned by multiprocessing. See _args_from_interpreter_flags() in Lib/subprocess.py. More generally, the idea that we

[issue31415] Add -X option to show import time

INADA Naoki added the comment: I'm +1 to remove -X option. Supporting both form makes code ugly and benefit is too small. And +1 to negative cache too. It's simple for environment varianle. -- ___ Python tracker

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: On Nov 3, 2017, at 14:23, Terry J. Reedy wrote: > > Does it set the EV for the entire session (which one likely would not want), > or just the one command (which has no Windows equivalent that I know of)? > Please

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: On Nov 3, 2017, at 14:41, Serhiy Storchaka wrote: > > Note that with environment variable you get more information. Fun! -- ___ Python tracker

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: On Windows you can create a 2-line bat-file that sets the environment variable and runs Python. -- ___ Python tracker

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: Note that with environment variable you get more information. $ ./python -X importtime -c pass import time: self [us] | cumulative | imported package import time:88 | 88 | _codecs import time: 789 |

[issue31415] Add -X option to show import time

Terry J. Reedy added the comment: > `PYTHONPROFILEIMPORTTIME=x python` isn’t *too* onerous It does not work on Windows. C:\Users\Terry>PYTHONPROFILEIMPORTTIME=x python 'PYTHONPROFILEIMPORTTIME' is not recognized as an internal or external command, operable program or batch

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: On Nov 3, 2017, at 11:05, Serhiy Storchaka wrote: > > But now, after adding the environment variable, do we still need the -X > option? From a user side I don't see much difference between specifying an > option

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: > I didn't think it's worth enough because import will be much slower than one > dict lookup. I agree. The main value of my patch is handling possible (but very unlike) errors. Implementing negative value caching adds not many

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: On Nov 2, 2017, at 22:50, INADA Naoki wrote: > > When adding environment variable option, it should be documented in > man page and `python -h`. I thought about that, but the problem is that none of the -X options

[issue31415] Add -X option to show import time

INADA Naoki added the comment: When adding environment variable option, it should be documented in man page and `python -h`. -- ___ Python tracker

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: New changeset 700d2e4755921d6c339ff20dacecde1aea64de34 by Barry Warsaw in branch 'master': bpo-31415: Support PYTHONPROFILEIMPORTTIME envvar equivalent to -X importtime (#4240)

[issue31415] Add -X option to show import time

Barry A. Warsaw added the comment: I like this a lot and while the issue is still open, I'm piggybacking my PR 4240 on this one. I have a use case for enabling this feature via environment variable, so PR 4240 adds PYTHONPROFILEIMPORTTIME which also enables this feature.

[issue31415] Add -X option to show import time

Change by Barry A. Warsaw : -- pull_requests: +4205 stage: resolved -> patch review ___ Python tracker ___

[issue31415] Add -X option to show import time

INADA Naoki added the comment: Xoptions is not environment variable. Some modules are imported before xoptions are parsed. So negative cache for xoptions is little more complex than negative cache for environment variable. --

[issue31415] Add -X option to show import time

STINNER Victor added the comment: > I didn't think it's worth enough because import will be much slower than one > dict lookup. I don't care much of performance. For the consistency with other environment variables, I like to only read the environment variable once

[issue31415] Add -X option to show import time

INADA Naoki added the comment: Does it worth enough? I didn't think it's worth enough because import will be much slower than one dict lookup. -- ___ Python tracker

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: The importtime option is cached only if it is set. If it is not set (the common case), it is not cached. PR 4138 makes it be cached in the common case and improves errors handling. -- status: closed -> open

[issue31415] Add -X option to show import time

Change by Serhiy Storchaka : -- pull_requests: +4101 ___ Python tracker ___ ___

[issue31415] Add -X option to show import time

STINNER Victor added the comment: Ok, the feature was implemented, the Windows clock resolution was fixed as well. It's time to close this issue. If you still have concerns about import time, see open discussions on the python-dev mailing list, or open new issues

[issue31415] Add -X option to show import time

STINNER Victor added the comment: > Why global _PyTime_GetWinPerfCounterWithInfo() is needed at all? It seems to > me that _PyTime_GetPerfCounterWithInfo() can be used instead. Smaller API is > better. I chose to expose _PyTime_GetWinPerfCounterWithInfo() to make my

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: Why global _PyTime_GetWinPerfCounterWithInfo() is needed at all? It seems to me that _PyTime_GetPerfCounterWithInfo() can be used instead. Smaller API is better. _PyTime_GetPerfCounter() is a simple wrapper around

[issue31415] Add -X option to show import time

Change by STINNER Victor : -- pull_requests: +3930 ___ Python tracker ___ ___

[issue31415] Add -X option to show import time

Change by STINNER Victor : -- pull_requests: +3914 ___ Python tracker ___ ___

[issue31415] Add -X option to show import time

STINNER Victor added the comment: New changeset a997c7b434631f51e00191acea2ba6097691e859 by Victor Stinner in branch 'master': bpo-31415: Add _PyTime_GetPerfCounter() and use it for -X importtime (#3936)

[issue31415] Add -X option to show import time

Change by STINNER Victor : -- pull_requests: +3910 stage: resolved -> patch review ___ Python tracker ___

[issue31415] Add -X option to show import time

STINNER Victor added the comment: > Could we change PyTime_GetMonotonicClock implementation to use it on Windows? No. See the PEP 418 for the rationale. But we could add a _PyTime_GetPerfCounter() to Python/pytime.c. --

[issue31415] Add -X option to show import time

INADA Naoki added the comment: https://github.com/python/cpython/blob/27c623c845dd6e4b8e1782666ca3a956636da266/Modules/timemodule.c#L80-L112 time module has perf_counter implementation based on QueryPerformanceCounter(). Could we change PyTime_GetMonotonicClock

[issue31415] Add -X option to show import time

Terry J. Reedy added the comment: This is pretty useless on Windows because you are using a clock with 15-16 milli (not micro) second resolution on Windows. Before Victor introduced time.perf_counter, with decent behavior on all systems, timeit used different time module

[issue31415] Add -X option to show import time

STINNER Victor added the comment: Thank you Naoki for this nice enhancement! Tooling always help to take smart decisions on optimizations. -- ___ Python tracker

[issue31415] Add -X option to show import time

INADA Naoki added the comment: New changeset 1a87de7fcfa3c19f08e29047337c350b4a32b259 by INADA Naoki in branch 'master': bpo-31415: Add `-X importtime` option (GH-3490) https://github.com/python/cpython/commit/1a87de7fcfa3c19f08e29047337c350b4a32b259 --

[issue31415] Add -X option to show import time

Change by INADA Naoki : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue31415] Add -X option to show import time

Łukasz Langa added the comment: Related: https://bugs.python.org/issue31574 -- nosy: +lukasz.langa ___ Python tracker ___

[issue31415] Add -X option to show import time

STINNER Victor added the comment: > The output in PR 3765 partially duplicates the output of the -v option. Right. But the idea is to be able to use "grep 'import time:'" to only extract importtime logs. "-v" logs different kind of informations. -- nosy: +haypo

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: The output in PR 3765 partially duplicates the output of the -v option. -- ___ Python tracker ___

[issue31415] Add -X option to show import time

Changes by STINNER Victor : -- pull_requests: +3750 ___ Python tracker ___ ___

[issue31415] Add -X option to show import time

INADA Naoki added the comment: how can I make it more machine readable? On 2017年9月25日(月) 19:39 Christian Heimes wrote: > > Christian Heimes added the comment: > > +1 for an option to profile import time. > > I have a minor complain: The output format is not

[issue31415] Add -X option to show import time

Christian Heimes added the comment: +1 for an option to profile import time. I have a minor complain: The output format is not machine-friendly, which makes it harder to perform automatic analysis. I created #31574 / PR3749 to add dtrace probes. -- nosy: +christian.heimes

[issue31415] Add -X option to show import time

INADA Naoki added the comment: I don't have any good idea for it. I usually run `python -X importtime -c 'import functools'` several times to ensure pyc files are created and it is cached by OS. On Linux, `strace -c` and `time` command can be used. But I don't know cross platform way to do

[issue31415] Add -X option to show import time

Raymond Hettinger added the comment: Naoki, is it possible to separate out how much of this is I/O vs CPU time? If the I/O dominates, optimizations tend toward zipimports, faster drives, disk caching etc. If the CPU time is dominant, different techniques are used (lazy evaluation,

[issue31415] Add -X option to show import time

Guido van Rossum added the comment: @rhettinger > Guido, is this something you want? I think it is useful given how much debate there has been around startup time in various contexts (not just pure interpreter startup time but also startup time when using specific libraries, packages or

[issue31415] Add -X option to show import time

Raymond Hettinger added the comment: Guido, is this something you want? Historically, we've shown a lot of restraint when it comes to adding command-line options. Also, I'm not sure we want to induce people to start moving their imports inside function calls. Just because we sometimes

[issue31415] Add -X option to show import time

INADA Naoki added the comment: > Actually it is easy. You need just one global integer accumulator and a local > variable for keeping a temporary copy of it. You're right! I updated my pull request. See current output: https://gist.github.com/methane/185d75a3c8da762d85317dd95918a623

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: > Stack management is difficult than inc/dec integer in multi threading case. Actually it is easy. You need just one global integer accumulator and a local variable for keeping a temporary copy of it. -- ___

[issue31415] Add -X option to show import time

INADA Naoki added the comment: > If this proposition be accepted I would want to see not only cumulated times, > but also pure times, without time of nested imports. It can be calculated by script. I don't want to add more complexity to import.c. > I guess this feature doesn't work correctly

[issue31415] Add -X option to show import time

Brett Cannon added the comment: Ah, OK. Then I wouldn't put it in square brackets as it seems to suggest it's somehow disconnected from the number. -- ___ Python tracker

[issue31415] Add -X option to show import time

Serhiy Storchaka added the comment: If this proposition be accepted I would want to see not only cumulated times, but also pure times, without time of nested imports. I guess this feature doesn't work correctly with threading. -- nosy: +serhiy.storchaka

[issue31415] Add -X option to show import time

INADA Naoki added the comment: It means μs (micro seconds), in ASCII. On 2017年9月20日(水) 3:35 Brett Cannon wrote: > > Brett Cannon added the comment: > > What does the "[us]" mean? > > -- > nosy: +brett.cannon > > ___ > Python

[issue31415] Add -X option to show import time

Brett Cannon added the comment: What does the "[us]" mean? -- nosy: +brett.cannon ___ Python tracker ___ ___

[issue31415] Add -X option to show import time

Changes by INADA Naoki : -- keywords: +patch pull_requests: +3483 stage: -> patch review ___ Python tracker ___

[issue31415] Add -X option to show import time

New submission from INADA Naoki: I used my local patch to profile import time. I think it's useful for 3rd party authors. For example: ./python -Ximportprofile -c 'import traceback' - _codecs 71 [us] - codecs 677 [us] - encodings.aliases 412 [us] - encodings 1688 [us] - encodings.utf_8