[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: I've added a second answer to the referenced Stack Overflow issue that attempts to more clearly explain what is going on: https://stackoverflow.com/questions/43393764/python-3-6-project-structure-leads-to-runtimewarning/45070583#45070583 (The problem there is

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Luke added the comment: I recently started getting this warning message (see bottom) that seems to be due to the changes from this issue. I'm running a submodule as main using the `-m` flag, but I'm not doing any modification to `sys.modules`, or even `sys.path`... I've taken a look at [The

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: Thanks Martin! Closing this as fixed, since the new warning should help nudge folks towards staying within the supported subset of the import system behaviour when it comes to interactions between package.__init__ and package.__main__. -- resolution:

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Roundup Robot added the comment: New changeset 43ae044eaccc by Martin Panter in branch '3.5': Issue #27487: Warn if submodule already imported before runpy execution https://hg.python.org/cpython/rev/43ae044eaccc New changeset fccd733aa78b by Martin Panter in branch 'default': Issue #27487:

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Martin Panter added the comment: Here is an updated patch. Since the message no longer applies to top-level modules, I went back to a version closer to Wolfgang’s suggestion, which should eliminate the problem with “any parent packages”. Now the messages look like this:

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: Running pre-imported top level packages like "runpy" or "site" with "-m" is supported behaviour, so that shouldn't emit a warning. ("python -m site" in particular is a debugging tool used to print out the default sys.path configuration) Otherwise, the warning

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Martin Panter added the comment: Here is a patch with the proposed warning. I think “Error finding module specification” might be a bit better than the current “finding spec”, so I included that change. With the patch, this is what the messages look like: $ ./python -m package.module

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: No changes to 2.7 - the import system logic there is fairly different, since it's still using the old pre-importlib implementation. For the "package.__main__" case, you're right that we don't want to emit the warning for "-m package", but we *do* want to emit

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Martin Panter added the comment: I can try to add the RuntimeWarning soon. Obviously we want the warning when executing a submodule like “package.module”. It also seems reasonable to warn when executing a top-level module that has already been imported. I presume we want these to go into the

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Brett Cannon added the comment: +1 from me -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: +1 for the more verbose version, since that will tend to nudge folks towards .__init__ as the likely culprit (and that does seem to be the most likely way for this particular double-import scenario to arise. Brett, does that resolution sound reasonable to you,

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Wolfgang Maier added the comment: A warning like this sounds good to me though I'd prefer it to be slightly more verbose, like: "RuntimeWarning: '' found in sys.modules after import of package '', but prior to execution of '' as '__main__'; this may result in unpredictable behaviour" or it

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: Perhaps a suitable outcome here would be to just add an unconditional RuntimeWarning when the target of "-m" is already in sys.modules after the parent package import, along the lines of: "RuntimeWarning: '' already in sys.modules prior to '__main__'

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: In http://bugs.python.org/issue27487#msg270300, Wolfgang noted: "In fact, for my own case I have now refactored things a bit so I can avoid the import from __init__.py." So I take that to mean that Wolfgang's particular project is now compatible with the new

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Changes by Ned Deily : -- priority: normal -> release blocker ___ Python tracker ___ ___

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Brett Cannon added the comment: So what I'm gathering is that Martin fixed a bug that accidentally introduced a new one in 3.5.2 (which Martin should not feel responsible for; this import stuff is all very subtle and people use every nuance of it). To me, the question is whether there is a

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: Sorry, I got the levels slightly confused there - a bit more experimentation shows it isn't the parent package that needs a __spec__ attribute, but only the package being executed. # Parent package needs __path__ rather than __spec__ >>>

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: Aye, the report is definitely appreciated - the interaction between "__main__" execution and normal module imports has always been tricky, so it's always good to be informed of new problems folks encounter. As an example illustrating the underlying problem here

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Wolfgang Maier added the comment: @Martin and regarding Python3.3: Right, you cannot do the replacement when running the module as __main__. For my use case that's not required though so in the module I can just do: if __name__ == '__main__': print('running module as a script') else:

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: Oops, that PEP reference was meant to be PEP 451 (which added the __spec__ attribute and the concept of module specs as something distinct from the modules themselves) -- ___ Python tracker

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: Breaking down a few of the moving parts here: * Yes, modules replacing themselves in sys.modules as a side effect of import is supported behaviour (it's why the import system looks them up by name in sys.modules as the final step, rather than just returning

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Martin Panter added the comment: In trying to understand this, I built a package with two simple files: $ cat package/*.py # package/__init__.py: from . import module # package/module.py: import sys sys.modules[__name__] = object() With this I can reproduce your __spec__ error, and I see it

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

Nick Coghlan added the comment: Since this worked in 3.5.1, and fails in 3.5.2, I think it's reasonable to consider if it makes sense to find a way to make it work again in 3.5.3 (and then decide separately whether or not we want to preserve the capability in 3.6.0). Specifically, restoring

[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)

New submission from Wolfgang Maier: As a result of Issue14285 Python 3.5.2 now imports packages in runpy. _get_module_details before calling importlib.util.find_spec. Although I'm not sure how important this is, I wanted to report that this new behaviour can have a side-effect under pretty