[issue14285] Traceback wrong on ImportError while executing a package

[Nick Coghlan]

Nick Coghlan added the comment:

Leaving Python 2.7 alone makes sense to me - runpy is heavily dependent on the 
import system, so there are going to be limits to the improvements that can be 
made there.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

I committed a slightly modified version to 3.5+. I stopped wrapping the new 
ImportError, and let the existing find_spec() handler wrap it instead. That way 
the existing error messages stay the same.

However I cannot figure out an easy way to do a similar fix for 2.7, where 
ImportError.name does not exist. Therefore I propose to leave 2.7 as it is, 
with just my first commit. This means that 2.7 still omits the traceback if a 
parent package fails to initialize:

$ ./python -m package.submodule
[. . .]/python: No module named missing_module

It also means we see an unnecessary traceback with broken *.pyc files, although 
the eventual error message is improved. This case was reported in 
:

$ mkdir bad_pyc
$ : > bad_pyc/__init__.pyc  # Create empty file
$ python2 -m bad_pyc  # Original behaviour
/sbin/python2: Bad magic number in bad_pyc/__init__.pyc; 'bad_pyc' is a package 
and cannot be directly executed
$ ./python -m bad_pyc  # Current behaviour
Traceback (most recent call last):
  File "[. . .]/Lib/runpy.py", line 163, in _run_module_as_main
mod_name, _Error)
  File "[. . .]/Lib/runpy.py", line 111, in _get_module_details
__import__(mod_name)  # Do not catch exceptions initializing package
ImportError: Bad magic number in bad_pyc/__init__.pyc

I propose to leave these two cases as they currently are in 2.7, unless someone 
has any ideas how to improve them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Roundup Robot]

Roundup Robot added the comment:

New changeset 3202d143a194 by Martin Panter in branch '3.5':
Issue #14285: Do not catch exceptions initializing any ancestor package
https://hg.python.org/cpython/rev/3202d143a194

New changeset a526ebcfd31d by Martin Panter in branch 'default':
Issue #14285: Merge runpy fix from 3.5
https://hg.python.org/cpython/rev/a526ebcfd31d

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Nick Coghlan]

Nick Coghlan added the comment:

+1, latest iteration looks good to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

The test suite does check run_module() with some relative names; see 
test_invalid_names() in test_runpy. But there does not appear to be a test for 
“python -m .relative”.

Changes in init-ancestor.3.patch:

* Added a comment explaining the exception handling
* Added check, tests and documentation for relative module names

--
Added file: http://bugs.python.org/file41258/init-ancestor.3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Nick Coghlan]

Nick Coghlan added the comment:

Ah, I think I understand the problem you're getting at now:

* if the parent package is missing entirely, we still want to suppress the 
traceback by throwing the runpy specific exception
* if the parent package is present, but one of the modules *that* tries to 
import is missing (or otherwise doesn't load), we want to let the traceback 
happen normally

The exception handler in the new code snippet addresses that by checking the 
name attribute on the ImportError, but could likely use a comment explaining 
the subtlety.

The way you've handled checking the result of the rpartition call also 
highlights a deficiency in the runpy docs, and likely also the test suite as 
well: runpy.run_module and the -m switch only work with absolute module names, 
but this isn't stated explicitly in the documentation, nor is there any early 
check in _get_module_details() to bail out if given a relative module name.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

init-ancestor.2.patch implements the single __import__ with ImportError.name 
checking

--
Added file: http://bugs.python.org/file41254/init-ancestor.2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

Yes, the package is all we need to import. I understand this bug is all about 
importing the parent package, and not __main__. If you read the original 
report, it is __init__.py that is trying to import a missing module.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Nick Coghlan]

Nick Coghlan added the comment:

That import will only import the parent package (if there is one), not the 
module itself. Imports from __main__ will still happen during the exec call 
later on.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

I think the problem with doing a blind import of the parent package is it would 
be hard to differentiate between the ImportError when the package (or an 
ancestor) is missing, versus a user-generated ImportError. Maybe you could 
inspect the exception as a workaround (untested):

try:
__import__(pkg_name)
except ImportError as err:
if err.name != pkg_name and not pkg_name.startswith(err.name + "."):
raise
raise error(format(err))

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Nick Coghlan]

Nick Coghlan added the comment:

I'm wondering if there might be a simpler option: use rpartition() to strip off 
any trailing segment (whether that's "__main__" or not), and then always do a 
plain dynamic import of that package (if any). Something like the following at 
the start of _get_module_details():

pkg_name, is_submodule, submodule = mod_name.rpartition(".")
if is_submodule:
__import__(pkg_name)

The key is that we *don't* want to be relying on the fact find_spec() will 
import parent packages implicitly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

Even with my committed fix, tracebacks triggered by initializing a parent 
package are still suppressed:

$ ./python -m hello
Traceback (most recent call last):
  File "/media/disk/home/proj/python/cpython/Lib/runpy.py", line 158, in 
_run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/media/disk/home/proj/python/cpython/Lib/runpy.py", line 116, in 
_get_module_details
__import__(mod_name)  # Do not catch exceptions initializing package
  File "/media/disk/home/proj/python/cpython/hello/__init__.py", line 1, in 

import random; random.dsgjdgj
AttributeError: module 'random' has no attribute 'dsgjdgj'
[Exit 1]
$ ./python -m hello.submodule
/media/disk/home/proj/python/cpython/python: Error while finding spec for 
'hello.submodule' (: module 'random' has no attribute 
'dsgjdgj')
[Exit 1]

Fixing this in general might require a loop around the find_spec() call as in 
init-ancestor.patch. But I’m unsure if it is worth the extra complication.

--
Added file: http://bugs.python.org/file41231/init-ancestor.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Roundup Robot]

Roundup Robot added the comment:

New changeset c4e950338e79 by Martin Panter in branch '2.7':
Issue #14285: Do not catch ImportError from __init__.py in runpy
https://hg.python.org/cpython/rev/c4e950338e79

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Roundup Robot]

Roundup Robot added the comment:

New changeset 784a64a21fd0 by Martin Panter in branch '3.5':
Issue #14285: Do not catch __init__.py exceptions in runpy
https://hg.python.org/cpython/rev/784a64a21fd0

New changeset 01397c11ebb8 by Martin Panter in branch 'default':
Issue #14285: Merge runpy exception fix from 3.5
https://hg.python.org/cpython/rev/01397c11ebb8

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Nick Coghlan]

Nick Coghlan added the comment:

Right, while I agree this is a bug fix that makes sense to apply to 2.7 and 
3.5, it's also a large enough change to runpy's control flow logic that I'm 
wary of including it in the final 3.4 binary release.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

Thanks for the review Nick. You removed Python 3.4 from the versions; do you 
think it is not worth the risk committing in 3.4? I understand the deadline for 
the final release of 3.4 is the end of this week.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Nick Coghlan]

Nick Coghlan added the comment:

Martin, your patch looks good to me, and is at the very least an improvement 
over the status quo (which clearly traps exceptions that it shouldn't), so I'd 
say go ahead and apply it.

Thanks for digging into this and figuring out a clean solution.

--
versions:  -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Changes by Martin Panter :


Added file: http://bugs.python.org/file41205/internal-error.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

Now I have a deeper understanding I think this can be handled separately to 
Issue 16217.

This patch pulls together my previous two patches, and adds a fix. There are 
two aspects of my fix:

1. Import the package before calling find_spec() on the __main__ submodule. 
This means exceptions raised by the initialization code can be differentiated 
from exceptions from find_spec().

2. Change all the special ImportError exceptions raised inside runpy [and also 
one raised by InspectLoader.get_code()] to an internal _Error exception known 
only to runpy. Now I can be sure that all _Error exceptions are not caused by 
the initialization code, and I can stop catching ImportError, but still catch 
_Error and suppress the traceback. When runpy is invoked from the documented 
run_module() or run_path() APIs, _Error is not used, and it still raises 
ImportError to maintain backwards compatibility.

I think my patch should avoid the main problem in Issue 19771 as well.

Please review my patch. There are so many error possibilities; it is hard to be 
sure I have got them all right.

--
components: +Library (Lib)
stage:  -> patch review
versions: +Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

I suspect a proper solution of this bug would also help with Issue 16217 
(provide only the relevant traceback for exceptions from user code).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

Posting finding-spec.patch which just has another test I wrote. It tests if 
AttributeError/ValueError/TypeError gets wrapped in the “finding spec” 
ImportError, though I’m not sure if this is a bug or a feature, hence I kept it 
separate. And again I’m not sure of a good way to fix it either.

--
Added file: http://bugs.python.org/file38439/finding-spec.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

The patches at Issue 19771 should remove the part of the message that 
incorrectly says “. . . is a package and cannot be directly executed”. However 
that still leaves the problem of the suppressed traceback.

I am posting runpy-traceback.patch here which adds some tests to check if the 
traceback is suppressed. The offending test is marked @expectedFailure. It also 
points out where the exception is being incorrectly caught, but I haven’t 
thought of a nice way to fix the problem.

Other changes in the runpy-traceback.patch:

* Removed the exception message rewriting in _run_module_as_main(), because it 
seems to be redundant with the _get_main_module_details() function
* Fixed test_cmd_line_script.CmdLineTest.test_dash_m_error_code_is_one(), which 
was only checking the Python exit status, and not verifying that the specific 
failure was the one anticipated

--
keywords: +patch
Added file: http://bugs.python.org/file38438/runpy-traceback.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

Closely related: Issue 19771, which seems to be complaining about a the error 
message when __main__.py generates an ImportError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

The relevant code is in the _get_module_details() function at Lib/runpy.py:101. 
There are a few of things going on:

1. The code is calling importlib.util.find_spec(".__main__"), and 
handles various kinds of exceptions by wrapping them in an ImportError, adding 
the “Error while finding spec” message. The call apparently causes the package 
to be imported, so some exceptions triggered by running __init__.py are 
wrapped. It would be nice if exceptions raised by running __init__.py were not 
caught, but I have no idea if this is practical or how to do it. It seems the 
exception handler is there to raise ImportError if you do something like 
“python -m os.path.something”, which seems rather unlikely to me.

2. The logic for handling the __main__ module in a package seems to assume that 
any ImportError (such as raised from step 1) is due to the __main__ module not 
being present. Then it wraps that exception in another ImportError, adding the 
“. . . cannot be directly executed” bit. Again, it would be good to separate 
the __init__.py running from this exception handling.

3. Finally in _run_module_as_main(), the ImportError is caught and printed, 
without any traceback. It would be good to only do this for internal errors 
loading the module, and not if ImportError is raised when the module (or 
__init__.py) is run.

I’m not sure what the best way to fix this is. Perhaps add an internal 
ImportError subclass (or subclasses) that is only used for the “No module named 
. . .” errors, or only used for errors appropriate for final handler in step 3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Rob Agar]

Rob Agar added the comment:

The message also needs to include the file and line number of the ImportError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Rob Agar]

Changes by Rob Agar :


--
nosy: +robagar

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Berker Peksag]

Changes by Berker Peksag :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Martin Panter]

Martin Panter added the comment:

A file called “package/__main__.py” is executed as a script by “python -m 
package”. See .

I’ve came across this issue myself. You don’t even need the __main__.py file to 
be doing anything special, as long as the __init__.py raises an ImportError, I 
think. On Python 3.4 the report is even more convoluted:

/sbin/python3: Error while finding spec for 'package.__main__' (: No module named 'missing_module'); 'package' is a package and 
cannot be directly executed

I dunno what “finding spec” means, and packages _can_ be directly executed if 
they have a __main__ module, so at least the last bit is definitely wrong.

--
nosy: +vadmium
versions: +Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Mark Lawrence]

Mark Lawrence added the comment:

I've no idea what having a file called __main__.py is likely to do so can 
someone comment please.

--
nosy: +BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Brett Cannon]

Changes by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Éric Araujo]

Changes by Éric Araujo :


--
nosy: +brett.cannon, eric.araujo, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14285] Traceback wrong on ImportError while executing a package

[Marc Schlaich]

New submission from Marc Schlaich :

It is very simple to reproduce this error. 
There is an executable package:

package/
__init__.py
__main__.py

The __init__ imports a missing module:

import missing_module

And the __main__ imports from it:

from . import missing_module

Now I get the following output which is not what I am expecting:

C:\Python27\python.exe: No module named missing_module; 'package' is 
a package and cannot be directly executed

--
messages: 155574
nosy: ms4py
priority: normal
severity: normal
status: open
title: Traceback wrong on ImportError while executing a package
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com