[issue18307] Relative path in co_filename for zipped modules

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

zipimport has been rewritten in pure Python (issue25711).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Brett Cannon]

Brett Cannon  added the comment:

So it sounds like that maybe we need to decide if we are going to replace 
zipimport with Serhiy's Python version. And if we do decide to go with the 
Python code then it should get updated to use _imp._fix_co_filename().

Regardless of the decision, obviously thanks to Vitaly for sparking this change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Nick Coghlan]

Nick Coghlan  added the comment:

Serhiy's analysis sounds right to me - for precompiled bytecode files, we 
really want co_filename to reflect the import time filename *not* the compile 
time filename.

While zipimport is one way to get the compile time and import time paths to 
differ, there are others:

- move an existing directory to another location
- copy a directory tree to a different machine
- move an implicitly cached pyc file to another location

Concrete example from a recent'ish build where I convert an implicitly cached 
pyc to a standalone one, but co_filename still references the original source 
file:

===
$ ./python -c "import contextlib; print(contextlib.__file__); 
print(contextlib.contextmanager.__code__.co_filename)"
/home/ncoghlan/devel/cpython/Lib/contextlib.py
/home/ncoghlan/devel/cpython/Lib/contextlib.py
$ mkdir -p /tmp/example
$ cd /tmp/example
$ cp /home/ncoghlan/devel/cpython/Lib/__pycache__/contextlib.cpython-38.pyc 
contextlib.pyc
$ /home/ncoghlan/devel/cpython/python -c "import contextlib; 
print(contextlib.__file__); 
print(contextlib.contextmanager.__code__.co_filename)"
/tmp/example/contextlib.pyc
/home/ncoghlan/devel/cpython/Lib/contextlib.py
===

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The patch for the Python implementation of zipimport (see issue25711) is much 
simpler (see the attached sample patch). But it requires adding a way of 
modifying co_filename. Currently code objects are immutable.

This issue looks as a part of the larger problem. Zipimport is not the only 
source of precompiled bytecode which needs updating co_filename. For example 
the tree of py and pyc files can be moved to other place. Also, since 
co_filename contains system depended path, it doesn't make sense when load pyc 
files on other system (for example if they were created on Linux and ran on 
Windows). On other side, the import machinery tries to load pyc and py files, 
therefore it should know the path of corresponding py file when load a pyc 
file. Maybe it be better to not save co_filename in a pyc file (note that all 
code object in a file have the same co_filename, but it is saved for every code 
object), but set co_filename after unmarshalling the module bytecode by the 
import machinery in all loaders.

--
Added file: https://bugs.python.org/file47775/pyzipimport-code-filename.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Brett Cannon]

Brett Cannon  added the comment:

This will be a semantic change to the value of co_filename so I don't think 
this can safely be backported.

--
versions:  -Python 2.7, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Brett Cannon]

Change by Brett Cannon :


--
nosy: +twouters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Vitaly Murashev]

Vitaly Murashev  added the comment:

> Only a PR for master is needed.
Serhiy Storchaka, thanks for advice,
I cancelled unnecessary PRs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Only a PR for master is needed. Changes to other branches will be ported after 
reviewing and merging the PR for master.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Vitaly Murashev]

Vitaly Murashev  added the comment:

Pull-requests for 2.7, 3.7 and master submitted on github,
all tests look passed, so

Python dev-team,
please, take a look.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Vitaly Murashev]

Change by Vitaly Murashev :


--
versions: +Python 2.7, Python 3.7, Python 3.8 -Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Vitaly Murashev]

Vitaly Murashev  added the comment:

> Vitaly, in the future please use gender-neutral words
Mariatta, OK, got it, I am sorry for that. I am not a native speaker.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +6774

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +6773

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +6772
stage: test needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:

Vitaly, in the future please use gender-neutral words such as "folks" and 
"y'all" instead of "guys". Thanks.

--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

In year 2018 you have to create a pull request on GitHub. See 
https://devguide.python.org/.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Vitaly Murashev]

Vitaly Murashev  added the comment:

Guys, a couple questions ...
I want to suggest new patches for python3.7 and python2.7 with regression tests 
included
What is proper way to do it now, in year 2018 ?
May I do it on github.com ? Should I submit new issue for that there ?
Or am I still supposed to attach new patches here - on bugs.python.org ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Nick Coghlan]

Changes by Nick Coghlan :


--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Vitaly Murashev]

Vitaly Murashev added the comment:

patch suggested (over 3.3.0 code base).
Without patch test fails, with patch - passed

--
components:  -Library (Lib)
keywords: +patch
Added file: http://bugs.python.org/file31781/zipimport.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Vitaly Murashev]

Vitaly Murashev added the comment:

unit-test attached.
There are 3 tests inside it: 2 failed, 1 succeeded.

According to this test results it becomes clear that specified issue is 
reproduced only with zip modules which contain precompiled bytecode inside 
(pyc-files)

--
Added file: http://bugs.python.org/file31451/test_zipimport_co_filename.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Christian Heimes]

Changes by Christian Heimes :


--
nosy: +brett.cannon, christian.heimes
stage:  -> test needed
versions: +Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18307] Relative path in co_filename for zipped modules

[Vitaly Murashev]

New submission from Vitaly Murashev:

Recently I found out that it not possible to debug python code if it is a part 
of zip-module.
Python version being used is 3.3.0

Well known GUI debuggers like Eclipse+PyDev or PyCharm are unable to start 
debugging and give the following warning:
---
pydev debugger: CRITICAL WARNING: This version of python seems to be 
incorrectly compiled (internal generated filenames are not absolute)
pydev debugger: The debugger may still function, but it will work slower and 
may miss breakpoints.
---
So I started my own investigation of this issue and results are the following.
At first I took traditional python debugger 'pdb' to analyze how it behaves 
during debug of zipped module.
'pdb' showed me some backtaces and filename part for stack entries looks 
malformed.
I expected something like 
'full-path-to-zip-dir/my_zipped_module.zip/subdir/test_module.py'
but realy it looks like 'full-path-to-current-dir/subdir/test_module.py'

Source code in pdb.py and bdb.py (which one are a part of python stdlib) gave 
me the answer why it happens.

The root cause are inside Bdb.format_stack_entry() + Bdb.canonic()

Please take a look at the following line inside 'format_stack_entry' method:
filename = self.canonic(frame.f_code.co_filename)

For zipped module variable 'frame.f_code.co_filename' holds _relative_ file 
path started from the root of zip archive like 'subdir/test_module.py'
And as relult Bdb.canonic() method gives what we have - 
'full-path-to-current-dir/subdir/test_module.py'
---
Looks like it is a bug in:
- in python core subsystem which one is responsible for loading zipped modules
- or in pdb debugger

--
components: Interpreter Core, Library (Lib)
messages: 191907
nosy: vmurashev
priority: normal
severity: normal
status: open
title: Relative path in co_filename for zipped modules
type: behavior
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com