[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:

if anyone needs this on their older 3.6 or 3.5 trees, the 3.7/3.8 patch is a 
trivial backport.

--
stage: patch review -> commit review
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 7c64726ced3d6b5d04537386d6a9ca6d179c3be4 by Gregory P. Smith in 
branch '3.7':
[3.7] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718) 
(GH-18725)
https://github.com/python/cpython/commit/7c64726ced3d6b5d04537386d6a9ca6d179c3be4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +18082
pull_request: https://github.com/python/cpython/pull/18725

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset ce720d3e0674d6ac6f1b950c20a89be4cfde7853 by Gregory P. Smith in 
branch '3.8':
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718)
https://github.com/python/cpython/commit/ce720d3e0674d6ac6f1b950c20a89be4cfde7853


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +18075
stage: commit review -> patch review
pull_request: https://github.com/python/cpython/pull/18718

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:

keeping this open while i investigate if fixing 3.8 and 3.7 is feasible.  if 
nothing else i'll try to add a note to the docs about the issue in 3.5-3.8 with 
a code sample suggested workaround.

--
resolution:  -> fixed
stage: patch review -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 02673352b5db6ca4d3dc804965facbedfe66425d by Gregory P. Smith in 
branch 'master':
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676)
https://github.com/python/cpython/commit/02673352b5db6ca4d3dc804965facbedfe66425d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Change by Gregory P. Smith :


--
keywords: +patch
pull_requests: +18036
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/18676

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:

belated postmortem: if there had been tests against the filename encoded in the 
pycs living in subdirs as generated by compile_dir, this regression would not 
have happened.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:

meaning this regression was introduced in 3.5.

--
versions: +Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:

The regression was caused by the parallelization enhancement implementation 
from https://bugs.python.org/issue16104

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Ironically 3.9a4 gains some compileall enhancements that could be used as an 
awkward workaround from https://bugs.python.org/issue38112:

python 2.7 compile_dir(d, ddir="") behavior can be had with
python 3.9a4 compile_dir(d, prependdir="", stripdir=d)

I still intend to fix the actual ddir behavior as it is wrong.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

Gregory P. Smith  added the comment:

This has probably been happening for many 3 releases.  I noticed it on 3.6 when 
investigating a problem with wrong paths in tracebacks coming from code which 
we used compileall.compile_dir(src_root, quiet=1, ddir=") on.

I'm guessing ddir= (-d on the cli) isn't widely used?

--
components: +Library (Lib)
stage:  -> needs patch
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39769] compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix

[Gregory P. Smith]

New submission from Gregory P. Smith :

Easiest to demonstrate as such:

```shell
#!/bin/bash

mkdir bug
touch bug/__init__.py
mkdir bug/foo
touch bug/foo/__init__.py
touch bug/foo/bar.py

python3 -m compileall -d "" bug
python2 -m compileall -d "" bug

echo "prefix embedded in PY3 pyc code object for lib.foo.bar:"
strings bug/foo/__pycache__/bar.cpython-3*.pyc | grep prefix
echo "prefix embedded in PY2 pyc code object for lib.foo.bar:"
strings bug/foo/bar.pyc | grep prefix
```

Run that script and you'll see:
Listing 'bug'...
Compiling 'bug/__init__.py'...
Listing 'bug/foo'...
Compiling 'bug/foo/__init__.py'...
Compiling 'bug/foo/bar.py'...
Listing bug ...
Compiling bug/__init__.py ...
Listing bug/__pycache__ ...
Listing bug/foo ...
Compiling bug/foo/__init__.py ...
Listing bug/foo/__pycache__ ...
Compiling bug/foo/bar.py ...
prefix embedded in PY3 pyc code object for lib.foo.bar:
/bar.py
prefix embedded in PY2 pyc code object for lib.foo.bar:
/foo/bar.pyt


Notice that the Python 3 pyc file contains a code.co_filename of 
"/bar.py" instead of the correct value (that Python 2 inserts) of 
"/foo/bar.py".

--
messages: 362767
nosy: gregory.p.smith
priority: normal
severity: normal
status: open
title: compileall.compile_dir(..., ddir="") omits the intermediate 
package paths when prepending the prefix
versions: Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com