[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Issue34738 fixes distutils.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
components: +Library (Lib) -Extension Modules
stage: patch review -> needs patch
type: behavior -> enhancement
versions: +Python 3.8 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

zipimport have been rewritten in pure Python (issue25711).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +superluser
versions: +Python 3.6 -Python 3.3, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +gregory.p.smith

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Phil Connell]

Phil Connell added the comment:

I've raised issue17633 to track the issue in my last message.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Phil Connell]

Changes by Phil Connell pconn...@gmail.com:


--
nosy: +pconnell

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Martin Morrison]

Changes by Martin Morrison m...@ensoft.co.uk:


--
nosy: +isoschiz

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Phil Connell]

Phil Connell added the comment:

The problem appears to be more general. zipimport fails for deeper hierarchies, 
even with directory entries.

With the supplied patch (zipimport-issue14905-2.patch) I see the following:

$ unzip -l foo.zip
Archive:  foo.zip
  Length  DateTimeName
-  -- -   
0  2013-04-03 17:28   a/b/c/foo.py
0  2013-04-03 17:34   a/
0  2013-04-03 17:34   a/b/
0  2013-04-03 17:34   a/b/c/
- ---
0 4 files
$ ls
foo.zip
$ PYTHONPATH=foo.zip ~/dev/cpython/python
Python 3.4.0a0 (default:3b1dbe7a2aa0+, Apr  3 2013, 17:31:54) 
[GCC 4.8.0] on linux
Type help, copyright, credits or license for more information.
 import a
 import a.b
Traceback (most recent call last):
  File stdin, line 1, in module
ImportError: No module named 'a.b'


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Phillip J. Eby]

Phillip J. Eby added the comment:

Just a note: the zip files produced by the distutils and friends (sdist, 
bdist_dumb, eggs) do not include entries for plain directories.  I would guess 
that this is also true for wheels at the moment, unless something was 
specifically done to work around this property of distutils-generated zip 
files.  So ISTM the right thing to do is to synthesize the entries at directory 
read time, when they're being looped over anyway.

Reviewing the patch, there is a performance optimization possible by making a 
slight change to the algorithm.  Currently the patch loops from the start of 
the string to the end, looking for path prefixes.  This means that the total 
overall performance is determined by the length of the strings and especially 
the average directory depth.

However, there is a significant shortcut possible: looping from the *end* of 
each string to the beginning, it's possible to break out of the loop if the 
prefix has already been seen -- thus saving (depth-1) dictionary lookups in the 
average case, and only looking at the characters in the base filename, unless a 
new directory is encountered... for a typical overhead of one unicode 
substring, dictionary lookup, and strrchr per zipfile directory entry.  (Which 
is very small compared to what else is going on at that point in the process.)

To elaborate, if you have paths of the form:

x/y/a
x/y/b
x/y/c/d

Then when processing 'x/y/a', you would first process x/y -- it's not in the 
dict, add it.  Then x -- not in the dict, add it.  Then you go to x/y/b, your 
first parent is x/y again -- but since it's in the dict you skip it, and don't 
even bother with the x.  Next you see x/y/c, which is not in the dict, so you 
add it, then x/y, which is, so you break out of the loop for that item.

Basically, about all that would change would be the for() loop starting at the 
end of the string and going to the beginning, with the loop position still 
representing the end of the prefix to be extracted.  And the PyDict_Contains 
check would result in a break rather than a continue.

So, if the only concern keeping the patch from being accepted is that it adds 
to startup time, this approach would cut down quite a bit on the overhead for 
generating the path information, in cases of repeated prefixes.  (And in the 
common cases for zipfile use on sys.path, one would expect to see a lot of 
common prefixes, if only for package names.)

--
nosy: +pje

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Ronald Oussoren]

Ronald Oussoren added the comment:

Why are zipfiles without entries for directories broken? When you don't care 
about directory permissions (such as when the zipfile won't be extracted at 
all) the entries for directories are not necessary. Also, AFAIK the zipfile 
specification http://www.pkware.com/documents/casestudies/APPNOTE.TXT does 
not require adding directory entries to the zipfile.

FWIW: the zipfiles created by py2app do no contain entries for directories at 
the moment.  I'll probably add entries for directories in the next update to 
work around this issue.

--
nosy: +ronaldoussoren

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Brett Cannon]

Changes by Brett Cannon br...@python.org:


--
nosy:  -brett.cannon

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Eric V. Smith]

Eric V. Smith added the comment:

I don't think such files are common: I've never seen such a file in the wild. 
I created one, by accident, while testing PEP 420.

OTOH, it was surprisingly easy to create the malformed file with zipfile.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +brett.cannon, ncoghlan
versions: +Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This can significant slowdown zipimport. I think we shouldn't support such 
broken zip files in zipimport.

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Nick Coghlan]

Nick Coghlan added the comment:

How common are such broken zip files? Like Serhiy, I'm concerned about the 
possible negative impact on the interpreter startup time as we try to second 
guess the contents of the zip file manifest.

It seems better to be explicit that we consider such zipfiles broken and they 
need to be regenerated with full manifests (perhaps providing a script in Tools 
that fixes them).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Nick Coghlan]

Nick Coghlan added the comment:

OTOH, the scan time should be short relative to the time needed to read the 
manifest in the first place - an appropriate microbenchmark may also be 
adequate to address my concerns.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Stephen Thorne]

Stephen Thorne added the comment:

Please see attached new patch, based on review comments.

--
Added file: http://bugs.python.org/file26894/zipimport-issue14905-2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
stage: needs patch - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Jonathan Paugh]

Changes by Jonathan Paugh jpa...@gmx.us:


--
nosy: +jpaugh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Stephen Thorne]

Stephen Thorne step...@thorne.id.au added the comment:

Here is a patch that synthesises the directory names at the point where file 
names are read in. The unit test now passes, and has had the expected failure 
removed.

Patch collaboration with Diarmuid Bourke diarmuidbou...@gmail.com at the 
europython sprint.

--
keywords: +patch
nosy: +jerub
Added file: http://bugs.python.org/file26302/zipimport-issue14905.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Eric V. Smith]

Eric V. Smith e...@trueblade.com added the comment:

See also test_namespace_pkgs.py ZipWithMissingDirectory.test_missing_directory 
which is currently marked as expectedFailure.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Eric V. Smith]

New submission from Eric V. Smith e...@trueblade.com:

If a zip file contains pkg/foo.py but no pkg/ entry, it will not be 
possible for pkg to be a namespace package portion.

--
components: Extension Modules
messages: 161543
nosy: eric.smith
priority: normal
severity: normal
stage: needs patch
status: open
title: zipimport.c needs to support namespace packages when no 'directory' 
entry exists
type: behavior
versions: Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14905] zipimport.c needs to support namespace packages when no 'directory' entry exists

[Eric V. Smith]

Eric V. Smith e...@trueblade.com added the comment:

For a (very) brief discussion on the strategy to implement this, see: 
http://mail.python.org/pipermail/import-sig/2012-May/000528.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14905
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com