[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Chris Jerdonek chris.jerdo...@gmail.com added the comment: FYI, I created issue 15299 and issue 15297 which also relate to pkgutil.walk_packages() not working correctly in Python 3.3 -- even with the caveat added by Brett. These two issues are in pkgutil.walk_packages()'s code path when passed path=None. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Chris Jerdonek chris.jerdo...@gmail.com added the comment: As someone who isn't an expert on Python's import mechanisms, I'm confused by this comment at the end of pkgutil.walk_packages(): Note: Only works for importers which define a iter_modules() method, which is non-standard but implemented by classes defined in this module. I'm confused because importers aren't mentioned anywhere in the context of walk_packages() (in particular, `importer` is not a parameter to the function), so it's not clear to me how the note comes into play. For example, is importers in reference to the import statements inside the modules in the package that one is trying to walk, or is it a reference to whether or not the user has overridden Python's standard import with a custom import function? In particular, does this mean that the normal case of trying to walk a directory using `pkgutil.walk_packages(['my_dir'])` won't work without doing something special, and if so, can that special thing be added? I think it would help if this were made clearer. I came to this after trying to start working on issue 14787. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Eric Snow ericsnowcurren...@gmail.com added the comment: Importers are a PEP 302 API. In the context of pkgutil, they are path hooks (see http://www.python.org/dev/peps/pep-0302/#id24), aka path importers. A path hook is a callable that takes a path and returns a finder. Path hooks are stored in sys.path_hooks and the finder resulting from a matching path importer is stored in sys.path_importer_cache. pkgutil also refers to PEP 302 finders as importers. To be honest, the import-related nomenclature is a mess. While you can use pkgutil.ImpImporter as a path hook, instances can also be used as metapath hooks; if you don't pass anything to ImpImporter, the resulting object acts more like a PEP 302 metapath hook that wraps the default import process. (See Lib/pkgutil.py) -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Chris Jerdonek chris.jerdo...@gmail.com added the comment: Thanks a lot, Eric. I'm going to create a new issue to clarify the note a bit, if that's okay. At minimum, I feel like enough information should be given to let one run the examples given. By the way, your comment, While you can use pkgutil.ImpImporter as a path hook, instances can also be used as metapath hooks seems to contradict the pkgutil documentation, which says, Note that ImpImporter does not currently support being used by placement on sys.meta_path. (According to PEP 302, To register a meta hook, simply add the finder object to sys.meta_path.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Chris Jerdonek chris.jerdo...@gmail.com added the comment: I created issue 15288 to improve the documentation around this. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset b92fa1c5a96b by Brett Cannon in branch 'default': Closes issue #14982: Document that pkgutil's walk_packages() and http://hg.python.org/cpython/rev/b92fa1c5a96b -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Changes by Brett Cannon br...@python.org: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Paul Nasrat pnas...@gmail.com added the comment: We've taken a simpler approach avoiding walk_packages in pip which we'll release for 3.3. I'd say if pkgutil doesn't work correctly with importers loaders outside of it we probably should make that very explicit in the docs, and potentially consider deprecating walk_packages. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Brett Cannon br...@python.org added the comment: Yes, the docs should get updated and I will do that before Python 3.3 goes out the door. -- assignee: - brett.cannon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Chris Jerdonek chris.jerdo...@gmail.com added the comment: See also issue 14787 re: walk_packages(), which also affects versions before 3.3. -- nosy: +cjerdonek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Brett Cannon br...@python.org added the comment: I should mention that Guido and others on python-dev mentioned coming up with an API for finders/loaders that allowed for file-like API and possibly being able to iterate over available modules when importlib's bootstrapping landed (sorry, don't have a link for it). If pip needs some specific introspection support from finders or loaders I would try asking on python-dev about exactly what you need to see if it can get into Python 3.3. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Marc Abramowitz msabr...@gmail.com added the comment: [last: 0] marca@scml-marca:~/dev/git-repos/pip$ python3.3 Python 3.3.0a4 (v3.3.0a4:7c51388a3aa7, May 30 2012, 16:58:42) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type help, copyright, credits or license for more information. from pip import vcs from pkgutil import walk_packages, iter_modules print(list(iter_modules(path=vcs.__path__, prefix=vcs.__name__+'.'))) [] print(list(iter_modules(path=vcs.__path__))) [] import pip.vcs.git pip.vcs.git module 'pip.vcs.git' from './pip/vcs/git.py' import pip.vcs.mercurial pip.vcs.mercurial module 'pip.vcs.mercurial' from './pip/vcs/mercurial.py' print(list(iter_modules(path=vcs.__path__, prefix=vcs.__name__+'.'))) [] print(list(iter_modules(path=vcs.__path__))) [] -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Changes by Ronan Lamy ronan.l...@gmail.com: -- nosy: +Ronan.Lamy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Brett Cannon br...@python.org added the comment: Basically pkgutil kind of handles importers properly, kind of doesn't. So if a module defined a __loader__ it will use it, but all the rest of its code assumes it uses only the loaders defined in pkgutil. The problem here is that pkgutil.walk_packages() ends up calling iter_importer_modules() which only returns anything of consequence if the loader has iter_modules() defined which is a non-standard API requirement that only pkgutil loaders has implemented. Basically the docs for pkgutil were incorrect in not specifying that the walk only works for loaders that define iter_modules(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Paul Nasrat pnas...@gmail.com added the comment: Ok so it seems I can't just use sys.meta_path in pip to work with ImpImporter due to according to the pydoc: Note that ImpImporter does not currently support being used by placement on sys.meta_path. I guess I can write a custom importer in pip for handling the vcs plugins or possibly change how we do that. Another option is to grossly hack around ala: i = ImpImporter(path=vcs.__path__[0]) sys.path_importer_cache.setdefault(vcs.__path__[0], i) Found another bug doing this though which I'll follow up with a patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
New submission from Marc Abramowitz msabr...@gmail.com:
I noticed that pip wasn't working properly on Python 3.3a - notably, it wasn't
able to load any of its own VCS modules -- it does this by using
pkgutil.walk_packages
I think the problem is that the behavior of pkgutil.walk_packages changed in
some incompatible way in 3.3 -- take a look at the following:
[last: 0] marca@scml-marca:~/dev/git-repos/lexicon$
../lexicon/.tox/py32/bin/python
Python 3.2.3 (v3.2.3:3d0686d90f55, Apr 10 2012, 11:25:50)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type help, copyright, credits or license for more information.
from pkgutil import walk_packages
list(walk_packages('/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/pip-1.1-py3.3.egg/pip/vcs'))
[(pkgutil.ImpImporter object at 0x1005bc710, 'lexicon', True),
(pkgutil.ImpImporter object at 0x100649410, 'lexicon.alias_dict', False),
(pkgutil.ImpImporter object at 0x100649410, 'lexicon.attribute_dict', False),
(pkgutil.ImpImporter object at 0x1005bc710, 'reg_settings', False),
(pkgutil.ImpImporter object at 0x1005bc710, 'setup', False)]
^D
[last: 10] marca@scml-marca:~/dev/git-repos/lexicon$
../lexicon/.tox/py33/bin/python
Python 3.3.0a3 (v3.3.0a3:0b53b70a40a0, May 1 2012, 11:39:35)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type help, copyright, credits or license for more information.
from pkgutil import walk_packages
list(walk_packages('/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/pip-1.1-py3.3.egg/pip/vcs'))
[]
--
components: Library (Lib)
messages: 162110
nosy: Marc.Abramowitz
priority: normal
severity: normal
status: open
title: pkgutil.walk_packages seems to not work properly on Python 3.3a
versions: Python 3.3
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14982
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Paul Nasrat pnas...@gmail.com added the comment: I stepped through in pdb (Pdb) p i _frozen_importlib.FileFinder object at 0x101066090 (Pdb) iter_importer_modules(i, prefix) [] (Pdb) p hasattr(importer, 'iter_modules') False 2.7 uses [pkgutil.ImpImporter instance at 0x107a4d830] (Pdb) pp dir(myimp) ['__doc__', '__init__', '__module__', 'find_module', 'iter_modules', 'path'] This seems related to http://www.python.org/dev/peps/pep-0420/ -- nosy: +pnasrat ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Eric V. Smith e...@trueblade.com added the comment: I don't think this is related to PEP 420. Adding Brett. -- nosy: +brett.cannon, eric.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Marc Abramowitz msabr...@gmail.com added the comment: Here's the pip issue: https://github.com/pypa/pip/issues/556 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14982] pkgutil.walk_packages seems to not work properly on Python 3.3a
Paul Nasrat pnas...@gmail.com added the comment: I was pretty tired when debugging last night and just quickly looked at hg logs, so that may be misattributed. I'll try come up with a clearer reproducer. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14982 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
