Re: Getting a list of all modules
On 01/08/2014 16:41, Steven D'Aprano wrote: On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote: Take a look at what has already been implemented in IPython: https://github.com/ipython/ipython/blob/master/IPython/core/ completerlib.py#L208 Awesome! Thank you! Is Lib/idlelib/AutoComplete.py of any use? I found it when looking at this http://bugs.python.org/issue18766 -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug with help (was Re: Getting a list of all modules)
Akira Li wrote:
Look at how `help('modules')` is implemented. Though it crashes on my
system.
Have you reported this at bugs.python.org or is there already an issue
for the problem that you see?
It is this issue for python2.7:
https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/896836
python3 doesn't crash with currently installed packages.
Python 2.7 on Windows XP Sp2 doesn't crash with this. So it seems to
be a python-OS-problem.
Regards,
H.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bug with help (was Re: Getting a list of all modules)
On 2014-08-02 09:33, Heinz Schmitz wrote:
Akira Li wrote:
Look at how `help('modules')` is implemented. Though it crashes on my
system.
Have you reported this at bugs.python.org or is there already an issue
for the problem that you see?
It is this issue for python2.7:
https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/896836
python3 doesn't crash with currently installed packages.
Python 2.7 on Windows XP Sp2 doesn't crash with this. So it seems to
be a python-OS-problem.
Well, it's just that `help('modules')` imports every module in the calling
process (at least in Python 2.7; I haven't checked Python 3). Some extension
modules conflict with each other and cause a crash when both are imported
together. It's possible that you just don't have such modules installed. While
the proximate cause of the crash is in the 3rd party modules, Python could (and
maybe Python 3 does) import each module in a separate subprocess and collect the
information that way.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bug with help (was Re: Getting a list of all modules)
On 8/2/2014 8:13 AM, Robert Kern wrote:
On 2014-08-02 09:33, Heinz Schmitz wrote:
Akira Li wrote:
Look at how `help('modules')` is implemented. Though it crashes on my
system.
Have you reported this at bugs.python.org or is there already an issue
for the problem that you see?
It is this issue for python2.7:
https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/896836
python3 doesn't crash with currently installed packages.
Python 2.7 on Windows XP Sp2 doesn't crash with this. So it seems to
be a python-OS-problem.
It is a local system problem.
Well, it's just that `help('modules')` imports every module in the
calling process (at least in Python 2.7; I haven't checked Python 3).
Some extension modules conflict with each other and cause a crash when
both are imported together. It's possible that you just don't have such
modules installed. While the proximate cause of the crash is in the 3rd
party modules, Python could (and maybe Python 3 does) import each module
in a separate subprocess and collect the information that way.
That would slow down help('modules') much more. The suggestion in
http://bugs.python.org/issue12902
help(modules) executes module code
is that help() should instead parse the module to get the docstring.
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Friday, 1 August 2014 16:41:41 UTC+1, Steven D'Aprano wrote: On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote: Take a look at what has already been implemented in IPython: https://github.com/ipython/ipython/blob/master/IPython/core/ completerlib.py#L208 Awesome! Thank you! -- Steven Also this, possibly: https://github.com/davidhalter/jedi Jon N -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On 02/08/2014 22:13, jonnicol...@googlemail.com wrote: On Friday, 1 August 2014 16:41:41 UTC+1, Steven D'Aprano wrote: On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote: Take a look at what has already been implemented in IPython: https://github.com/ipython/ipython/blob/master/IPython/core/ completerlib.py#L208 Awesome! Thank you! -- Steven Also this, possibly: https://github.com/davidhalter/jedi Jon N Would you please read and action this https://wiki.python.org/moin/GoogleGroupsPython to prevent us seeing the double line spacing above, thanks. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On 2014-07-31 11:41, Steven D'Aprano wrote: On Wed, 30 Jul 2014 21:22:18 +0800, Leo Jay wrote: On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano st...@pearwood.info wrote: I'm looking for a programmatic way to get a list of all Python modules and packages. Not just those already imported, but all those which *could* be imported. If you don't actually import it, how can you know it could be imported? Not all .so files are valid python modules. Not all .py files could be imported by all python interpreters. You're right, of course, but I'm not concerned by whether or not the module is error-free and can be imported successfully. I'm working on tab completion for module names. I have some alpha-quality code working, so if I hit TAB after typing import ma I get this: py import ma macpath macurl2path mailbox mailcap mangle markupbase math For what it's worth, importing mangle fails with a SyntaxError. But that's okay, I don't expect tab completion to only show *valid* modules :-) Over the next few days I'll make an official announcement, but if anyone wants a sneak-peek, check out: http://code.google.com/p/tabhistory/source/browse/tabhistory.py where I have indenting, code completion, filename completion, and module completion all working to some degree or another. Take a look at what has already been implemented in IPython: https://github.com/ipython/ipython/blob/master/IPython/core/completerlib.py#L208 -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote: Take a look at what has already been implemented in IPython: https://github.com/ipython/ipython/blob/master/IPython/core/ completerlib.py#L208 Awesome! Thank you! -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug with help (was Re: Getting a list of all modules)
Mark Lawrence breamore...@yahoo.co.uk writes:
On 31/07/2014 19:55, Akira Li wrote:
Steven D'Aprano st...@pearwood.info writes:
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
...
Is this problem already solved? Can anyone make any suggestions?
Look at how `help('modules')` is implemented. Though it crashes on my
system.
Have you reported this at bugs.python.org or is there already an issue
for the problem that you see?
It is this issue for python2.7:
https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/896836
python3 doesn't crash with currently installed packages.
--
Akira
--
https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Wed, 30 Jul 2014 21:22:18 +0800, Leo Jay wrote: On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano st...@pearwood.info wrote: I'm looking for a programmatic way to get a list of all Python modules and packages. Not just those already imported, but all those which *could* be imported. If you don't actually import it, how can you know it could be imported? Not all .so files are valid python modules. Not all .py files could be imported by all python interpreters. You're right, of course, but I'm not concerned by whether or not the module is error-free and can be imported successfully. I'm working on tab completion for module names. I have some alpha-quality code working, so if I hit TAB after typing import ma I get this: py import ma macpath macurl2path mailbox mailcap mangle markupbase math For what it's worth, importing mangle fails with a SyntaxError. But that's okay, I don't expect tab completion to only show *valid* modules :-) Over the next few days I'll make an official announcement, but if anyone wants a sneak-peek, check out: http://code.google.com/p/tabhistory/source/browse/tabhistory.py where I have indenting, code completion, filename completion, and module completion all working to some degree or another. Thanks to everyone for their help. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
In article 53da1d5a$0$29974$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: I'm working on tab completion for module names. I have some alpha-quality code working, so if I hit TAB after typing import ma I get this: py import ma macpath macurl2path mailbox mailcap mangle markupbase math That's kind of neat. What do you do if the same name appears multiple places in your path? It would be useful to call this out. More than once, I've done import x and gotten a different x than I expected. Those kinds of problems can be tricky to debug. -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Thu, Jul 31, 2014 at 9:55 PM, Roy Smith r...@panix.com wrote: In article 53da1d5a$0$29974$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: I'm working on tab completion for module names. I have some alpha-quality code working, so if I hit TAB after typing import ma I get this: py import ma macpath macurl2path mailbox mailcap mangle markupbase math That's kind of neat. What do you do if the same name appears multiple places in your path? It would be useful to call this out. More than once, I've done import x and gotten a different x than I expected. Those kinds of problems can be tricky to debug. Immaterial to the tab completion, as it just means one of them is shadowed by the other. But yes, that could be extremely useful - or maybe completely useless, if there's a lot of intentional shadowing happening. At any rate, it'd be nice to be able to say Show me everything that could be found for this name, which the same code could answer. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
Steven D'Aprano st...@pearwood.info writes:
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
...
Is this problem already solved? Can anyone make any suggestions?
Look at how `help('modules')` is implemented. Though it crashes on my
system.
See also,
How can I get a list of locally installed Python modules? [1]
python - get available modules [2]
[1]
http://stackoverflow.com/questions/739993/how-can-i-get-a-list-of-locally-installed-python-modules
[2]
http://stackoverflow.com/questions/3952513/python-get-available-modules
--
Akira
--
https://mail.python.org/mailman/listinfo/python-list
Bug with help (was Re: Getting a list of all modules)
On 31/07/2014 19:55, Akira Li wrote:
Steven D'Aprano st...@pearwood.info writes:
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
...
Is this problem already solved? Can anyone make any suggestions?
Look at how `help('modules')` is implemented. Though it crashes on my
system.
Have you reported this at bugs.python.org or is there already an issue
for the problem that you see?
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
--
https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On 7/31/2014 6:41 AM, Steven D'Aprano wrote: I'm working on tab completion for module names. I have some alpha-quality code working, so if I hit TAB after typing import ma I get this: py import ma macpath macurl2path mailbox mailcap mangle markupbase math This is an interesting idea for Idle. Idle currently completes names from builtins and, in interactive mode, the current globals. So in the situation above, Idle offers 'map' and 'max' in a listbox with all builtins (and globals). But after '.', it completes attribute names instead and within strings, filenames, starting from the current directory of the user process for relative names. It should not be too hard to recognize 'import' and 'from' instead of or in addition to '.' as context markers. Even just offering stdlib name (or current directory names or . imports) would be a help. Builtin names can never be correct in this context. For what it's worth, importing mangle fails with a SyntaxError. But that's okay, I don't expect tab completion to only show *valid* modules :-) Over the next few days I'll make an official announcement, but if anyone wants a sneak-peek, check out: http://code.google.com/p/tabhistory/source/browse/tabhistory.py where I have indenting, code completion, filename completion, and module completion all working to some degree or another. Thanks to everyone for their help. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug with help (was Re: Getting a list of all modules)
On 7/31/2014 3:19 PM, Mark Lawrence wrote:
On 31/07/2014 19:55, Akira Li wrote:
Steven D'Aprano st...@pearwood.info writes:
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
...
Is this problem already solved? Can anyone make any suggestions?
Look at how `help('modules')` is implemented. Though it crashes on my
system.
Have you reported this at bugs.python.org or is there already an issue
for the problem that you see?
I strongly suspect this issue:
http://bugs.python.org/issue12902
help('modules') imports all modules, and crashes if any import crashes
the system.
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Getting a list of all modules
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
I have a quick-and-dirty function which half does the job:
def get_modules():
extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
matches = set()
for location in sys.path:
if location == '': location = '.'
if os.path.isdir(location):
for name in os.listdir(location):
base, ext = os.path.splitext(name)
if ext in extensions:
matches.add(base)
return sorted(matches)
but I know it's wrong (it doesn't handle packages correctly, or zip
files, doesn't follow .pth files, has a very naive understanding of cross-
platform issues, fails to include built-in modules that don't live in the
file system, and probably more).
Is this problem already solved? Can anyone make any suggestions?
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Wed, Jul 30, 2014 at 5:43 PM, Steven D'Aprano st...@pearwood.info wrote: Is this problem already solved? Can anyone make any suggestions? I don't know of an actual solution, but I know where I'd look for one, and that's importlib. If nothing else, you can use importlib.machinery.all_suffixes() rather than hard-coding them, for instance. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
Steven D'Aprano wrote:
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
I have a quick-and-dirty function which half does the job:
def get_modules():
extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
matches = set()
for location in sys.path:
if location == '': location = '.'
if os.path.isdir(location):
for name in os.listdir(location):
base, ext = os.path.splitext(name)
if ext in extensions:
matches.add(base)
return sorted(matches)
but I know it's wrong (it doesn't handle packages correctly, or zip
files, doesn't follow .pth files, has a very naive understanding of cross-
platform issues, fails to include built-in modules that don't live in the
file system, and probably more).
Is this problem already solved? Can anyone make any suggestions?
$ python3 -m pydoc -b
shows a page with modules that I think is more complete than what you have.
A quick glance at the implementation suggests that the hard work is done by
pkgutil.iter_modules()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On 2014-07-30 09:46, Peter Otten wrote:
Steven D'Aprano wrote:
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
I have a quick-and-dirty function which half does the job:
def get_modules():
extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
matches = set()
for location in sys.path:
if location == '': location = '.'
if os.path.isdir(location):
for name in os.listdir(location):
base, ext = os.path.splitext(name)
if ext in extensions:
matches.add(base)
return sorted(matches)
but I know it's wrong (it doesn't handle packages correctly, or zip
files, doesn't follow .pth files, has a very naive understanding of cross-
platform issues, fails to include built-in modules that don't live in the
file system, and probably more).
Is this problem already solved? Can anyone make any suggestions?
$ python3 -m pydoc -b
shows a page with modules that I think is more complete than what you have.
A quick glance at the implementation suggests that the hard work is done by
pkgutil.iter_modules()
There are two niggles to this answer: it omits builtin modules, but those are
easily discovered through sys.builtin_module_names. It can also include spurious
script .py files that cannot be imported because their names are not Python
identifiers: e.g. check-newconfigs.py. Those are easy to filter out, fortunately.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
--
https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano st...@pearwood.info wrote:
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
If you don't actually import it, how can you know it could be imported?
Not all .so files are valid python modules.
Not all .py files could be imported by all python interpreters.
I have a quick-and-dirty function which half does the job:
def get_modules():
extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
matches = set()
for location in sys.path:
if location == '': location = '.'
if os.path.isdir(location):
for name in os.listdir(location):
base, ext = os.path.splitext(name)
if ext in extensions:
matches.add(base)
return sorted(matches)
but I know it's wrong (it doesn't handle packages correctly, or zip
files, doesn't follow .pth files, has a very naive understanding of cross-
platform issues, fails to include built-in modules that don't live in the
file system, and probably more).
Is this problem already solved? Can anyone make any suggestions?
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
--
Best Regards,
Leo Jay
--
https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Wed, Jul 30, 2014 at 11:22 PM, Leo Jay python.leo...@gmail.com wrote: On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano st...@pearwood.info wrote: I'm looking for a programmatic way to get a list of all Python modules and packages. Not just those already imported, but all those which *could* be imported. If you don't actually import it, how can you know it could be imported? Not all .so files are valid python modules. Not all .py files could be imported by all python interpreters. What if you define it as modules you could attempt to import? Sure, any module might fail during importing, but you've still taken a statement of import spam and turned it into an attempt to read some specific file from the disk. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Jul 30, 2014 4:37 AM, Robert Kern robert.k...@gmail.com wrote:
On 2014-07-30 09:46, Peter Otten wrote:
Steven D'Aprano wrote:
I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.
I have a quick-and-dirty function which half does the job:
def get_modules():
extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
matches = set()
for location in sys.path:
if location == '': location = '.'
if os.path.isdir(location):
for name in os.listdir(location):
base, ext = os.path.splitext(name)
if ext in extensions:
matches.add(base)
return sorted(matches)
but I know it's wrong (it doesn't handle packages correctly, or zip
files, doesn't follow .pth files, has a very naive understanding of
cross-
platform issues, fails to include built-in modules that don't live in
the
file system, and probably more).
Is this problem already solved? Can anyone make any suggestions?
$ python3 -m pydoc -b
shows a page with modules that I think is more complete than what you
have.
A quick glance at the implementation suggests that the hard work is done
by
pkgutil.iter_modules()
There are two niggles to this answer: it omits builtin modules, but those
are easily discovered through sys.builtin_module_names. It can also include
spurious script .py files that cannot be imported because their names are
not Python identifiers: e.g. check-newconfigs.py. Those are easy to filter
out, fortunately.
It will also omit any modules provided by a custom module finder that
doesn't implement iter_modules, which is not a required part of the
interface.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Getting a list of all modules
On Wed, Jul 30, 2014 at 2:43 AM, Steven D'Aprano st...@pearwood.info wrote: I'm looking for a programmatic way to get a list of all Python modules and packages. Not just those already imported, but all those which *could* be imported. I wrote a modified dir(), which I inject into builtins in interactive sessions. When getting a directory of an object, it attempts to identify not-yet-imported modules it contains. Such modules are enclosed in square brackets. Modules that smell like packages also get a trailing '/'. For example: % python Python 2.7.5+ (2.7:2921f6c2009e, Apr 30 2014, 14:00:13) [GCC 4.4.6 [TWW]] on linux2 Type help, copyright, credits or license for more information. import dateutil dir(dateutil) ['[easter]', '[parser]', '[relativedelta]', '[rrule]', '[tz]', '[tzwin]', '[zoneinfo/]', '__author__', '__builtins__', '__doc__', '__file__', '__license__', '__name__', '__package__', '__path__', '__version__'] It's not perfect, but works for my needs. Perhaps it will give you some ideas. Skip import os _dir = dir def dir(o=globals(), hidden=False): if not hidden and hasattr(o, __all__): contents = o.__all__ else: contents = _dir(o) if hasattr(o, __file__): dname = os.path.dirname(o.__file__) # look for not-yet-imported modules within packages if /__init__.py in o.__file__: try: stuff = os.listdir(dname) except OSError: # Searching eggs lands here. Introspect. import zipfile d = os.path.dirname(dname) if not zipfile.is_zipfile(d): return sorted(contents) base = os.path.basename(dname) stuff = [f[len(base)+1:] for f in zipfile.ZipFile(d).namelist() if f.startswith(base)] for p in stuff: m = os.path.splitext(p)[0] if ( # not already known m not in contents and # isn't a package file p != __init__.py and # is a python or ... (p.endswith(.py) or # c module or ... p.endswith(.so) or # a subpackage (os.path.isdir(os.path.join(dname, p)) and os.path.exists(os.path.join(dname, p, __init__.py): if os.path.isdir(os.path.join(dname, p)): # tack on trailing / to distinguish packages from # modules m += / if not m.startswith(_) or hidden: # [...] shows it hasn't been imported yet contents.append([%s] % m) return sorted(contents) -- https://mail.python.org/mailman/listinfo/python-list
