Re: [Python-Dev] New relative import issue
Hi Guido, On Thu, Sep 21, 2006 at 07:22:04AM -0700, Guido van Rossum wrote: sys.path exists to stitch together the toplevel module/package namespace from diverse sources. Import hooks and sys.path hackery exist so that module/package sources don't have to be restricted to the filesystem (as well as to allow unbridled experimentation by those so inclined :-). This doesn't match my experience, which is that sys.path hackery is required in any project that is larger than one directory, but is not itself a library. The basic assumption is that I don't want to put whole applications in 'site-packages' or in my $PYTHONPATH; I would like them to work in a self-contained, zero-installation way, much like they do if they are built from several modules in a single directory. For example, consider an application with the following structure: myapp/ main.py a/ __init__.py b.py test_b.py c/ __init__.py This theoretical example shows main.py (the main entry point) at the root of the package directories - it is the only place where it can be if it needs to import the packages a and c. The module a.b can import c, too (and this is not bad design - think about c as a package regrouping utilities that make sense for the whole application). But then the testing script test_b.py cannot import the whole application any more. Imports of a or c will fail, and even a relative import of b will crash when b tries to import c. The only way I can think of is to insert the root directory in sys.path from within test_b.py, and then use absolute imports. (For example, to support this way of organizing applications, the 'py' lib provides a call py.magic.autopath() that can be dropped at the start of test_b.py. It hacks sys.path by guessing the real root according to how many levels of __init__.py there are...) A bientot, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Armin Rigo wrote: This doesn't match my experience, which is that sys.path hackery is required in any project that is larger than one directory, but is not itself a library. [...] myapp/ main.py a/ __init__.py b.py test_b.py c/ __init__.py This theoretical example shows main.py (the main entry point) at the root of the package directories - it is the only place where it can be if it needs to import the packages a and c. The module a.b can import c, too (and this is not bad design - think about c as a package regrouping utilities that make sense for the whole application). But then the testing script test_b.py cannot import the whole application any more. Imports of a or c will fail, and even a relative import of b will crash when b tries to import c. The only way I can think of is to insert the root directory in sys.path from within test_b.py, and then use absolute imports. This also matches my experience, but I never used sys.path hackery for this kind of things. I either set PYTHONPATH while I work on myapp (which I consider not such a big trouble after all, and surely much less invasive than adding specific Python code tweaking sys.path into all the tests), or, even more simply, I run the test from myapp main directory (manually typing myapp/b/test_b.py). There is also another possibility, which is having a smarter test framework where you can specify substrings of test names: I don't know py.test in detail, but in my own framework I can say something like ./run_tests.py PAT, which basically means recursively discover and run all files named test_NAME, and where PAT is a substring of NAME). (For example, to support this way of organizing applications, the 'py' lib provides a call py.magic.autopath() that can be dropped at the start of test_b.py. It hacks sys.path by guessing the real root according to how many levels of __init__.py there are...) Since I consider this more of an environmental problem, I would not find satisfying any kind of solution at the single module level (and even less so one requiring so much guess-work as this one). Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Phillip J. Eby [EMAIL PROTECTED] wrote: At 08:44 PM 9/21/2006 -0700, Josiah Carlson wrote: This can be implemented with a fairly simple package registry, contained within a (small) SQLite database (which is conveniently shipped in Python 2.5). There can be a system-wide database that all users use as a base, with a user-defined package registry (per user) where the system-wide packages can be augmented. As far as I can tell, you're ignoring that per-user must *also* be per-version, and per-application. Each application or runtime environment needs its own private set of information like this. Having a different database per Python version is not significantly different than having a different Python binary for each Python version. About the only (annoying) nit is that the systemwide database needs to be easily accessable to the Python runtime, and is possibly volatile. Maybe a symlink in the same path as the actual Python binary on *nix, and the file located next to the binary on Windows. I didn't mention the following because I thought it would be superfluous, but it seems that I should have stated it right out. My thoughts were that on startup, Python would first query the 'system' database, caching its results in a dictionary, then query the user's listing, updating the dictionary as necessary, then unload the databases. On demand, when code runs packages.register(), if both persist and systemwide are False, it just updates the dictionary. If either are true, it opens up and updates the relevant database. With such a semantic, every time Python gets run, every instance gets its own private set of paths, derived from the system database, user database, and runtime-defined packages. Next, putting the installation data inside a database instead of per-installation-unit files presents problems of its own. While some system packaging tools allow install/uninstall scripts to run, they are often frowned upon, and can be unintentionally bypassed. This is easily remedied with a proper 'packages' implementation: python -Mpackages name path Note that Python could auto-insert standard library and site-packages 'packages' on startup (creating the initial dictionary, then the systemwide, then the user, ...). These are just a few of the issues that come to mind. Realistically speaking, .pth files are currently the most effective mechanism we have, and there actually isn't much that can be done to improve upon them. Except that .pth files are only usable in certain (likely) system paths, that the user may not have write access to. There have previously been proposals to add support for .pth files in the path of the run .py file, but they don't seem to have gotten any support. What's more needed are better mechanisms for creating and managing Python environments (to use a term coined by Ian Bicking and Jim Fulton over on the distutils-sig), which are individual contexts in which Python applications run. Some current tools in development by Ian and Jim include: Anyway, system-wide and per-user environment information isn't nearly sufficient to address the issues that people have when developing and deploying multiple applications on a server, or even using multiple applications on a client installation (e.g. somebody using both the Enthought Python IDE and Chandler on the same machine). These relatively simple use cases rapidly demonstrate the inadequacy of system-wide or per-user configuration of what packages are available. It wouldn't be terribly difficult to add environment switching and environment derivation (copying or linked, though copying would be simpler). packages.derive_environment(parent_environment) packages.register(name, path, env=environment) packages.use(environment) It also wouldn't be terribly difficult to set up environments that required certain packages... packages.new_environment(environment, *required_packages, test=True) To verify that the Python installation has the required packages, then later... packages.new_environment(environment, *required_packages, persist=True) I believe that most of the concerns that you have brought up can be addressed, and I think that it could be far nicer to deal with than the current sys.path hackery. The system database location is a bit annoying, but I lack the *nix experience to say where such a database could or should be located. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Brett Cannon wrote: But either way I will be messing with the import system in the relatively near future. If you want to help, Paul (or anyone else), just send me an email and we can try to coordinate something (plan to do the work in the sandbox as a separate thing from my security stuff). Starting with pkgutil.get_loader and removing the current dependency on imp.find_module and imp.load_module would probably be a decent way to start. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
At 12:08 AM 9/22/2006 -0700, Josiah Carlson wrote: Phillip J. Eby [EMAIL PROTECTED] wrote: At 08:44 PM 9/21/2006 -0700, Josiah Carlson wrote: This can be implemented with a fairly simple package registry, contained within a (small) SQLite database (which is conveniently shipped in Python 2.5). There can be a system-wide database that all users use as a base, with a user-defined package registry (per user) where the system-wide packages can be augmented. As far as I can tell, you're ignoring that per-user must *also* be per-version, and per-application. Each application or runtime environment needs its own private set of information like this. Having a different database per Python version is not significantly different than having a different Python binary for each Python version. You misunderstood me: I mean that the per-user database must be able to store information for *different Python versions*. Having a single per-user database without the ability to include configuration for more than one Python version (analagous to the current situation with the distutils per-user config file) is problematic. In truth, a per-user configuration is just a special case of the real need: to have per-application environments. In effect, a per-user environment is a fallback for not having an appplication environment, and the system environment is a fallback for not having a user environment. About the only (annoying) nit is that the systemwide database needs to be easily accessable to the Python runtime, and is possibly volatile. Maybe a symlink in the same path as the actual Python binary on *nix, and the file located next to the binary on Windows. I didn't mention the following because I thought it would be superfluous, but it seems that I should have stated it right out. My thoughts were that on startup, Python would first query the 'system' database, caching its results in a dictionary, then query the user's listing, updating the dictionary as necessary, then unload the databases. On demand, when code runs packages.register(), if both persist and systemwide are False, it just updates the dictionary. If either are true, it opens up and updates the relevant database. Using a database as the primary mechanism for managing import locations simply isn't workable. You might as well suggest that each environment consist of a single large zipfile containing the packages in question: this would actually be *more* practical (and fast!) in terms of Python startup, and is no different from having a database with respect to the need for installation and uninstallation to modify a central file! I'm not proposing we do that -- I'm just pointing out why using an actual database isn't really workable, considering that it has all of the disadvantages of a big zipfile, and none of the advantages (like speed, having code already written that supports it, etc.) This is easily remedied with a proper 'packages' implementation: python -Mpackages name path Note that Python could auto-insert standard library and site-packages 'packages' on startup (creating the initial dictionary, then the systemwide, then the user, ...). I presume here you're suggesting a way to select a runtime environment from the command line, which would certainly be a good idea. These are just a few of the issues that come to mind. Realistically speaking, .pth files are currently the most effective mechanism we have, and there actually isn't much that can be done to improve upon them. Except that .pth files are only usable in certain (likely) system paths, that the user may not have write access to. There have previously been proposals to add support for .pth files in the path of the run .py file, but they don't seem to have gotten any support. Setuptools works around this by installing an enhancement for the 'site' module that extends .pth support to include all PYTHONPATH directories. The enhancement delegates to the original site module after recording data about sys.path that the site module destroys at startup. I believe that most of the concerns that you have brought up can be addressed, Well, as I said, I've already dealt with them, using .pth files, for the use cases I care about. Ian Bicking and Jim Fulton have also gone farther with work on tools to create environments with greater isolation or more fixed version linkages than what setuptools does. (Setuptools-generated environments dynamically select requirements based on available versions at runtime, while Ian and Jim's tools create environments whose inter-package linkages are frozen at installation time.) and I think that it could be far nicer to deal with than the current sys.path hackery. I'm not sure of that, since I don't yet know how your approach would deal with namespace packages, which are distributed in pieces and assembled later. For example, many PEAK and Zope distributions live in the peak.*
Re: [Python-Dev] New relative import issue
Phillip J. Eby [EMAIL PROTECTED] wrote:
At 12:08 AM 9/22/2006 -0700, Josiah Carlson wrote:
Phillip J. Eby [EMAIL PROTECTED] wrote:
At 08:44 PM 9/21/2006 -0700, Josiah Carlson wrote:
[snip]
You misunderstood me: I mean that the per-user database must be able to
store information for *different Python versions*. Having a single
per-user database without the ability to include configuration for more
than one Python version (analagous to the current situation with the
distutils per-user config file) is problematic.
Just like having different systemwide databases for each Python version
makes sense, why wouldn't we have different user databases for each
Python version? Something like ~/.python_packages.2.6 and
~/.python_packages.3.0
Also, by separating out the files per Python version, we can also
guarantee database compatability for any fixed Python series (2.5.x, etc.).
I don't know if the internal organization of SQLite databases changes
between revisions in a backwards compatible way, so this may not
actually be a concern (it is with bsddb).
In truth, a per-user configuration is just a special case of the real need:
to have per-application environments. In effect, a per-user environment is
a fallback for not having an appplication environment, and the system
environment is a fallback for not having a user environment.
I think you are mostly correct. The reason you are not completely
correct is that if I were to install psyco, and I want all applications
that could use it to use it (they guard the psyco import with a
try/except), I merely need to register the package in the systemwide (or
user) package registery. No need to muck about with each environment I
(or my installed applications) have defined, it just works. Is it a
fallback? Sure, but I prefer to call them convenient defaults.
I didn't mention the following because I thought it would be superfluous,
but it seems that I should have stated it right out. My thoughts were
that on startup, Python would first query the 'system' database, caching
its results in a dictionary, then query the user's listing, updating the
dictionary as necessary, then unload the databases. On demand, when
code runs packages.register(), if both persist and systemwide are False,
it just updates the dictionary. If either are true, it opens up and
updates the relevant database.
Using a database as the primary mechanism for managing import locations
simply isn't workable.
Why? Remember that this database isn't anything other than a
persistance mechanism that has pre-built locking semantics for
multi-process opening, reading, writing, and closing. Given proper
cross-platform locking, we could use any persistance mechanism as a
replacement; miniconf, Pickle, marshal; whatever.
You might as well suggest that each environment
consist of a single large zipfile containing the packages in question: this
would actually be *more* practical (and fast!) in terms of Python startup,
and is no different from having a database with respect to the need for
installation and uninstallation to modify a central file!
We should remember that the sizes of databases that (I expect) will be
common, we are talking about maybe 30k if a user has installed every
package in pypi. And after the initial query, everything will be stored
in a dictionary or dictionary-like object, offering faster query times
than even a zip file (though loading the module/package from disk won't
have its performance improved).
I'm not proposing we do that -- I'm just pointing out why using an actual
database isn't really workable, considering that it has all of the
disadvantages of a big zipfile, and none of the advantages (like speed,
having code already written that supports it, etc.)
SQLite is pretty fast. And for startup, we are really only performing a
single query per database SELECT * FROM package_registry. It will end
up reading the entire database, but these databases will be generally
small, perhaps a few dozen rows, maybe a few thousand if we have set up
a bunch of installation-time application environments.
This is easily remedied with a proper 'packages' implementation:
python -Mpackages name path
Note that Python could auto-insert standard library and site-packages
'packages' on startup (creating the initial dictionary, then the
systemwide, then the user, ...).
I presume here you're suggesting a way to select a runtime environment from
the command line, which would certainly be a good idea.
Actually, I'm offering a way of *registering* a package with the
repository from the command line. I'm of the opinion that setting the
environment via command line for the subsequent Python runs is a bad
idea, but then again, I have been using wxPython's wxversion method for
a while to select which wxPython installation I want to use, and find
things like:
import wxversion
wxversion.ensureMinimal('2.6-unicode',
Re: [Python-Dev] New relative import issue
At 12:42 PM 9/22/2006 -0700, Josiah Carlson wrote:
You might as well suggest that each environment
consist of a single large zipfile containing the packages in question:
this
would actually be *more* practical (and fast!) in terms of Python startup,
and is no different from having a database with respect to the need for
installation and uninstallation to modify a central file!
We should remember that the sizes of databases that (I expect) will be
common, we are talking about maybe 30k if a user has installed every
package in pypi. And after the initial query, everything will be stored
in a dictionary or dictionary-like object, offering faster query times
than even a zip file
Measure it. Be sure to include the time to import SQLite vs. the time to
import the zipimport module.
SQLite is pretty fast. And for startup, we are really only performing a
single query per database SELECT * FROM package_registry. It will end
up reading the entire database, but these databases will be generally
small, perhaps a few dozen rows, maybe a few thousand if we have set up
a bunch of installation-time application environments.
Again, seriously, compare this against a zipfile. You'll find that there's
absolutely no comparison between reading this and reading a zipfile central
directory -- which also results in an in-memory cache that can then be used
to seek() directly to the module.
Actually, I'm offering a way of *registering* a package with the
repository from the command line. I'm of the opinion that setting the
environment via command line for the subsequent Python runs is a bad
idea, but then again, I have been using wxPython's wxversion method for
a while to select which wxPython installation I want to use, and find
things like:
import wxversion
wxversion.ensureMinimal('2.6-unicode', optionsRequired=True)
To be exactly the amount of control I want, where I want it.
Well, that's already easy to do for arbitrary packages and arbitrary
versions with setuptools. Eggs installed in multi-version mode are added
to sys.path at runtime if/when they are requested.
With a package registry (perhaps as I have been describing, perhaps
something different), all of the disparate ways of choosing a version of
a library during import can be removed in favor of a single mechanism.
This single mechanism could handle things like the wxPython
'ensureMinimal', perhaps even 'ensure exact' or 'use latest'.
This discussion is mostly making me realize that sys.path is exactly the
right thing to have, and that the only thing that actually need fixing is
universal .pth support, and maybe some utility functions for better
sys.path manipulation within .pth files. I suggest that there is no way an
arbitrary registry implementation is going to be faster than reading
lines from a text file.
Setuptools works around this by installing an enhancement for the 'site'
module that extends .pth support to include all PYTHONPATH
directories. The enhancement delegates to the original site module after
recording data about sys.path that the site module destroys at startup.
But wasn't there a recent discussion describing how keeping persistant
environment variables is a PITA both during install and runtime?
Yes, exactly.
Extending .pth files to PYTHONPATH seems to me like a hack meant to work
around the fact that Python doesn't have a package registry. And really,
all of the current sys.path + .pth + PYTHONPATH stuff could be subsumed
into a *single* mechanism.
Sure -- I suggest that the single mechanism is none other than
*sys.path*. The .pth files, PYTHONPATH, and a new command-line option
merely being ways to set it.
All of the discussion that's taken place here has sufficed at this point to
convince me that sys.path isn't broken at all, and doesn't need
fixing. Some tweaks to 'site' and maybe a new command-line option will
suffice to clean everything up quite nicely.
I say this because all of the version and dependency management things that
people are asking about can already be achieved by setuptools, so clearly
the underlying machinery is fine. It wasn't until this message of yours
that I realized that you are trying to solve a bunch of problems that are
quite solvable within the existing machinery. I was mainly interested in
cleaning up the final awkwardness that's effectively caused by lack of .pth
support for the startup script directory.
I'm not sure of that, since I don't yet know how your approach would deal
with namespace packages, which are distributed in pieces and assembled
later. For example, many PEAK and Zope distributions live in the peak.*
and zope.* package namespaces, but are installed separately, and glued
together via __path__ changes (see the pkgutil docs).
packages.register('zope', '/path/to/zope')
And if the installation path is different:
packages.register('zope.subpackage', '/different/path/to/subpackage/')
Otherwise the
Re: [Python-Dev] New relative import issue
Phillip J. Eby [EMAIL PROTECTED] wrote:
At 12:42 PM 9/22/2006 -0700, Josiah Carlson wrote:
[snip]
Measure it. Be sure to include the time to import SQLite vs. the time to
import the zipimport module.
[snip]
Again, seriously, compare this against a zipfile. You'll find that there's
absolutely no comparison between reading this and reading a zipfile central
directory -- which also results in an in-memory cache that can then be used
to seek() directly to the module.
They are not directly comparable. The registry of packages can do more
than zipimport in terms of package naming and hierarchy, but it's not an
importer; it's a conceptual replacement of sys.path. I have already
stated that the actual imports from this registry won't be any faster,
as it will still need to read modules/packages from disk *after* it has
decided on a list of paths to check for the package/module. Further,
whether we use SQLite, or any one of a number of other persistance
mechanisms, such a choice should depend on a few things (speed being one
of them, though maybe not the *only* consideration). Perhaps even a zip
file whose 'files' are named with the desired package hierarchy, and
whose contents are something like:
import imp
globals.update(imp.load_XXX(...).__dict__)
del imp
Actually, I'm offering a way of *registering* a package with the
repository from the command line. I'm of the opinion that setting the
environment via command line for the subsequent Python runs is a bad
idea, but then again, I have been using wxPython's wxversion method for
a while to select which wxPython installation I want to use, and find
things like:
import wxversion
wxversion.ensureMinimal('2.6-unicode', optionsRequired=True)
To be exactly the amount of control I want, where I want it.
Well, that's already easy to do for arbitrary packages and arbitrary
versions with setuptools. Eggs installed in multi-version mode are added
to sys.path at runtime if/when they are requested.
Why do we have to use eggs or setuptools to get a feature that
*arguably* should have existed a decade ago in core Python?
The core functionality I'm talking about is:
packages.register(name, path, env=None, system=False, persist=False)
#system==True implies persist==True
packages.copy_env(fr_env, to_env)
packages.use_env(env)
packages.check(name, version=None)
packages.use(name, version)
With those 5 functions and a few tricks, we can replace all user-level .pth
and PYTHONPATH use, and sys.path manipulation done in other 3rd party
packages (setuptools, etc.) are easily handled and supported.
With a package registry (perhaps as I have been describing, perhaps
something different), all of the disparate ways of choosing a version of
a library during import can be removed in favor of a single mechanism.
This single mechanism could handle things like the wxPython
'ensureMinimal', perhaps even 'ensure exact' or 'use latest'.
This discussion is mostly making me realize that sys.path is exactly the
right thing to have, and that the only thing that actually need fixing is
universal .pth support, and maybe some utility functions for better
sys.path manipulation within .pth files. I suggest that there is no way an
arbitrary registry implementation is going to be faster than reading
lines from a text file.
Setuptools works around this by installing an enhancement for the 'site'
module that extends .pth support to include all PYTHONPATH
directories. The enhancement delegates to the original site module after
recording data about sys.path that the site module destroys at startup.
But wasn't there a recent discussion describing how keeping persistant
environment variables is a PITA both during install and runtime?
Yes, exactly.
You have confused me, because not only have you just said we use
PYTHONPATH as a solution, but you have just acknowledged that using
PYTHONPATH is not reasonable as a solution. You have also just said
that we need to add features to .pth support so that it is more usable.
So, sys.path is exactly the right thing to have, but we need to add
more features to make it better.
Ok, here's a sample .pth file if we are willing to make it better (in my
opinion):
zope,/path/to/zope,3.2.1,netserver
zope.subpackage,/path/to/subpackage,.1.1,netserver
That's a CSV file with rows defining packages, and columns in order:
package name, path to package, version, and a semicolon-separated list
of environments that this package is available in (a leading semicolon,
or a double semicolon says that it is available when no environment is
specified).
With a base sys.path, a dictionary of environment - packages created
from .pth files, and a simple function, one can generally develop an
applicable sys.path on demand to some choose_environment() call.
This is, effectively, a variant of what I was suggesting, only with
a different persistance
Re: [Python-Dev] New relative import issue
Guido van Rossum wrote: On 9/19/06, Greg Ewing [EMAIL PROTECTED] wrote: I haven't really thought it through in detail. It just seems as though it would be a lot less confusing if you could figure out from static information which module will get imported by a given import statement, instead of having it depend on the history of run-time modifications to sys.path. One such kind of static information is the layout of the filesystem. Eek? If there are two third-party top-level packages A and B, by different third parties, and A depends on B, how should A find B if not via sys.path or something that is sufficiently equivalent as to have the same problems? Surely every site shouldn't be required to install A and B in the same location (or in the same location relative to each other). I sympathize with the problems that exist with the current import mechanism, really, I do. Google feels the pain every day (alas, Google's requirements are a bit unusual, so they alone can't provide much guidance for a solution). But if you combine the various requirements: zip imports, import hooks of various sorts, different permissions for the owners of different packages that must cooperate, versioning issues (Python versions as well as package versions), forwards compatibility, backwards compatibility, ease of development, ease of packaging, ease of installation, supporting the conventions of vastly different platforms, data files mixed in with the source code (sometimes with their own search path), and probably several other requirements that I'm forgetting right now, it's just not an easy problem. But you're the BDFL! You mean to tell me there are some problems you can't solve?!?!?!?!? shocked-and-amazed-ly y'rs - steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Oleg Broytmann wrote: There really shouldn't be any such thing as sys.path -- the view that any given module has of the package namespace should depend only on where it is I do not understand this. Can you show an example? Imagine I have two servers, Linux and FreeBSD, and on Linux python is in /usr/bin, home is /home/phd, on BSD these are /usr/local/bin and /usr/home/phd. I have some modules in site-packages and some modules in $HOME/lib/python. How can I move programs from one server to the other without rewriting them (how can I not to put full paths to modules)? I use PYTHONPATH manipulation - its enough to write a shell script that starts daemons once and use it for many years. How can I do this without sys.path?! My idea (and interpretation of Greg's statement) is that a module/package should be able to live with either relative imports within itself, or fully absolute imports. No sys.path *hackery* should ever be necessary to access modules in sibling namespaces. Either it's an absolute import, or a relative (internal) import. A sibling import is a symptom of wrong design of the packages. This is how I usually design my packages at least. There might be valid use cases for doing sys.path hackery, but I have yet to find them. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/21/06, Giovanni Bajo [EMAIL PROTECTED] wrote: Greg Eqing wrote: There really shouldn't be any such thing as sys.path -- the view that any given module has of the package namespace should depend only on where it is My idea (and interpretation of Greg's statement) is that a module/package should be able to live with either relative imports within itself, or fully absolute imports. No sys.path *hackery* should ever be necessary to access modules in sibling namespaces. Either it's an absolute import, or a relative (internal) import. A sibling import is a symptom of wrong design of the packages. This is how I usually design my packages at least. There might be valid use cases for doing sys.path hackery, but I have yet to find them. While I agree with your idea(l), I don't think that's what Greg meant. He clearly say sys.path should not exist at all. I do think it's fair to use sibling imports (using from ..sibling import module) from inside subpackages of the same package; I couldn't tell if you were against that or not. sys.path exists to stitch together the toplevel module/package namespace from diverse sources. Import hooks and sys.path hackery exist so that module/package sources don't have to be restricted to the filesystem (as well as to allow unbridled experimentation by those so inclined :-). I think one missing feature is a mechanism whereby you can say THIS package (gives top-level package name) lives HERE (gives filesystem location of package) without adding the parent of HERE to sys.path for all module searches. I think Phillip Eby's egg system might benefit from this. Another missing feature is a mechanism whereby you can use a particular file as the main script without adding its directory to the front of sys.path. Other missing features have to do with versioning constraints. But that quickly gets extremely messy. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/21/06, Guido van Rossum [EMAIL PROTECTED] wrote: I think one missing feature is a mechanism whereby you can say THIS package (gives top-level package name) lives HERE (gives filesystem location of package) without adding the parent of HERE to sys.path for all module searches. I think Phillip Eby's egg system might benefit from this. This is pretty easy to do with a custom importer on sys.meta_path. Getting the details right is a touch fiddly, but it's conceptually straightforward. Hmm, I might play with this - a set of PEP 302 importers to completely customise the import mechanism. The never-completed phase 2 of the PEP included a reimplementation of the built in import mechanism as hooks. Is there any interest in this actually happening? I've been looking for an interesting coding project for a while (although I never have any free time...) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/21/06, Guido van Rossum [EMAIL PROTECTED] wrote:
On 9/21/06, Paul Moore [EMAIL PROTECTED] wrote:
On 9/21/06, Guido van Rossum [EMAIL PROTECTED] wrote:
I think one missing feature is a mechanism whereby you can say THIS
package (gives top-level package name) lives HERE (gives filesystem
location of package) without adding the parent of HERE to sys.path
for all module searches. I think Phillip Eby's egg system might
benefit from this.
This is pretty easy to do with a custom importer on sys.meta_path.
Getting the details right is a touch fiddly, but it's conceptually
straightforward.
Isn't the main problem how to specify a bunch of these in the
environment? Or can this be done through .pkg files? Those aren't
cheap either though -- it would be best if the work was only done when
the package is actually needed.
Hmm, I wasn't thinking of the environment. I pretty much never use
PYTHONPATH, so I tend to forget about that aspect. I was assuming an
importer object with a register(package_name, filesystem_path)
method. Then register the packages you want in your code, or in
site.py.
I've attached a trivial proof of concept.
But yes, you'd need to consider the environment. Possibly just have an
initialisation function called at load time (I'm assuming the new hook
is defined in a system module of some sort - I mean when that system
module is loaded) which parses an environment variable and issues a
set of register() calls.
Paul.
import sys
import imp
class Importer(object):
def __init__(self):
self.registered_modules = {}
def register(self, fullname, filename):
self.registered_modules[fullname] = filename
def find_module(self, fullname, path=None):
if path is None:
return Loader(self)
class Loader(object):
def __init__(self, importer):
self.importer = importer
def load_module(self, fullname):
print Trying, fullname
filename = self.importer.registered_modules.get(fullname)
if filename:
print fullname, was registered - load from, filename
# Create a dummy for now, as I don't have time to code the full
# filesystem lookup.
mod = sys.modules.setdefault(fullname, imp.new_module(fullname))
mod.__file__ = %s % self.__class__.__name__
mod.__loader__ = self
mod.__path__ = [filename]
exec pass in mod.__dict__
return mod
importer = Importer()
sys.meta_path.append(importer)
# Put code here to register module/path pairs from an environment variable
# ...
if __name__ == '__main__':
# Trivial test
importer.register(a, C:\\Whatever)
# C:\Whatever needs to contain a subdir b, containing __init__.py and c.py
import a.b.c
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
At 01:54 PM 9/21/2006 -0700, Guido van Rossum wrote: On 9/21/06, Paul Moore [EMAIL PROTECTED] wrote: On 9/21/06, Guido van Rossum [EMAIL PROTECTED] wrote: I think one missing feature is a mechanism whereby you can say THIS package (gives top-level package name) lives HERE (gives filesystem location of package) without adding the parent of HERE to sys.path for all module searches. I think Phillip Eby's egg system might benefit from this. This is pretty easy to do with a custom importer on sys.meta_path. Getting the details right is a touch fiddly, but it's conceptually straightforward. Isn't the main problem how to specify a bunch of these in the environment? Yes, that's exactly the problem, assuming that by environment you mean the operating environment, as opposed to e.g. os.environ. (Environment variables are problematic for installation purposes, as on Unix-y systems there is no one obvious place to set them, and on Windows, the one obvious place is one that the user may have no permissions for!) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/21/06, Phillip J. Eby [EMAIL PROTECTED] wrote: Isn't the main problem how to specify a bunch of these in the environment? Yes, that's exactly the problem, assuming that by environment you mean the operating environment, as opposed to e.g. os.environ. Hmm, now I don't understand again. What operating environment might there be, other than - os.environ - code that gets executed before the import ? There are clearly application design issues involved here (application configuration via initialisation files, plugin registries, etc etc). But in purely technical terms, don't they all boil down to executing a registration function (either directly by the user, or by the application on behalf of the user)? I don't think I'd expect anything at the language (or base library) level beyond a registration function and possibly an OS environment check. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/21/06, Guido van Rossum [EMAIL PROTECTED] wrote: On 9/21/06, Paul Moore [EMAIL PROTECTED] wrote:[SNIP] Hmm, I might play with this - a set of PEP 302 importers to completely customise the import mechanism. The never-completed phase 2 of the PEP included a reimplementation of the built in import mechanism as hooks. Is there any interest in this actually happening? I've been looking for an interesting coding project for a while (although I never have any free time...)There's a general desire to reimplement import entirely in Python formore flexibility. I believe Brett Cannon is working on this.Since I need to control imports to the point of being able to deny importing built-in and extension modules, I was planning on re-implementing the import system to use PEP 302 importers. Possibly do it in pure Python for ease-of-use. Then that can be worked off of for possible Py3K improvements to the import system. But either way I will be messing with the import system in the relatively near future. If you want to help, Paul (or anyone else), just send me an email and we can try to coordinate something (plan to do the work in the sandbox as a separate thing from my security stuff). -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/21/06, Paul Moore [EMAIL PROTECTED] wrote: On 9/21/06, Guido van Rossum [EMAIL PROTECTED] wrote: Isn't the main problem how to specify a bunch of these in the environment? Or can this be done through .pkg files? Those aren't cheap either though -- it would be best if the work was only done when the package is actually needed. Hmm, I wasn't thinking of the environment. I pretty much never use PYTHONPATH, so I tend to forget about that aspect. As Phillip understood, I meant the environment to include the filesystem (and on Windows, the registry -- in fact, Python on Windows *has* exactly such a mechanism in the registry, although I believe it's rarely used these days -- it was done by Mark Hammond to support COM servers I believe.) I was assuming an importer object with a register(package_name, filesystem_path) method. Then register the packages you want in your code, or in site.py. Neither is an acceptable method for an installer tool (e.g. eggs) to register new packages. it needs to be some kind of data file or set of data files. But yes, you'd need to consider the environment. Possibly just have an initialisation function called at load time (I'm assuming the new hook is defined in a system module of some sort - I mean when that system module is loaded) which parses an environment variable and issues a set of register() calls. os.environ is useless because there's no way for a package installer to set it for all users. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Guido van Rossum wrote: Eek? If there are two third-party top-level packages A and B, by different third parties, and A depends on B, how should A find B if not via sys.path or something that is sufficiently equivalent as to have the same problems? Some kind of configuration mechanism is needed, but I don't see why it can't be a static, declarative one rather than computed at run time. Whoever installs package A should be responsible for setting up whatever environment is necessary around it for it to find package B and anything else it directly depends on. The program C which uses package A needs to be told where to find it. But C doesn't need to know where to find B, the dependency on which is an implementation detail of A, because A already knows where to find it. In the Eiffel world, there's a thing called and ACE (Assembly of Classes in Eiffel), which is a kind of meta-language for describing the shape of the class namespace, and it allows each source file to have its own unique view of that namespace. I'm groping towards something equivalent for the Python module namespace. (AMP - Assembly of Modules in Python?) -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Another thought on static module namespace configuration: It would make things a *lot* easier for py2exe, py2app and the like that have to figure out what packages a program depends on without running the program. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
At 12:07 PM 9/22/2006 +1200, Greg Ewing wrote: Another thought on static module namespace configuration: It would make things a *lot* easier for py2exe, py2app and the like that have to figure out what packages a program depends on without running the program. Setuptools users already explicitly declare what projects their projects depend on; this is how easy_install can then find and install those dependencies. So, there is at least one system already available for Python that manages this type of thing already, and my understanding is that the py2exe and py2app developers plan to support using this dependency information in the future. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/21/06, Greg Ewing [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Eek? If there are two third-party top-level packages A and B, by different third parties, and A depends on B, how should A find B if not via sys.path or something that is sufficiently equivalent as to have the same problems? Some kind of configuration mechanism is needed, but I don't see why it can't be a static, declarative one rather than computed at run time. That would preclude writing the code that interprets the static data in Python itself. Despite the good use cases I think that's a big showstopper. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Giovanni Bajo wrote: My idea (and interpretation of Greg's statement) is that a module/package should be able to live with either relative imports within itself, or fully absolute imports. I think it goes further than that -- each module should (potentially) have its own unique view of the module namespace, defined at the time the module is installed, that can't be disturbed by anything that any other module does. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/21/06, Greg Ewing [EMAIL PROTECTED] wrote: Giovanni Bajo wrote: My idea (and interpretation of Greg's statement) is that a module/package should be able to live with either relative imports within itself, or fully absolute imports. I think it goes further than that -- each module should (potentially) have its own unique view of the module namespace, defined at the time the module is installed, that can't be disturbed by anything that any other module does. Well, maybe. But there's also the requirement that if packages A and B both import C, they should get the same C. Having multiple versions of the same package loaded simultaneously sounds like a recipe for disaster. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Guido van Rossum wrote: While I agree with your idea(l), I don't think that's what Greg meant. He clearly say sys.path should not exist at all. Refining that a bit, I don't think there should be a *single* sys.path for the whole program -- more like each module having its own sys.path. And, at least in most cases, its contents should be set up from static information that exists outside the program, established when the module is installed. One reason for this is the lack of any absolute notion of a program. What is a program on one level can be part of a larger program on another level. For example, a module with test code that is run when it's invoked as a main script. Sometimes it's a program of its own, other times it's not. And it shouldn't *matter* whether it's a program or not when it comes to being able to find other modules that it needs to import. So using a piece of program-wide shared state for this seems wrong. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Guido van Rossum wrote: On 9/21/06, Greg Ewing [EMAIL PROTECTED] wrote: I think it goes further than that -- each module should (potentially) have its own unique view of the module namespace, defined at the time the module is installed, that can't be disturbed by anything that any other module does. Well, maybe. But there's also the requirement that if packages A and B both import C, they should get the same C. Having multiple versions of the same package loaded simultaneously sounds like a recipe for disaster. I have exactly this scenario in my current codebase for a server. It was absolutely necessary for me to update a module in Twisted because all other solutions I could come up with were less desirable. Either I send my patch upstream and wait (can't wait), or I fork out another version and place it at the top of sys.path (this seems ok). Except an even better solution is to maintain my own subset of Twisted, because I am localized to a particularly small corner of the codebase. I can continue to use upstream updates to the rest of Twisted without any fussing about merging changes and so forth. And if Twisted was allowed to decide how it saw its own world, then I would have to go back to maintaining my own complete branch. While I don't strictly need to be able to do this, I wanted to at least raise my hand and say, I abuse this facet of the current import mechanism. -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Guido writes: As Phillip understood, I meant the environment to include the filesystem (and on Windows, the registry -- in fact, Python on Windows *has* exactly such a mechanism in the registry, although I believe it's rarely used these days -- it was done by Mark Hammond to support COM servers I believe.) It is rarely used these days due to the fact it is truly global to the machine. These days, it is not uncommon to have multiple copies of the same Python version installed on the same box - generally installed privately into an application by the vendor - eg, Komodo and Open Office both do this to some degree. The problem with a global registry is that *all* Python installations honoured it. This meant bugs in the vendor applications, as their 'import foo' did *not* locate the foo module inside their directory, but instead loaded a completely unrelated one, which promptly crashed. A mechanism similar to .pth files, where the declaration of a module's location is stored privately to the installation seems a more workable approach. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Greg Ewing [EMAIL PROTECTED] wrote:
Guido van Rossum wrote:
Eek? If there are two third-party top-level packages A and B, by
different third parties, and A depends on B, how should A find B if
not via sys.path or something that is sufficiently equivalent as to
have the same problems?
Some kind of configuration mechanism is needed, but
I don't see why it can't be a static, declarative one
rather than computed at run time.
I could be missing something, or be completely off-topic, but why not
both, or really a mechanism to define:
1. Installation time package location (register package X in the package
registry at path Y and persist across Python sessions).
2. Runtime package location (package X is in path Y, do not persist
across Python sessions).
With 1 and 2, we remove the need for .pth files, all packages to be
installed into Lib/site-packages, and sys.path manipulation. You want
access to package X?
packages.register('X', '~/mypackages/X')
packages.register('X', '~/mypackages/X', persist=True)
packages.register('X', '~/mypackages/X', systemwide=True)
This can be implemented with a fairly simple package registry, contained
within a (small) SQLite database (which is conveniently shipped in
Python 2.5). There can be a system-wide database that all users use as
a base, with a user-defined package registry (per user) where the
system-wide packages can be augmented.
With a little work, it could even be possible to define importers during
registration (filesystem, zip, database, etc.) or include a tracing
mechanism for py2exe/distutils/py2app/cx_freeze/etc. (optionally writing
to a simplified database-like file for distribution so that SQLite
doesn't need to be shipped).
- Josiah
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
At 08:44 PM 9/21/2006 -0700, Josiah Carlson wrote: This can be implemented with a fairly simple package registry, contained within a (small) SQLite database (which is conveniently shipped in Python 2.5). There can be a system-wide database that all users use as a base, with a user-defined package registry (per user) where the system-wide packages can be augmented. As far as I can tell, you're ignoring that per-user must *also* be per-version, and per-application. Each application or runtime environment needs its own private set of information like this. Next, putting the installation data inside a database instead of per-installation-unit files presents problems of its own. While some system packaging tools allow install/uninstall scripts to run, they are often frowned upon, and can be unintentionally bypassed. These are just a few of the issues that come to mind. Realistically speaking, .pth files are currently the most effective mechanism we have, and there actually isn't much that can be done to improve upon them. What's more needed are better mechanisms for creating and managing Python environments (to use a term coined by Ian Bicking and Jim Fulton over on the distutils-sig), which are individual contexts in which Python applications run. Some current tools in development by Ian and Jim include: http://cheeseshop.python.org/pypi/workingenv.py/ http://cheeseshop.python.org/pypi/zc.buildout/ I don't know that much about either, but I do know that for zc.buildout, Jim's goal was to be able to install Python scripts with pre-baked sys.path based on package dependencies from setuptools, and as far as I know, he achieved it. Anyway, system-wide and per-user environment information isn't nearly sufficient to address the issues that people have when developing and deploying multiple applications on a server, or even using multiple applications on a client installation (e.g. somebody using both the Enthought Python IDE and Chandler on the same machine). These relatively simple use cases rapidly demonstrate the inadequacy of system-wide or per-user configuration of what packages are available. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Paul Moore wrote: On 9/21/06, Guido van Rossum [EMAIL PROTECTED] wrote: I think one missing feature is a mechanism whereby you can say THIS package (gives top-level package name) lives HERE (gives filesystem location of package) without adding the parent of HERE to sys.path for all module searches. I think Phillip Eby's egg system might benefit from this. This is pretty easy to do with a custom importer on sys.meta_path. Getting the details right is a touch fiddly, but it's conceptually straightforward. Hmm, I might play with this - a set of PEP 302 importers to completely customise the import mechanism. The never-completed phase 2 of the PEP included a reimplementation of the built in import mechanism as hooks. Is there any interest in this actually happening? I've been looking for an interesting coding project for a while (although I never have any free time...) My interest in such a project would be in replacing a bunch of legacy C code with varying implementations of the import mechanism in pure Python strictly according to the dictats of PEP 302, using sys.path_hooks and sys.path (meta_path is for future consideration ;-). Some readers may remember a lightning talk I gave at OSCON about three years ago. In that I discussed a database structure that allowed different implementations of modules to be loaded according to compatibility requirements established as a result of testing. Although I now have a working database import mechanism based on PEP 302 it's by no means obvious how that can be used exclusively (in other words, replacing the current import mechanism: the present implementation relies on an import of MySQLdb, which has many dependencies that clearly must be importable before the DB mechanism is in place). And I certainly haven't followed up by establishing the compatibility data that such an implementation would require. Has anyone done any work on (for example) zip-only distributions? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/19/06, Greg Ewing [EMAIL PROTECTED] wrote: I haven't really thought it through in detail. It just seems as though it would be a lot less confusing if you could figure out from static information which module will get imported by a given import statement, instead of having it depend on the history of run-time modifications to sys.path. One such kind of static information is the layout of the filesystem. Eek? If there are two third-party top-level packages A and B, by different third parties, and A depends on B, how should A find B if not via sys.path or something that is sufficiently equivalent as to have the same problems? Surely every site shouldn't be required to install A and B in the same location (or in the same location relative to each other). I sympathize with the problems that exist with the current import mechanism, really, I do. Google feels the pain every day (alas, Google's requirements are a bit unusual, so they alone can't provide much guidance for a solution). But if you combine the various requirements: zip imports, import hooks of various sorts, different permissions for the owners of different packages that must cooperate, versioning issues (Python versions as well as package versions), forwards compatibility, backwards compatibility, ease of development, ease of packaging, ease of installation, supporting the conventions of vastly different platforms, data files mixed in with the source code (sometimes with their own search path), and probably several other requirements that I'm forgetting right now, it's just not an easy problem. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On Tue, Sep 19, 2006 at 03:46:59PM +1200, Greg Ewing wrote: There really shouldn't be any such thing as sys.path -- the view that any given module has of the package namespace should depend only on where it is I do not understand this. Can you show an example? Imagine I have two servers, Linux and FreeBSD, and on Linux python is in /usr/bin, home is /home/phd, on BSD these are /usr/local/bin and /usr/home/phd. I have some modules in site-packages and some modules in $HOME/lib/python. How can I move programs from one server to the other without rewriting them (how can I not to put full paths to modules)? I use PYTHONPATH manipulation - its enough to write a shell script that starts daemons once and use it for many years. How can I do this without sys.path?! Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Josiah Carlson wrote: As it stands, in order to work around this particular feature, one would need to write a 'loader' to handle importing and/or main() calling in subpackage1/module1.py . Yup. At the moment, you can rely on PEP 328, or an PEP 338, but not both at the same time. This was previously discussed back in June/July with Anthony convincing me that the solution to the current poor interaction shouldn't be rushed [1]. It is, however, pretty trivial to write a runpy.run_module based launcher that will execute your module and use something other than __name__ == '__main__' to indicate that the module is the main module. By letting run_module set __name__ normally, relative imports will just work. For example: #mypkg/launch.py # Runs a script, using the global _launched to indicate whether or not # the module is the main module if _launched not in globals(): _launched = False if (__name__ == __main__) or _launched: import runpy # Run the module specified as the next command line argument if len(sys.argv) 2: print sys.stderr, No module specified for execution else: del sys.argv[0] # Make the requested module sys.argv[0] run_module(sys.argv[0], init_globals=dict(_launched=True), alter_sys=True) Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2006-July/067077.html -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Greg Ewing wrote: Armin Rigo wrote: My (limited) understanding of the motivation for relative imports is that they are only here as a transitional feature. Fully-absolute imports are the official future. Guido does seem to have a dislike for relative imports, but I don't really understand why. The usefulness of being able to make a package self-contained and movable to another place in the package hierarchy without hacking it seems self-evident to me. What's happening in Py3k? Will relative imports still exist? there is no clean way from a test module 'foo.bar.test.test_hello' to import 'foo.bar.hello': the top-level directory must first be inserted into sys.path magically. I've felt for a long time that problems like this wouldn't arise so much if there were a closer connection between the package hierarchy and the file system structure. There really shouldn't be any such thing as sys.path -- the view that any given module has of the package namespace should depend only on where it is, not on the history of how it came to be invoked. This does, of course, assume that you're importing modules from the filestore, which assumption is no longer valid in the presence of PEP 302 importers. The current initialization code actually looks for os.py as a means of establishing path elements. This should really be better integrated with the PEP 302 mechanism: ideally Python should work on systems that don't rely on filestore for import (even though for the foreseeable future all systems will continue to do this). regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Steve Holden wrote: This does, of course, assume that you're importing modules from the filestore, which assumption is no longer valid in the presence of PEP 302 importers. Well, you need to allow for a sufficiently abstract notion of filesystem. I haven't really thought it through in detail. It just seems as though it would be a lot less confusing if you could figure out from static information which module will get imported by a given import statement, instead of having it depend on the history of run-time modifications to sys.path. One such kind of static information is the layout of the filesystem. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Greg Ewing wrote: Steve Holden wrote: This does, of course, assume that you're importing modules from the filestore, which assumption is no longer valid in the presence of PEP 302 importers. Well, you need to allow for a sufficiently abstract notion of filesystem. For some value of sufficiently ... I haven't really thought it through in detail. It just seems as though it would be a lot less confusing if you could figure out from static information which module will get imported by a given import statement, instead of having it depend on the history of run-time modifications to sys.path. One such kind of static information is the layout of the filesystem. Less confusing, but sadly also less realistic. I suspect what's really needed is *more* importer behavior rather than less but, like you, I haven't yet thought it through in detail. All I *can* tell you is once you start importing modules for a database the whole import mechanism starts to look a bit under-specified an over-complicated. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Hi Fabio, On Sun, Sep 17, 2006 at 03:38:42PM -0300, Fabio Zadrozny wrote: I've been playing with the new features and there's one thing about the new relative import that I find a little strange and I'm not sure this was intended... My (limited) understanding of the motivation for relative imports is that they are only here as a transitional feature. Fully-absolute imports are the official future. Neither relative nor fully-absolute imports address the fact that in any multi-package project I've been involved with, there is some kind of sys.path hackery required (or even custom import hooks). Indeed, there is no clean way from a test module 'foo.bar.test.test_hello' to import 'foo.bar.hello': the top-level directory must first be inserted into sys.path magically. /foo/bar/imp1.py -- has a from . import imp2 /foo/bar/imp2.py if I now put a test-case (or any other module I'd like as the main module) at: /foo/bar/mytest.py if it imports imp1, it will always fail. Indeed: foo/bar/mytest.py must do 'import foo.bar.imp1' or 'from foo.bar import imp1', and then it works (if sys.path was properly hacked first, of course). (I'm not sure, but I think that this not so much a language design decision as a consequence of the complexities of import.c, which is the largest C source file of CPython and steadily growing.) A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Fabio Zadrozny wrote: I've been playing with the new features and there's one thing about the new relative import that I find a little strange and I'm not sure this was intended... When you do a from . import xxx, it will always fail if you're in a top-level module, and when executing any module, the directory of the module will automatically go into the pythonpath, thus making all the relative imports in that structure fail. Correct. Relative imports are based on __name__ and don't work properly if __name__ does not properly reflect the module's position in the package hierarchy (usually because the module is the main module, so name is set to '__main__'). This is noted briefly in PEP 328 [1], with the current workarounds explained in more detail in PEP 338 [2]. Cheers, Nick. [1] http://www.python.org/dev/peps/pep-0328/#relative-imports-and-name [2] http://www.python.org/dev/peps/pep-0338/#import-statements-and-the-main-module -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Armin Rigo wrote: My (limited) understanding of the motivation for relative imports is that they are only here as a transitional feature. Fully-absolute imports are the official future. Guido does seem to have a dislike for relative imports, but I don't really understand why. The usefulness of being able to make a package self-contained and movable to another place in the package hierarchy without hacking it seems self-evident to me. What's happening in Py3k? Will relative imports still exist? there is no clean way from a test module 'foo.bar.test.test_hello' to import 'foo.bar.hello': the top-level directory must first be inserted into sys.path magically. I've felt for a long time that problems like this wouldn't arise so much if there were a closer connection between the package hierarchy and the file system structure. There really shouldn't be any such thing as sys.path -- the view that any given module has of the package namespace should depend only on where it is, not on the history of how it came to be invoked. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Greg Ewing [EMAIL PROTECTED] wrote: Armin Rigo wrote: there is no clean way from a test module 'foo.bar.test.test_hello' to import 'foo.bar.hello': the top-level directory must first be inserted into sys.path magically. I've felt for a long time that problems like this wouldn't arise so much if there were a closer connection between the package hierarchy and the file system structure. There really shouldn't be any such thing as sys.path -- the view that any given module has of the package namespace should depend only on where it is, not on the history of how it came to be invoked. Wait, wait, wait. If I remember correctly, one of the use-cases cited was for sub-packages of a single larger package to be able to import other sub-packages, via 'from ..subpackage2 import module2'. That is to say, given a package structure like... .../__init__.py .../subpackage1/module1.py .../subpackage1/__init__.py .../subpackage2/module2.py .../subpackage2/__init__.py Running module1.py, with an import line that read: from ..subpackage2 import module2 ... would import module2 from subpackage2 Testing this in the beta I have installed tells me: Traceback (most recent call last): File module1.py, line 1, in module from ..subpackage2 import module2 ValueError: Relative importpath too deep While I can understand why this is the case (if one is going to be naming modules relative to __main__ or otherwise, unless one preserves the number of leading '.', giving module2 a __name__ of __main__..subpackage2.module2 or ..subpackage2.module2, naming can be confusing), it does remove a very important feature. Guido suggested I make up a PEP way back in March or so, but I was slowed by actually implementing __main__-relative naming (which is currently incomplete). As it stands, in order to work around this particular feature, one would need to write a 'loader' to handle importing and/or main() calling in subpackage1/module1.py . - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On 9/18/06, Greg Ewing [EMAIL PROTECTED] wrote: Armin Rigo wrote: My (limited) understanding of the motivation for relative imports is that they are only here as a transitional feature.Fully-absolute imports are the official future. Guido does seem to have a dislike for relative imports,but I don't really understand why. The usefulness ofbeing able to make a package self-contained and movableto another place in the package hierarchy without hacking it seems self-evident to me.It is more of how relative imports used to be inherent and thus have no clear way to delineate that an import was being done using a relative path compared to an absolute one. What's happening in Py3k? Will relative imports stillexist? Using the dot notation, yes they will exist in Py3K. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] New relative import issue
I've been playing with the new features and there's one thing about the new relative import that I find a little strange and I'm not sure this was intended... When you do a from . import xxx, it will always fail if you're in a top-level module, and when executing any module, the directory of the module will automatically go into the pythonpath, thus making all the relative imports in that structure fail. E.g.: /foo/bar/imp1.py -- has a from . import imp2 /foo/bar/imp2.py if I now put a test-case (or any other module I'd like as the main module) at: /foo/bar/mytest.py if it imports imp1, it will always fail. The solutions I see would be: - only use the pythonpath actually defined by the user (and don't put the current directory in the pythonpath) - make relative imports work even if they reach some directory in the pythonpath (making it work as an absolute import that would only search the current directory structure) Or is this actually a bug? (I'm with python 2.5 rc2) I took another look at http://docs.python.org/dev/whatsnew/pep-328.html and the example shows: pkg/ pkg/__init__.py pkg/main.py pkg/string.py with the main.py doing a from . import string, which is what I was trying to accomplish... Cheers, Fabio ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
