[Distutils] Importable wheels using distlib/distil
> I'm not top-posting, but trying to keep GMane happy ...
Since wheels are .zip files, they can sometimes be used to provide
functionality without needing to be installed. Whereas .zip files contain no
convention for indicating compatibility with a particular Python, wheels do
contain this compatibility information. Thus, it is possible to check if a
wheel can be directly imported from, and the wheel support in distlib allows
you to take advantage of this using the mount() and unmount() methods. When you
mount a wheel, its absolute path name is added to sys.path, allowing the Python
code in it to be imported. (A DistlibException is raised if the wheel isn't
compatible with the Python which calls the mount() method.)
You don't need mount() just to add the wheel's name to sys.path, or to import
pure-Python wheels. The mount() method goes further than just enabling Python
imports - any C extensions in the wheel are also made available for import. For
this to be possible, the wheel has to be built with additional metadata about
extensions - a JSON file called EXTENSIONS which serialises an extension
mapping dictionary. This maps extension module names to the names in the wheel
of the shared libraries which implement those modules.
Running unmount() on the wheel removes its absolute pathname from sys.path and
makes its C extensions, if any, also unavailable for import.
Wheels built with distil contain the EXTENSIONS metadata, so can be mounted
complete with C extensions:
$ distil download -d /tmp simplejson
Downloading simplejson-3.1.2.tar.gz to /tmp/simplejson-3.1.2
63KB @ 73 KB/s 100 % Done: 00:00:00
Unpacking ... done.
$ distil package --fo=wh -d /tmp /tmp/simplejson-3.1.2/
The following packages were built:
/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl
$ python
Python 2.7.2+ (default, Jul 20 2012, 22:15:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distlib.wheel import Wheel
>>> w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
>>> w.mount()
>>> import simplejson._speedups
>>> dir(simplejson._speedups)
['__doc__', '__file__', '__loader__', '__name__', '__package__',
'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring']
>>> simplejson._speedups.__file__
'/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so'
>>>
This, IMO, makes the wheel format more useful than it already is :-)
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Wed, Mar 27, 2013 at 3:21 PM, Vinay Sajip wrote:
>> I'm not top-posting, but trying to keep GMane happy ...
>
> Since wheels are .zip files, they can sometimes be used to provide
> functionality without needing to be installed. Whereas .zip files contain no
> convention for indicating compatibility with a particular Python, wheels do
> contain this compatibility information. Thus, it is possible to check if a
> wheel can be directly imported from, and the wheel support in distlib allows
> you to take advantage of this using the mount() and unmount() methods. When
> you
> mount a wheel, its absolute path name is added to sys.path, allowing the
> Python
> code in it to be imported. (A DistlibException is raised if the wheel isn't
> compatible with the Python which calls the mount() method.)
>
> You don't need mount() just to add the wheel's name to sys.path, or to import
> pure-Python wheels. The mount() method goes further than just enabling Python
> imports - any C extensions in the wheel are also made available for import.
> For
> this to be possible, the wheel has to be built with additional metadata about
> extensions - a JSON file called EXTENSIONS which serialises an extension
> mapping dictionary. This maps extension module names to the names in the wheel
> of the shared libraries which implement those modules.
>
> Running unmount() on the wheel removes its absolute pathname from sys.path and
> makes its C extensions, if any, also unavailable for import.
>
> Wheels built with distil contain the EXTENSIONS metadata, so can be mounted
> complete with C extensions:
>
> $ distil download -d /tmp simplejson
> Downloading simplejson-3.1.2.tar.gz to /tmp/simplejson-3.1.2
> 63KB @ 73 KB/s 100 % Done: 00:00:00
> Unpacking ... done.
> $ distil package --fo=wh -d /tmp /tmp/simplejson-3.1.2/
> The following packages were built:
> /tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl
> $ python
> Python 2.7.2+ (default, Jul 20 2012, 22:15:08)
> [GCC 4.6.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
from distlib.wheel import Wheel
w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
w.mount()
import simplejson._speedups
dir(simplejson._speedups)
> ['__doc__', '__file__', '__loader__', '__name__', '__package__',
> 'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring']
simplejson._speedups.__file__
> '/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so'
>
> This, IMO, makes the wheel format more useful than it already is :-)
It's a trap!
At least on Unix systems:
- Extensions in zip files that get magically extracted to a user's
home directory lead to tragic deployment failures for services that
run as special users.
- Zip files are a pain in the ass during development or debugging.
- Zip files are slower to import from (at least in my experience)
It would be far better IMO to just unzip the wheel and put that in
your path. (I'm hoping that wheels used this way are a suitable
replacement for eggs.)
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Jim Fulton zope.com> writes: > It's a trap! > > At least on Unix systems: > > - Extensions in zip files that get magically extracted to a user's > home directory lead to tragic deployment failures for services that > run as special users. I can see how it would lead to problems, but the home directory location is just as a proof of concept - the cache doesn't need to be in any private place. > - Zip files are a pain in the ass during development or debugging. Of course, but wheels are for deployment, not development, and this is one possibility for deployment (several people have mentioned wanting to sometimes just add wheels to sys.path rather than installing them, which got me thinking about this functionality). > - Zip files are slower to import from (at least in my experience) It's just another option for a user of wheels. Caveat emptor, and all that. > It would be far better IMO to just unzip the wheel and put that in > your path. (I'm hoping that wheels used this way are a suitable > replacement for eggs.) Well that's tantamount to installing the wheel, which is fine. I was thinking along the line of egg replacement - AFAIK eggs allow you to import extensions from zip in a similar fashion. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Wed, Mar 27, 2013 at 4:50 PM, Vinay Sajip wrote: > Jim Fulton zope.com> writes: > >> It's a trap! >> >> At least on Unix systems: >> >> - Extensions in zip files that get magically extracted to a user's >> home directory lead to tragic deployment failures for services that >> run as special users. > > I can see how it would lead to problems, but the home directory location is > just as a proof of concept - the cache doesn't need to be in any private > place. Anywhere you extract them is likely going to lead to access control or security issues and generally cause pain, IMO. > >> - Zip files are a pain in the ass during development or debugging. > > Of course, but wheels are for deployment, not development, and this is one > possibility for deployment (several people have mentioned wanting to sometimes > just add wheels to sys.path rather than installing them, which got me thinking > about this functionality). I expect to use wheels during development just like I use eggs now. Not for development of the wheel/egg, but for development of something that uses it. You're in pdb and you land in a zipped egg/wheel that the package under development invoked and now you're screwed. > >> - Zip files are slower to import from (at least in my experience) > > It's just another option for a user of wheels. Caveat emptor, and all that. It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment. I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony. > >> It would be far better IMO to just unzip the wheel and put that in >> your path. (I'm hoping that wheels used this way are a suitable >> replacement for eggs.) > > Well that's tantamount to installing the wheel, Not really. If you just unzip the wheel and add it to your path, you can stop using it by just removing from your path. If you install the wheel, it's contents will be poured into site-packages (and other places).It's much heavier than just adding the wheel (zipped or unzipped) to your path. > which is fine. I was thinking > along the line of egg replacement Me too. > - AFAIK eggs allow you to import extensions > from zip in a similar fashion. Importing from zipped eggs has proved itself to be an anti pattern. Buildout (as of buildout 2) always unzips eggs. It can then generate scripts with just the packages they need by adding (unzipped) eggs to sys.path. Various plugin systems (including buildout itself with extensions and recipes) do this dynamically at run time. It's very useful. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Jim Fulton writes: > Anywhere you extract them is likely going to lead to access control > or security issues and generally cause pain, IMO. right! search the web for PYTHON_EGG_CACHE. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On 27 March 2013 21:06, Jim Fulton wrote: >> - AFAIK eggs allow you to import extensions >> from zip in a similar fashion. > > Importing from zipped eggs has proved itself to be an > anti pattern. I don't like the idea of making wheels work like eggs in this respect. As Jim said, (zipped) eggs have a very bad reputation and associating wheels with that type of functionality would be a very bad idea. Wheels are, and should remain, a binary installer format. On the other hand, zipimport is a very cool feature, and seriously under-used. But it has specific benefits and limitations, and in particular zipimport does not support binary extensions for very good reasons. Zip files on sys.path are practical for pure Python code only, IMO. Having said all that, the fact that wheels are zipfiles, and can be used on sys.path, *can* be useful. But it's an incidental benefit and *not* a core feature. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Jim Fulton zope.com> writes: > >> It would be far better IMO to just unzip the wheel and put that in > >> your path. (I'm hoping that wheels used this way are a suitable > >> replacement for eggs.) > > > > Well that's tantamount to installing the wheel, > > Not really. If you just unzip the wheel and add it to your path, > you can stop using it by just removing from your path. If you > install the wheel, it's contents will be poured into site-packages > (and other places).It's much heavier than just adding the > wheel (zipped or unzipped) to your path. > [snip] > by adding (unzipped) eggs to sys.path. Various plugin > systems (including buildout itself with extensions and recipes) > do this dynamically at run time. It's very useful. Thanks for the feedback. How about if I change mount()/unmount() to: def mount(self, append=False, destdir=None): """ Unzip the wheel's contents to the specified directory, or to a temporary directory if destdir is None. Add this directory to sys.path, either appending or prepending according to whether append is True or False. Before doing this, check that the wheel is compatible with the Python making the call to mount(). If successful, this makes the contents of the wheel's root directory - both Python packages and C extensions - importable via normal Python import mechanisms. """ def unmount(self): """ Remove the directory that was used for mounting from sys.path, thus making the wheel's code no longer importable. Return this directory. Note that the caller is responsible for deleting this directory and its contents, which might not be possible - e.g. in Windows, if a shared library has been imported and is linked to the running Python process, there will be an open handle to the shared library which will prevent its deletion. """ Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On 28 March 2013 16:02, Vinay Sajip wrote: > Return this directory. Note that the caller is responsible for > deleting this directory and its contents, which might not be > possible - e.g. in Windows, if a shared library has been > imported and is linked to the running Python process, there will be > an open handle to the shared library which will prevent its deletion. > """ That's the big issue I have with *any* approach like this. It's entirely possible that the directory cannot be deleted, and as a result the user ends up with the problem of managing clutter caused by this mechanism. Even if the directory is in %TEMP% the user still has the issue of clearing up. Consider a buildslave that continually runs tests - temp directory clutter is a definite issue in a situation like that. And of course, if an application user chooses to use this mechanism, I don't have an option to opt out unless we start getting into complex "if the package is installed use it, otherwise mount our internal wheel" logic. I'd like to hold off on this feature until there are actual requests for the functionality. It's not easy to argue against the idea purely on a "it might go wrong" basis without actual use cases to look at and see if/how they would handle the problem situations. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Thu, Mar 28, 2013 at 12:02 PM, Vinay Sajip wrote: > Jim Fulton zope.com> writes: > > >> >> It would be far better IMO to just unzip the wheel and put that in >> >> your path. (I'm hoping that wheels used this way are a suitable >> >> replacement for eggs.) >> > >> > Well that's tantamount to installing the wheel, >> >> Not really. If you just unzip the wheel and add it to your path, >> you can stop using it by just removing from your path. If you >> install the wheel, it's contents will be poured into site-packages >> (and other places).It's much heavier than just adding the >> wheel (zipped or unzipped) to your path. >> > [snip] >> by adding (unzipped) eggs to sys.path. Various plugin >> systems (including buildout itself with extensions and recipes) >> do this dynamically at run time. It's very useful. > > Thanks for the feedback. Thanks for trying to provide a useful feature. I hope my comments aren't too much of a downer. > How about if I change mount()/unmount() to: > > def mount(self, append=False, destdir=None): > """ > Unzip the wheel's contents to the specified directory, or to > a temporary directory if destdir is None. Add this directory to > sys.path, either appending or prepending according to whether > append is True or False. > > Before doing this, check that the wheel is compatible with the > Python making the call to mount(). > > If successful, this makes the contents of the wheel's root directory > - both Python packages and C extensions - importable via normal Python > import mechanisms. > """ > > def unmount(self): > """ > Remove the directory that was used for mounting from sys.path, > thus making the wheel's code no longer importable. > > Return this directory. Note that the caller is responsible for > deleting this directory and its contents, which might not be > possible - e.g. in Windows, if a shared library has been > imported and is linked to the running Python process, there will be > an open handle to the shared library which will prevent its deletion. > """ I'm not sure which users or use cases you're trying to serve here, so I'm not sure what to think of this. For buildout users, buildout would download and extract the wheel the first time it's used and keep it in a cache and then add it to a path at script generation time. For buildout's own uses (extensions and recipes) it would simply add the extracted wheel's location to sys.path at run time (downloading and extracting it first if necessary). So the win for buildout and it's users is to be able to have extracted (but not "installed" wheels) around to be mixed and matched either for script generation or run-time use. If I wasn't using buildout, I kinda doubt I'd want to use something like this rather than just installing wheels with pip. Jim P.S. I'm happy to see all the work you've done on distlib. I'm sorry to say I haven't had time to dig into it yet. I assume that buildout 3 will be based on it at some point. -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Am 28.03.2013 17:42, schrieb Paul Moore: On 28 March 2013 16:02, Vinay Sajip wrote: Return this directory. Note that the caller is responsible for deleting this directory and its contents, which might not be possible - e.g. in Windows, if a shared library has been imported and is linked to the running Python process, there will be an open handle to the shared library which will prevent its deletion. """ That's the big issue I have with *any* approach like this. It's entirely possible that the directory cannot be deleted, and as a result the user ends up with the problem of managing clutter caused by this mechanism. Even if the directory is in %TEMP% the user still has the issue of clearing up. Consider a buildslave that continually runs tests - temp directory clutter is a definite issue in a situation like that. I made an experiment some time ago: It is possible to delete shared libs containing extension modules imported by Python if the Python process (after Py_Finalize()) calls FreeLibrary(hmod) in a loop for every extension until FreeLibrary returns zero; then the shared lib file can be deleted. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Paul Moore gmail.com> writes: > That's the big issue I have with *any* approach like this. It's > entirely possible that the directory cannot be deleted, and as a > result the user ends up with the problem of managing clutter caused by > this mechanism. Even if the directory is in %TEMP% the user still has > the issue of clearing up. Consider a buildslave that continually runs > tests - temp directory clutter is a definite issue in a situation like > that. > > And of course, if an application user chooses to use this mechanism, I > don't have an option to opt out unless we start getting into complex > "if the package is installed use it, otherwise mount our internal > wheel" logic. Well, if you use the feature because it has its uses, you have to work around any costs that it has. At least the problem isn't being ducked. Plus, given that the wheel format is open, it's not a lot of work for an application developer to do a zipfile.extractall() followed by a sys.path.append(), whether distlib's Wheel has a mount() or not. Having mount() might be facilitating a useful feature in a (slightly more) controlled fashion. > I'd like to hold off on this feature until there are actual requests > for the functionality. It's not easy to argue against the idea purely > on a "it might go wrong" basis without actual use cases to look at and > see if/how they would handle the problem situations. Didn't Jim Fulton say in a post in this thread that it was a useful feature? I'm presuming he based this on real-world experience, but perhaps he would care to clarify. In terms of "actual requests" - there haven't been any actual requests for anything, other than suggestions to improve features that I've unilaterally introduced. This is another instance of the same thing, it seems to me. This is a feature that eggs have but nothing else does, so it seems reasonable to see if we can have alternatives. AFAICT, your worries would apply to eggs too, it's nothing to do with wheels or distlib in particular ... Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Jim Fulton zope.com> writes: > So the win for buildout and it's users is to be able to have extracted > (but not "installed" wheels) around to be mixed and matched either for > script generation or run-time use. > > If I wasn't using buildout, I kinda doubt I'd want to use something > like this rather than just installing wheels with pip. Ok, thanks for the clarification. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Fri, Mar 29, 2013 at 2:02 AM, Vinay Sajip wrote: > Thanks for the feedback. How about if I change mount()/unmount() to: > > def mount(self, append=False, destdir=None): > """ > Unzip the wheel's contents to the specified directory, or to > a temporary directory if destdir is None. Add this directory to > sys.path, either appending or prepending according to whether > append is True or False. No, mutating sys.path for versioned imports is a broken design. You end up with two possibilities: * If you append, then you can't override modules that have a default version available on sys.path. This is not an acceptable restriction, which is why pkg_resources doesn't do it that way * If you prepend, then you have the existing pkg_resources failure mode where it can accidentally shadow more modules than intended. This is a nightmare to debug when it goes wrong (it took me months to realise this was why a system install of the main application I work on was shadowing the version in source checkout when running the test suite or building the documentation). The correct way to do it is with a path hook that processes a special "" marker label in sys.path (probably placed after the standard library but before site-packages by default). Any mounted directories would be tracked by that path hook, but never included directly in sys.path itself. See http://mail.python.org/pipermail/distutils-sig/2013-March/020207.html for more on how this could be handled (consider mount/unmount as the lower level API for actually adding new path entries directly to the dynamic importer). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Fri, Mar 29, 2013 at 3:55 PM, Nick Coghlan wrote: > No, mutating sys.path for versioned imports is a broken design. You > end up with two possibilities: > > * If you append, then you can't override modules that have a default > version available on sys.path. This is not an acceptable restriction, > which is why pkg_resources doesn't do it that way > * If you prepend, then you have the existing pkg_resources failure > mode where it can accidentally shadow more modules than intended. This > is a nightmare to debug when it goes wrong (it took me months to > realise this was why a system install of the main application I work > on was shadowing the version in source checkout when running the test > suite or building the documentation). > > The correct way to do it is with a path hook that processes a special > "" marker label in sys.path (probably placed after > the standard library but before site-packages by default). Any mounted > directories would be tracked by that path hook, but never included > directly in sys.path itself. How is that different from replacing "" with the path of the versioned package being added? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Sat, Mar 30, 2013 at 6:42 AM, PJ Eby wrote: > On Fri, Mar 29, 2013 at 3:55 PM, Nick Coghlan wrote: >> No, mutating sys.path for versioned imports is a broken design. You >> end up with two possibilities: >> >> * If you append, then you can't override modules that have a default >> version available on sys.path. This is not an acceptable restriction, >> which is why pkg_resources doesn't do it that way >> * If you prepend, then you have the existing pkg_resources failure >> mode where it can accidentally shadow more modules than intended. This >> is a nightmare to debug when it goes wrong (it took me months to >> realise this was why a system install of the main application I work >> on was shadowing the version in source checkout when running the test >> suite or building the documentation). >> >> The correct way to do it is with a path hook that processes a special >> "" marker label in sys.path (probably placed after >> the standard library but before site-packages by default). Any mounted >> directories would be tracked by that path hook, but never included >> directly in sys.path itself. > > How is that different from replacing "" with the > path of the versioned package being added? You don't lose the place where you want the inserts to happen. Without the marker, you end up having to come up with a heuristic for "make insertions here" and that gets messy as you modify the path (particularly since other code may also modify the path without your involvement). Using a path hook to say "process these additional path entries here" cleans all that up and lets you precisely control the relative precedence of the additions as a group, without needing to care about their contents, or the other contents of sys.path. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Fri, Mar 29, 2013 at 4:50 PM, Nick Coghlan wrote: > On Sat, Mar 30, 2013 at 6:42 AM, PJ Eby wrote: >> On Fri, Mar 29, 2013 at 3:55 PM, Nick Coghlan wrote: >>> No, mutating sys.path for versioned imports is a broken design. You >>> end up with two possibilities: >>> >>> * If you append, then you can't override modules that have a default >>> version available on sys.path. This is not an acceptable restriction, >>> which is why pkg_resources doesn't do it that way >>> * If you prepend, then you have the existing pkg_resources failure >>> mode where it can accidentally shadow more modules than intended. This >>> is a nightmare to debug when it goes wrong (it took me months to >>> realise this was why a system install of the main application I work >>> on was shadowing the version in source checkout when running the test >>> suite or building the documentation). >>> >>> The correct way to do it is with a path hook that processes a special >>> "" marker label in sys.path (probably placed after >>> the standard library but before site-packages by default). Any mounted >>> directories would be tracked by that path hook, but never included >>> directly in sys.path itself. >> >> How is that different from replacing "" with the >> path of the versioned package being added? > > You don't lose the place where you want the inserts to happen. Without > the marker, you end up having to come up with a heuristic for "make > insertions here" and that gets messy as you modify the path > (particularly since other code may also modify the path without your > involvement). But at least you can tell exactly what the order is by inspecting sys.path. > Using a path hook to say "process these additional path > entries here" cleans all that up and lets you precisely control the > relative precedence of the additions as a group, without needing to > care about their contents, or the other contents of sys.path. Then can't you just bracket the entries with "" and "" then? ;-) (That would actually work right now without a path hook, since strings that refer to non-existent paths are optimized away even in Python 2.5+.) Also, btw, pkg_resources's sys.path munging is far more aggressive than anything with wheels needs to be, because pkg_resources is *always* working with individual eggs, not plain install directories. If you are only using standalone wheels to handle alternate versions, then you *definitely* want those standalone wheels to override other things, so a strategy of always placing them immediately before their containing directory is actually safe. (In effect, treating the containing directory as the insertion marker.) So, if the standalone wheel is in site-packages, then activating it would place it just ahead of site-packages -- i.e., the same place you're saying it should go. And as I believe I mentioned before, a single marker for insertion points doesn't address the user site-packages, app directory or app plugin directories, etc. AFAICT your proposal only addresses the needs of system packages, and punts on everything else. At the least, may I suggest that instead of using a marker, if you must use a path hook, simply install the path hook as a wrapper for a specific directory (e.g. site-packages), and let it process insertions for *that directory only*, rather than having a single global notion of "all the versioned packages". ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Sat, Mar 30, 2013 at 8:52 AM, PJ Eby wrote: > On Fri, Mar 29, 2013 at 4:50 PM, Nick Coghlan wrote: >> You don't lose the place where you want the inserts to happen. Without >> the marker, you end up having to come up with a heuristic for "make >> insertions here" and that gets messy as you modify the path >> (particularly since other code may also modify the path without your >> involvement). > > But at least you can tell exactly what the order is by inspecting sys.path. Agreed that introspection support for metapath importers and path hooks is currently lacking. >> Using a path hook to say "process these additional path >> entries here" cleans all that up and lets you precisely control the >> relative precedence of the additions as a group, without needing to >> care about their contents, or the other contents of sys.path. > > Then can't you just bracket the entries with "" and > "" then? ;-) > > (That would actually work right now without a path hook, since strings > that refer to non-existent paths are optimized away even in Python > 2.5+.) Sure, you *could*, but then you're effectively embedding a list inside another list and it would be a lot cleaner to just say "at this point, go consult that other dynamically modified list over there". > Also, btw, pkg_resources's sys.path munging is far more aggressive > than anything with wheels needs to be, because pkg_resources is > *always* working with individual eggs, not plain install directories. > If you are only using standalone wheels to handle alternate versions, > then you *definitely* want those standalone wheels to override other > things, so a strategy of always placing them immediately before their > containing directory is actually safe. (In effect, treating the > containing directory as the insertion marker.) Yes, that was the other notion I had - insert the extra directory immediately before the directory where the metadata was located. > So, if the standalone wheel is in site-packages, then activating it > would place it just ahead of site-packages -- i.e., the same place > you're saying it should go. > > And as I believe I mentioned before, a single marker for insertion > points doesn't address the user site-packages, app directory or app > plugin directories, etc. AFAICT your proposal only addresses the > needs of system packages, and punts on everything else. No, it doesn't - all it does is provide a clear demarcation between the default system path provided by the interpreter and the ad hoc runtime modifications used to gain access to additional packages that aren't available by default. At the moment, if you print out sys.path, you have *no idea* what it originally looked like before the application started modifying it (process global shared state is always fun that way). *What* directories can be added is then entirely up to the manipulation API. I guess we could easily enough snapshot the path during the interpreter initialisation to help diagnose issues with post-startup modifications. > At the least, may I suggest that instead of using a marker, if you > must use a path hook, simply install the path hook as a wrapper for a > specific directory (e.g. site-packages), and let it process insertions > for *that directory only*, rather than having a single global notion > of "all the versioned packages". It's not really "all the versioned packages", it's "all the packages found through this particular path manipulation API". Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
