Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 4:32 PM, Paul Moore wrote: > On 25 August 2013 20:53, PJ Eby wrote: >> >> FWIW, I would also note that if you use easy_install to install >> anything, you are quite possibly using multi-version installs without >> realizing it. (The __main__.__requires__ API is used in >> easy_install-generated script wrappers, so there isn't any way you'd >> know about it without paying specific attention.) > > > Unless I'm missing something, I suspect that this over-counts the number of > people using multi-version, in the sense that many (the majority?) of > wrapper scripts using multi-version do not actually need to,because the > users never install more than one version. And quite likely don't even know > that they could. That's just it: if you install two programs, one of which needs CherryPy 2 and the other CherryPy 3, then with easy_install this just works, without you having any idea that you even have more than one version installed, unless you for some reason choose to look into it. Thus, you don't have to know you have multiple versions installed; it can trivially happen by way of dependencies you aren't paying attention to. The more things you install, the more likely it is you have two versions hanging around. (The main limiting factor on conflicts isn't a choice to install multiple versions, it's the relative dearth of pinned versions and upper limits on version numbers. If everything just specifies minimum versions, you'll end up using the latest version for everything as the default version. It's only if a package pins or limits a dependency that any conflict is possible to begin with.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 26 Aug 2013 07:57, "Jim Fulton" wrote: > > On Sun, Aug 25, 2013 at 5:21 PM, Nick Coghlan wrote: > > > > On 26 Aug 2013 07:00, "Donald Stufft" wrote: > >> > >> > >> On Aug 25, 2013, at 4:51 PM, Nick Coghlan wrote: > >> > >>> Anyway, I like Paul's suggestion of defining a specific runtime format > >>> for this, even if it's just "wheel layout plus a RECORD file". I'm currently > >>> thinking of using the ".dist" suffix, matching the existing egg vs egg-info > >>> naming convention. > >> > >> > >> It seems to me the easiest thing to do is just continue using eggs for > >> this feature for now especially if the proposal is just standardizing what > >> eggs do and doesn't offer any benefits besides standardization. That gets > >> you all the benefits sans standardization and doesn't spend time putting a > >> PEP through (and all the back and forth that entails) for something that > >> already works when we can spend the time on stuff that still needs actual > >> design work. > > > > Egg based multi-version installs still suffer from the problem of lacking a > > RECORD file so you need an external tool to manage them properly. > > Well, I'd argue that eggs are effectively also records. You can find out what's > installed by simply looking at the names in whatever directory you put eggs. > > The harder part, of course, is deciding when an egg is no longer needed. > I assume the RECORD file doesn't address that either. > > Note that with multi-version support, uninstalling things is an optimization, > not a necessity. The only harm a never-uninstalled egg does is take up > space and maybe make tools that scan for what's installed take more time. And make you fail security audits due to the "use" of unpatched software :) Cheers, Nick. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 5:21 PM, Nick Coghlan wrote: > > On 26 Aug 2013 07:00, "Donald Stufft" wrote: >> >> >> On Aug 25, 2013, at 4:51 PM, Nick Coghlan wrote: >> >>> Anyway, I like Paul's suggestion of defining a specific runtime format >>> for this, even if it's just "wheel layout plus a RECORD file". I'm currently >>> thinking of using the ".dist" suffix, matching the existing egg vs egg-info >>> naming convention. >> >> >> It seems to me the easiest thing to do is just continue using eggs for >> this feature for now especially if the proposal is just standardizing what >> eggs do and doesn't offer any benefits besides standardization. That gets >> you all the benefits sans standardization and doesn't spend time putting a >> PEP through (and all the back and forth that entails) for something that >> already works when we can spend the time on stuff that still needs actual >> design work. > > Egg based multi-version installs still suffer from the problem of lacking a > RECORD file so you need an external tool to manage them properly. Well, I'd argue that eggs are effectively also records. You can find out what's installed by simply looking at the names in whatever directory you put eggs. The harder part, of course, is deciding when an egg is no longer needed. I assume the RECORD file doesn't address that either. Note that with multi-version support, uninstalling things is an optimization, not a necessity. The only harm a never-uninstalled egg does is take up space and maybe make tools that scan for what's installed take more time. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 26 Aug 2013 07:00, "Donald Stufft" wrote: > > > On Aug 25, 2013, at 4:51 PM, Nick Coghlan wrote: > >> Anyway, I like Paul's suggestion of defining a specific runtime format for this, even if it's just "wheel layout plus a RECORD file". I'm currently thinking of using the ".dist" suffix, matching the existing egg vs egg-info naming convention. > > > It seems to me the easiest thing to do is just continue using eggs for this feature for now especially if the proposal is just standardizing what eggs do and doesn't offer any benefits besides standardization. That gets you all the benefits sans standardization and doesn't spend time putting a PEP through (and all the back and forth that entails) for something that already works when we can spend the time on stuff that still needs actual design work. Egg based multi-version installs still suffer from the problem of lacking a RECORD file so you need an external tool to manage them properly. They also aren't integrated into pip's listing and upgrading capabilities. This is another problem in the "important but not urgent" category, though. This discussion covered enough of the big issues that I'm happy I can come up with a new standard that pip and pkg_resources will be willing to support at some point in the future, but in the meantime we can continue using the egg based approach. Cheers, Nick. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Aug 25, 2013, at 4:51 PM, Nick Coghlan wrote: > Anyway, I like Paul's suggestion of defining a specific runtime format for > this, even if it's just "wheel layout plus a RECORD file". I'm currently > thinking of using the ".dist" suffix, matching the existing egg vs egg-info > naming convention. > It seems to me the easiest thing to do is just continue using eggs for this feature for now especially if the proposal is just standardizing what eggs do and doesn't offer any benefits besides standardization. That gets you all the benefits sans standardization and doesn't spend time putting a PEP through (and all the back and forth that entails) for something that already works when we can spend the time on stuff that still needs actual design work. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 26 Aug 2013 05:53, "PJ Eby" wrote: > > On Sun, Aug 25, 2013 at 12:58 PM, Jim Fulton wrote: > > On Sun, Aug 25, 2013 at 3:06 AM, Nick Coghlan wrote: > >> The clumsiness of the __main__.__requires__ workaround aside, the main > >> advantage this offers is that it *should* result in a relatively > >> straightforward addition to pkg_resources to make it work with wheel > >> files as well as eggs. That's important, because anyone that is > >> currently doing side-by-side multi-versioning in Python is using the > >> pkg_resources API to do it, since that's the only option currently > >> available. > > > > No. It isn't. Buildout doesn't use pks_resources to do it. > > (Buildout used pkg_resources at build time to manage package meta > > data, but I think that's orthogonal to what you're talking about.) > > > > I'd also hazard to guess that most of the folks with multi-version > > installs are using buildout to do it, as buildout does have a > > fair number of users. > > FWIW, I would also note that if you use easy_install to install > anything, you are quite possibly using multi-version installs without > realizing it. (The __main__.__requires__ API is used in > easy_install-generated script wrappers, so there isn't any way you'd > know about it without paying specific attention.) > > I don't know how big the "buildout users w/known multi-version" vs. > "easy_install users w/implicit multi-version" groups are, but I > imagine the combined group has got to be pretty darn big. ;-) I'd be willing to bet the number of Linux installs relying on multi-version imports without the end user's knowledge trumps both :) Anyway, I like Paul's suggestion of defining a specific runtime format for this, even if it's just "wheel layout plus a RECORD file". I'm currently thinking of using the ".dist" suffix, matching the existing egg vs egg-info naming convention. The likely vehicle for defining it will be the next generation installation database format. Cheers Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 25 August 2013 20:53, PJ Eby wrote: > FWIW, I would also note that if you use easy_install to install > anything, you are quite possibly using multi-version installs without > realizing it. (The __main__.__requires__ API is used in > easy_install-generated script wrappers, so there isn't any way you'd > know about it without paying specific attention.) > Unless I'm missing something, I suspect that this over-counts the number of people using multi-version, in the sense that many (the majority?) of wrapper scripts using multi-version do not actually need to,because the users never install more than one version. And quite likely don't even know that they could. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 12:58 PM, Jim Fulton wrote: > On Sun, Aug 25, 2013 at 3:06 AM, Nick Coghlan wrote: >> The clumsiness of the __main__.__requires__ workaround aside, the main >> advantage this offers is that it *should* result in a relatively >> straightforward addition to pkg_resources to make it work with wheel >> files as well as eggs. That's important, because anyone that is >> currently doing side-by-side multi-versioning in Python is using the >> pkg_resources API to do it, since that's the only option currently >> available. > > No. It isn't. Buildout doesn't use pks_resources to do it. > (Buildout used pkg_resources at build time to manage package meta > data, but I think that's orthogonal to what you're talking about.) > > I'd also hazard to guess that most of the folks with multi-version > installs are using buildout to do it, as buildout does have a > fair number of users. FWIW, I would also note that if you use easy_install to install anything, you are quite possibly using multi-version installs without realizing it. (The __main__.__requires__ API is used in easy_install-generated script wrappers, so there isn't any way you'd know about it without paying specific attention.) I don't know how big the "buildout users w/known multi-version" vs. "easy_install users w/implicit multi-version" groups are, but I imagine the combined group has got to be pretty darn big. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 3:06 AM, Nick Coghlan wrote: > On 25 August 2013 16:41, Donald Stufft wrote: >> >> On Aug 25, 2013, at 1:57 AM, Nick Coghlan wrote: >> >>> [snip] >> >> I'll look at this closer, my off the cuff response isn't good but before I >> commit to a side I want to dig into how it actually works currently. > > The clumsiness of the __main__.__requires__ workaround aside, the main > advantage this offers is that it *should* result in a relatively > straightforward addition to pkg_resources to make it work with wheel > files as well as eggs. That's important, because anyone that is > currently doing side-by-side multi-versioning in Python is using the > pkg_resources API to do it, since that's the only option currently > available. No. It isn't. Buildout doesn't use pks_resources to do it. (Buildout used pkg_resources at build time to manage package meta data, but I think that's orthogonal to what you're talking about.) I'd also hazard to guess that most of the folks with multi-version installs are using buildout to do it, as buildout does have a fair number of users. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 1:57 AM, Nick Coghlan wrote: > I'm currently working on the docs for the __main__.__requires__ > feature of pkg_resources, and have been generally poking around inside > pkg_resources before I started on that. It gave me an idea for a > question that has come up a few times: how should we support parallel > installation of multiple versions of the same distribution in the same > Python installation, *without* requiring complete isolation in the > form of virtual environments. > > The current solution (at least in Fedora), is to use the multi-version > support in pkg_resources by installing unpacked egg files. This is also the approach used by buildout. (It's also the approach (except for the unpacked part) used in modern Java-ecosystem-based deployments, FWIW. Collect jar files, typically in a cache, and set application-specific classpaths to point to the right ones.) ... > CherryPy 2 is *not* the default, and is instead installed as an > unpacked "CherryPy-2.3.0-py2.7.egg" directory. You can force this > directory to be added to sys.path by doing the following in your > __main__ module: > > __requires__ = ["CherryPy < 3"] > import pkg_resources I'd never see this. Interesting. > While I'm not a fan (to put it mildly) of non-trivial side effects > when importing a module, Me neither. > this approach to multi-version imports *does* > work well (and, as noted, I'm currently working on improving the docs > for it), and I think the approach to the filesystem layout in > particular makes sense - the alternative versions are installed to the > usual location, but pushed down a level in a subdirectory or zip > archive. Note that buildout takes a different approach that I think is worth keeping in mind. In buildout, of course, there's a specific "build" step that assembles an application from its parts. In the case of Python distributions, this means creating an application specific Python path as a list of installed eggs. This works well. It's explicit and pretty non-invasive. No import magic, .pth files or funny site.py files*. Buildout really wants to have self-contained distribution installations, whether they be eggs or wheels or whatever, to function properly. Jim * There was a well-intention but unfortunate deviation in later buildout 1 versions. This was, somewhat ironically in pursuit of better integration with system Python installations. -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 12:06 AM, Nick Coghlan wrote: > anyone that is > currently doing side-by-side multi-versioning in Python is using the > pkg_resources API to do it, since that's the only option currently > available. I'm not sure it changes anything, but there are other options - ones specific to particular packages. For instance, wxPython has wx.version. And at least a few years ago, some other major packages (pyGTK?) also had roll-your-own approaches. I have no idea how active any of those are (wxPython's still works, though not on the Mac, and I don't think sees much use). -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 25 August 2013 15:02, Donald Stufft wrote: > On Aug 25, 2013, at 10:00 AM, Paul Moore wrote: > > > As regards Nick's proposal: > > 1. "An option (or options) to pip, telling it to just drop a wheel file (or > the unpacked contents of the wheel as a directory) into site-packages > instead of installing the distribution directly as the default version." > Well, if it's in site-packages, it *is* importable/installed. That's how > site-packages works. So I don't understand what "instead of installing" > means. But if --target does what you want then go for it. If you want > something different then I think it's likely a bad idea, but I'd need > details of how it would work to be sure. > > > As I understand it Nick means to take the .whl unzip it into a folder > named foo-whatever….whl and put that folder into site-packages. Basically > the exact same structure as happens with .egg folders except with Wheels. > OK, I get that. But I want to avoid referring to it as "wheel format" because that's what eggs did, having 3 distinct formats, all used for subtly different things and it meant that people had a confused view of "what an egg was". I'd rather not promote the same confusion for wheels. Thanks for the various comments that have been made. I think I'm clear where I stand now - I have no objection to the idea of the proposal, but (1) I'd rather the on-disk format was clearly distinguished from wheels as it's a runtime format rather than a distribution format, (2) I don't really want to add an option to pip to install in this format, but if we do it should work for sdists as well as wheels (notwithstanding any longer-term goal to deprecate installing from sdists) and (3) I don't want to see adding wheel files to sys.path as ever becoming a mainstream way of working (that's what "pkg_resources supports the whl format" means to me - which is why I want to see the unpacked format referred to as something other than "wheel"). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Aug 25, 2013, at 10:00 AM, Paul Moore wrote: > > As regards Nick's proposal: > > 1. "An option (or options) to pip, telling it to just drop a wheel file (or > the unpacked contents of the wheel as a directory) into site-packages instead > of installing the distribution directly as the default version." Well, if > it's in site-packages, it *is* importable/installed. That's how site-packages > works. So I don't understand what "instead of installing" means. But if > --target does what you want then go for it. If you want something different > then I think it's likely a bad idea, but I'd need details of how it would > work to be sure. As I understand it Nick means to take the .whl unzip it into a folder named foo-whatever….whl and put that folder into site-packages. Basically the exact same structure as happens with .egg folders except with Wheels. > > 2. "Support for the ".whl" filename format and internal layout in > pkg_resources". No, very definitely -1. The wheel format is a *distribution* > format and pkg_resources is a *runtime* mechanism. Misxing the two is the key > mistake (in my mind) that setuptools made, and we do not want to do so again. > If you need a multiversion runtime layout for pkg_resources, then let's > define it in a PEP, and write installers to put packages into that format if > we need to. But let's not just reuse wheel as that format - that was not its > intent. If a distribution format better than wheel comes along in the future, > we should be able to replace it without needing to break everyone's runtime > code because pkg_resources doesn't know wbout it. > > Paul - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 25 August 2013 14:07, Donald Stufft wrote: > As far as I can tell there's nothing preventing Installing a Wheel *into* > and .egg > directory which will give you exactly the same situation as you have today > without needing to do *anything* except make a tool that will install a > Wheel > into an .egg directory. This solves your immediate desire without hanging > more things onto Wheel. > I think that relating this to wheels is a mistake. As Donald says, wheels are just a distribution format. Looking at this from a different perspective, I don't see any immediate issue with being able to install distributions outside of site-packages. That's what pip's --target option does. It seems like the multi-versioning support that is being discussed is nothing much more than a means of managing such "not on site-packages" directory in a way that allows them to be added to sys.path at runtime. On that basis, I don't have an issue with the idea. It's supported in pip (via --target) already, and the runtime aspects can be handled in pkg_resources or wherever without affecting me. If, on the other hand, there is a proposal in here to change pip, then (a) I'd like to see what the explicit proposal is before commenting, (b) it should apply equally to all means by which you can install using pip (wheel and sdist - AFAIK --target has no meaning for --develop installs, and the same should be true here), and (c) it should not need any changes to the wheel format (again, I see no indication that it does, but I'd like to see that explicitly stated). The key point here is that I *don't* want wheel associated with all of the extra baggage that goes along with the egg concept. Even putting wheels on sys.path gives me a nervous feeling that we're reinventing eggs. I feel strongly that wheel is inventing bdist_simple rather than egg, and should continue to do so. As regards Nick's proposal: 1. "An option (or options) to pip, telling it to just drop a wheel file (or the unpacked contents of the wheel as a directory) into site-packages instead of installing the distribution directly as the default version." Well, if it's in site-packages, it *is* importable/installed. That's how site-packages works. So I don't understand what "instead of installing" means. But if --target does what you want then go for it. If you want something different then I think it's likely a bad idea, but I'd need details of how it would work to be sure. 2. "Support for the ".whl" filename format and internal layout in pkg_resources". No, very definitely -1. The wheel format is a *distribution* format and pkg_resources is a *runtime* mechanism. Misxing the two is the key mistake (in my mind) that setuptools made, and we do not want to do so again. If you need a multiversion runtime layout for pkg_resources, then let's define it in a PEP, and write installers to put packages into that format if we need to. But let's not just reuse wheel as that format - that was not its intent. If a distribution format better than wheel comes along in the future, we should be able to replace it without needing to break everyone's runtime code because pkg_resources doesn't know wbout it. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Aug 25, 2013, at 1:57 AM, Nick Coghlan wrote: > I'm currently working on the docs for the __main__.__requires__ > feature of pkg_resources, and have been generally poking around inside > pkg_resources before I started on that. It gave me an idea for a > question that has come up a few times: how should we support parallel > installation of multiple versions of the same distribution in the same > Python installation, *without* requiring complete isolation in the > form of virtual environments. > > The current solution (at least in Fedora), is to use the multi-version > support in pkg_resources by installing unpacked egg files. To use > CherryPy as an example, Fedora currently provides RPMs for both > CherryPy 2 and 3. > > CherryPy 3 is the default and installed directly into site-packages, > with an appropriate .egg-info directory > > CherryPy 2 is *not* the default, and is instead installed as an > unpacked "CherryPy-2.3.0-py2.7.egg" directory. You can force this > directory to be added to sys.path by doing the following in your > __main__ module: > >__requires__ = ["CherryPy < 3"] >import pkg_resources > > (__main__.__requires__ *has* to be set before importing pkg_resources > or the default CherryPy 3 module will be activated automatically, and > conflict with a later call to pkg_resources.requires that asks for > CherryPy 2) > > While I'm not a fan (to put it mildly) of non-trivial side effects > when importing a module, this approach to multi-version imports *does* > work well (and, as noted, I'm currently working on improving the docs > for it), and I think the approach to the filesystem layout in > particular makes sense - the alternative versions are installed to the > usual location, but pushed down a level in a subdirectory or zip > archive. > > So, it seems to me that only two new pieces are needed to gain > multi-version import support for wheel files: > > 1. An option (or options) to pip, telling it to just drop a wheel file > (or the unpacked contents of the wheel as a directory) into > site-packages instead of installing the distribution directly as the > default version. The "root_is_purelib" setting in the wheel metadata > would indicate whether the destination was purelib or platlib. A wheel > installed this way wouldn't have script wrappers generated, etc - it > would only allow the contents to be used as an importable library. > > 2. Support for the ".whl" filename format and internal layout in > pkg_resources, modelling after the existing support for the ".egg" > filename format. For wheels that include both purelib and platlib, > this would involved adding both directories to the path. > > That means there wouldn't be an major design work to be done - just > replicating what easy_install and pkg_resources already support for > the egg format using the wheel format instead. > > Regardless of my personal feelings about the current *front end* API > for multi-version imports (and PJE has put up with my ranting about > that with remarkably good grace!), the current egg-based back end > design looks solid to me and worth keeping rather than trying to > invent something new for the sake of being different. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig I think I am against this. Part of the beauty of Wheel is that it is simply a package format. This means it does not need to concern itself with situations that Egg had to which bloat the spec and make it harder to implement. I feel like tacking too much onto the Wheel format is going to end us up in the exact same place we are at with Eggs today. The more use cases we force onto Wheel the more we have to consider anytime we make a change. For instance with the proposed change now we have to worry about the importability of a Wheel file if an unpacked Wheel is added directly to sys.path which is something we currently do not need to worry about. As far as I can tell there's nothing preventing Installing a Wheel *into* and .egg directory which will give you exactly the same situation as you have today without needing to do *anything* except make a tool that will install a Wheel into an .egg directory. This solves your immediate desire without hanging more things onto Wheel. Additionally I think the way it munges sys.path is completely backwards. I believe, as you know, that the current order of the sys.path is somewhat nonsensical by default it goes ".", then standard library, then user site packages, then regular site packages. I believe that the order should be ".", user-packages, site-packages, and then standard library to provide a consistent ordering and allowing people to shadow packages using a hierarchy of specificity (the packages installed by a particular are more specific than the packages instal
Re: [Distutils] Multi-version import support for wheel files
On 25 August 2013 21:02, Paul Moore wrote: > Essentially, that is my question - and I'd like to know what Nick's proposal > is here because I am not happy with the existing pkg_resources solution of > using .pth files. I know there's a new feature being discussed on import-sig > which may replace the need for pth files, but I don't want to see pth files > being needed if that isn't available. Specific reasons I have for this: > > 1. The extra sys.path entries that get added (not for performance reasons, > but because having a long sys.path is difficult for the interactive user to > interpret when trying to see what's going on). > 2. The whole "if I'm not in a site packages directory I won't work" issue > 3. Questions of where things go on sys.path (if it depends on the order you > "declare" things, it gets nasty when there are name clashes) I'm not proposing copying over any of the implicit .pth file magic. As far as I am aware, that's unrelated to the explicit multi-versioning feature - I believe it's only needed if you want to install something as an egg archive or subdirectory *and* have it available for import by default. The namespace package related .pth files are also unrelated. Explicit multi-versioning is different - you just install the directory, and you can't import from it by default. However, pkg_resources can be used to activate it if it is needed to satisfy a runtime requirement (or a dependency of a runtime requirement). If you don't import pkg_resources, none of the parallel installed versions will ever activate - you'll just get the version that is on sys.path by default. > The other thing I want to understand is how things would work if, for > example, I wanted to use CherryPy 3 99% of the time, but occasionally needed > CherryPy 2. Could I install CherryPy 3 as a normal (not multi-versioned) > package, and then override it when needed? So only applications needing > CherryPy 2 had to declare a version requirement? Yep, this is exactly how Fedora & EPEL use it. CherryPy 3 is installed as normal in site-packages (with an adjacent egg-info directory), while CherryPy 2 is installed as an egg file containing the earlier version of the module and the relevant metadata. > More generally, how does a project choose whether to use run-time > multi-versioning, vs metadata declaring a dependency for install time? Why > would I ever write code that uses a run-time dependency on CherryPy 2, > rather than saying in my metadata "requires: CherryPy = 2"? Or conversely > why would I ever declare an install-time dependency rather than declaring my > requirements at runtime "because that's more flexible"? For Beaker, we use runtime dependencies because we build system packages that run in the system Python, but need to run on RHEL 6 (CherryPy 2 as default) and also on recent versions of Fedora (CherryPy 3 as default, CherryPy 2 available for multi-version import). We also use it in our doc build scripts to handle the different versions of Sphinx (Sphinx 1.x as default in Fedora, but only available as a multi-version import on RHEL 6). The reason to declare install time dependencies as well is that pkg_resources works recursively - the dependencies of any distribution you activate will be checked to ensure you're importing a consistent set of packages into your application. If you use virtual environments to create isolated stacks for each application, these issues don't come up. On the other hand, if you're trying to make thousands of packages play nice in a single Python installation (as Linux distros do), then you need to deal with the "parallel installation of mutually incompatible versions" problem. The current Linux distro solution is to use pkg_resources, and having been working with it for several months now, while I still think there are some significant rough edges in the way the pkg_resources front end works, I'm happy that the *file layout* the explicit multi-versioning feature uses is a decent one that should translate to wheel files fairly easily. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 25 August 2013 10:58, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > > > 1. An option (or options) to pip, telling it to just drop a wheel file > > (or the unpacked contents of the wheel as a directory) into > > site-packages instead of installing the distribution directly as the > > default version. The "root_is_purelib" setting in the wheel metadata > > would indicate whether the destination was purelib or platlib. A wheel > > installed this way wouldn't have script wrappers generated, etc - it > > would only allow the contents to be used as an importable library. > > ISTM you would still need path-munging for this to work, or is there some > other way? > Essentially, that is my question - and I'd like to know what Nick's proposal is here because I am not happy with the existing pkg_resources solution of using .pth files. I know there's a new feature being discussed on import-sig which may replace the need for pth files, but I don't want to see pth files being needed if that isn't available. Specific reasons I have for this: 1. The extra sys.path entries that get added (not for performance reasons, but because having a long sys.path is difficult for the interactive user to interpret when trying to see what's going on). 2. The whole "if I'm not in a site packages directory I won't work" issue 3. Questions of where things go on sys.path (if it depends on the order you "declare" things, it gets nasty when there are name clashes) The other thing I want to understand is how things would work if, for example, I wanted to use CherryPy 3 99% of the time, but occasionally needed CherryPy 2. Could I install CherryPy 3 as a normal (not multi-versioned) package, and then override it when needed? So only applications needing CherryPy 2 had to declare a version requirement? More generally, how does a project choose whether to use run-time multi-versioning, vs metadata declaring a dependency for install time? Why would I ever write code that uses a run-time dependency on CherryPy 2, rather than saying in my metadata "requires: CherryPy = 2"? Or conversely why would I ever declare an install-time dependency rather than declaring my requirements at runtime "because that's more flexible"? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
Nick Coghlan gmail.com> writes: > 1. An option (or options) to pip, telling it to just drop a wheel file > (or the unpacked contents of the wheel as a directory) into > site-packages instead of installing the distribution directly as the > default version. The "root_is_purelib" setting in the wheel metadata > would indicate whether the destination was purelib or platlib. A wheel > installed this way wouldn't have script wrappers generated, etc - it > would only allow the contents to be used as an importable library. ISTM you would still need path-munging for this to work, or is there some other way? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Aug 25, 2013, at 4:02 AM, Paul Moore wrote: > My main concern is that for people who *don't* need multi-versioning, or who > only use it for one or two special cases, there is no detrimental impact on > user interface or runtime (the "if you don't use it, you don't pay for it" > principle). A lot of the FUD around setuptools' multi-versioning was around > precisely this point (path munging, etc) and I want to understand how the > proposed solution avoids repeating those problems (to the extent that they > were real and not imagined). Yea my primary concerns is that the cost is paid only by those who use the feature and to make sure it's not something that's the way it works isn't going to leave us regretting the decision down the road. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 25 August 2013 06:57, Nick Coghlan wrote: > I'm currently working on the docs for the __main__.__requires__ > feature of pkg_resources, and have been generally poking around inside > pkg_resources before I started on that. It gave me an idea for a > question that has come up a few times: how should we support parallel > installation of multiple versions of the same distribution in the same > Python installation, *without* requiring complete isolation in the > form of virtual environments. > > The current solution (at least in Fedora), is to use the multi-version > support in pkg_resources by installing unpacked egg files. To use > CherryPy as an example, Fedora currently provides RPMs for both > CherryPy 2 and 3. > > CherryPy 3 is the default and installed directly into site-packages, > with an appropriate .egg-info directory > > CherryPy 2 is *not* the default, and is instead installed as an > unpacked "CherryPy-2.3.0-py2.7.egg" directory. You can force this > directory to be added to sys.path by doing the following in your > __main__ module: > > __requires__ = ["CherryPy < 3"] > import pkg_resources > > (__main__.__requires__ *has* to be set before importing pkg_resources > or the default CherryPy 3 module will be activated automatically, and > conflict with a later call to pkg_resources.requires that asks for > CherryPy 2) > > While I'm not a fan (to put it mildly) of non-trivial side effects > when importing a module, this approach to multi-version imports *does* > work well (and, as noted, I'm currently working on improving the docs > for it), and I think the approach to the filesystem layout in > particular makes sense - the alternative versions are installed to the > usual location, but pushed down a level in a subdirectory or zip > archive. > > So, it seems to me that only two new pieces are needed to gain > multi-version import support for wheel files: > > 1. An option (or options) to pip, telling it to just drop a wheel file > (or the unpacked contents of the wheel as a directory) into > site-packages instead of installing the distribution directly as the > default version. The "root_is_purelib" setting in the wheel metadata > would indicate whether the destination was purelib or platlib. A wheel > installed this way wouldn't have script wrappers generated, etc - it > would only allow the contents to be used as an importable library. > > 2. Support for the ".whl" filename format and internal layout in > pkg_resources, modelling after the existing support for the ".egg" > filename format. For wheels that include both purelib and platlib, > this would involved adding both directories to the path. > > That means there wouldn't be an major design work to be done - just > replicating what easy_install and pkg_resources already support for > the egg format using the wheel format instead. > > Regardless of my personal feelings about the current *front end* API > for multi-version imports (and PJE has put up with my ranting about > that with remarkably good grace!), the current egg-based back end > design looks solid to me and worth keeping rather than trying to > invent something new for the sake of being different. > Like Donald, I'm going to need to look into the proposal a bit more before commenting. My main concern is that for people who *don't* need multi-versioning, or who only use it for one or two special cases, there is no detrimental impact on user interface or runtime (the "if you don't use it, you don't pay for it" principle). A lot of the FUD around setuptools' multi-versioning was around precisely this point (path munging, etc) and I want to understand how the proposed solution avoids repeating those problems (to the extent that they were real and not imagined). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Aug 25, 2013, at 3:06 AM, Nick Coghlan wrote: > On 25 August 2013 16:41, Donald Stufft wrote: >> >> On Aug 25, 2013, at 1:57 AM, Nick Coghlan wrote: >> >>> [snip] >> >> I'll look at this closer, my off the cuff response isn't good but before I >> commit to a side I want to dig into how it actually works currently. > > The clumsiness of the __main__.__requires__ workaround aside, the main > advantage this offers is that it *should* result in a relatively > straightforward addition to pkg_resources to make it work with wheel > files as well as eggs. That's important, because anyone that is > currently doing side-by-side multi-versioning in Python is using the > pkg_resources API to do it, since that's the only option currently > available. > > If Fedora is going to switch completely to a wheel based build > process, we need to be able to do it in a way that allows side-by-side > imports through pkg_resources to continue to work. > > I'd previously considered writing a pkg_resources replacement that > worked with wheels instead of eggs, but I've now realised that would > be repeating one of the mistakes made with the distutils2 effort: just > as we needed to account for the fact that a lot of people are > currently building their projects with setuptools, we also need to > account for the fact that anyone doing side-by-side imports is using > pkg_resources. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia Yea I understand the motivations for it. The main thing i'm worried about is codifying a solution that ends up being a misfeature instead of a feature. This may be perfectly fine (hence why I want to dig into further having never used it greatly before) but in general I think we should be conservative in copying things over from the old way into the new way. The new tools in many ways are removing features that are less than optimal or overall a bad idea and that's good. We don't need to copy over every single feature if someone really depends on a feature we don't bring over they can continue to use the existing tooling. That being said I don't know for sure that this falls into that category which is why I want to dig into it more. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On 25 August 2013 16:41, Donald Stufft wrote: > > On Aug 25, 2013, at 1:57 AM, Nick Coghlan wrote: > >> [snip] > > I'll look at this closer, my off the cuff response isn't good but before I > commit to a side I want to dig into how it actually works currently. The clumsiness of the __main__.__requires__ workaround aside, the main advantage this offers is that it *should* result in a relatively straightforward addition to pkg_resources to make it work with wheel files as well as eggs. That's important, because anyone that is currently doing side-by-side multi-versioning in Python is using the pkg_resources API to do it, since that's the only option currently available. If Fedora is going to switch completely to a wheel based build process, we need to be able to do it in a way that allows side-by-side imports through pkg_resources to continue to work. I'd previously considered writing a pkg_resources replacement that worked with wheels instead of eggs, but I've now realised that would be repeating one of the mistakes made with the distutils2 effort: just as we needed to account for the fact that a lot of people are currently building their projects with setuptools, we also need to account for the fact that anyone doing side-by-side imports is using pkg_resources. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
