[Distutils] Explicitly constructing SOABI for Python 2.7
Currently, bdist_wheel doesn't set the ABI tag properly on Python 2.x - the SOABI symbol it expects to find isn't present. PEP 425 also disclaims dealing with the problem. However, PEP 3149 (which defined SOABI) explains how to define that based on 3 other syconfig flags: Py_DEBUG, Py_UNICODE_SIZE and WITH_PYMALLOC sysconfig.get_config_var(SOABI) sysconfig.get_config_var(Py_DEBUG) 0 sysconfig.get_config_var(Py_UNICODE_SIZE) 4 sysconfig.get_config_var(WITH_PYMALLOC) 1 With those settings and the platform module (to get the current implementation), it is possible to figure out that the wheel ABI tag should be: cp27mu If Py_DEBUG was true, then it would be cp27dmu If WITH_PYMALLOC was false for some reason: cp27u And for a narrow Unicode build: cp27m That way, the ABI tagging would work properly in 2.x as well, rather than people getting the impression that wheels can't mark the ABI compatibility requirements properly. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP 440 (versioning) nearing completion (I hope)
I just pushed a new draft of PEP 440 live: http://www.python.org/dev/peps/pep-0440/ Relative to the version I posted just before Christmas, the main significant change is the inclusion of version epochs as part of the spec. In 99.9% of cases, those won't be needed and can be ignored. However, I'm swayed by the fact that at least Debian and Fedora found them useful enough to add to their versioning schemes, and they're a relatively elegant solution to the How do I switch from date based versioning to semantic versioning without renaming my project? problem. There are also two clarifications: - I changed the pseudo-syntax for the release segment from N[.N]+ to N(.N)* to make it clearer that single value version numbers are acceptable - I point out in the rationale section that - makes local version identifiers more readable in isolation, and the existing escaping rules in PEP 427 take care of converting them to hyphens to avoid ambiguity in wheel filenames. (Paul, you can consider this a belated reply to your post from last week...) With this update, I believe PEP 440 is functionally complete. However, there are three additional requirements that need to be met before I'm willing to declare it accepted: - a reference implementation (I'm hoping Vinay will be willing to step up to the plate with distlib again on that front) - rerunning the compatibility analysis against the PyPI of early 2014 - surviving distutils-sig review without too much grumbling or virtual tomato throwing ;) I know it's a rather complex spec, but robust and comprehensive versioning is a rather complex problem :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Explicitly constructing SOABI for Python 2.7
+1 However, on Windows, I don't see most of the config vars. At least the Unicode width is easy to get at in other ways. We also need to update the PEP/implementation to allow/add to the list of supported tags py2.py3-none-(arch) for cffi-type doesn't use the ABI extensions. This might just be a bug in the implementation... On Tue, Jan 28, 2014 at 5:40 AM, Nick Coghlan ncogh...@gmail.com wrote: Currently, bdist_wheel doesn't set the ABI tag properly on Python 2.x - the SOABI symbol it expects to find isn't present. PEP 425 also disclaims dealing with the problem. However, PEP 3149 (which defined SOABI) explains how to define that based on 3 other syconfig flags: Py_DEBUG, Py_UNICODE_SIZE and WITH_PYMALLOC sysconfig.get_config_var(SOABI) sysconfig.get_config_var(Py_DEBUG) 0 sysconfig.get_config_var(Py_UNICODE_SIZE) 4 sysconfig.get_config_var(WITH_PYMALLOC) 1 With those settings and the platform module (to get the current implementation), it is possible to figure out that the wheel ABI tag should be: cp27mu If Py_DEBUG was true, then it would be cp27dmu If WITH_PYMALLOC was false for some reason: cp27u And for a narrow Unicode build: cp27m That way, the ABI tagging would work properly in 2.x as well, rather than people getting the impression that wheels can't mark the ABI compatibility requirements properly. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Explicitly constructing SOABI for Python 2.7
On 28 Jan 2014 23:29, Daniel Holth dho...@gmail.com wrote: +1 However, on Windows, I don't see most of the config vars. At least the Unicode width is easy to get at in other ways. It's less of an issue on Windows, since we're already assuming python.orgbinary compatibility in most cases anyway. However, I'll have a look and see if this info is exposed any other way. We also need to update the PEP/implementation to allow/add to the list of supported tags py2.py3-none-(arch) for cffi-type doesn't use the ABI extensions. This might just be a bug in the implementation... That's definitely supposed to be allowed by the spec, specifically for that ctypes/cffi style use case. Now, whether the tools currently provide a way to *generate* a tag like that is a different question :) Cheers, Nick. On Tue, Jan 28, 2014 at 5:40 AM, Nick Coghlan ncogh...@gmail.com wrote: Currently, bdist_wheel doesn't set the ABI tag properly on Python 2.x - the SOABI symbol it expects to find isn't present. PEP 425 also disclaims dealing with the problem. However, PEP 3149 (which defined SOABI) explains how to define that based on 3 other syconfig flags: Py_DEBUG, Py_UNICODE_SIZE and WITH_PYMALLOC sysconfig.get_config_var(SOABI) sysconfig.get_config_var(Py_DEBUG) 0 sysconfig.get_config_var(Py_UNICODE_SIZE) 4 sysconfig.get_config_var(WITH_PYMALLOC) 1 With those settings and the platform module (to get the current implementation), it is possible to figure out that the wheel ABI tag should be: cp27mu If Py_DEBUG was true, then it would be cp27dmu If WITH_PYMALLOC was false for some reason: cp27u And for a narrow Unicode build: cp27m That way, the ABI tagging would work properly in 2.x as well, rather than people getting the impression that wheels can't mark the ABI compatibility requirements properly. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Using Wheel with zipimport
I noticed yesterday that Nick made a commit to PEP427 that Wheels were designed to be compatible with zipimport so that you could directly add them to sys.path (http://hg.python.org/peps/rev/811e34eda04c). I was not aware that this was the case, and had thought otherwise, and have been telling people that although Wheels could technically be used that way that doing so was considered unsupported. I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. One of the major problems with Egg was that it confused the difference between a distribution format and a runtime installation format. I believe that if we have this as a supported option, that ultimately it will lead towards the same kind of problems that Egg had trying to fulfill two roles at once. Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). If a single file importable item for a project is something that end users want, then an additional format should be developed that does not concern itself with the ability to unpack and install the project and instead solves the problems that come with trying to stick arbitrary things in a zipped file and import them. So yea, I think that the goals of an importable format and an distribution format are not completely overlapped, and if Wheel tries to be both, then both sides of that will suffer with a less useful and less suitable - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation). The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically. All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the Root-is-purelib setting, rather than requiring separate purelib and platlib subdirectories). Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 28, 2014, at 6:38 PM, Nick Coghlan ncogh...@gmail.com wrote: I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation). The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically. All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the Root-is-purelib setting, rather than requiring separate purelib and platlib subdirectories). Cheers, Nick. Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft don...@stufft.io wrote: On Jan 28, 2014, at 6:38 PM, Nick Coghlan ncogh...@gmail.com wrote: I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation). The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically. All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the Root-is-purelib setting, rather than requiring separate purelib and platlib subdirectories). Cheers, Nick. Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so. Hey, as long as they are zipfiles that don't totally scramble the layout of the internal Python code you can add them to sys.path. Did you know you can even add subpaths of a zipfile to sys.path? /mind blown I'm opposed to making wheel generic as in package PostgreSQL itself generic. There are other ways to do that. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP 427
Hi, I saw that people from this list are responsible for Wheel related PEP. I’m comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. I’m trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... So, i would like to ask few questions about it : 1. I’m coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn’t it be way easier to deal with the dependencies and imports? 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can’t be used the same exact way? I would appreciate any insight about the development infrastructure the wheel designers have in mind. Thanks in advance, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Tue, Jan 28, 2014 at 8:04 PM, Evgeny Sazhin eug...@sazhin.us wrote: Hi, I saw that people from this list are responsible for Wheel related PEP. I'm comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. I'm trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... So, i would like to ask few questions about it : 1. I'm coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn't it be way easier to deal with the dependencies and imports? 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can't be used the same exact way? I would appreciate any insight about the development infrastructure the wheel designers have in mind. Thanks in advance, Eugene The main reason the packaging format is not explicitly importable is simply because Python hasn't had jar-like deployment for as long or as consistently as Java. So while Java code universally uses a get resource API to get stuff on the classpath, a a lot of Python code will try to open a file. C extensions can't be loaded from inside zip files. And it's a lot harder to edit .py files once they are zipped up, unlike Java where only the compiled and non-editable .class files are zipped. ZIP import is a great feature but wheels aren't really designed for it. It's more reliable that way. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
Donald Stufft donald at stufft.io writes: I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. I know that people have had bad experiences in the past with eggs, for the reasons Nick outlined in his response. However, you don't say *with specifics* why you think putting wheels on sys.path is a bad idea - I don't think it's convincing enough just to hand-wavingly reference problems with the egg format. When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC. One can't say that having the same packaging and importable formats is inherently bad, since Java shows otherwise. So if there are specific problems, they should be identified. Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). I don't yet see a technical impediment here. For example, distlib's wheel implementation uses a mount method to add a wheel to sys.path. This uses the tags to check for compatibility - there's no figuring out that the user has to do. If additional metadata about C extensions is available (which it is, if the wheel is built from source using distil), then those extensions are made available for import, too. I realise that's an extension to the current spec, but then no one is forcing any one to use it. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 Jan 2014 11:19, Daniel Holth dho...@gmail.com wrote: On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft don...@stufft.io wrote: On Jan 28, 2014, at 6:38 PM, Nick Coghlan ncogh...@gmail.com wrote: I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation). The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically. All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the Root-is-purelib setting, rather than requiring separate purelib and platlib subdirectories). Cheers, Nick. Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so. Hey, as long as they are zipfiles that don't totally scramble the layout of the internal Python code you can add them to sys.path. Did you know you can even add subpaths of a zipfile to sys.path? /mind blown Yep, the only requirement is there will be a non empty subset of valid wheel files that can be used directly with zipimport. Wheels that include C extensions, or otherwise need to be unpacked to disk in order to work correctly aren't in that subset, and running directly from a wheel does prevent bytecode caching, but those are both OK - supported when practical isn't the same thing as recommended. Cheers, Nick. I'm opposed to making wheel generic as in package PostgreSQL itself generic. There are other ways to do that. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 440 (versioning) nearing completion (I hope)
Nick Coghlan ncoghlan at gmail.com writes: - a reference implementation (I'm hoping Vinay will be willing to step up to the plate with distlib again on that front) First cut, done :-) I updated distlib's implementation to: - include the local component in NormalizedVersion - allow single numeric component versions in NormalizedVersion - update the NormalizedMatcher to include it in matches when required - convert between - and _ in the version for the wheel filename I've added/updated tests for these changes, but additional reviews/test results are always welcome. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 28, 2014, at 11:54 PM, Nick Coghlan ncogh...@gmail.com wrote: On 29 Jan 2014 11:19, Daniel Holth dho...@gmail.com wrote: On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft don...@stufft.io wrote: On Jan 28, 2014, at 6:38 PM, Nick Coghlan ncogh...@gmail.com wrote: I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation). The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically. All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the Root-is-purelib setting, rather than requiring separate purelib and platlib subdirectories). Cheers, Nick. Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so. Hey, as long as they are zipfiles that don't totally scramble the layout of the internal Python code you can add them to sys.path. Did you know you can even add subpaths of a zipfile to sys.path? /mind blown Yep, the only requirement is there will be a non empty subset of valid wheel files that can be used directly with zipimport. Wheels that include C extensions, or otherwise need to be unpacked to disk in order to work correctly aren't in that subset, and running directly from a wheel does prevent bytecode caching, but those are both OK - supported when practical isn't the same thing as recommended. Cheers, Nick. I'm opposed to making wheel generic as in package PostgreSQL itself generic. There are other ways to do that. And yet on another read through of PEP427 the first item in “Comparison to Egg” is Wheel is an installation format; egg is importable.”. The only other mention of importing any of them in that PEP is the section you just added saying that Wheels are designed to be importable. The original text of the PEP does not provide any evidence that Wheels were meant to be importable and instead makes it a point to call that a difference from Eggs. Further more there is a comment from Daniel on python-dev [1] which states that “Jim Fulton is right that weird failures are a characteristic of zipped eggs, so one of the #1 requests for setuptools is how to prohibit zipping from ever happening. This is an important reason why wheel is billed as an installation format -- fewer users with pitchforks. It's very cool that it works though. Debugging is slightly easier than it was in the old days because pdb can now read the source code from the zip.” Which further shows that at the time it was “cool that it worked” but that the “weird failures” being a reason that Wheel was an installation format. I believe that adding this “feature” to the PEP ex post facto is a bad idea. Had the PEP contained anything to indicate that Wheels were intended to be importable as part of the design philosophy I would have spoken out about it then instead it’s been added after it was accepted with no discussion that I can see. If I missed where this discussion happened during the PEP please direct me to it because I’ve just spent 15 minutes googling attempting to find any information on it, and all I’ve been able to find is Vinay’s experiment with Wheel.mount(), You mentioning the possibility of using Wheels in the sense of Multi Version Installs (which ends up talking about a Wheel inspired directory layout instead), and the thread I mentioned above. Not that it’s entirely relevant because really regardless of if it was discussed if it wasn’t encoded in the PEP than people following along by watching the PEPs had no ability to step in to speak for or against it. [1] https://groups.google.com/d/msg/dev-python/BS28mF7mb6c/o8jOo1NcousJ - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___
Re: [Distutils] Using Wheel with zipimport
On Jan 28, 2014, at 11:14 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. I know that people have had bad experiences in the past with eggs, for the reasons Nick outlined in his response. However, you don't say *with specifics* why you think putting wheels on sys.path is a bad idea - I don't think it's convincing enough just to hand-wavingly reference problems with the egg format. When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC. One can't say that having the same packaging and importable formats is inherently bad, since Java shows otherwise. So if there are specific problems, they should be identified. Last time this topic came up Jim Fulton did point things out. 1. That unpacking into the home directory is problematic because users that run services often don’t have home directories. This you waved away by saying that the home directory isn’t a required place, to which Jim responded that unpacking them anywhere else was likely to have access control / security issues and “generally cause pain”. Which I wholly agree with, intact Mitre has assigned CVE’s to applications that did switch their Egg Cache to directories other than HOME. 2. Zipped Eggs are more difficult to work with for debugging purposes, which you responded saying that Wheels are a “deployment format” (which I don’t believe is supported by the PEP at all) and Jim responded with his experience in developing against things that got installed as zipped Eggs and the pain that has personally caused him. 3. Zipped imports (In Jim’s experience) are slower than unpacked imports, which you replied (quite literally) Caveat emptor”. Jim’s response was that this had been tried with Egg and had been found to be more pain than it was worth. In his exact words: It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment. I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony.” He closes up with, Importing from zipped eggs has proved itself to be an anti pattern. Buildout (as of buildout 2) always unzips eggs.” So you have pip that doesn’t use zipped Eggs, buildout which made sure in buildout 2 (presumably drawing from the lessons of buildout 1) to always unzip eggs, and easy_install which does support zipped eggs (sometimes, if it thinks that particular Egg will support it). Why do we insist upon repeating the mistakes of the past? Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). I don't yet see a technical impediment here. For example, distlib's wheel implementation uses a mount method to add a wheel to sys.path. This uses the tags to check for compatibility - there's no figuring out that the user has to do. If additional metadata about C extensions is available (which it is, if the wheel is built from source using distil), then those extensions are made available for import, too. I realise that's an extension to the current spec, but then no one is forcing any one to use it. Zipped Eggs have a long long history of causing very weird failures conditions, you say your Wheel.mount checks for compatibility, does it also check that the library to be installed isn’t using __file__ shenanigans instead of pkgutil.get_data()? easy_install tried to do this with varying degrees of success, I’ve never seen much except pain from zipped Eggs. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic. However, for people that are comfortable with the import system and the limitations of direct zip imports, it's a very useful feature. I wouldn't have accepted PEP 427 without the zipimport compatibility that meant a developer *could* use it as a direct replacement for eggs if they really wanted to. Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities). I clarified PEP 427 specifically because Armin Ronacher's wheel article showed that he was unaware of this *deliberately included* feature of the wheel format. People are free not to like it - the default tools deliberately make it less convenient to run things that way, and that's as it should be. However, it's not a super-secret capability only to be used by us to implement things like ensurepip - it's a defined capability of the format that if your software is capable of running correctly from a zip archive in the first place, then that archive can be also be a valid wheel file. Running directly from a wheel is a power tool - that's a reason to put handle with care warnings on it, not to refuse to support a feature that was deliberately designed into the format. You can do a lot more damage with a badly written meta-importer, yet we have no intention of deprecating that capability either. Even *.pth files have turned out to have a valid use case for sharing packages between virtual environments. We have lots of features like that elsewhere in Python - when people ask about metaclasses, the first reaction is You probably don't want to use them. However, sometimes developers *do* need them, and that's why the feature exists. Most of the time developers won't want to make use of the zipimport compatibility of wheel files, either, but advanced use cases like ensurepip are exactly why the capability exists. I can make the new note in the PEP more explicit that while this is a supported use case that ensures the feature set provided by wheels is a strict superset of that provided by eggs, that's not the same thing as *recommending* that wheels be used that way. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
