Re: [Distutils] pip on windows experience
On 29 January 2014 18:06, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 05:06, Chris Barker chris.bar...@noaa.gov wrote: This isn't just for users of the SciPy Stack -- there are LOT of use-cases for just numpy by itself. Not that I don't want folks to have easy access of the rest of the stack as well -- just sayin' Agreed - my main use for NumPy is with Pandas for data analysis tasks, which is a truly great combination. I confess I tend to lump the many and varied data analysis tools that depend on NumPy under the phrase Scientific Python Stack, rather than intending to refer specifically to SciPy :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
Donald Stufft donald at stufft.io writes: 1. That unpacking into the home directory is problematic because users that run services often don’t have home directories. This you waved away by saying that the home directory isn’t a required place, to which Jim responded that unpacking them anywhere else was likely to have access control / security issues and “generally cause pain”. $HOME is just a sensible default location for caches, but as long as other locations can be used in situations where there's no HOME directory, I don't see how that's waving away. If a service has no access to writeable disk locations at all, I can see how that might pose a problem, but that hardly seems like a use case which should dictate policies for every other scenario. OTOH anywhere else was likely to have access control / security issues and 'generally cause pain' is light on details. wholly agree with, intact Mitre has assigned CVE’s to applications that did switch their Egg Cache to directories other than HOME. Are you referring to the situation where users of setuptools set PYTHON_EGG_CACHE to a world-writeable location? If so, the problem would appear to be with the using code, not setuptools itself. AFAICT it was felt that while CVEs could be assigned against individual applications that did this, and setuptools itself was not assigned a CVE. If a HOME-less application uses an alternative cache location which isn't world-writeable, does a security issue arise? 2. Zipped Eggs are more difficult to work with for debugging purposes, which you responded saying that Wheels are a “deployment format” (which I don’t believe is supported by the PEP at all) and Jim responded with his experience in developing against things that got installed as zipped Eggs and the pain that has personally caused him. 3. Zipped imports (In Jim’s experience) are slower than unpacked imports, which you replied (quite literally) Caveat emptor”. Jim’s response was that this had been tried with Egg and had been found to be more pain than it was worth. In his exact words: It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment. I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony.” He closes up with, Importing from zipped eggs has proved itself to be an anti pattern. Buildout (as of buildout 2) always unzips eggs.” So you have pip that doesn’t use zipped Eggs, buildout which made sure in buildout 2 (presumably drawing from the lessons of buildout 1) to always unzip eggs, and easy_install which does support zipped eggs (sometimes, if it thinks that particular Egg will support it). Why do we insist upon repeating the mistakes of the past? Because there are scenarios where that functionality is desirable, and the downsides (such as greater difficulty in debugging, no caching of byte code, lower performance of imports) are tolerable. Just because something can cause problems in some scenarios is no reason for a wholesale ban. C makes segfaulting easy, but that doesn't mean that C extensions are right out ;-) Python is generally a consenting adults kind of system. An important difference from the setuptools/easy_install/zipped eggs experience is that nothing happens automatically - a specific action has to be taken to add wheels to sys.path, whereas in the zipped eggs case, sys.path manipulation happens under the hood. Since you don't know it's happening, failures are bound to seem weird - it's nothing I've done, I'm sure! Making something possible != recommending its use wholesale. Zipped Eggs have a long long history of causing very weird failures conditions, you say your Wheel.mount checks for compatibility, does it also check that the library to be installed isn’t using __file__ shenanigans instead of pkgutil.get_data()? easy_install tried to do this with varying degrees of success, I’ve never seen much except pain from zipped Eggs. Weird failure conditions are only weird because they haven't been investigated to find the exact cause. There's a lot of software around that can't run from zips because of __file__ shenanigans - that doesn't negate the very real usefulness of zipimport for other software. If I want to write software that runs from zip, I could take care to use pkgutil rather than __file__. Presumably, others can do the same. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On Sat, Jan 25, 2014 at 4:29 PM, Nick Coghlan ncogh...@gmail.com wrote: To put the but what if the user doesn't have SSE2 support? concern in context, it should only affect Intel users with CPUs older than a Pentium 4 (released 2001), and AMD users with a CPU older than an Opteron or Athlon 64 (both released 2003). All x86/x86_64 CPUs released in the past decade should be able to handle SSE2 binaries, so our caveat can be if your computer is more than a decade old, 'pip install numpy' may not work for you, but it should do the right thing on newer systems. Exactly However, from my perspective, having NumPy readily available to users using the python.org Windows installers for Python 3.4 would *significantly* lower the barrier to entry to the Scientific Python stack for new users on relatively modern systems when compared to the 4 current options +1 with a note: This isn't just for users of the SciPy Stack -- there are LOT of use-cases for just numpy by itself. Not that I don't want folks to have easy access of the rest of the stack as well -- just sayin' -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 05:52, Donald Stufft don...@stufft.io wrote: Which further shows that at the time it was cool that it worked but that the weird failures being a reason that Wheel was an installation format. Having cool features in a format is not a bad thing. Zip imports are the cool feature here - they have been used for lots of really impressive things (like py2exe, for a less controversial example). But they have limitations (which become weird failures when people try to use the feature without knowing or catering for the limitations) and it's the responsibility of the user to understand and address those limitations - or not use the feature if the limitations are too great for them. I believe that adding this feature to the PEP ex post facto is a bad idea. Had the PEP contained anything to indicate that Wheels were intended to be importable as part of the design philosophy I would have spoken out about it then instead it's been added after it was accepted with no discussion that I can see. As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the wheel install command. I do *not* recommend using wheels as a runtime format if the package in the wheel hits any of the limitations of zipimport (C extensions, data files that are not or cannot be accessed via pkgutil.package_data, etc). I also would not recommend using wheels except in *very* specialised circumstances (where you have the wheel already and cannot easily just install it before use - pip, ensurepip, virtualenv and similar distribution-oriented environments are the key use cases here). But I do think the feature is useful and should be supported. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 04:14, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC. From what I recall, the topic came up before in relation to distlib's wheel mount functionality. My specific concerns about that API (none of which have been addressed, AFAIK, but most of which are somewhat non-specific, I concede, for reasons see below) are: 1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and accepted limitation of zipimport, and as a workaround, it has issues. If C extensions in zipfiles could work reliably, this should go into zipimport itself, and *not* into 3rd party code. Then everyone would benefit. 2. It makes what should be a rare use case, to be used only when the code in the wheel has been carefully checked to work from a zipfile, seem like a common and straightforward operation. (The attractive nuisance argument). I believe that people using this API will typically *not* check the code, and will blame the wheel format, or distlib, when their application does not work as expected. 3. It is no easier than sys.path.insert(0, wheelname). All it adds over that is compatibility checking and the ability to import C extensions (see above on why I think that's a bad thing). As for compatibility checking, I'd prefer a distlib.wheel.check_compatibility API that people could call *before* manually inserting the wheel onto sys.path. That's a better separation of concerns, in my view. I can't give specific examples of failure modes because I don't use the wheel mount functions, nor do I typically add wheels to sys.path. When I did (in virtualenv) I hit a number of issues, but all of these were ones I fixed in user code (either in virtualenv itself, or in pip, which was the wheel I was importing). So you could reasonably dismiss these as not related to the mount API to which all I can say is that if I'd been able to do wheel.mount(wheelname) I would likely have put less thought into whether what I was doing was a good idea - and *that's* what I think is the bad aspect of the API. Ultimately, I'll just never use the distlib mount functionality, and I'll recommend not using it when (if) people ask. But I'd rather it were not there to prompt the question. I hope this helps, Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
Paul Moore p.f.moore at gmail.com writes: 1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and accepted limitation of zipimport, and as a workaround, it has issues. If C extensions in zipfiles could work reliably, this should go into zipimport itself, and *not* into 3rd party code. Then everyone would benefit. (a) The mount method does not do the extraction of C extensions *unconditionally* - only when there is suitable indication in the metadata. (b) What are these issues to which you refer? As I said in my other response, I don't see the zipped-egg problem as the same, because that sys.path manipulation is under the hood/not under the developer's control. (c) Why can't third-party code break new ground, which may or may not prove fertile? If considered beneficial, it could always be added to zipimport at a later date. That's how a lot of functionality has entered the stdlib, after all. 2. It makes what should be a rare use case, to be used only when the code in the wheel has been carefully checked to work from a zipfile, seem like a common and straightforward operation. (The attractive nuisance argument). I believe that people using this API will typically *not* check the code, and will blame the wheel format, or distlib, when their application does not work as expected. Doesn't the same argument apply to zipimport? Any code which is zipimported shouldn't use __file__ manipulation to access package resources, for example. Are you saying that you also feel wheels should never be importable, since one can't guarantee whether any code in general will work correctly from a zip? And yet, we have PEP 441, which is intended to encourage use of zipimport. Do we say, no point - there's bound to be people out there who'll use __file__ instead of pkgutil? 3. It is no easier than sys.path.insert(0, wheelname). All it adds over that is compatibility checking and the ability to import C extensions (see above on why I think that's a bad thing). As for compatibility checking, I'd prefer a distlib.wheel.check_compatibility API that people could call *before* manually inserting the wheel onto sys.path. That's a better separation of concerns, in my view. There's no problem with providing a compatibility check API, but the other POV is that people might forget to call that API, so it seems to make sense to call it from the mount method too. And I don't see why you say all it adds - surely the check is important to prevent a certain class of weird things happening. I can't give specific examples of failure modes because I don't use the wheel mount functions, nor do I typically add wheels to sys.path. When I did (in virtualenv) I hit a number of issues, but all of these were ones I fixed in user code (either in virtualenv itself, or in pip, which was the wheel I was importing). So you could reasonably When I looked into running pip from a zip, it was clear that the issues were related to pip's use of __file__ and such, and nothing to do with shortcomings in the wheel format or zipimport. dismiss these as not related to the mount API to which all I can say is that if I'd been able to do wheel.mount(wheelname) I would likely have put less thought into whether what I was doing was a good idea - and *that's* what I think is the bad aspect of the API. Again, this feels like saying zipimport is bad because people use __file__ instead of pkgutil. Having any API available doesn't absolve people from the responsibility of thinking about what they're doing and what the implications of using that API are. Ultimately, I'll just never use the distlib mount functionality, and I'll recommend not using it when (if) people ask. But I'd rather it were not there to prompt the question. You don't want it to be there, even if it might be useful to others, just because it isn't useful to you? It's not as if distlib is forcing that functionality on anyone. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 20:36, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: You don't want it to be there, even if it might be useful to others, just because it isn't useful to you? It's not as if distlib is forcing that functionality on anyone. I believe Paul's concern is with anything that suggests that arbitrary *third party* code can be run from wheel files, when the reality is that it is fairly easy to accidentally write code that assumes it is installed on the filesystem in a way that isn't easy for a quick scan of the files in the zip archive to detect (especially since the PEP 376 installation database PEP doesn't include any support for arbitrary metapath importers). By contrast PEP 441 is a *distribution* utility - the creator of the application is expected to ensure that doing so actually works correctly before publishing their app that way, just as we would expect py2exe, py2app and cx-freeze users to do. With the reference implementation position that distlib is likely to occupy in a post-PEP-426/440/459 world, though, there's an additional legitimate concern about allowing end users to easily distinguish between this API is fully supported by the PyPA as part of the reference implementation for metadata 2.0 and this is an experimental packaging related API that may or may not be useful in general, and some members of the PyPA may still have grave reservations about it. At the moment, distlib contains both kinds of API, and it confuses *us*, let alone anyone else that isn't closely following along on distutils-sig. As long as distlib is serving the dual role of providing both the reference implementation for metadata 2.0 and some experimental packaging related APIs, we're going to get concerns like this one arising. If there was a clear way to distinguish them (ideally with a separate project for either the reference implementation or the experimental stuff, but even a distinct namespace within the distlib project would help a great deal), I suspect there would be less concern. In the specific case of distlib.mount, if it's eventually combined with a metadata extension like distlib.mount which packages must export in order for the command to allow them to be automatically used that way, then I don't see anything wrong with it *in general* - it's a natural extension of the setuptools zip_safe flag, but with the ability to include additional details (like whether or not there are C extensions that need to be automatically extracted). Note that this goes further than the current EXTENSIONS approach - this proposal would be akin to *requiring* an empty EXTENSIONS file, and/or the setuptools zip_safe flag in order to allow mounting of even the pure Python wheel. Such a conservative approach is also the antithesis of the setuptools attempt to guess: if the package publisher doesn't explicitly opt in to zip support, then distlib.mount would assume that it is *not* supported (but may provide an API for the caller to override that, like assume_zip_safe=True or force=True). However, like Paul, I have some concerns about a still experimental API like that being in the metadata 2.0 reference implementation, since that will likely end up having to deal with stdlib-like levels of backwards compatibility requirements, and removing experimental APIs that we later decided we weren't happy with could prove problematic. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels, metadata
On 24 January 2014 06:18, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: certainly mention the distlib implementations, but also let's be clear if there is a pypa-recommend tool that is user-facing (like pip), that is using those parts of distlib. In most cases, that is not true currently. As for mentioning distil, I'm inclined to say no. Up to this point, you've presented it as a proof of concept.If you're wanting to mention distil as a real option for users, I'm concerned about fracturing the mind of users, but it's something to discuss I guess. I hear what you're saying. I've positioned distil as a proof of concept purely because it hasn't had widespread use, but I certainly expect it to fulfil the same role as pip functionally (which it must do to be an effective test-bed for distlib). I understand that pip is the officially recommended tool, and don't want to muddy the waters, there being enough confusion about packaging in the wider community. It seems a shame that some of the improvements over pip won't be more widely available, but such is life. I have the use of them, so there's that :-) The fact we're still working on PEP 426/440/459 so distlib and distil are chasing a moving target also makes it a little difficult to recommend them to end users at this point :) I actually expect that we'll see many of the internals of pip significantly refactored in the next 12 months or so - in addition to metadata 2.0, there's also The Update Framework support as a result of PEP 458, and once we get proper metadata publication on PyPI, then adopting a real dependency solver becomes a far more viable option than it is today (and both conda and Fedora's hawkey have tackled the problem of making a depsolver available to a Python based installation tool). pip, ultimately, is just a CLI - so long as that remains the case, then the internals can change radically (which is why I agree with the idea of it *not* having a public Python API, and instead leaving that to distlib). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the wheel install command. I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick’s most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 10:36, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: 1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and accepted limitation of zipimport, and as a workaround, it has issues. If C extensions in zipfiles could work reliably, this should go into zipimport itself, and *not* into 3rd party code. Then everyone would benefit. (a) The mount method does not do the extraction of C extensions *unconditionally* - only when there is suitable indication in the metadata. Suitable indication of what, exactly? It's not possible (at an OS level) to link to a DLL in a zipfile, so if there's a DLL in there and it's used, it needs to be extracted. I don't really see much point hairsplitting over whether that counts as unconditional or not. (b) What are these issues to which you refer? As I said in my other response, I don't see the zipped-egg problem as the same, because that sys.path manipulation is under the hood/not under the developer's control. I did point out that I knew I was being non-specific. But I know people have reported problems in the past with attempts (and there have been many) to do this. Getting cleanup right is one area. Picking a suitable location to extract to is another. But conceded, this is anecdotal. Feel free to ignore it if that's what you want to do. (c) Why can't third-party code break new ground, which may or may not prove fertile? If considered beneficial, it could always be added to zipimport at a later date. That's how a lot of functionality has entered the stdlib, after all. This is *not* new ground. Gordon McMillan's importer did it years ago, back when zipimport was first implemented. Eggs did it in pkg_resources. I think py2exe explicitly decided *not* to do it because of known issues with Gordon McMillan's implementation (feel free to call this anecdotal again if you want - but you may be able to find the details with some research). If this was really something that had never been tried before, I'd be happy with it as an experimental API. But it's neither marked as experimental, nor is it new. Your implementation *may* have found some new approach - but from your descriptions, I'm not sure it has, and you seem not to have been aware of or researched the previous attempts, so I'm guessing you're not aware of the prior art here. 2. It makes what should be a rare use case, to be used only when the code in the wheel has been carefully checked to work from a zipfile, seem like a common and straightforward operation. (The attractive nuisance argument). I believe that people using this API will typically *not* check the code, and will blame the wheel format, or distlib, when their application does not work as expected. Doesn't the same argument apply to zipimport? Any code which is zipimported shouldn't use __file__ manipulation to access package resources, for example. Are you saying that you also feel wheels should never be importable, since one can't guarantee whether any code in general will work correctly from a zip? And yet, we have PEP 441, which is intended to encourage use of zipimport. Do we say, no point - there's bound to be people out there who'll use __file__ instead of pkgutil? No, I'm saying that hiding the subtleties may be doing people a disservice. That's why I described it as an attractive nuisance. I'm fully aware that this is a judgement call, and one person's convenience API is another's disaster waiting to happen. Again, I'me fine with your opinion not matching mine. 3. It is no easier than sys.path.insert(0, wheelname). All it adds over that is compatibility checking and the ability to import C extensions (see above on why I think that's a bad thing). As for compatibility checking, I'd prefer a distlib.wheel.check_compatibility API that people could call *before* manually inserting the wheel onto sys.path. That's a better separation of concerns, in my view. There's no problem with providing a compatibility check API, but the other POV is that people might forget to call that API, so it seems to make sense to call it from the mount method too. And I don't see why you say all it adds - surely the check is important to prevent a certain class of weird things happening. OK, I misspoke. All that mount adds over a compatibility API is a sys.path.insert call (*if* you agree with my point that the C extension stuff shouldn't be used - and I know you don't). Breaking mount down: 1. Compatibility check - I support this and would like a separate API 2. sys.path modification - trivial for anyone who knows enough to be using this API 3. C extension support - we'll have to agree to disagree here I can't give specific examples of failure modes because I don't use the wheel mount functions, nor do I typically
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 2:32 AM, Nick Coghlan ncogh...@gmail.com wrote: All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic. However, for people that are comfortable with the import system and the limitations of direct zip imports, it's a very useful feature. I wouldn't have accepted PEP 427 without the zipimport compatibility that meant a developer *could* use it as a direct replacement for eggs if they really wanted to. Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities). I clarified PEP 427 specifically because Armin Ronacher's wheel article showed that he was unaware of this *deliberately included* feature of the wheel format. People are free not to like it - the default tools deliberately make it less convenient to run things that way, and that's as it should be. However, it's not a super-secret capability only to be used by us to implement things like ensurepip - it's a defined capability of the format that if your software is capable of running correctly from a zip archive in the first place, then that archive can be also be a valid wheel file. Running directly from a wheel is a power tool - that's a reason to put handle with care warnings on it, not to refuse to support a feature that was deliberately designed into the format. You can do a lot more damage with a badly written meta-importer, yet we have no intention of deprecating that capability either. Even *.pth files have turned out to have a valid use case for sharing packages between virtual environments. We have lots of features like that elsewhere in Python - when people ask about metaclasses, the first reaction is You probably don't want to use them. However, sometimes developers *do* need them, and that's why the feature exists. Most of the time developers won't want to make use of the zipimport compatibility of wheel files, either, but advanced use cases like ensurepip are exactly why the capability exists. I can make the new note in the PEP more explicit that while this is a supported use case that ensures the feature set provided by wheels is a strict superset of that provided by eggs, that's not the same thing as *recommending* that wheels be used that way. Cheers, Nick. I don’t think it particularly matters wether you would have accepted the PEP without that ability or not. You *did* accept the PEP when the text of the PEP directly pointed out that one of the differences of Wheel and Egg were that “Wheel is an installation format, Egg is importable” which points to that fact that Wheel was not designed to be importable. As far as I can tell you’ve added this “feature” to the PEP by fiat without any chance for anyone to be able to discuss it after it’s already been accepted. I’m well aware that we cannot prevent Wheels from being imported, but that’s another thing all together from directly supporting it. For one thing by supporting it we forever lock ourselves into dumping what would be in site-packages directly in the root of the Wheel because adding the .whl file directly to sys.path is now a documented feature. This behavior has personally caused me annoyance and one of the things I was hoping on doing in a Wheel 1.2 was revise the layout a bit to make installation simpler and make inspecting a Wheel file easier. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 6:47 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 2:32 AM, Nick Coghlan ncogh...@gmail.com wrote: All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic. However, for people that are comfortable with the import system and the limitations of direct zip imports, it's a very useful feature. I wouldn't have accepted PEP 427 without the zipimport compatibility that meant a developer *could* use it as a direct replacement for eggs if they really wanted to. Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities). I clarified PEP 427 specifically because Armin Ronacher's wheel article showed that he was unaware of this *deliberately included* feature of the wheel format. People are free not to like it - the default tools deliberately make it less convenient to run things that way, and that's as it should be. However, it's not a super-secret capability only to be used by us to implement things like ensurepip - it's a defined capability of the format that if your software is capable of running correctly from a zip archive in the first place, then that archive can be also be a valid wheel file. Running directly from a wheel is a power tool - that's a reason to put handle with care warnings on it, not to refuse to support a feature that was deliberately designed into the format. You can do a lot more damage with a badly written meta-importer, yet we have no intention of deprecating that capability either. Even *.pth files have turned out to have a valid use case for sharing packages between virtual environments. We have lots of features like that elsewhere in Python - when people ask about metaclasses, the first reaction is You probably don't want to use them. However, sometimes developers *do* need them, and that's why the feature exists. Most of the time developers won't want to make use of the zipimport compatibility of wheel files, either, but advanced use cases like ensurepip are exactly why the capability exists. I can make the new note in the PEP more explicit that while this is a supported use case that ensures the feature set provided by wheels is a strict superset of that provided by eggs, that's not the same thing as *recommending* that wheels be used that way. Cheers, Nick. I don’t think it particularly matters wether you would have accepted the PEP without that ability or not. You *did* accept the PEP when the text of the PEP directly pointed out that one of the differences of Wheel and Egg were that “Wheel is an installation format, Egg is importable” which points to that fact that Wheel was not designed to be importable. As far as I can tell you’ve added this “feature” to the PEP by fiat without any chance for anyone to be able to discuss it after it’s already been accepted. I’m well aware that we cannot prevent Wheels from being imported, but that’s another thing all together from directly supporting it. For one thing by supporting it we forever lock ourselves into dumping what would be in site-packages directly in the root of the Wheel because adding the .whl file directly to sys.path is now a documented feature. This behavior has personally caused me annoyance and one of the things I was hoping on doing in a Wheel 1.2 was revise the layout a bit to make installation simpler and make inspecting a Wheel file easier. As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the chance to discuss it. It should not be added ex post facto by a committer to the PEP repo just because they want Wheel to have it. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Wed, Jan 29, 2014 at 3:41 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the wheel install command. I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick's most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable. FWIW, Nick clarified this aspect of wheels on this list at least once -- back in Sep 2012 when the PEP was first introduced. See this email: https://mail.python.org/pipermail/distutils-sig/2012-September/018959.html It's also not quite true that the contents of a wheel aren't importable - since they're still just a zipfile, they can still be added to an import path in the same manner as egg files. It's just not *recommended* to use them that way, as the format is designed primarily for use in distribution rather than runtime imports. --Chris ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
That doesn't really speak to the fact they were designed for that. If I read that it looks like some commenting that they are importable (which as it stands they are) and not someone calling it a supported feature of the format. It even states that the format is designed primarily for distribution. Given that the PEPs text had always been contrary to the position that wheels were designed to be importable I don't think you can draw any other conclusion. On Jan 29, 2014, at 6:52 AM, Chris Jerdonek chris.jerdo...@gmail.com wrote: On Wed, Jan 29, 2014 at 3:41 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the wheel install command. I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick's most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable. FWIW, Nick clarified this aspect of wheels on this list at least once -- back in Sep 2012 when the PEP was first introduced. See this email: https://mail.python.org/pipermail/distutils-sig/2012-September/018959.html It's also not quite true that the contents of a wheel aren't importable - since they're still just a zipfile, they can still be added to an import path in the same manner as egg files. It's just not *recommended* to use them that way, as the format is designed primarily for use in distribution rather than runtime imports. --Chris ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 11:41, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the wheel install command. I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick's most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable. Apologies. It was something Daniel pointed out a few times very early on - I hadn't realised it wasn't in the spec directly. What is in the spec - which effectively constrains the format to *allowing* (rather than encouraging) direct import - are the facts that wheels are zip format, and that one of purelib/platlib is at the root. The concept of separating unpack and spread and the comment Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages doesn't leave any room for wheels *not* to be importable in the majority of pure-python, no package data, cases. Debating how we present this in later versions of the wheel spec is fine. But deliberately making wheels not importable would break backward compatibility in a way that would have other, likely more serious, implications. Nevertheless, I understand your concerns, and I think we should be very careful not to let people get the impression that this is in any way similar to importable eggs, which had a very bad press. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 6:59 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 11:41, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the wheel install command. I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick's most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable. Apologies. It was something Daniel pointed out a few times very early on - I hadn't realised it wasn't in the spec directly. What is in the spec - which effectively constrains the format to *allowing* (rather than encouraging) direct import - are the facts that wheels are zip format, and that one of purelib/platlib is at the root. The concept of separating unpack and spread and the comment Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages doesn't leave any room for wheels *not* to be importable in the majority of pure-python, no package data, cases. Debating how we present this in later versions of the wheel spec is fine. But deliberately making wheels not importable would break backward compatibility in a way that would have other, likely more serious, implications. Nevertheless, I understand your concerns, and I think we should be very careful not to let people get the impression that this is in any way similar to importable eggs, which had a very bad press. Paul I read that statement differently, in that it doesn’t guarantee that the files would be at the *root* of the Wheel, just that you could unpack a Wheel into a site-packages directory using unzip and have it be “installed”. I can see how it could be read otherwise though. In either case this is something that is more able to be removed in later versions of Wheel because unpacking by hand is something I don’t believe will ever be commonplace, and any installer that doesn’t check it’s Wheel version before doing things with the Wheel is wrong. But more importantly, while I’m against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in without any chance for discussion. I don’t think anyone can call me a casual observer of distutils-sig or the various PEP processes and this was the first time that I had heard of Wheels being importable as anything other than a sometimes useful hack that wasn’t a design goal but instead just an accident of implementation. I followed the PEP closely trying to make sure that we weren’t going to add a supported feature that locked us into any corners while the format was still new and this is something I would have fought against had it been in the original PEP. Nick may have, in his head, considered this to be a feature of Wheel all along and not an implementation detail, however that is not what the PEP stated. I don’t believe that as BDFL-Delegate for packaging issues Nick should be able to add supported features to an already accepted PEP without discussion especially when the existing text of that PEP is directly contrary to that feature being part of the PEP. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 12:18, Donald Stufft don...@stufft.io wrote: But more importantly, while I'm against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in without any chance for discussion. Fair point. I'm sure Nick did just think of it as clarification - I know I did - but the process point is reasonable. I'm happy to leave off arguing with your position till the point comes for proper debate ;-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 7:43 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:18, Donald Stufft don...@stufft.io wrote: But more importantly, while I'm against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in without any chance for discussion. Fair point. I'm sure Nick did just think of it as clarification - I know I did - but the process point is reasonable. I'm happy to leave off arguing with your position till the point comes for proper debate ;-) Paul Let me be explicit that I don’t think Nick is doing anything nefarious here, I do believe that he probably did think of it as a clarification. I’m concerned with the fact that the PEP did not support this when it was accepted and contained language that, to me at least, is contrary to the fact that this was supported, and then the new text was addd to an already accepted PEP giving it a new feature without any chance for discussion or disagreement. If I wasn’t anal about reading every email that hits my inbox *and* I wasn’t on python-checkins I never would have noticed this addition until it was likely too late to do anything about it because it had been documented as part of the PEP long enough that people started to depend on it. So hopefully nobody thinks I’m calling Nick a bad actor in this case. Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then in the next version of the Wheel spec it should be added when everyone has a chance to properly discuss it. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 21:50, Donald Stufft don...@stufft.io wrote: As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the chance to discuss it. It should not be added ex post facto by a committer to the PEP repo just because they want Wheel to have it. It was discussed during the approval process for the wheel format, I approved it on the basis that wheels provided a superset of the functionality of eggs. I merely missed the fact it wasn't properly recorded in the PEP (which is now rectified). You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 7:58 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 21:50, Donald Stufft don...@stufft.io wrote: As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the chance to discuss it. It should not be added ex post facto by a committer to the PEP repo just because they want Wheel to have it. It was discussed during the approval process for the wheel format, I approved it on the basis that wheels provided a superset of the functionality of eggs. I merely missed the fact it wasn't properly recorded in the PEP (which is now rectified). You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia Can you point to this discussion? Because I cannot find it. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a special exception here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 22:57, Donald Stufft don...@stufft.io wrote: Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then in the next version of the Wheel spec it should be added when everyone has a chance to properly discuss it. Whether you like it or not, the wheel spec *does* already support being used that way, and changing it *would* be a backwards incompatible change to the format (and hence not acceptable without a compelling justification, and there isn't one that wouldn't equally well apply to other powerful but potentially confusing features like metaclasses and monkeypatching). I am also not willing to allow the perception that the wheel format does not offer a strict superset of the feature of the egg format to persist until we have a defined wheel 1.1 spec (which we don't even have a volunteer to work on yet). So, rather than removing it, I have instead updated the note in PEP 427 to read: = Is it possible to import Python code directly from a wheel file? Yes, the wheel format is deliberately designed to be compatible with Python's support for importing from zip files, ensuring that it provides a superset of the functionality provided by the preceding egg format. However, this is generally not a *recommended* approach to using wheel files, as importing from a zip file rather than an ordinary filesystem directory imposes a number of additional constraints that will often break the assumptions of Python developers (for example, C extensions cannot be imported directly from a zip archive, and the ``__file__`` attribute no longer refers to an ordinary filesystem path, but to a combination path that includes both the location of the zip archive on the filesystem and the relative path to the module inside the archive). Like metaclasses and metapath importers, if you're not sure if you need to take advantage of this feature, you almost certainly don't need it. = Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:14 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 22:57, Donald Stufft don...@stufft.io wrote: Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then in the next version of the Wheel spec it should be added when everyone has a chance to properly discuss it. Whether you like it or not, the wheel spec *does* already support being used that way, and changing it *would* be a backwards incompatible change to the format (and hence not acceptable without a compelling justification, and there isn't one that wouldn't equally well apply to other powerful but potentially confusing features like metaclasses and monkeypatching). I am also not willing to allow the perception that the wheel format does not offer a strict superset of the feature of the egg format to persist until we have a defined wheel 1.1 spec (which we don't even have a volunteer to work on yet). So, rather than removing it, I have instead updated the note in PEP 427 to read: = Is it possible to import Python code directly from a wheel file? Yes, the wheel format is deliberately designed to be compatible with Python's support for importing from zip files, ensuring that it provides a superset of the functionality provided by the preceding egg format. However, this is generally not a *recommended* approach to using wheel files, as importing from a zip file rather than an ordinary filesystem directory imposes a number of additional constraints that will often break the assumptions of Python developers (for example, C extensions cannot be imported directly from a zip archive, and the ``__file__`` attribute no longer refers to an ordinary filesystem path, but to a combination path that includes both the location of the zip archive on the filesystem and the relative path to the module inside the archive). Like metaclasses and metapath importers, if you're not sure if you need to take advantage of this feature, you almost certainly don't need it. = Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia So basically even though the text of the PEP specifically points out that a difference of Wheel and Egg is that Eggs are importable it somehow supports that? Can you point to a single line in the PEP that supports this besides the ones you’ve added? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 23:10, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a special exception here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Not just PyPA - this feature is at the heart of how the ensurepip module in the standard library works. This approach is explicitly documented in PEP 453: http://www.python.org/dev/peps/pep-0453/#implementation-strategy Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:17 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:10, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a special exception here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Not just PyPA - this feature is at the heart of how the ensurepip module in the standard library works. This approach is explicitly documented in PEP 453: http://www.python.org/dev/peps/pep-0453/#implementation-strategy Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia There is a very important distinction. Both of those are designed to work with very specific Wheels and relied on, until this change, private, undocumented, and undiscussed features of Wheel. It’s not unusual for the Python standard library to make use of internal APIs to make it’s features work nor is it unusual for Virtualenv to do that either because of the nature of it bolting things on to Python that Python wasn’t designed to do. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a special exception here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Paul Here’s Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 23:31, Donald Stufft don...@stufft.io wrote: Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html Donald, I was *there*. I know what we discussed, and I know what PEP I approved. Trying to play gotcha with links *isn't going to work*. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:31 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:16, Donald Stufft don...@stufft.io wrote: So basically even though the text of the PEP specifically points out that a difference of Wheel and Egg is that Eggs are importable it somehow supports that? Can you point to a single line in the PEP that supports this besides the ones you've added? I added the clarification based on the facts that: 1. We discussed this extensively before PEP 427 was approved, and this was an accepted feature of the design 2. Root-is-purelib only makes sense in the context of supporting the feature 3. Both ensurepip and virtualenv rely on the feature 4. PEP 453 explicitly documents ensurepip's reliance on the feature, with no caveat about this being unsupported in the spec 5. I wouldn't have accepted PEP 427 if wheels didn't provide a strict superset of the features provided by eggs We make mistakes, and things that were discussed and agreed don't always get properly captured in the corresponding PEPs. When that happens, it's a judgement call as to whether properly documenting that is a new feature requiring a new PEP, or merely a clarification to the existing one. For standard library PEPs, we often don't do either - we just fix the implementation without going back for another round of PEP discussions (for smaller tweaks, sometimes we don't even go back to python-dev and instead just resolve things on the tracker). In this case, as BDFL-Delegate, I decided it was a case that merely called for clarification, because I *know* what spec I accepted, and it was the one where wheels offer all the features that eggs do and more. I added the new text specifically because people like Armin Ronacher and yourself had gained an idea from the PEP text that emphatically does *not* align with the design discussions that occurred prior to the acceptance of the PEP. Now, if you'd like to campaign to *remove* this support, then explain your rationale, and make the case for why you think providing the feature is so dangerous that removing it is worth breaking backwards compatibility over. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia This was supposedly extensively discussed prior to PEP427 being accepted yet I have no memory of this being discussed, am unable to find any discussion of it (other than one offs saying it’s possible but not a core feature), and you’ve been apparently unwilling to point to any discussion. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:32 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:31, Donald Stufft don...@stufft.io wrote: Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html Donald, I was *there*. I know what we discussed, and I know what PEP I approved. Trying to play gotcha with links *isn't going to work*. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia If I’m wrong, then by all means show me where it was discussed so I can admit I was wrong. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 28, 2014, at 9:54 PM, Daniel Holth dho...@gmail.com wrote: On Tue, Jan 28, 2014 at 8:04 PM, Evgeny Sazhin eug...@sazhin.us wrote: Hi, I saw that people from this list are responsible for Wheel related PEP. I'm comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. I'm trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... So, i would like to ask few questions about it : 1. I'm coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn't it be way easier to deal with the dependencies and imports? 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can't be used the same exact way? I would appreciate any insight about the development infrastructure the wheel designers have in mind. Thanks in advance, Eugene The main reason the packaging format is not explicitly importable is simply because Python hasn't had jar-like deployment for as long or as consistently as Java. Thanks a lot for taking time to respond! Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? If wheel is promoted as the ultimate solution to the packaging problems (at least that’s my impression ) aren’t the abilities to import packages and classes from it as well as make it runnable a kind of top priority and something that is right on the surface? So while Java code universally uses a get resource API to get stuff on the classpath, a a lot of Python code will try to open a file. C extensions can't be loaded from inside zip files. I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? I’d think wheel should do everything in its power to provide pain free workflow for pure python, and if there is a need to add some C extension then solve that problem separately? Is the possibility to add C extension into the wheel so critical, that it should prohibit useful feature for pure python modules? And it's a lot harder to edit .py files once they are zipped up, unlike Java where only the compiled and non-editable .class files are zipped. Why would i want to edit the zipped .py file from a wheel? I want to import a package or a class from it, reuse a function. Nothing more. ZIP import is a great feature but wheels aren't really designed for it. It's more reliable that way. More reliable in what regard? My impression is that the whole thing about pip, virtualenv and wheels is great - it takes into consideration a lot of end user problems, aka getting proper module and installing it to RUN applications. But for whatever reason it misses some important problems on the DEVELOPMENT side. I’m trying to find a sane way for developer to work, so if wheel is not an answer - what is? Deprecated Egg? Or may be then egg is not going to go anywhere and pip has to learn to deal with them as well as it does with wheels? Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On the one hand, it's easy to see the allure of a zipimport-able format. Grab a file, import it, use functionality straight away. It has an attraction of self-containedness. On the other hand, python's dynamic nature means that things are not as simple as Java jar's as Donald points out. I agree that wheels shouldn't officially support this feature wholesale. However, perhaps they could do so conditionally? E.g. designate some mechanism by which it is possible to inspect a wheel and determine that it intends to explicitly be zipimport compatible. A prime candidate would be that the wheel is of a purelib in nature, another would be to add a top-level file or metadata indicating such (e.g. '.zipimportable'). But other wise I agree that we should explicitly seek to avoid any language that could lead users to expect this functionality of wheels wholesale in any way, even though they happen to be regular zip files. On 29 Jan 2014 17:09, Donald Stufft don...@stufft.io wrote: On Jan 28, 2014, at 11:14 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. I know that people have had bad experiences in the past with eggs, for the reasons Nick outlined in his response. However, you don't say *with specifics* why you think putting wheels on sys.path is a bad idea - I don't think it's convincing enough just to hand-wavingly reference problems with the egg format. When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC. One can't say that having the same packaging and importable formats is inherently bad, since Java shows otherwise. So if there are specific problems, they should be identified. Last time this topic came up Jim Fulton did point things out. 1. That unpacking into the home directory is problematic because users that run services often don't have home directories. This you waved away by saying that the home directory isn't a required place, to which Jim responded that unpacking them anywhere else was likely to have access control / security issues and generally cause pain. Which I wholly agree with, intact Mitre has assigned CVE's to applications that did switch their Egg Cache to directories other than HOME. 2. Zipped Eggs are more difficult to work with for debugging purposes, which you responded saying that Wheels are a deployment format (which I don't believe is supported by the PEP at all) and Jim responded with his experience in developing against things that got installed as zipped Eggs and the pain that has personally caused him. 3. Zipped imports (In Jim's experience) are slower than unpacked imports, which you replied (quite literally) Caveat emptor. Jim's response was that this had been tried with Egg and had been found to be more pain than it was worth. In his exact words: It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment. I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony. He closes up with, Importing from zipped eggs has proved itself to be an anti pattern. Buildout (as of buildout 2) always unzips eggs. So you have pip that doesn't use zipped Eggs, buildout which made sure in buildout 2 (presumably drawing from the lessons of buildout 1) to always unzip eggs, and easy_install which does support zipped eggs (sometimes, if it thinks that particular Egg will support it). Why do we insist upon repeating the mistakes of the past? Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). I don't yet see a technical impediment here. For example, distlib's wheel implementation uses a mount method to add a wheel to sys.path. This uses the tags to check for compatibility - there's no figuring out that the user has to do. If additional metadata about C extensions is available (which it is, if the wheel is built from source using distil), then those extensions are made available for import, too. I realise that's an extension to the current spec, but then no one is forcing any one to use it. Zipped Eggs have a long long history of causing very weird failures conditions, you say your Wheel.mount checks for compatibility, does it also check that the library to be installed isn't using __file__
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:34 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:32 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:31, Donald Stufft don...@stufft.io wrote: Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html Donald, I was *there*. I know what we discussed, and I know what PEP I approved. Trying to play gotcha with links *isn't going to work*. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia If I’m wrong, then by all means show me where it was discussed so I can admit I was wrong. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig Ironically, in another thread [1] on distutils-sig we have Daniel Holth, the author of the PEP stating: The main reason the packaging format is not explicitly importable is simply because Python hasn't had jar-like deployment for as long or as consistently as Java. So while Java code universally uses a get resource API to get stuff on the classpath, a a lot of Python code will try to open a file. C extensions can't be loaded from inside zip files. And it's a lot harder to edit .py files once they are zipped up, unlike Java where only the compiled and non-editable .class files are zipped. ZIP import is a great feature but wheels aren't really designed for it. It's more reliable that way. [1] https://mail.python.org/pipermail/distutils-sig/2014-January/023554.html - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:43 AM, Daniel Holth dho...@gmail.com wrote: On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a special exception here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Paul Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html I designed it very intentionally to be compatible with zip import. Otherwise wheels would probably not even be zip files (like pypm packages which are tar) and would achieve greater compression. It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. Then why did you tell someone on this very list a few hours ago that Wheels were not really designed to be zip imported. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Wed, Jan 29, 2014 at 8:44 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:43 AM, Daniel Holth dho...@gmail.com wrote: On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a special exception here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Paul Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html I designed it very intentionally to be compatible with zip import. Otherwise wheels would probably not even be zip files (like pypm packages which are tar) and would achieve greater compression. It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. Then why did you tell someone on this very list a few hours ago that Wheels were not really designed to be zip imported. It's shorter. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:46 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:34, Donald Stufft don...@stufft.io wrote: If I'm wrong, then by all means show me where it was discussed so I can admit I was wrong. You have the burden of proof backwards there. You're the one asking me to break backwards compatibility, and to let people continue to believe wheels are only a partial replacement for eggs - it's up to you to make the case that waiting until wheel 1.1 (which, again, has nobody committed to writing it and a completely unspecified timeline) is a superior approach to clarifying something I thought was already documented when I accepted the PEP (and is certainly inherent in the design of the format). The zipimport compatibility didn't need to be discussed much because it was there in Daniel's original wheel design - there was never any proposal to use a zipimport *incompatible* approach, so nobody had to campaign in favour of zipimport compatibility. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia So what did you mean when you said “We discussed it extensively before PEP 427 was approved” if you’re now saying that it wasn’t discussed. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a special exception here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Paul Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html I designed it very intentionally to be compatible with zip import. Otherwise wheels would probably not even be zip files (like pypm packages which are tar) and would achieve greater compression. It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 23:34, Donald Stufft don...@stufft.io wrote: If I'm wrong, then by all means show me where it was discussed so I can admit I was wrong. You have the burden of proof backwards there. You're the one asking me to break backwards compatibility, and to let people continue to believe wheels are only a partial replacement for eggs - it's up to you to make the case that waiting until wheel 1.1 (which, again, has nobody committed to writing it and a completely unspecified timeline) is a superior approach to clarifying something I thought was already documented when I accepted the PEP (and is certainly inherent in the design of the format). The zipimport compatibility didn't need to be discussed much because it was there in Daniel's original wheel design - there was never any proposal to use a zipimport *incompatible* approach, so nobody had to campaign in favour of zipimport compatibility. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
Nick Coghlan ncoghlan at gmail.com writes: I believe Paul's concern is with anything that suggests that arbitrary *third party* code can be run from wheel files, when the reality is that it is fairly easy to accidentally write code that assumes it is installed on the filesystem in a way that isn't easy for a quick scan of the files in the zip archive to detect (especially since the PEP 376 installation database PEP doesn't include any support for arbitrary metapath importers). That is a valid concern, but no one is suggesting that arbitrary third party code can run from wheel files, just as zipimport makes no guarantees about zipped code working. By contrast PEP 441 is a *distribution* utility - the creator of the application is expected to ensure that doing so actually works correctly before publishing their app that way, just as we would expect py2exe, py2app and cx-freeze users to do. True, but there's no reason why wheels couldn't have some metadata indicating that this diligence has been exercised by the wheel creator. With the reference implementation position that distlib is likely to occupy in a post-PEP-426/440/459 world, though, there's an additional legitimate concern about allowing end users to easily distinguish between this API is fully supported by the PyPA as part of the reference implementation for metadata 2.0 and this is an experimental packaging related API that may or may not be useful in general, and some members of the PyPA may still have grave reservations about it. At the moment, distlib contains both kinds of API, and it confuses *us*, let alone anyone else that isn't closely following along on distutils-sig. As long as distlib is serving the dual role of providing both the reference implementation for metadata 2.0 and some experimental packaging related APIs, we're going to get concerns like this one arising. If there was a clear way to distinguish them (ideally with a separate project for either the reference implementation or the experimental stuff, but even a distinct namespace within the distlib project would help a great deal), I suspect there would be less concern. These are social concerns perhaps more than technical concerns, and to me they lack specificity. Of course some of the APIs in distlib are new and untried-except-by-me, but the way to allay concerns is to focus on specifics, force out the details of the concerns and then see how best they can be addressed. This is not doable with zipped-eggs-were-bad rhetoric. Details generally help to identify what the real problem is. For example, Donald raised the spectre of security vulnerabilities with his mention of Mitre and CVEs, but there were no specifics beyond that. I found a discussion where someone had set PYTHON_EGG_CACHE to /tmp. I can certainly see the negative security implications of that, but the finger was pointed at the using applications rather than setuptools. Even though setuptools specifically added code as a remedy to warn when the env var pointed to a world-writeable directory, this was seen as trying to be helpful rather than patching a vulnerability. Of course, if I've misunderstood something in that discussion or missed some other security issue, then some pointers would help move the discussion along. In the specific case of distlib.mount, if it's eventually combined with a metadata extension like distlib.mount which packages must export in order for the command to allow them to be automatically used that way, then I don't see anything wrong with it *in general* - it's a natural extension of the setuptools zip_safe flag, but with the ability to include additional details (like whether or not there are C extensions that need to be automatically extracted). Are you talking just about adding wheels to sys.path, or do you mean the extension-extraction stuff? Note that distlib's Wheel.mount does a compatibility check and addition to sys.path, which I feel is not especially controversial and better than just adding to sys.path, which user code can now do, anyway. But nothing else happens, unless specific metadata is provided in the wheel to enable it. While it's not specifically a distlib.mount export, there is a facility to ask for extensions to be extracted, and the in absence of metadata asking for this, no extraction occurs. goes further than the current EXTENSIONS approach - this proposal would be akin to *requiring* an empty EXTENSIONS file, and/or the setuptools zip_safe flag in order to allow mounting of even the pure Python wheel. Such a conservative approach is also the antithesis of the setuptools attempt to guess: if the package publisher doesn't explicitly opt in to zip support, then distlib.mount would assume that it is *not* supported (but may provide an API for the caller to override that, like assume_zip_safe=True or force=True). I have no problem with adding wheel metadata to allow/disallow even adding to sys.path - it's
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 23:48, Donald Stufft don...@stufft.io wrote: So what did you mean when you said We discussed it extensively before PEP 427 was approved if you're now saying that it wasn't discussed. Explicitly would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html Like I said, that particular aspect wasn't controversial, so while it was noted a few times (a few other examples of which you found), it was the overall discussions that were extensive. Both Daniel and I knew the zipimport compatibility for packages that were themselves zip compatible was a deliberate feature, so it was a surprise to me when Armin Ronacher said in his recent article that it wasn't supported (and hence the clarification). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On 29 January 2014 13:41, Evgeny Sazhin eug...@sazhin.us wrote: I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? Most interesting Python projects will end up with C dependencies through things like NumPy, SQL Alchemy, GUI toolkits, image libraries, etc. You can do a lot with pure Python and the standard library, but interfacing with C is the norm rather than the exception the way it is in Java. I'd think wheel should do everything in its power to provide pain free workflow for pure python, and if there is a need to add some C extension then solve that problem separately? Is the possibility to add C extension into the wheel so critical, that it should prohibit useful feature for pure python modules? The Python culture and the Java culture is very different in that regard - the JVM allows Java developers to almost completely ignore the C runtime on a system. CPython, by contrast, integrates with that C runtime directly, and this is then reflected in the ecosystem that has built up around it. So while there is some support for running without unpacking things to disk, we generally prefer approaches that work seamlessly with C extensions as well. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? There is a parallel discussion going on, with the title Using Wheel with zipimport, which is relevant to this question, and other questions you raised (e.g. about supporting C extensions/pure-Python modules. I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? A lot of Python users depend on C extensions - and while it is a subset of all Python users, it is a large (and important) subset. Example: any usage of Python in numerical analysis or scientific applications involves use of C extensions. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 8:59 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:48, Donald Stufft don...@stufft.io wrote: So what did you mean when you said We discussed it extensively before PEP 427 was approved if you're now saying that it wasn't discussed. Explicitly would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html Like I said, that particular aspect wasn't controversial, so while it was noted a few times (a few other examples of which you found), it was the overall discussions that were extensive. Both Daniel and I knew the zipimport compatibility for packages that were themselves zip compatible was a deliberate feature, so it was a surprise to me when Armin Ronacher said in his recent article that it wasn't supported (and hence the clarification). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia That also mentions the fact that he was considering allowing other compression algorithms which would make it incompatible with zip import. I don’t agree that this points to any promise that Wheels are themselves importable as I can guarantee I poured over that spec over and over making sure we weren’t adding something from Egg that had been problematic in the past. I have no plans to support adding Wheels to sys.path in a way that is supported for end users to do that, in any tool or project I work on. But whatever, I feel like you’ve placed yourself above the PEP process and are adding features by fiat instead of through the process. Obviously you feel different and this discussion has gone toxic. So carry on. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to system behaviour and because the ramifications of changing it under the hood are unpredictable. But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way. What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about pain and weirdness. I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound naïve to some - but there it is. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 9:25 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to system behaviour and because the ramifications of changing it under the hood are unpredictable. But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way. What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about pain and weirdness. I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound naïve to some - but there it is. Regards, Vinay Sajip It’s hard to pin down because the failure modes of zipped eggs are nebulous themselves. For instance take pip. I just recently redid the get-pip.py installer to use a zip file (not a Wheel or Egg) that contained pip and add that zip file straight to sys.path instead of unzipping it. One of the failure modes was that it was suddenly unable to validate TLS certificates. The reason why, was because it bundles it’s own cacerts.pem which it passes to the ssl library to validate the connection. In this case it’s impossible as far as I can tell to use pkgutil.get_data directly because the ssl library does not directly support pkgutil.get_data nor does it support passing the certificates as a string or other in memory. That’s really the biggest problem with strictly pure python Zipped Eggs. That the failure modes are ill defined and depend greatly on the application/library itself. For some you’ll get exceptions, for others some features will just stop working, for even others you’ll just get subtle bugs that only happen under zip import. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Wed, 29/1/14, Donald Stufft don...@stufft.io wrote: Mitre’s rules for CVEs are not entirely obvious to people who are not familiar with them. Generally if the feature *can* be used securely or there was no evidence that the author intended that the code be secure they will not issue a CVE. The issue is that the feature makes a very attractive footgun for people using it to do the wrong thing and have it be a very bad idea. So, was a CVE issued against setuptools? My understanding is that it wasn't - have I misunderstood? tool = setuptools footgun = configurability of egg cache using PYTHON_EGG_CACHE trigger = setuptools user sets PYTHON_EGG_CACHE to a world writeable directory shot = malicious user replaces eggs in the cache with malicious code BTW when no HOME directory is available, distlib uses tempfile.mkdtemp() which IIUC provides a directory with permissions of 0700, which should be safe from tampering. Do you see a security problem with this? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Wed, Jan 29, 2014 at 9:40 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 9:25 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to system behaviour and because the ramifications of changing it under the hood are unpredictable. But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way. What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about pain and weirdness. I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound naïve to some - but there it is. Regards, Vinay Sajip It’s hard to pin down because the failure modes of zipped eggs are nebulous themselves. For instance take pip. I just recently redid the get-pip.py installer to use a zip file (not a Wheel or Egg) that contained pip and add that zip file straight to sys.path instead of unzipping it. One of the failure modes was that it was suddenly unable to validate TLS certificates. The reason why, was because it bundles it’s own cacerts.pem which it passes to the ssl library to validate the connection. In this case it’s impossible as far as I can tell to use pkgutil.get_data directly because the ssl library does not directly support pkgutil.get_data nor does it support passing the certificates as a string or other in memory. That’s really the biggest problem with strictly pure python Zipped Eggs. That the failure modes are ill defined and depend greatly on the application/library itself. For some you’ll get exceptions, for others some features will just stop working, for even others you’ll just get subtle bugs that only happen under zip import. Just a heads-up that if I ever get around to re-implementing zipfile importing using importlib (http://bugs.python.org/issue17630) I will ask this list for feature feedback to make sure I don't miss something that's critical. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] problems with eggs?
Here are some of the problems with eggs that I tried to solve: - Cannot be unzipped on top of each other due to the EGG-INFO directory. Wheels repeat the package name in the dist-info directory and are more like their installed layout, hopefully taking all the mystery out of the format. - The egg naming scheme doesn't let you distinguish between Python interpreter implementations (not an important issue at the time). - They include .pyc and so are always [at least somewhat] Python version specific. Wheels don't. - Eggs do not have categories of files. Wheel uses subdirectories of the .data/ directory, and are therefore a complete replacement for most other kinds of bdist_x - And of course wheels use .dist-info which is also something that is newer than eggs, and generally try to bring the useful packaging PEP work into reality. I remember dealing with the zipped eggs hassle, and I remember having to wait too long for binary packages to be uploaded to pypi as eggs when a new Python version was released. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 30 Jan 2014 00:28, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to system behaviour and because the ramifications of changing it under the hood are unpredictable. But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way. What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about pain and weirdness. I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound naïve to some - but there it is. I went through this with Chris McDonough back when packaging was dropped from 3.3, and he really helped me focus on what I found to be the two closely related core problems: - implicit sys.path manipulation - installing as eggs by default That was due to easy_install defaults being chosen for the Chandler use case rather than installing into a shared environment, but the egg format (and zip imports in general) still ended up being tainted by association. More recently, I've become concerned about the way setuptools/easy_install assumes that an egg is zip import compatible in the absence of clear markers that it isn't. pip's wheel support avoids those issues, because the normal workflows just install things normally. It takes special effort to go to the trouble of keeping a wheel file *as* a wheel and then putting it on sys.path later. distlib.mount already has the virtue of always being opt-in from the user point of view, and the C extension support is opt-in from the publisher point of view, but it would be improved by needing the publisher to opt in to declaring support for zipimport (or meta imports in general), rather than letting end users deal with debugging obscure errors from code that assumes it has been fully installed. Cheers, Nick. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Wed, 29/1/14, Donald Stufft don...@stufft.io wrote:
It’s hard to pin down because the failure modes of zipped
eggs are nebulous themselves.
For instance take pip. I just recently redid the get-pip.py
installer to use a zip file (not a Wheel or Egg) that contained pip
and add that zip file straight to sys.path instead of unzipping it.
One of the failure modes was that it was suddenly unable to validate
TLS certificates. The reason why, was because it bundles it’s
own cacerts.pem which it passes to the ssl library to validate the
connection. In this case it’s impossible as far as I can tell to use
pkgutil.get_data directly because the ssl library does not directly
support pkgutil.get_data nor does it support passing the certificates
as a string or other in memory.
That’s really the biggest problem with strictly pure python Zipped Eggs.
If that's all, I'm not overly worried as long as there's a mountability flag on
wheels. I looked at running pip from a zip a while ago, the cacerts thing
was easy to diagnose and fix - a five-or-ten-minute job. (I'm not trying to
come across as a show-off.) Here's the code I added as a fix in
pip/locations.py (no doubt it could be improved):
def _check_for_zip():
import zipimport
global default_cert_path
try:
if isinstance(__loader__, zipimport.zipimporter):
# running from a zip file. Save data to a handy location.
data = __loader__.get_data(default_cert_path)
destdir = os.path.expanduser('~/.pip')
if not os.path.isdir(destdir):
os.mkdir(destdir)
default_cert_path = os.path.join(destdir, 'cacert.pem')
with open(default_cert_path, 'wb') as f:
f.write(data)
except NameError:
# __loader__ not defined, so not in zip
pass
_check_for_zip()
del _check_for_zip
Of course, pip wasn't originally designed to run from zip, so one would
expect to find these kinds of issues. OTOH, if importable zips were more
commonplace, then presumably people would think about these kinds of
issues more when writing their code. Of course, in the above case, using
pkgutil wouldn't have worked, and there are bound to be other similar cases,
but I still don't see any real reason to fear the consequences of importable
wheels.
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
I went through this with Chris McDonough back when packaging was dropped from 3.3, and he really helped me focus on what I found to be the two closely related core problems: - implicit sys.path manipulation - installing as eggs by default That was due to easy_install defaults being chosen for the Chandler use case rather than installing into a shared environment, but the egg format (and zip imports in general) still ended up being tainted by association. But now that we have identified that it was by association, we can hopefully make more informed decisions and not throw the baby out with the bath-water. pip's wheel support avoids those issues, because the normal workflows just install things normally. It takes special effort to go to the trouble of keeping a wheel file *as* a wheel and then putting it on sys.path later. distlib.mount already has the virtue of always being opt-in from the user point of view, and the C extension support is opt-in from the publisher point of view, but it would be improved by needing the publisher to opt in to declaring support for zipimport (or meta imports in general), rather than letting end users deal with debugging obscure errors from code that assumes it has been fully installed. That's a measured response to a specific concern: it makes sense to codify the publisher's intent in the wheel (with a sensible default) and to have distlib's Wheel.mount() take into account the publisher's intent when considering whether to allow mounting of a wheel to take place. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 13:31, Donald Stufft don...@stufft.io wrote: Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html That's quoted out of context. It was in a thread about enabling importing C extensions from zipfiles, which I'm against. I don't always write emails as if they were documents to be saved for posterity. Sometimes I even say things that are contradictory, or not right. Heck, I'm not the authority on wheels, so what I'm saying is at best opinion. Sorry if I confused you or gave you the wrong impression, but honestly I don't have the time to review and vet my emails any more than I currently do :-( Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 29, 2014, at 10:00 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
On Wed, 29/1/14, Donald Stufft don...@stufft.io wrote:
It’s hard to pin down because the failure modes of zipped
eggs are nebulous themselves.
For instance take pip. I just recently redid the get-pip.py
installer to use a zip file (not a Wheel or Egg) that contained pip
and add that zip file straight to sys.path instead of unzipping it.
One of the failure modes was that it was suddenly unable to validate
TLS certificates. The reason why, was because it bundles it’s
own cacerts.pem which it passes to the ssl library to validate the
connection. In this case it’s impossible as far as I can tell to use
pkgutil.get_data directly because the ssl library does not directly
support pkgutil.get_data nor does it support passing the certificates
as a string or other in memory.
That’s really the biggest problem with strictly pure python Zipped Eggs.
If that's all, I'm not overly worried as long as there's a mountability flag
on
wheels. I looked at running pip from a zip a while ago, the cacerts thing
was easy to diagnose and fix - a five-or-ten-minute job. (I'm not trying to
come across as a show-off.) Here's the code I added as a fix in
pip/locations.py (no doubt it could be improved):
def _check_for_zip():
import zipimport
global default_cert_path
try:
if isinstance(__loader__, zipimport.zipimporter):
# running from a zip file. Save data to a handy location.
data = __loader__.get_data(default_cert_path)
destdir = os.path.expanduser('~/.pip')
if not os.path.isdir(destdir):
os.mkdir(destdir)
default_cert_path = os.path.join(destdir, 'cacert.pem')
with open(default_cert_path, 'wb') as f:
f.write(data)
except NameError:
# __loader__ not defined, so not in zip
pass
_check_for_zip()
del _check_for_zip
Of course, pip wasn't originally designed to run from zip, so one would
expect to find these kinds of issues. OTOH, if importable zips were more
commonplace, then presumably people would think about these kinds of
issues more when writing their code. Of course, in the above case, using
pkgutil wouldn't have worked, and there are bound to be other similar cases,
but I still don't see any real reason to fear the consequences of importable
wheels.
Regards,
Vinay Sajip
FWIW I'm not particularly that upset about this feature itself. I think it's a
bad idea and I think the way it locks the format into a particular method is a
poor trade off and that there is possibly a better form for this feature, but
if I had engaged the process argued my side, and then lost I could deal with
that.
What I'm upset about is that I can find no support that the *PEP* text (not the
intentions of the PEP) contains explicit support for this feature (as in, it
exists because we want it to work and we think people should use it) and little
to no evidence that the fact that this was (apparently) a design goal was
properly communicated (which includes multiple statements by various people
including
Daniel that it wasn't really designed for this, though Daniel is apparently
recanting
that meaning). According to PEP1 a PEP must be a clear and complete description
of the proposed enhancement.
I believe that what was proposed for discussion, if it was indeed intended that
Wheels should be generally importable by adding them to sys path, was not clear
about the fact that this was part of the proposal (and in fact contains text
that reads contrary to that intent), and because the proposal and the resulting
discussion was unclear, that people, such as myself, were not given the chance
to discuss this particular aspect of the proposal.
I feel that as BDFL delegate Nick is abusing the fact that he ultimately
accepted the PEP to circumvent the lack of a clear proposal of a particular
design goal (albeit likely unwittingly) to include this part of the proposal
without discussion. *This* is what I have a problem with. The PEP process
exists to gather everyone's input on a proposal and that ability was not
afforded to people who do not believe that this particular feature should be
supported.
This is why I started this thread (which has someone gone off the rails trying
to discuss whether or not this is a good feature or not which is immaterial to
the actual issue), because I believe that this change should be reverted, and
if this is something that Nick, Daniel, Vinay, or anyone else feels is
important then it should be proposed as part of Wheel 1.1 and everyone should
be given a fair chance to discuss it before it is accepted.
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed
Re: [Distutils] Using Wheel with zipimport
On 29 January 2014 14:25, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: What *else* have we learned from eggs? That designing modules to support zipimport correctly is non-trivial. And that assuming that things work unless told otherwise is a bad default behaviour. That packaging solutions should not depend on runtime support modules. How you package your code should not imply runtime dependencies. That package installation utilities should not dabble in sys.path manipulation. The import path is the user's responsibility. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Wed, 29/1/14, Paul Moore p.f.mo...@gmail.com wrote: That designing modules to support zipimport correctly is non-trivial. It's not trivial, but it's not especially hard, either. Mostly, it's about remembering to consider zipimport, since that hasn't been a mainstream way of deploying Python software. And that assuming that things work unless told otherwise is a bad default behaviour. Assumptions in general are bad, if they are made without due consideration. That's not especially egg-related, is it? That packaging solutions should not depend on runtime support modules. How you package your code should not imply runtime dependencies. Do you mean pkg_resources here? Certainly, distlib/distil don't expect to be present in stuff that they install. That package installation utilities should not dabble in sys.path manipulation. The import path is the user's responsibility. User as in developer (rather than end user). Right, and distlib's wheel code does no sys.path manipulation unless explicitly asked to. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 30/01/14 00:59, Nick Coghlan wrote: On 29 January 2014 23:48, Donald Stufft don...@stufft.io wrote: So what did you mean when you said We discussed it extensively before PEP 427 was approved if you're now saying that it wasn't discussed. Explicitly would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html Like I said, that particular aspect wasn't controversial, so while it was noted a few times (a few other examples of which you found), it was the overall discussions that were extensive. Both Daniel and I knew the zipimport compatibility for packages that were themselves zip compatible was a deliberate feature, so it was a surprise to me when Armin Ronacher said in his recent article that it wasn't supported (and hence the clarification). Cheers, Nick. In My Humble Opinion, that just isn't good enough. I am sure that everyone on this list has a passionate interest in making sure that the decisions for python packaging are as right as possible, and we know there are processes to follow that help us achieve that. One of those is that one should always be explicit about what features and functionality anything supports, which means that all features and functionality can be discussed, agreed upon, improved, etc, before it becomes something that everyone wilfully agrees to support. Wheels are either *documented* to be a binary format that *officially supports* zipimport-ability (fully, conditionally, or whatever), or the default answer is, like any other functionality not explicitly mentioned, no they are not. That's how you get a standard where issues can be discussed without tension. Two people agreeing to themselves that this is an uncontroversial implicit feature is not part of the documentation process, and whether or not they are the creators and acceptors of such standards, it leaves others out of the process until this late stage of the game. The unfortunate disagreements are exactly what can happen when things aren't made explicit from the start and are what are preventable when they are. Docs or it doesn't exist is popular developer mantra for a reason. Even as mostly an kibitzer on this group I hope it can learn from and avoid this sort of discussion into the future. -- Matt Iversen PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973 signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] problems with eggs?
btw, there's a stub in the PUG for Wheel vs Egg, if anyone wants to help fill that out. https://python-packaging-user-guide.readthedocs.org/en/latest/technical.html#wheel-vs-egg On Wed, Jan 29, 2014 at 7:07 AM, Vinay Sajip vinay_sa...@yahoo.co.ukwrote: On Wed, 29/1/14, Daniel Holth dho...@gmail.com wrote: Here are some of the problems with eggs that I tried to solve: [snipped] Right, and the wheel design has addressed those issues. I suppose what I was after was the kinds of problems that would arise from the importability of wheels, which seems to be causing so much fear, uncertainty and doubt :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
Nick Coghlan wrote: Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities). Would it help if zipimportable wheels were given a different extension? The format could be exactly the same (except perhaps for a flag in the metadata) but users would be able to immediately tell whether a particular wheel was zipimportable without having to run a tool. We would need a suitably catchy name for them too. Zippy wheels? Fast wheels? Hotwheels? -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On 30 Jan 2014 07:50, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Nick Coghlan wrote: Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities). Would it help if zipimportable wheels were given a different extension? The format could be exactly the same (except perhaps for a flag in the metadata) but users would be able to immediately tell whether a particular wheel was zipimportable without having to run a tool. We would need a suitably catchy name for them too. Zippy wheels? Fast wheels? Hotwheels? 1.1 will likely have a flag in WHEEL (off by default), and distlib.mount will then refuse to add wheels that don't have it set (including 1.0 wheels) to sys.path. Cheers, Nick. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On Sun, Jan 26, 2014 at 12:29 AM, Nick Coghlan ncogh...@gmail.com wrote: Paul's position exactly mirrors my own - I an perfectly fine with the recommended advice to scientific users continuing to be NumPy doesn't officially support pip and virtualenv because of the way it is built and installed, so you will have to get one of the curated scientific stacks, bootstrap conda, use the Windows installers or use the version provided by your Linux distro vendor. The metadata 2.0 standards *will not* be accepted until the pip 1.6 or 1.7 time frame, and it's more likely the latter, since I don't want to distract anyone from the current security work (I know I said otherwise recently, but I managed to temporarily forget that the Warehouse transition and implementing PEP 458 was next on the to do list when I said that). So, if the NumPy community choose to wait for general post-install script support in wheel files, they're likely to be waiting at least until the release of pip 1.7. In the meantime, the failure mode for people attempting to try out the Scientific Python stack via pip install numpy in an existing Python installation or virtualenv will remain a failure to build with a likely cryptic error. I do see a few possible workarounds, but none of them would change the metadata 2.0 standards: 1. Add explicit NumPy support *directly* to pip. This would be the quick hack, private API support that Oscar is requesting, since it would be a special arrangement between the pip devs and the numpy devs, and eventually replaced by a general purpose post-install mechanism in metadata 2.0. I am not speaking for the whole numpy team, but as the former maintainer of numpy.distutils, I think this will be more hurtful than helpful. I think the SSE issue is a bit of a side discussion: most people who care about performance already know how to install numpy. What we care about here are people who don't care so much about fast eigenvalue decomposition, but want to use e.g. pandas. Building numpy in a way that supports every architecture is both doable and acceptable IMO. 2. Add support to pip to request the conversion of available wininst installers (and bdist_dumb?) to wheels for installation with pip. Vinay has this working from a technical perspective, so it may be something the pip devs are interested in exploring for pip 1.6. Building numpy wheels is not hard, we can do that fairly easily (I have already done so several times, the hard parts have nothing to do with wheel or even python, and are related to mingw issues on win 64 bits). 3. Both of the above options require waiting for pip 1.6 (at the earliest), which means neither will improve the behaviour in CPython 3.4 (which will ship pip 1.5.1). The only folks with the power to improve *that* situation are the NumPy devs, who have the ability to choose between the doesn't work for anyone except experienced build engineers status quo and works for a large percentage of users, but will still fail at runtime for users on hardware without SSE2 support. To put the but what if the user doesn't have SSE2 support? concern in context, it should only affect Intel users with CPUs older than a Pentium 4 (released 2001), and AMD users with a CPU older than an Opteron or Athlon 64 (both released 2003). All x86/x86_64 CPUs released in the past decade should be able to handle SSE2 binaries, so our caveat can be if your computer is more than a decade old, 'pip install numpy' may not work for you, but it should do the right thing on newer systems. Now, the NumPy devs may feel that persisting with the status quo for another 6 to 12 months while waiting for still hypothetical additional changes in pip specifically to accommodate NumPy's current installation practices is a better alternative than taking option 3. However, from my perspective, having NumPy readily available to users using the python.orgWindows installers for Python 3.4 would *significantly* lower the barrier to entry to the Scientific Python stack for new users on relatively modern systems when compared to the 4 current options (while we accept the Linux distro problem is on distutils-sig to deal with, that's far from being a NumPy specific problem). Just to clarify: you actually can install numpy on windows with python.orginstallers fairly easily by using easy_install already (we upload a bdist_wininst compatible binary which should not use any CPU-specific instructions). It looks like those are missing for 1.8.0, but we can fix this fairly easily. David Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] wheels on sys.path clarification (reboot)
I have clearly done a bad job so far of explaining the clarification in PEP 427, so here's a new attempt that relies solely on the PEP text and the way the import system works, rather than the fact that the discussions around the PEP show that the import system compatibility was a deliberate feature that I took into account when accepting the PEP. Here is the key quote from PEP 427: Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time. That feature (which *is* explicitly documented in the accepted PEP), ensures that there will always be a subset of wheel files which can be used without needing a customised installer - just unzip them into a directory on sys.path and go. Not all wheels will work that way, and there are some downsides when you do it (e.g. launch scripts won't be created and bytecode files won't be compiled with elevated permissions if that is needed). However, it's explicitly covered in the PEP and is an apparently non-controversial design goal for the format. Once you have that can be installed just by unzipping the archive feature, then it is *an inherent property of how zipimport works* that there will also be an even smaller subset of wheels that will work correctly without even unpacking them first - you can just add the archive directly to sys.path and let zipimport do its thing. Aside from deliberately choosing a zipimport incompatible archive format (which we didn't do), you cannot design a format that meets the first requirement without inevitably also supporting the latter behaviour. However, this fact isn't obvious (as has become eminently clear to me since reading Armin Ronacher's recent wheel article), as not everyone is intimately with the subtleties of the import system. That is why I added the new text to the PEP - to make this capability clear without needing to make that connection independently. I do now plan to reword the new FAQ entry to make it clear that the import compatibility is mainly a side effect of the no specialised installer needed feature, but hopefully the above helps make it clear why this compatibility is already an inherent feature of the accepted PEP rather than something that can be discussed and accepted independently in a new version - it is pointing out a non-obvious consequence of an existing capability, not actually adding anything new. Regards, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On Wed, Jan 29, 2014 at 2:04 PM, David Cournapeau courn...@gmail.comwrote: I think the SSE issue is a bit of a side discussion: most people who care about performance already know how to install numpy. What we care about here are people who don't care so much about fast eigenvalue decomposition, but want to use e.g. pandas. Building numpy in a way that supports every architecture is both doable and acceptable IMO. Exactly -- I'm pretty sure SSE2 is being suggested because that's the lowest common denominator that we expect to see a lot of -- if their really are a lot of non-SSE-2 machines out there we could leave that off, too. Building numpy wheels is not hard, we can do that fairly easily (I have already done so several times, the hard parts have nothing to do with wheel or even python, and are related to mingw issues on win 64 bits). David, Where is numpy as with building out of the box with the python.org binary for Windows, and the standard MS compilers that are used with those builds. That used to be an easy python setup.py install away -- has that changed? If so, is this a known bug, or a known we-aren't-supporting-that? i.e. it would be nice if anyone setup to build C extensions could just build numpy. -Chris Just to clarify: you actually can install numpy on windows with python.orginstallers fairly easily by using easy_install already (we upload a bdist_wininst compatible binary which should not use any CPU-specific instructions). It looks like those are missing for 1.8.0, but we can fix this fairly easily. presumably just as easy to do a binary wheel then -- I vote for that. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 29, 2014, at 5:24 PM, Nick Coghlan ncogh...@gmail.com wrote: I have clearly done a bad job so far of explaining the clarification in PEP 427, so here's a new attempt that relies solely on the PEP text and the way the import system works, rather than the fact that the discussions around the PEP show that the import system compatibility was a deliberate feature that I took into account when accepting the PEP. Here is the key quote from PEP 427: Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time. That feature (which *is* explicitly documented in the accepted PEP), ensures that there will always be a subset of wheel files which can be used without needing a customised installer - just unzip them into a directory on sys.path and go. Not all wheels will work that way, and there are some downsides when you do it (e.g. launch scripts won't be created and bytecode files won't be compiled with elevated permissions if that is needed). However, it's explicitly covered in the PEP and is an apparently non-controversial design goal for the format. Once you have that can be installed just by unzipping the archive feature, then it is *an inherent property of how zipimport works* that there will also be an even smaller subset of wheels that will work correctly without even unpacking them first - you can just add the archive directly to sys.path and let zipimport do its thing. Aside from deliberately choosing a zipimport incompatible archive format (which we didn't do), you cannot design a format that meets the first requirement without inevitably also supporting the latter behaviour. However, this fact isn't obvious (as has become eminently clear to me since reading Armin Ronacher's recent wheel article), as not everyone is intimately with the subtleties of the import system. That is why I added the new text to the PEP - to make this capability clear without needing to make that connection independently. I do now plan to reword the new FAQ entry to make it clear that the import compatibility is mainly a side effect of the no specialised installer needed feature, but hopefully the above helps make it clear why this compatibility is already an inherent feature of the accepted PEP rather than something that can be discussed and accepted independently in a new version - it is pointing out a non-obvious consequence of an existing capability, not actually adding anything new. Regards, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig I understand what you're saying. My point is there is a difference between supported because we didn't go out of our way to break it, and supported because it's an API contract we intend to honor. The first one is fine, because it's an implementation detail and anyone relying on it cannot expect it to have backwards compatibility concerns. It's the second one that concerned me. By including this in the PEP we've moved the fact that Wheel's are directly importable by adding them to the sys.path from an implementation detail to a documented feature (maybe documented is a better word than supported here) which means we've made some level of promise that this is going to continue working in the future. It's that second part that I have a problem with. It means that any change to Wheels in the future must be weighed against the fact that we can't break compatibility with zip import without incrementing to a new major rev of the Wheel format. It means that we're promising that this feature will work. It's essentially the difference between naming a function _foo instead of foo. Yes you can use a function named _foo just fine, but there's no promise that it won't break in the future if you rely on that. Again let me stress that this is separate from the issue of *if* we should officially support zip importing Wheels as a feature. I have reservations about that but I would not be angry if that made it into the spec *after* discussion. What has me upset is that this API promise is being added without discussion with the rationale of We didn't actively attempt to break it so now it's an officially supported feature”. I'm upset that the ability to discuss and home and perhaps exclude a certain API promise was kept from the population at large because of the fact that the format didn't go out it's way to prevent itself from being used that way. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist -
Re: [Distutils] pip on windows experience
I don’t see any reason why SSE couldn’t be added as tags in the Wheel filename fwiw. That doesn’t help for things like MKL though. On Jan 29, 2014, at 5:50 PM, David Cournapeau courn...@gmail.com wrote: On Wed, Jan 29, 2014 at 10:27 PM, Chris Barker chris.bar...@noaa.gov wrote: On Wed, Jan 29, 2014 at 2:04 PM, David Cournapeau courn...@gmail.com wrote: I think the SSE issue is a bit of a side discussion: most people who care about performance already know how to install numpy. What we care about here are people who don't care so much about fast eigenvalue decomposition, but want to use e.g. pandas. Building numpy in a way that supports every architecture is both doable and acceptable IMO. Exactly -- I'm pretty sure SSE2 is being suggested because that's the lowest common denominator that we expect to see a lot of -- if their really are a lot of non-SSE-2 machines out there we could leave that off, too. The failure mode is fairly horrible though, and the gain is not that substantial anyway compared to really optimized installation (MKL, etc... as provided by Continuum or us). Building numpy wheels is not hard, we can do that fairly easily (I have already done so several times, the hard parts have nothing to do with wheel or even python, and are related to mingw issues on win 64 bits). David, Where is numpy as with building out of the box with the python.org binary for Windows, and the standard MS compilers that are used with those builds. That used to be an easy python setup.py install away -- has that changed? If so, is this a known bug, or a known we-aren't-supporting-that? i.e. it would be nice if anyone setup to build C extensions could just build numpy. This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). Numpy is actually fairly easy to build if you have a C Compiler (which is the obvious pain point on windows). Scipy, and fortran is where things fall apart. David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
But that's what I'm saying, there are only three ways to break this behaviour: 1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme The first two aren't going to happen, which leaves only the third. You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the no specialised installer needed feature. People saying I didn't realise that the current design implied zipimport compatibility is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate. Regards, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On Wed, Jan 29, 2014 at 10:52 PM, Donald Stufft don...@stufft.io wrote: I don’t see any reason why SSE couldn’t be added as tags in the Wheel filename fwiw. You still need to decide when to install what, but I would be interested in talking more about that part. That doesn’t help for things like MKL though. Nope, but MKL is actually easy in the sense that it deals with architectures at runtime. OSS numerical libraries generally don't (lots of work, and often a non issue when you can build stuff by yourself :) ). David On Jan 29, 2014, at 5:50 PM, David Cournapeau courn...@gmail.com wrote: On Wed, Jan 29, 2014 at 10:27 PM, Chris Barker chris.bar...@noaa.govwrote: On Wed, Jan 29, 2014 at 2:04 PM, David Cournapeau courn...@gmail.comwrote: I think the SSE issue is a bit of a side discussion: most people who care about performance already know how to install numpy. What we care about here are people who don't care so much about fast eigenvalue decomposition, but want to use e.g. pandas. Building numpy in a way that supports every architecture is both doable and acceptable IMO. Exactly -- I'm pretty sure SSE2 is being suggested because that's the lowest common denominator that we expect to see a lot of -- if their really are a lot of non-SSE-2 machines out there we could leave that off, too. The failure mode is fairly horrible though, and the gain is not that substantial anyway compared to really optimized installation (MKL, etc... as provided by Continuum or us). Building numpy wheels is not hard, we can do that fairly easily (I have already done so several times, the hard parts have nothing to do with wheel or even python, and are related to mingw issues on win 64 bits). David, Where is numpy as with building out of the box with the python.orgbinary for Windows, and the standard MS compilers that are used with those builds. That used to be an easy python setup.py install away -- has that changed? If so, is this a known bug, or a known we-aren't-supporting-that? i.e. it would be nice if anyone setup to build C extensions could just build numpy. This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). Numpy is actually fairly easy to build if you have a C Compiler (which is the obvious pain point on windows). Scipy, and fortran is where things fall apart. David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 29, 2014, at 5:59 PM, Nick Coghlan ncogh...@gmail.com wrote: But that's what I'm saying, there are only three ways to break this behaviour: 1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme The first two aren't going to happen, which leaves only the third. You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the no specialised installer needed feature. Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official support for zip import means that the change would have to be weighed against also breaking that backwards compatibility. People saying I didn't realise that the current design implied zipimport compatibility is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate. Regards, Nick. I completely realized that the current design implied zip import compatibility, but implicit features do not make features that we intend to support going into the future. You're making the decision for us that we're not going to be making these kinds of changes and even then it's really not about whether or not this change can be removed or not. For instance, pip supports installing from Wheels how long until we get a bug report asking for the ability to install zipped Wheels? If we refuse to accept it are we violating the spec? If we implement the spec as it stands are we handing the users potential footguns? It's more than just backwards compatibility concerns as well. I care about providing the end users of these tools comprehensive solutions to the problems they solve. By simply tacking a documented feature on after the PEP has been accepted you're also removing the ability of this sig to arrive at such a comprehensive solution. Instead what you get is Well you can add a Wheel directly to sys.path, except when you can't. This is user hostile. Essentially a feature is more than the raw ability to do something, it also includes documentation, support features, support infrastructure, etc. You've removed the chance for this sig to participate in defining how we're going to present this feature, what APIs around it we want to provide etc. Python is a very dynamic language, just because you *can* do something doesn't mean it's supported to actually do it. The PEP process exists so that important decisions can be discussed and honed. I want this change reverted and deferred to 1.1 so that we can follow the PEP process and have this discussion. These seems like the most reasonable thing to do and the thing that is most in spirit of the PEP process. It may be that ultimately zip importable Wheels are added as a feature, hopefully with the proper infrastructure to handle it sanely, but it is *wrong* to make the decision that this a feature that Wheels officially support with first having that discussion. Finally I don't understand why reverting is such a controversial change with you. As far as I can tell your rationale for adding this change is that it would be difficult to remove support for it. In that case, and if it's truly a thing that this sig wants to support as a desired feature, then I'm sure the proponents of this particular feature will have no trouble getting it formally accepted as part of Wheel 1.1 whereas inversely if it ends up being a thing that this sig does *not* want to support, removing that feature is much harder than adding it. The dialog is essentially being forced to be Why should we break this for people who depended on our documented feature which is a much harder thing to justify than Why should we formally define this feature. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 29, 2014, at 2:59 PM, Nick Coghlan ncogh...@gmail.com wrote: But that's what I'm saying, there are only three ways to break this behaviour: 1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme The first two aren't going to happen, which leaves only the third. You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the no specialised installer needed feature. People saying I didn't realise that the current design implied zipimport compatibility is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate. If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as well it works right now without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
Guys, I'm still unable to see my emails reaching the list for unknown reason... Any ideas what can be the problem? FWIW I have tested it by adding the same __main__.py i used for the egg variant of the distribution to the wheel root and specifying $ PYTHONPATH=projectA.whl; python executable_projectB.whl I was able to run the program! So it seems like the stuff does behave like a jar for pure python modules! Don't know who to thank for that - because nobody seems to have claimed this functionality;) Well Nick said it supposed to be that way, So please don't break it!!;) It also means that I have the full picture in place without the necessity to use the buildout or anything else other then the pip and wheel from a flat folder structure! I'd appreciate if you could let me know about any problems you could foresee (we are talking pure python only) Thanks! Eugene On Wed, Jan 29, 2014 at 5:36 PM, Evgeny Sazhin eug...@sazhin.us wrote: I'm sorry for possible dup, but for whatever reason i don't see this email reaching the list, so i'm resending. On Wed, Jan 29, 2014 at 12:50 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? There is a parallel discussion going on, with the title Using Wheel with zipimport, which is relevant to this question, and other questions you raised (e.g. about supporting C extensions/pure-Python modules. I read all of it and got a bit lost in between the distil API and PEP process discussion;) I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? A lot of Python users depend on C extensions - and while it is a subset of all Python users, it is a large (and important) subset. Example: any usage of Python in numerical analysis or scientific applications involves use of C extensions. Regards, Vinay Sajip I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the release program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip vinay_sa...@yahoo.co.ukwrote: Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? There is a parallel discussion going on, with the title Using Wheel with zipimport, which is relevant to this question, and other questions you raised (e.g. about supporting C extensions/pure-Python modules. I read all of it and got a bit lost in between the distil API and PEP process discussion;) I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? A lot of Python users depend on C extensions - and while it is a subset of all Python users, it is a large (and important) subset. Example: any usage of Python in numerical analysis or scientific applications involves use of C extensions. Regards, Vinay Sajip I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the release program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? Thanks a lot! Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 9:50 AM, Evgeny Sazhin eug...@sazhin.us wrote: On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? There is a parallel discussion going on, with the title Using Wheel with zipimport, which is relevant to this question, and other questions you raised (e.g. about supporting C extensions/pure-Python modules. I read all of it and got a bit lost in between the distil API and PEP process discussion;) I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? A lot of Python users depend on C extensions - and while it is a subset of all Python users, it is a large (and important) subset. Example: any usage of Python in numerical analysis or scientific applications involves use of C extensions. Regards, Vinay Sajip I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the release program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 6:34 PM, Evgeny Sazhin eug...@sazhin.us wrote: Guys, I'm still unable to see my emails reaching the list for unknown reason... Any ideas what can be the problem? FWIW I have tested it by adding the same __main__.py i used for the egg variant of the distribution to the wheel root and specifying $ PYTHONPATH=projectA.whl; python executable_projectB.whl I was able to run the program! So it seems like the stuff does behave like a jar for pure python modules! Don't know who to thank for that - because nobody seems to have claimed this functionality;) Well Nick said it supposed to be that way, So please don't break it!!;) Wether this was “supposed” to be there is a matter that’s up for debate. If you’re just using these for runtime you don’t need the Wheel format at all. Just regular old pyzips will do just fine. http://docs.python.org/2/library/zipfile#pyzipfile-objects It also means that I have the full picture in place without the necessity to use the buildout or anything else other then the pip and wheel from a flat folder structure! I'd appreciate if you could let me know about any problems you could foresee (we are talking pure python only) Thanks! Eugene On Wed, Jan 29, 2014 at 5:36 PM, Evgeny Sazhin eug...@sazhin.us wrote: I'm sorry for possible dup, but for whatever reason i don't see this email reaching the list, so i'm resending. On Wed, Jan 29, 2014 at 12:50 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? There is a parallel discussion going on, with the title Using Wheel with zipimport, which is relevant to this question, and other questions you raised (e.g. about supporting C extensions/pure-Python modules. I read all of it and got a bit lost in between the distil API and PEP process discussion;) I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? A lot of Python users depend on C extensions - and while it is a subset of all Python users, it is a large (and important) subset. Example: any usage of Python in numerical analysis or scientific applications involves use of C extensions. Regards, Vinay Sajip I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the release program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin eug...@sazhin.us wrote: Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. I didn’t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You’ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 10:49 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin eug...@sazhin.us wrote: Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. I didn’t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You’ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I’m 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it’s not entirely up to me I would be opposed to it getting the ability to do so. 1. use minimum of “standard” tools, 2. *manage dependencies* in a unified and standardized way 3. use only *one format* and *one artifact* per project for everything related to the code development, shipment, deployment and execution. What would be a motivation to learn yet another tool like buildout or anything, pollute infrastructure with more libraries, manage several different ways of working with the code, overcomplicate the production area by requiring stuff to be unzipped and whatnot? I hope nobody would want to strip any of those nice features from the wheel, but improve the running from it!;) I don’t understand what any of the rest of this have to do with wether you add zipped Wheels to your sys.path or not. Thanks, Eugene Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 11:17 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 10:49 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin eug...@sazhin.us wrote: Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. I didn’t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You’ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I’m 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it’s not entirely up to me I would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you’re missing the point a bit. I’m totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! 1. use minimum of “standard” tools, 2. *manage dependencies* in a unified and standardized way 3. use only *one format* and *one artifact* per project for everything related to the code development, shipment, deployment and execution. What would be a motivation to learn yet another tool like buildout or anything, pollute infrastructure with more libraries, manage several different ways of working with the code, overcomplicate the production area by requiring stuff to be unzipped and whatnot? I hope nobody would want to strip any of those nice features from the wheel, but improve the running from it!;) I don’t understand what any of the rest of this have to do with wether you add zipped Wheels to your sys.path or not. see above Thanks, Eugene Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/29/2014 06:55 PM, Noah Kantrowitz wrote: If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as well it works right now without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior. Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the no special installers design choice. - - Given that there is a non-empty set of wheels which can be unpacked to a filesystem-directory tree in a directory on sys.path, and that some of those wheels are already known not to otherwise break zip- importability, it is a logical necessity that such wheels can be put onto sys.path without unpacking. We already have existence proof for this in software being released *by the folks who made these specs*. Noting is as such is the *point* of Nick's change. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLp2hIACgkQ+gerLs4ltQ6VGACgjnYRbgVSs8ceTXNeWTH0zCls yHwAn0/nyDUMRjNl7ARi0bVtkBOeO1nJ =2pvt -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 29, 2014, at 8:50 PM, Tres Seaver tsea...@palladion.com wrote: Signed PGP part On 01/29/2014 06:55 PM, Noah Kantrowitz wrote: If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as well it works right now without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior. Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the no special installers design choice. No, thats a side effect of various other technologies that are beyond the scope of that spec. The spec should either say this is undefined behavior or commit that it is part of the spec. We can't pick this middle ground of well right now we just get it for free because of X, Y, and Z because that doesn't express to the reader the true semantics of is this something they can use. Don't assume the reader has as much information as we do about the pros and cons, and about how the implementations work. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 29, 2014, at 11:50 PM, Tres Seaver tsea...@palladion.com wrote: Signed PGP part On 01/29/2014 06:55 PM, Noah Kantrowitz wrote: If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as well it works right now without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior. Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the no special installers design choice. - Given that there is a non-empty set of wheels which can be unpacked to a filesystem-directory tree in a directory on sys.path, and that some of those wheels are already known not to otherwise break zip- importability, it is a logical necessity that such wheels can be put onto sys.path without unpacking. We already have existence proof for this in software being released *by the folks who made these specs*. Noting is as such is the *point* of Nick's change. Tres. -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig I can't believe folks are unable to differentiate between the difference of It's possible to do so because we didn't actively attempt to prevent it and This is a documented goal of the format and thus must be considered as part of the backward compatibility contract that the format has whenever changes are made to the format. The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of supports. At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep the process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're providing for that is. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] wheels on sys.path clarification (reboot)
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official support for zip import means that the change would have to be weighed against also breaking that backwards compatibility. Eh, please don’t break it!! Improve! Keep the ability to add wheels to python path. I don’t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i’m probably alone in the desert…l;) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 29, 2014, at 11:50 PM, Tres Seaver tsea...@palladion.com wrote: Signed PGP part On 01/29/2014 06:55 PM, Noah Kantrowitz wrote: If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as well it works right now without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior. Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the no special installers design choice. - Given that there is a non-empty set of wheels which can be unpacked to a filesystem-directory tree in a directory on sys.path, and that some of those wheels are already known not to otherwise break zip- importability, it is a logical necessity that such wheels can be put onto sys.path without unpacking. We already have existence proof for this in software being released *by the folks who made these specs*. Noting is as such is the *point* of Nick's change. Tres. -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig Also it’s entirely inaccurate that Zip import must be maintained in order to support using unzip as a simple installer for simple Wheels. For instance if Wheels instead used directories at the root named after sysconfig scheme names then with your standard unzip tool you could simple do:: unzip example.whl 'purelib/*' Which is every bit as simple and not requiring a specialized installer but ends up breaking zip import compatibility (Techincally you'd just need to add the example.whl/purelib to the sys.path). However this is a change that wasn't off the table in the original PEP, it is now in the PEP-by-fiat because we now have to maintain zip import compatibility. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 30, 2014, at 12:33 AM, Evgeny Sazhin eug...@sazhin.us wrote: Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official support for zip import means that the change would have to be weighed against also breaking that backwards compatibility. Eh, please don’t break it!! Improve! Keep the ability to add wheels to python path. I don’t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i’m probably alone in the desert…l;) OTOH if this is such a bad idea - what would be the least intrusive way for me to deploy the code? What about unzipped wheel? If I have wheels in one location and then my release procedure knows to unzip them in the folder that matches the wheel name and structure completely, would it be possible to add this unzipped folder to the PYTHONPATH and run the program? Is that closer to what you guys are preaching?? Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On 29 January 2014 22:50, David Cournapeau courn...@gmail.com wrote: i.e. it would be nice if anyone setup to build C extensions could just build numpy. This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). I don't know if you saw my comment earlier in this thread: A quick test later: No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the rest of the error stack (which I'm frankly not interested in investing the time to diagnose and fix) I get RuntimeError: Broken toolchain: cannot link a simple C program. Which is utter rubbish - I routinely build extensions with this installation. This is a straight pip install numpy run in a virtualenv on Windows 7 64-bit with MSVC 2010 (full edition) installed. I still don't have the time to do detailed diagnosis or digging, but if you cannot generate the same error yourself I can rerun the command and put the output into a bug report if that helps. (Please provide a link to the bug tracker, I didn't see it mentioned on www.numpy.org or on the pypi page for numpy). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
