Re: [Distutils] wheels on sys.path clarification (reboot)
On 30 January 2014 05:33, Evgeny Sazhin eug...@sazhin.us wrote: Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official support for zip import means that the change would have to be weighed against also breaking that backwards compatibility. Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;) This is the biggest concern I see with promoting wheels being directly importable via zipimport (I say promoting and not specifying deliberately, but I don't want to get back into the process debate). People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars. This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive. Pointing out the implication that wheels can be put on sys.path in specialised circumstances is entirely reasonable when talking to a Python audience, but it triggers entirely the wrong associations when a person with a Java background sees the statement. Evgeny - for your purposes, you'd be better advised to see wheels as similar in principle to Windows MSI installers (they aren't, but it will avoid triggering incorrect analogies). I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list. (Be careful, you may hear suggestions that you use eggs because they are similar to jars - while that is true, the way Python works means that the experience with eggs is far from being as natural as the jar experience you're used to). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Wed, Jan 29, 2014 at 5:22 PM, Vinay Sajip vinay_sa...@yahoo.co.ukwrote: On Wed, 29/1/14, Paul Moore p.f.mo...@gmail.com wrote: That package installation utilities should not dabble in sys.path manipulation. The import path is the user's responsibility. User as in developer (rather than end user). Right, and distlib's wheel code does no sys.path manipulation unless explicitly asked to. Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying. Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Thu, 30/1/14, Ralf Gommers ralf.gomm...@gmail.com wrote: Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying. If tool developers want to offer end users the option to control how they work with sys.path, that's up to them. For example, once the details are worked out, the distil tool will probably get a --mountable option for the package command, which will write metadata into the built wheel indicating whether the wheel is addable to sys.path or not (based on the builder's knowledge of the wheel's contents). Distlib, when asked to mount a wheel (add it to sys.path) will check the mountability metadata and honour the wheel publisher's intent. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 22:50, David Cournapeau courn...@gmail.com wrote: i.e. it would be nice if anyone setup to build C extensions could just build numpy. This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). I don't know if you saw my comment earlier in this thread: A quick test later: No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the rest of the error stack (which I'm frankly not interested in investing the time to diagnose and fix) I get RuntimeError: Broken toolchain: cannot link a simple C program. Which is utter rubbish - I routinely build extensions with this installation. This is a straight pip install numpy run in a virtualenv on Windows 7 64-bit with MSVC 2010 (full edition) installed. which version of python ? For 2.x, it is expected to fail at that point, since you can't build C extensions with something different than 2008 (with python.org builds). For 3.3, it means there is a bug that needs to be fixed. David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, 30/1/14, Paul Moore p.f.mo...@gmail.com wrote: This is the biggest concern I see with promoting wheels being directly importable via zipimport (I say promoting and not specifying deliberately, but I don't want to get back into the process debate). In my view, we can have our cake and eat it. Those who don't want their wheels to be mountable can say so in their wheel metadata. I expect that distlib will honour such metadata (once the details have been worked out) and not mount wheels which aren't supposed to be mounted. However, it's perfectly easy to write code that runs from zip, as long as you know that's a deployment option you want to support. So it doesn't make sense to prevent that useful functionality - even pip is using it, I believe ;-) - just because some people think it's a bad idea, for reasons that are hardly compelling. To say that we should keep packaging separate from importing is in some sense a religious argument, unless sound technical reasons are given for it. Java proves otherwise otherwise - and for those who hate Java, that's a religious viewpoint, too ... The argument that importing wheels will cause problems is trumped by allowing the mountability to be configurable by wheel metadata: the builder of a wheel, by saying it is mountable, is agreeing to all that entails in terms of handling package resources appropriately, etc. So the support burden shifts to them, and not to wheel, distlib or any packaging tool. So much heat over process, but so little light over exactly why *appropriately designed* software deployed in wheels shouldn't be importable. No detailed analysis of any problem with wheels taking into account the differences from eggs (the things Daniel mentioned in the other thread problems with eggs?, plus the fact that there's no sys.path manipulation *magic*). Just blanket statements to the effect that eggs were importable and bad, so wheels must be too smacks of superstition, not engineering. I would like the detractors of importable wheels to put their technical reasoning on the table, not use process debates to try to revert situations they're not happy with. That technical reasoning should address wheels as they are now and avoid referring to eggs as the bogeyman. The only reason I've heard from detractors so far is that software not designed to run from zip can give unexpected results when run from zip, which could give the wheel format a bad name. Given that a wheel publisher can have a say on importability, this issue seems to me to be adequately addressed. No detractor has come up with any other solid reasoning as to why the useful feature of wheel importability is bad. I invite them to put up, or ... we'll still be in the dark the next time this debate comes around ;-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On 30 January 2014 09:12, David Cournapeau courn...@gmail.com wrote: On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 22:50, David Cournapeau courn...@gmail.com wrote: i.e. it would be nice if anyone setup to build C extensions could just build numpy. This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). I don't know if you saw my comment earlier in this thread: A quick test later: No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the rest of the error stack (which I'm frankly not interested in investing the time to diagnose and fix) I get RuntimeError: Broken toolchain: cannot link a simple C program. Which is utter rubbish - I routinely build extensions with this installation. This is a straight pip install numpy run in a virtualenv on Windows 7 64-bit with MSVC 2010 (full edition) installed. which version of python ? For 2.x, it is expected to fail at that point, since you can't build C extensions with something different than 2008 (with python.org builds). For 3.3, it means there is a bug that needs to be fixed. Doh. Sorry, I knew I'd forget something. 3.3. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On Thu, Jan 30, 2014 at 10:20 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 09:12, David Cournapeau courn...@gmail.com wrote: On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 22:50, David Cournapeau courn...@gmail.com wrote: i.e. it would be nice if anyone setup to build C extensions could just build numpy. This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). I don't know if you saw my comment earlier in this thread: A quick test later: No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the rest of the error stack (which I'm frankly not interested in investing the time to diagnose and fix) I get RuntimeError: Broken toolchain: cannot link a simple C program. Which is utter rubbish - I routinely build extensions with this installation. This is a straight pip install numpy run in a virtualenv on Windows 7 64-bit with MSVC 2010 (full edition) installed. which version of python ? For 2.x, it is expected to fail at that point, since you can't build C extensions with something different than 2008 (with python.org builds). For 3.3, it means there is a bug that needs to be fixed. Doh. Sorry, I knew I'd forget something. 3.3. Here we go: https://github.com/numpy/numpy/issues/4245 David Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On 30 January 2014 09:45, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: So much heat over process, but so little light over exactly why *appropriately designed* software deployed in wheels shouldn't be importable. Good point. In my view, *appropriately designed* software is fine, and a metadata flag is a good idea. Although obviously that makes mounting a little more complex (check the flag) and argues for a mount method rather than people just fiddling sys.path. There's a lot of leeway for dispute over appropriate design and who gets to say, but that's not a technical issue. My one technical issue is with going beyond zipimport behaviour to the point of extracting DLLs to the filesystem. I remain -1 on that feature, and I believe I have explained why I think there are issues (and why I think that any solution should be part of zipimport and not added on in library or user code). But I'm happy to go through the details again, if you like - or just to accept that I don't need to use the feature. (Would you be willing to add some sort of never extract C extensions regardless of what the metadata might say option to wheel mount? It's not critical, as mounting random wheels without knowing how they work is clearly bad, but it does add a level of assurance that might be helpful,) The remaining issues seem to be mainly of education and expectations - people coming from non-Python (particularly Java) backgrounds have a set of assumptions that aren't easy to translate into Python terms. That's definitely not a technical problem. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
Thanks. I've added the installation details and output from a test run. Paul On 30 January 2014 10:25, David Cournapeau courn...@gmail.com wrote: On Thu, Jan 30, 2014 at 10:20 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 09:12, David Cournapeau courn...@gmail.com wrote: On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 22:50, David Cournapeau courn...@gmail.com wrote: i.e. it would be nice if anyone setup to build C extensions could just build numpy. This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). I don't know if you saw my comment earlier in this thread: A quick test later: No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the rest of the error stack (which I'm frankly not interested in investing the time to diagnose and fix) I get RuntimeError: Broken toolchain: cannot link a simple C program. Which is utter rubbish - I routinely build extensions with this installation. This is a straight pip install numpy run in a virtualenv on Windows 7 64-bit with MSVC 2010 (full edition) installed. which version of python ? For 2.x, it is expected to fail at that point, since you can't build C extensions with something different than 2008 (with python.org builds). For 3.3, it means there is a bug that needs to be fixed. Doh. Sorry, I knew I'd forget something. 3.3. Here we go: https://github.com/numpy/numpy/issues/4245 David Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On 30 Jan 2014 15:27, Donald Stufft don...@stufft.io wrote: I can't believe folks are unable to differentiate between the difference of It's possible to do so because we didn't actively attempt to prevent it and This is a documented goal of the format and thus must be considered as part of the backward compatibility contract that the format has whenever changes are made to the format. Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional ( https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format. This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it. I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design. For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40 This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made. The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of supports. At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep the process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're providing for that is. The problem with believing that we still have a choice about this topic is twofold: - firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here: - those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported
Re: [Distutils] wheels on sys.path clarification (reboot)
My one technical issue is with going beyond zipimport behaviour to the point of extracting DLLs to the filesystem. I remain -1 on that feature, and I believe I have explained why I think there are issues (and why I think that any solution should be part of zipimport and not added on in library or user code). But I'm happy to go through the details again, if you like - or just to accept that I don't Yes please, let's get into some details. Of course I understand that you might not want to use the feature, but I don't understand the -1 on the feature per se - whether it is in distlib or in zipimport is a secondary consideration. I agree that zipimport is the logical place for it, but ISTM the reason why it can't go in there just yet is also the reason why one might have some reservations about the feature: binary compatibility. I accept that this not yet a fully resolved issue in general (cf. the parallel discussion about numpy), but if we can isolate these issues, we can perhaps tackle them. But for me, that's the main reason why this part of the distlib API is experimental. Since zipimport makes no prescription about the contents of a zip (beyond describing how importing works), there is no agreed place to place metadata information such as about binary compatibility. Nor is binary compatibility between third-party packages of core concern to python-dev, so I'm not sure discussions there will be fruitful. However, such compatibility is a valid concern here, so I would expect useful input to come from interested parties. Also, the wheel format already caters for a limited set of binary compatibility info to be communicated, but this information is incomplete, which results in reservations about the dangers of extracting C extensions. In a sense, the contentiousness of extraction of these from a wheel is a red herring, because the exact same issues would arise if you installed from the wheel and then tried to use software which purported to be binary compatible with your system, but wasn't. That's why we don't put Linux wheels on PyPI, right? It's to avoid binary wheels compiled on e.g. CentOS ending up on an e.g. Ubuntu system which is not quite binary compatible. But that doesn't solve the problem at source, so much as act as a prophylactic. Much of the Python community works in a POSIX environment where build- from-source is the norm and binary compatibility is a non-issue. In a corporate environment with a homogeneous infrastructure, the same applies; but in a heterogeneous environment, not so much. I think some benefit would accrue to packaging as a whole if we had better definitions of binary compatibility, the ability to express it in more detail, how to test for it at installation time and so on. I think this is one of the areas where we can and should improve WHEEL metadata. Perhaps the NumPy/SciPy readers would care to comment, as we're talking beyond the realms of Python version compatibility. If you have other reasons for your -1, I'd like to hear them. need to use the feature. (Would you be willing to add some sort of never extract C extensions regardless of what the metadata might say option to wheel mount? It's not critical, as mounting random wheels without knowing how they work is clearly bad, but it does add a level of assurance that might be helpful,) Currently, extraction only happens if there is metadata in the wheel to indicate that extraction should occur, and it's up to the wheel builder to put it there. It doesn't make sense in general to prohibit just the C extensions but allow importing pure-Python modules in a wheel: the pure-Python bits mightn't work properly if the C extensions aren't available. So it seems safer in general to have all-or-nothing, or else have an additional flag in the metadata which indicates that the C extensions have to be extracted for the wheel to be useful. End user control of mounting should be at the discretion of the developer of the software which invokes the mount method. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))
On 30 January 2014 21:57, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: My one technical issue is with going beyond zipimport behaviour to the point of extracting DLLs to the filesystem. I remain -1 on that feature, and I believe I have explained why I think there are issues (and why I think that any solution should be part of zipimport and not added on in library or user code). But I'm happy to go through the details again, if you like - or just to accept that I don't Yes please, let's get into some details. Of course I understand that you might not want to use the feature, but I don't understand the -1 on the feature per se - whether it is in distlib or in zipimport is a secondary consideration. I agree that zipimport is the logical place for it, but ISTM the reason why it can't go in there just yet is also the reason why one might have some reservations about the feature: binary compatibility. I accept that this not yet a fully resolved issue in general (cf. the parallel discussion about numpy), but if we can isolate these issues, we can perhaps tackle them. But for me, that's the main reason why this part of the distlib API is experimental. I actually think this is a useful thing to experiment with, I'm just not sure distlib is the best place for that experiment. With appropriately secure tempfile handling and the right sys.path (and module __path__) manipulation it's not obviously *impossible* to handle C extensions at arbitrary positions in the module namespace this way, just difficult. zipimport itself is a bad place to experiment though, since not only is it currently a complex ball of C code, but adding such a feature without clear evidence of robust support in a third party project would be irresponsible. In the case of distlib, the potential complexity of ensuring that such a scheme works consistently across multiple platforms and as part of various complex package layouts is enough to make me nervous about having it in the same library as the metadata 2.0 reference implementation :) Now, if you were to split that functionality out from distlib into a separate wheeltab project (or a name of your choice), I'd be substantially less nervous, because endorsing distlib as the metadata 2.0 reference implementation wouldn't carry any implications of endorsing a feature I consider potentially interesting but rather challenging to implement in a robust manner. mount() would become something I could explore when I had some additional free time (hah!), rather than something I felt obliged to help get to a more robust state before releasing metadata 2.0. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))
Changing the subject to clearly focus the discussion. On 30 January 2014 11:57, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: If you have other reasons for your -1, I'd like to hear them. OK. Note that this is not, in my view, an issue with wheels, but rather about zipfiles on sys.path, and (deliberate) design limitations of the module loader and zipimport implementations.[1] First of all, it is not possible to load a DLL into a process' memory [2, 3] unless it is stored as a file in the filesystem. So any attempt to import a C extension from a zipfile must, by necessity, involve extracting that DLL to the filesystem. That's where I see the problems. None are deal-breaking issues, but they consist of a number of niggling issues that cumulatively chip away at the reliability of the concept until the end result has enough corner cases and risks to make it unacceptable (depending on your tolerance for risks - there's a definite judgement call involved). The issues I can see are: [4] 1. You need to choose a location to put the extracted file. On Windows in particular, there is no guaranteed-available filesystem location that can be used without risk. Some accounts have no home directory, some (locked down) users have no permissions anywhere but very specific places, even TEMP may not be usable if there's an aggressive housekeeping routine in place - but TEMP is probably the best choice of a bad lot. 2. There are race conditions to consider. If the extraction is not completely isolated per-process, what if 2 processes want to use different versions of the same DLL? How will these be distinguished? [5] So to avoid corner cases you have to assume only the one process uses a given extracted DLL. 3. Clean-up is an issue. How will the extracted files be removed? You can't unload the DLLs from Python, and you can't delete open files in Windows. So do you simply leave the files lying round? Or do you do some sort of atexit dance to run a separate process after the Python process terminates which will do the cleanup? What happens to that process when virus checkers hold the file open? Leaving the files around is probably the most robust answer, but it's not exactly friendly. As I've said elsewhere, these are fundamental issues with importing DLLs from zipfiles, and have no direct relationship to wheels. The only place where having a wheel rather than a general zipfile makes a difference is that a wheel *might* at some point contain metadata that allows the wheel to claim that it's OK to load its contents from a zipfile. But my points above are not something that the author of the C extension can address, so there's no way that I can see that an extension author can justifiably set that flag. So: as wheels don't give any additional reliability over any other zipfile, I don't see this (loading C extensions) as a wheel-related feature. Ideally, if these problems can be solved, the solution should be included in the core zipimport module so that all users can benefit. If there are still issues to iron out and experience to be gained, a 3rd party enhanced zip importer module would be a reasonable test-bed for the solution. A 3rd party solution could also be appropriate if the caveats and/or limitations were generally acceptable, but sufficient to prohibit stdlib inclusion. The wheel mount API could, if you wanted, look for the existence of that enhanced zipimport module and use it when appropriate, but baking the feature into wheel mount just limits your user base (and hence your audience for raising bug reports, etc) needlessly. I hope this explains my reasoning in sufficient detail. FINAL DISCLAIMER: I have no objection to this feature being provided per se, any more than I object to the existence of (say) Zope. Just because I'm not a member of the target audience doesn't mean that it's not a feature that some might benefit from. All I'm trying to do here is offer my input as someone who was involved in the initial implementation of zipimport, and who has kept an interested eye on how it has been used in the 11 years since its introduction - and in particular how people have tried to overcome the limitations we felt we had to impose when designing it. Ultimately, I would be overjoyed if someone could find a solution to this issue (in much the same way as I'm delighted by what Brett has done with importlib). Paul Footnotes: [1] Historical footnote - I was directly involved with the design of PEP 302 and the zipimport implementation, and we made a deliberate choice to only look at pure Python files, because the platform issues around C extensions were too hard. [2] I'm talking from a Windows perspective here. I do not have sufficient low-level knowledge of Unix to comment on that case. I suspect that the issues are similar but I defer to the platform experts. [3] There is, I believe, code out there on the internet to map a DLL image into a process based purely in memory, but I think it's a fairly gross hack.
Re: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))
On 30 January 2014 12:29, Nick Coghlan ncogh...@gmail.com wrote: I actually think this is a useful thing to experiment with, I'm just not sure distlib is the best place for that experiment. With appropriately secure tempfile handling and the right sys.path (and module __path__) manipulation it's not obviously *impossible* to handle C extensions at arbitrary positions in the module namespace this way, just difficult. zipimport itself is a bad place to experiment though, since not only is it currently a complex ball of C code, but adding such a feature without clear evidence of robust support in a third party project would be irresponsible. I just sent a long message that essentially gave a chunk of history and suggested a similar thing - an enhanced zipimport module to experiment with solutions in this space. With importlib available, an experimental implementation shouldn't even be too hard. The only difference is that I see very little reason why such a solution can't apply to all zipfiles, and not just wheels. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
On 30 January 2014 10:56, Paul Moore p.f.mo...@gmail.com wrote: Here we go: https://github.com/numpy/numpy/issues/4245 Thanks. I've added the installation details and output from a test run. That bug report was just closed blaming a distutils issue which apparently numpy aren't going to work around :-( I don't know if you want to pick up on the issue and argue the case with the guy who closed it. So no, numpy won't build on Windows. Maybe in Python 3.5, if someone aggressively pushes for a distutils fix. But I'm not holding my breath. Sigh. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On 30 January 2014 11:57, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: ISTM the reason why it can't go in there just yet is also the reason why one might have some reservations about the feature: binary compatibility Sorry, I didn't comment on this point. To me, binary compatibility is *not* the issue. So my other email ignored it. It is a reasonable point, but not one that bothers me, oddly enough :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 11:17 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 10:49 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin eug...@sazhin.us wrote: Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. I didn’t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You’ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I’m 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it’s not entirely up to me I would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you’re missing the point a bit. I’m totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! Just use pip and virtualenv in production. It’s bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it’ll lead to hard to diagnose errors and bugs that you’ll never be able to reproduce in development. Even though the spec apparently allows it, using Wheels like this is an anti pattern. 1. use minimum of “standard” tools, 2. *manage dependencies* in a unified and standardized way 3. use only *one format* and *one artifact* per project for everything related to the code development, shipment, deployment and execution. What would be a motivation to learn yet another tool like buildout or anything, pollute infrastructure with more libraries, manage several different ways of working with the code, overcomplicate the production area by requiring stuff to be unzipped and whatnot? I hope nobody would want to strip any of those nice features from the wheel, but improve the running from it!;) I don’t understand what any of the rest of this have to do with wether you add zipped Wheels to your sys.path or not. see above Thanks, Eugene Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 30, 2014, at 9:30 AM, Donald Stufft don...@stufft.io wrote: On Jan 30, 2014, at 6:43 AM, Nick Coghlan ncogh...@gmail.com wrote: On 30 Jan 2014 15:27, Donald Stufft don...@stufft.io wrote: I can't believe folks are unable to differentiate between the difference of It's possible to do so because we didn't actively attempt to prevent it and This is a documented goal of the format and thus must be considered as part of the backward compatibility contract that the format has whenever changes are made to the format. Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional (https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format. This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it. I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design. For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40 This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made. The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of supports. At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep the process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're providing for that is. The problem with believing that we still have a choice about this topic is twofold: - firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here: - those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support
Re: [Distutils] wheels on sys.path clarification (reboot)
On 30 January 2014 15:03, Donald Stufft don...@stufft.io wrote: For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270. Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Jan 30, 2014, at 10:12 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 15:03, Donald Stufft don...@stufft.io wrote: For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270. Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear. Paul Sure, using a library via zip import when it wasn’t designed to be used as such is an anti pattern. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/30/2014 10:23 AM, Donald Stufft wrote: On Jan 30, 2014, at 10:12 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 15:03, Donald Stufft don...@stufft.io wrote: For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270. Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear. Paul Sure, using a library via zip import when it wasn’t designed to be used as such is an anti pattern. The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path. There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too. Which is a long way of saying: I think calling it zip_safe is a misnomer. The flag is really conveying I don't need to be on a filesystem. Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand. Eric. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
I see you've noticed wheel was released in an imperfect state. Let's add a Zip-Safe flag to the WHEEL file, values of true and false same as Root-Is-Purelib. Daniel On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith e...@trueblade.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/30/2014 10:23 AM, Donald Stufft wrote: On Jan 30, 2014, at 10:12 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 15:03, Donald Stufft don...@stufft.io wrote: For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270. Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear. Paul Sure, using a library via zip import when it wasn't designed to be used as such is an anti pattern. The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path. There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too. Which is a long way of saying: I think calling it zip_safe is a misnomer. The flag is really conveying I don't need to be on a filesystem. Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand. Eric. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))
On Thu, 30/1/14, Paul Moore p.f.mo...@gmail.com wrote: Subject: Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) To: Vinay Sajip vinay_sa...@yahoo.co.uk Cc: Distutils distutils-sig@python.org Date: Thursday, 30 January, 2014, 13:23 OK. Note that this is not, in my view, an issue with wheels, but rather about zipfiles on sys.path, and (deliberate) design limitations of the module loader and zipimport implementations.[1] Okay, I'm glad that's clarified. Otherwise, there's a danger of it being conflated with an importing wheels is bad viewpoint which relates even to pure-Python code. First of all, it is not possible to load a DLL into a process' memory [2, 3] unless it is stored as a file in the filesystem. So any attempt to import a C extension from a zipfile must, by necessity, involve extracting that DLL to the filesystem. That's where I see the problems. None are deal-breaking issues, but they consist of a number of niggling issues that cumulatively chip away at the reliability of the concept until the end result has enough corner cases and risks to make it unacceptable (depending on your tolerance for risks - there's a definite judgement call involved). Okay, let's work through the issues you raise. 1. You need to choose a location to put the extracted file. On Windows in particular, there is no guaranteed-available filesystem location that can be used without risk. Some accounts have no home directory, some (locked down) users have no permissions anywhere but very specific places, even TEMP may not be usable if there's an aggressive housekeeping routine in place - but TEMP is probably the best choice of a bad lot. There are always going to be environments where you can't do stuff, say because of corporate lock-down policies. There is no requirement on any solution to do the impossible; merely to fail with an informative error message. There is lots of other functionality that fails in these environments, too (e.g. access to the Internet). So my view is that this should not be an obstacle to developing such functionality for environments where it can work, as long as it fails fast and informatively when it fails. 2. There are race conditions to consider. If the extraction is not completely isolated per-process, what if 2 processes want to use different versions of the same DLL? How will these be distinguished? Processes are isolated from each other, so that doesn't stop different processes using different versions of DLLs. Software in those DLLs needs to be designed to avoid stepping on its own toes, but that's orthogonal to whether it came from a zip or not (.NET SxS assemblies, for example - if they have files they write to, they need to not overwrite each other's stuff). Distlib covers this by placing the DLL in a location which is based on the absolute pathname of the wheel it came from. So any software which uses the exact same wheel will use the same DLL, but other software which uses a wheel with a different version (which by definition will have a different absolute path) will use a different DLL. Perhaps there are holes in this approach - if so, please point out any that you see. 3. Clean-up is an issue. How will the extracted files be removed? You can't unload the DLLs from Python, and you can't delete open files in Windows. So do you simply leave the files lying round? Or do you do some sort of atexit dance to run a separate process after the Python process terminates which will do the cleanup? What happens to that process when virus checkers hold the file open? Leaving the files around is probably the most robust answer, but it's not exactly friendly. But it's a drawback of the underlying platform, and it seems to me OK to do the best that's possible (like we do with pip updating itself on Windows). Also, it's not clear if you always want to clean up: perhaps you don't want to extract DLLs every time if they're already there (let's not go down a cache invalidation rabbit-hole - later is definitely better than right now ;-) My view is that cleanup belongs with the application, not the library - the application developer is best placed to know what the right thing to do is for that particular application. This is currently covered in distlib by having an API which provides the root directory path for the cache. Cache cleanup can be done on start-up before any wheels are mounted. By the way, surely you've seen how much cruft accumulates in TEMP on Windows machines? It's not as if Windows users' expectations can be particularly high here ;-) I'm all for keeping things tidy, of course. The only place where having a wheel rather than a general zipfile makes a difference is that a wheel *might* at some point contain metadata that allows the wheel to claim that it's OK to load its contents from a zipfile. But my points above are not something
Re: [Distutils] wheels on sys.path clarification (reboot)
Just an FYI, I've begun compiling notes and plan on proposing PEPs that will be aimed at supplanting PEP 425 and PEP 427 with the goal of nailing down undefined behavior, including missing functionality, and removing misfeatures. On Jan 30, 2014, at 10:56 AM, Daniel Holth dho...@gmail.com wrote: I see you've noticed wheel was released in an imperfect state. Let's add a Zip-Safe flag to the WHEEL file, values of true and false same as Root-Is-Purelib. Daniel On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith e...@trueblade.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/30/2014 10:23 AM, Donald Stufft wrote: On Jan 30, 2014, at 10:12 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 15:03, Donald Stufft don...@stufft.io wrote: For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270. Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear. Paul Sure, using a library via zip import when it wasn't designed to be used as such is an anti pattern. The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path. There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too. Which is a long way of saying: I think calling it zip_safe is a misnomer. The flag is really conveying I don't need to be on a filesystem. Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand. Eric. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, 30/1/14, Eric V. Smith e...@trueblade.com wrote: The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path. Software can run from zip even if some things need filesystem access, so it's not as clear-cut as __file__ having to be a filesystem path. It's probably safe to assume that any C extension will need to be on a filesystem path, at least on the mainstream platforms, since that's how their run-time loaders work. For example, both pip and distil use cacert.pem to verify SSL hostnames, and the SSL module's API requires provision of a filesystem path for the .pem file. The solution is of course to use __file__ directly when running from the file system, and extracting cacert.pem to a cache location, and using that when running from zip. My view is that there are probably two different assertions: one is will run from wheel (I say wheel rather than zip, since wheel implies a place to put that metadata) and the other is The C extensions are mandatory. If this flag is set, the wheel is asserted to be not worth mounting without the extensions. If clear, it means that C extensions, if present, are optional (for example, an accelerator). If there are extensions, there would need to be separate metadata giving the details so that extraction could happen (since we should refuse the temptation to guess by merely looking at file extensions of entries in the archive). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 05:33, Evgeny Sazhin eug...@sazhin.us wrote: Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official support for zip import means that the change would have to be weighed against also breaking that backwards compatibility. Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;) This is the biggest concern I see with promoting wheels being directly importable via zipimport (I say promoting and not specifying deliberately, but I don't want to get back into the process debate). People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars. No offense taken - i'm happy to be a use case to add some perspective;)) This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive. For pure python wheels - you're right - very attractive! And while i hear you guys saying it is not advisable to use wheels in sys.path i'm still far from seeing good reasoning. Please, correct me, but so far i saw 2: 1. It is impossible to load C extensions 2. Error handling is bad None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge): 1. Yes there is a group of elite interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops? 2. IF we are talking about error handling it seems that people generally do not consider jar-like behavior to be bad for pure python modules - It's just doesn't work very well (yet?). Is there really a problem with error handling for pure python? What if the error handling is improved? Is it still going to be bad? Why? Pointing out the implication that wheels can be put on sys.path in specialised circumstances is entirely reasonable when talking to a Python audience, but it triggers entirely the wrong associations when a person with a Java background sees the statement. Evgeny - for your purposes, you'd be better advised to see wheels as similar in principle to Windows MSI installers (they aren't, but it will avoid triggering incorrect analogies). That is an awesome analogy, thank you! I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list. And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there. Thanks, Eugene (Be careful, you may hear suggestions that you use eggs because they are similar to jars - while that is true, the way Python works means that the experience with eggs is far from being as natural as the jar experience you're used to). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 11:17 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 10:49 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin eug...@sazhin.us wrote: Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. I'm totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. I'm happy to concede the run from zip thing if somebody could explain how should i use pip and virtualenv in production? Currently it seems to be very clean and clear. I can have one folder where i can dump multiple versions of the same project in wheel format and having requirements.txt in each of them i can construct PYTHONPATH and run things. Simple! How am i supposed to manage that using pip and virtual envs in production? Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 11:17 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 10:49 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin eug...@sazhin.us wrote: Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. I'm totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. I'm happy to concede the run from zip thing if somebody could explain how should i use pip and virtualenv in production? Currently it seems to be very clean and clear. I can have one folder where i can dump multiple versions of the same project in wheel format and having requirements.txt in each of them i can construct PYTHONPATH and run things. Simple! How am i supposed to manage that using pip and virtual envs in production? The same way you’d use them in development? Hell I believe you can even do: $ virtualenv my_virtualenv $ my_virtualenv/bin/pip install path/to/wheelhouse/* The point is it’s just installed libraries, asking this question doesn’t really make much sense, it’s like asking how do you use apt-get or yum in production. Thanks, Eugene - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Thu, Jan 30, 2014 at 11:48 AM, Donald Stufft don...@stufft.io wrote: On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 11:17 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 10:49 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin eug...@sazhin.us wrote: Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. I'm totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. I'm happy to concede the run from zip thing if somebody could explain how should i use pip and virtualenv in production? Currently it seems to be very clean and clear. I can have one folder where i can dump multiple versions of the same project in wheel format and having requirements.txt in each of them i can construct PYTHONPATH and run things. Simple! How am i supposed to manage that using pip and virtual envs in production? The same way you'd use them in development? Hell I believe you can even do: $ virtualenv my_virtualenv $ my_virtualenv/bin/pip install path/to/wheelhouse/* The point is it's just installed libraries, asking this question doesn't really make much sense, it's like asking how do you use apt-get or yum in production. Could you please reread this https://mail.python.org/pipermail/distutils-sig/2014-January/023636.html I'm explaining there what is the task at hand and why I'm favoring the jar-like workflow for deployment. Thanks, Eugnee ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))
Am 30.01.2014 14:23, schrieb Paul Moore: First of all, it is not possible to load a DLL into a process' memory [2, 3] unless it is stored as a file in the filesystem. [...] [2] I'm talking from a Windows perspective here. I do not have sufficient low-level knowledge of Unix to comment on that case. I suspect that the issues are similar but I defer to the platform experts. [3] There is, I believe, code out there on the internet to map a DLL image into a process based purely in memory, but I think it's a fairly gross hack. I have a suspicion that someone - possibly Thomas Heller - experimented with it at one time, but never came up with a viable implementation. There's also TCL's tclkit technology, which *may* support binary extensions, and may be worth a look, but TCL has virtual filesystem support built in quite deep in the core, so how it works may not be applicable to Python. Well, py2exe uses code that loads dlls from zip-files directly into process memory and executes it. This is used for 'single-file' applications that py2exe creates; it is triggered by the appropriate 'bundle_files' option. It works quite well for some applications (wx apps for example), but not for others. Applications that use numpy for example crashes, but the reason is that py2exe *also* puts the mkl-library (or maybe even more stuff like windows dlls) that numpy needs into the zip. So, the trick is to only load python-extensions and not arbitrary dlls from the zip. Changing the packing process so that py2exe leave these non-python dlls in the file system makes the exe work again. At least now the range of apps that work reliably is much larger. Anyway; the py2exe installer installs the python-extension and correspinding py-module that implements this magic, so anyone can experiment with it; it is called 'zipextimporter.py'. Here's the start of the docstring: rzipextimporter - an importer which can import extension modules from zipfiles This file and also _memimporter.pyd is part of the py2exe package. Overview zipextimporter.py contains the ZipExtImporter class which allows to load Python binary extension modules contained in a zip.archive, without unpacking them to the file system. Call the zipextimporter.install() function to install the import hook, add a zip-file containing .pyd or .dll extension modules to sys.path, and import them. It uses the _memimporter extension which uses code from Joachim Bauch's MemoryModule library. This library emulates the win32 api function LoadLibrary. I wanted to post this to clarify the current state. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, Jan 30, 2014 at 11:59 AM, Donald Stufft don...@stufft.io wrote: On Jan 30, 2014, at 11:33 AM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 05:33, Evgeny Sazhin eug...@sazhin.us wrote: Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official support for zip import means that the change would have to be weighed against also breaking that backwards compatibility. Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;) This is the biggest concern I see with promoting wheels being directly importable via zipimport (I say promoting and not specifying deliberately, but I don't want to get back into the process debate). People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars. No offense taken - i'm happy to be a use case to add some perspective;)) This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive. For pure python wheels - you're right - very attractive! And while i hear you guys saying it is not advisable to use wheels in sys.path i'm still far from seeing good reasoning. Please, correct me, but so far i saw 2: 1. It is impossible to load C extensions 2. Error handling is bad None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge): 1. Yes there is a group of elite interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops? Because C extensions are not elite, they are pretty common. It's somewhat rare in my experience to see a non trivial project that doesn't have a C extension somewhere in it. Isn't this a self contradictory statement? C extensions are quite common for non-trivial project. non-trivial is not common. Doesn't it mean that C extensions are rarely used for common trivial stuff? You also implying that there were non-trivial projects that didn't need C extensions. Or are you trying to make an argument that the overall majority of python projects have C extensions? Is python such a bad language it cannot deal with simple stuff without C being involved? 2. IF we are talking about error handling it seems that people generally do not consider jar-like behavior to be bad for pure python modules - It's just doesn't work very well (yet?). Is there really a problem with error handling for pure python? What if the error handling is improved? Is it still going to be bad? Why? i consider the fact that they'll only work in a subset of situations bad, because it's non obvious without digging through the code if any particular project will support working from a place other than a traditional install. The necessity to dig through the code to understand if the project is runnable from zip is a responsibility of project provider - i.e. poor documentation. If the project is designed to be run that way the doc should say so. This isn't a new feature, Eggs have had this for a long time and when confronted with bugs caused by zipped Eggs the common solution is simply adding zip_safe=False to their project which disables zipped eggs for that project. The name for the kind of feature this is, is an attractive nuisance. They look like something that might work, especially to people coming to Python from other languages such as Java, but if actually use the feature in a non trivial fashion you're going to end up with more problems then you've solved when random projects simply don't work and the likelihood that their authors have no desire to support running through a zipped file even if they *could* make it possible. OK, you're saying if actually use the feature in a non trivial fashion But was is non-trivial in putting pure python wheel in sys.path? Non-trivial would probably mean trying to load c extension from it, but that's obviously not supported, so i wouldn't even try. Thanks,
Re: [Distutils] PEP 427
On Thu, Jan 30, 2014 at 12:23 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 11:48 AM, Donald Stufft don...@stufft.io wrote: On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 11:17 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Jan 29, 2014, at 10:49 PM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin eug...@sazhin.us wrote: Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. I'm totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. I'm happy to concede the run from zip thing if somebody could explain how should i use pip and virtualenv in production? Currently it seems to be very clean and clear. I can have one folder where i can dump multiple versions of the same project in wheel format and having requirements.txt in each of them i can construct PYTHONPATH and run things. Simple! How am i supposed to manage that using pip and virtual envs in production? The same way you'd use them in development? Hell I believe you can even do: $ virtualenv my_virtualenv $ my_virtualenv/bin/pip install path/to/wheelhouse/* The point is it's just installed libraries, asking this question doesn't really make much sense, it's like asking how do you use apt-get or yum in production. Could you please reread this https://mail.python.org/pipermail/distutils-sig/2014-January/023636.html I'm explaining there what is the task at hand and why I'm favoring the jar-like workflow for deployment. It's clear that you favor the jar-like workflow. You will have discover for yourself why it works or does not work for you. There is more mature existing jar-like support for eggs (which will continue to work for some time, but not be improved). We do actually need a better way to turn libraries on and off individually, for example, if I install the wrong set of Python libraries on Ubuntu the plugin system for somesuch package crashes; having everything installed importable all the time is a problem. Even if you don't consider the unrelated packages that have chosen the same module name.
[Distutils] PEX - Twitter's multi-platform executable archive format for Python
I searched this list for PEX and didn't find any hits, so I thought some people here might be interested in the PEX software from Twitter: https://www.youtube.com/watch?v=NmpnGhRwsu0 is a 15-minute presentation by Brian Wickman about it and other topics. It uses zipimport and is used to do packaging of Python software in Twitter across multiple platforms, so it could have some relevance to recent discussions we've been having about wheels, binary compatibility and the like. Brian has posted here (re. PEP 425 tags most recently) but not about PEX AFAICT. I don't yet know if there's anything particularly revolutionary about the tech (I haven't dug into it yet) but of course it's interesting because of Twitter's size and the usage in production. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On 30 January 2014 16:33, Evgeny Sazhin eug...@sazhin.us wrote: I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list. And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there. There's 2 distinct concepts in the Python world, that you should be aware of. Application deployment and library deployment. Library deployment is simple. Build your library and upload it as a source distribution (sdist) and one or more wheels to PyPI (if it's a public project) or a local index (if it's a private project). People or other projects will use the library however they deem appropriate, typically by doing pip install projectname to make the code accessible. That unpacks the wheel into a local site-packages, but you have no need to care about that. Application deployment is not as straightforward - there are a number of common approaches and at least as many opinions on what you should do. You may bundle Python and your code and its dependencies together into a standalone executable, using a tool like cx_Freeze. or you may use something like buildout (I can't give details as I've never used this). You can also, in some cases, package your application main code in the same way as a library, and use entry points to define the executables - that will then be installed by the end user into an existing Python installation. As an application developer, you typically wouldn't care about how the end user deployed your code for execution, unless you were bundling via something like cx_Freeze. You might choose to bundle your application code *without* a Python interpreter. That's basically what executable zipfiles are about, and it's probably the closest equivalent to Java jars. But that's not a particularly well-travelled route in the Python community (yet) - there are projects like pyzzer that automate certain aspects of that process, but in general, you have to roll your own solution here. Hopefully that makes some sense - I'm getting pretty burned out with the various threads going on at the moment, so I'm probably glossing over a lot. Cheers, Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))
On 30 January 2014 17:51, Thomas Heller thel...@ctypes.org wrote: I wanted to post this to clarify the current state. Thanks. I had no idea (or I'd forgotten - one or the other!) that this was actually being done. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] platform specific python only wheels
Your best bet currently is to execute the mv command to change the filename of the wheel. I think bdist_wheel should probably accept a setup() argument to set the tags. Currently you can just set a universal flag which means pure 2+3 python. On Thu, Jan 30, 2014 at 1:14 PM, Ronald Oussoren ronaldousso...@mac.com wrote: Hi, Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? In particular, I'm looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot cross-compile application bundles. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, Jan 30, 2014 at 12:56 PM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 16:33, Evgeny Sazhin eug...@sazhin.us wrote: I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list. And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there. There's 2 distinct concepts in the Python world, that you should be aware of. Application deployment and library deployment. Library deployment is simple. Build your library and upload it as a source distribution (sdist) and one or more wheels to PyPI (if it's a public project) or a local index (if it's a private project). People or other projects will use the library however they deem appropriate, typically by doing pip install projectname to make the code accessible. That unpacks the wheel into a local site-packages, but you have no need to care about that. Application deployment is not as straightforward - there are a number of common approaches and at least as many opinions on what you should do. You may bundle Python and your code and its dependencies together into a standalone executable, using a tool like cx_Freeze. or you may use something like buildout (I can't give details as I've never used this). You can also, in some cases, package your application main code in the same way as a library, and use entry points to define the executables - that will then be installed by the end user into an existing Python installation. As an application developer, you typically wouldn't care about how the end user deployed your code for execution, unless you were bundling via something like cx_Freeze. You might choose to bundle your application code *without* a Python interpreter. That's basically what executable zipfiles are about, and it's probably the closest equivalent to Java jars. But that's not a particularly well-travelled route in the Python community (yet) - there are projects like pyzzer that automate certain aspects of that process, but in general, you have to roll your own solution here. Hopefully that makes some sense - I'm getting pretty burned out with the various threads going on at the moment, so I'm probably glossing over a lot. Cheers, Paul Thanks Paul - you are very helpful! this is good distinction and i think i'm coming to it as well. I just suggested one approach that might be feasible in my case in another thread;) Everybody - I'm really grateful for the time you are willing to spend in helping me to understand the state of the things. I hope this discussion might be helpful for both sides and may be people like me can be referenced to it in future;) PS That doesn't mean i will stop arguing with you;) Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] platform specific python only wheels
Hi, Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? In particular, I’m looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot “cross-compile” application bundles. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 30, 2014, at 1:09 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: On Thu, 30/1/14, Ralf Gommers ralf.gomm...@gmail.com wrote: Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying. If tool developers want to offer end users the option to control how they work with sys.path, that's up to them. For example, once the details are worked out, the distil tool will probably get a --mountable option for the package command, which will write metadata into the built wheel indicating whether the wheel is addable to sys.path or not (based on the builder's knowledge of the wheel's contents). Distlib, when asked to mount a wheel (add it to sys.path) will check the mountability metadata and honour the wheel publisher's intent. For everyone following along, the PEP has been updated. http://hg.python.org/peps/rev/26983acc9c11 If anyone has comments on the next text you can find it at http://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file I hope we can discuss further changes as a group before they are pushed live. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
One thing that might be useful would be to develop the unpacked wheel which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue. I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can). But as has been repeated you won't find robust support for this in the existing code. On Thu, Jan 30, 2014 at 1:19 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 12:56 PM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 16:33, Evgeny Sazhin eug...@sazhin.us wrote: I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list. And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there. There's 2 distinct concepts in the Python world, that you should be aware of. Application deployment and library deployment. Library deployment is simple. Build your library and upload it as a source distribution (sdist) and one or more wheels to PyPI (if it's a public project) or a local index (if it's a private project). People or other projects will use the library however they deem appropriate, typically by doing pip install projectname to make the code accessible. That unpacks the wheel into a local site-packages, but you have no need to care about that. Application deployment is not as straightforward - there are a number of common approaches and at least as many opinions on what you should do. You may bundle Python and your code and its dependencies together into a standalone executable, using a tool like cx_Freeze. or you may use something like buildout (I can't give details as I've never used this). You can also, in some cases, package your application main code in the same way as a library, and use entry points to define the executables - that will then be installed by the end user into an existing Python installation. As an application developer, you typically wouldn't care about how the end user deployed your code for execution, unless you were bundling via something like cx_Freeze. You might choose to bundle your application code *without* a Python interpreter. That's basically what executable zipfiles are about, and it's probably the closest equivalent to Java jars. But that's not a particularly well-travelled route in the Python community (yet) - there are projects like pyzzer that automate certain aspects of that process, but in general, you have to roll your own solution here. Hopefully that makes some sense - I'm getting pretty burned out with the various threads going on at the moment, so I'm probably glossing over a lot. Cheers, Paul Thanks Paul - you are very helpful! this is good distinction and i think i'm coming to it as well. I just suggested one approach that might be feasible in my case in another thread;) Everybody - I'm really grateful for the time you are willing to spend in helping me to understand the state of the things. I hope this discussion might be helpful for both sides and may be people like me can be referenced to it in future;) PS That doesn't mean i will stop arguing with you;) Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] platform specific python only wheels
On Thu, 30/1/14, Ronald Oussoren ronaldousso...@mac.com wrote: Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? Perhaps not yet, but the idea is there in the 'supports_environments' metadata in PEP 426. This is currently supported by distil, but I don't believe pip does yet. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] platform specific python only wheels
On 30 Jan 2014, at 19:19, Daniel Holth dho...@gmail.com wrote: Your best bet currently is to execute the mv command to change the filename of the wheel. I think bdist_wheel should probably accept a setup() argument to set the tags. Currently you can just set a universal flag which means pure 2+3 python. There’s also —plat-name argument for the setuptools command, but that doesn’t work. I’d have to rename the platform tag from “any” to a list of supported platforms, is that correct? That would give a pretty convoluted filename, because the platform tag doesn’t allow for wildcards. That is, there is no way to specify “any linux”, only something like “linux_x86_64” (to borrow a tag from PEP 425). Ronald On Thu, Jan 30, 2014 at 1:14 PM, Ronald Oussoren ronaldousso...@mac.com wrote: Hi, Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? In particular, I'm looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot cross-compile application bundles. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] platform specific python only wheels
On 30 Jan 2014, at 19:23, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: On Thu, 30/1/14, Ronald Oussoren ronaldousso...@mac.com wrote: Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? Perhaps not yet, but the idea is there in the 'supports_environments' metadata in PEP 426. This is currently supported by distil, but I don't believe pip does yet. supports_environment would work for me. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] platform specific python only wheels
On Thu, Jan 30, 2014 at 1:29 PM, Ronald Oussoren ronaldousso...@mac.com wrote: On 30 Jan 2014, at 19:19, Daniel Holth dho...@gmail.com wrote: Your best bet currently is to execute the mv command to change the filename of the wheel. I think bdist_wheel should probably accept a setup() argument to set the tags. Currently you can just set a universal flag which means pure 2+3 python. There's also --plat-name argument for the setuptools command, but that doesn't work. I'd have to rename the platform tag from any to a list of supported platforms, is that correct? That would give a pretty convoluted filename, because the platform tag doesn't allow for wildcards. That is, there is no way to specify any linux, only something like linux_x86_64 (to borrow a tag from PEP 425). My position has been that if there is no alternative that works on a different platform, then at least the wheel metadata should say nothing. It's only there to allow the installer to pick among the best of several alternatives. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] platform specific python only wheels
You are probably a perfect customer for supports_environment On Thu, Jan 30, 2014 at 1:40 PM, Ronald Oussoren ronaldousso...@mac.com wrote: On 30 Jan 2014, at 19:36, Daniel Holth dho...@gmail.com wrote: On Thu, Jan 30, 2014 at 1:29 PM, Ronald Oussoren ronaldousso...@mac.com wrote: On 30 Jan 2014, at 19:19, Daniel Holth dho...@gmail.com wrote: Your best bet currently is to execute the mv command to change the filename of the wheel. I think bdist_wheel should probably accept a setup() argument to set the tags. Currently you can just set a universal flag which means pure 2+3 python. There's also --plat-name argument for the setuptools command, but that doesn't work. I'd have to rename the platform tag from any to a list of supported platforms, is that correct? That would give a pretty convoluted filename, because the platform tag doesn't allow for wildcards. That is, there is no way to specify any linux, only something like linux_x86_64 (to borrow a tag from PEP 425). My position has been that if there is no alternative that works on a different platform, then at least the wheel metadata should say nothing. It's only there to allow the installer to pick among the best of several alternatives. The py2app setup.py file currently hardcodes a platform check because users did install py2app on other platforms (mostly Windows) and then complained that it didn't work. It is much nicer to tell the user that a package won't work during installation. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] platform specific python only wheels
On 30 Jan 2014, at 19:36, Daniel Holth dho...@gmail.com wrote: On Thu, Jan 30, 2014 at 1:29 PM, Ronald Oussoren ronaldousso...@mac.com wrote: On 30 Jan 2014, at 19:19, Daniel Holth dho...@gmail.com wrote: Your best bet currently is to execute the mv command to change the filename of the wheel. I think bdist_wheel should probably accept a setup() argument to set the tags. Currently you can just set a universal flag which means pure 2+3 python. There's also --plat-name argument for the setuptools command, but that doesn't work. I'd have to rename the platform tag from any to a list of supported platforms, is that correct? That would give a pretty convoluted filename, because the platform tag doesn't allow for wildcards. That is, there is no way to specify any linux, only something like linux_x86_64 (to borrow a tag from PEP 425). My position has been that if there is no alternative that works on a different platform, then at least the wheel metadata should say nothing. It's only there to allow the installer to pick among the best of several alternatives. The py2app setup.py file currently hardcodes a platform check because users did install py2app on other platforms (mostly Windows) and then complained that it didn’t work. It is much nicer to tell the user that a package won’t work during installation. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth dho...@gmail.com wrote: One thing that might be useful would be to develop the unpacked wheel which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue. I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can). But as has been repeated you won't find robust support for this in the existing code. I just tested it - it works! Yeah! I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse. After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl It worked! Is this workflow OK to rely on in future? Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On 30 Jan 2014, at 17:27, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: On Thu, 30/1/14, Eric V. Smith e...@trueblade.com wrote: The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path. […] My view is that there are probably two different assertions: one is will run from wheel (I say wheel rather than zip, since wheel implies a place to put that metadata) and the other is The C extensions are mandatory”. It’s not just wheels, the same information can be used by application bundlers like py2exe and py2app to recognise if a particular component can be included in a zipfile. Py2app currently includes all python code in a zip file within an application bundle, unless a python package is on a manually maintained list of software that won’t work that way. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth dho...@gmail.com wrote: One thing that might be useful would be to develop the unpacked wheel which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue. I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can). But as has been repeated you won't find robust support for this in the existing code. I just tested it - it works! Yeah! I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse. After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl It worked! Is this workflow OK to rely on in future? Thanks, Eugene The only request is that you don't name the directory .whl because that is a little bit confusing. Directories with only one package installed into them will continue to work. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, Jan 30, 2014 at 2:32 PM, Daniel Holth dho...@gmail.com wrote: On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth dho...@gmail.com wrote: One thing that might be useful would be to develop the unpacked wheel which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue. I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can). But as has been repeated you won't find robust support for this in the existing code. I just tested it - it works! Yeah! I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse. After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl It worked! Is this workflow OK to rely on in future? Thanks, Eugene The only request is that you don't name the directory .whl because that is a little bit confusing. Directories with only one package installed into them will continue to work. pip can't find the distribution then - so it must have the name totally matching wheel spec, i guess. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, Jan 30, 2014 at 12:46 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 11:59 AM, Donald Stufft don...@stufft.io wrote: On Jan 30, 2014, at 11:33 AM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 05:33, Evgeny Sazhin eug...@sazhin.us wrote: Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official support for zip import means that the change would have to be weighed against also breaking that backwards compatibility. Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;) This is the biggest concern I see with promoting wheels being directly importable via zipimport (I say promoting and not specifying deliberately, but I don't want to get back into the process debate). People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars. No offense taken - i'm happy to be a use case to add some perspective;)) This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive. For pure python wheels - you're right - very attractive! And while i hear you guys saying it is not advisable to use wheels in sys.path i'm still far from seeing good reasoning. Please, correct me, but so far i saw 2: 1. It is impossible to load C extensions 2. Error handling is bad None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge): 1. Yes there is a group of elite interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops? Because C extensions are not elite, they are pretty common. It's somewhat rare in my experience to see a non trivial project that doesn't have a C extension somewhere in it. Isn't this a self contradictory statement? C extensions are quite common for non-trivial project. non-trivial is not common. Doesn't it mean that C extensions are rarely used for common trivial stuff? You also implying that there were non-trivial projects that didn't need C extensions. Or are you trying to make an argument that the overall majority of python projects have C extensions? He's trying to argue that enough projects use C extensions that to have special support for projects that don't use them is supporting a special case. The Zen of Python says Special cases aren't special enough to break the rules and explicitly having wheels support direct execution only when they contain pure Python is supporting a special case. Obviously you can argue over at what point something stops being a special case, but considering venvs isolate code well and are the officially supported way to isolate code then that makes stuff that goes passed what venv can do a special case. Is python such a bad language it cannot deal with simple stuff without C being involved? The use of a C extension does not necessarily mean a shortcoming of Python. People can write performance-critical code in C so as to get that last bit of speed in heavily executed code in a tight loop, etc. It has nothing to do with Python not being able to handle something and more that C code can simply be faster when coded correctly; it's a feature and not a band-aid. -Brett 2. IF we are talking about error handling it seems that people generally do not consider jar-like behavior to be bad for pure python modules - It's just doesn't work very well (yet?). Is there really a problem with error handling for pure python? What if the error handling is improved? Is it still going to be bad? Why? i consider the fact that they'll only work in a subset of situations bad, because it's non obvious without digging through the code if any particular project will support working from a place other than a traditional install. The necessity to dig through the code to understand if the project is runnable
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, Jan 30, 2014 at 2:52 PM, Brett Cannon br...@python.org wrote: still far from seeing good reasoning. Please, correct me, but so far i saw 2: 1. It is impossible to load C extensions 2. Error handling is bad None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge): 1. Yes there is a group of elite interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops? Because C extensions are not elite, they are pretty common. It's somewhat rare in my experience to see a non trivial project that doesn't have a C extension somewhere in it. Isn't this a self contradictory statement? C extensions are quite common for non-trivial project. non-trivial is not common. Doesn't it mean that C extensions are rarely used for common trivial stuff? You also implying that there were non-trivial projects that didn't need C extensions. Or are you trying to make an argument that the overall majority of python projects have C extensions? He's trying to argue that enough projects use C extensions that to have special support for projects that don't use them is supporting a special case. The Zen of Python says Special cases aren't special enough to break the rules and explicitly having wheels support direct execution only when they contain pure Python is supporting a special case. Obviously you can argue over at what point something stops being a special case, but considering venvs isolate code well and are the officially supported way to isolate code then that makes stuff that goes passed what venv can do a special case. yes, you got it absolutely right - I'm coming from the point of view that the C extension is a special case. It's even called extension! And pure python is never special case, therefore should take priority. Is python such a bad language it cannot deal with simple stuff without C being involved? The use of a C extension does not necessarily mean a shortcoming of Python. People can write performance-critical code in C so as to get that last bit of speed in heavily executed code in a tight loop, etc. It has nothing to do with Python not being able to handle something and more that C code can simply be faster when coded correctly; it's a feature and not a band-aid. -Brett I understand that, i was sarcastic;) But thank you! Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
Wow, it really can install from a directory-named-like-a-wheel with the --no-index flag passed. I'm sad to say that is totally unintentional, but bizarrely interesting. On Thu, Jan 30, 2014 at 2:52 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 2:32 PM, Daniel Holth dho...@gmail.com wrote: On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin eug...@sazhin.us wrote: On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth dho...@gmail.com wrote: One thing that might be useful would be to develop the unpacked wheel which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue. I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can). But as has been repeated you won't find robust support for this in the existing code. I just tested it - it works! Yeah! I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse. After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl It worked! Is this workflow OK to rely on in future? Thanks, Eugene The only request is that you don't name the directory .whl because that is a little bit confusing. Directories with only one package installed into them will continue to work. pip can't find the distribution then - so it must have the name totally matching wheel spec, i guess. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, 30/1/14, Daniel Holth dho...@gmail.com wrote: Wow, it really can install from a directory-named-like-a-wheel with the --no-index flag passed. I'm sad to say that is totally unintentional, but bizarrely interesting. What, *more* unintentional drive-by behaviour? Now, where's that pitchfork? ;-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Jan 30, 2014, at 1:21 PM, Noah Kantrowitz n...@coderanger.net wrote: On Jan 30, 2014, at 1:09 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: On Thu, 30/1/14, Ralf Gommers ralf.gomm...@gmail.com wrote: Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying. If tool developers want to offer end users the option to control how they work with sys.path, that's up to them. For example, once the details are worked out, the distil tool will probably get a --mountable option for the package command, which will write metadata into the built wheel indicating whether the wheel is addable to sys.path or not (based on the builder's knowledge of the wheel's contents). Distlib, when asked to mount a wheel (add it to sys.path) will check the mountability metadata and honour the wheel publisher's intent. For everyone following along, the PEP has been updated. http://hg.python.org/peps/rev/26983acc9c11 If anyone has comments on the next text you can find it at http://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file I hope we can discuss further changes as a group before they are pushed live. --Noah ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig The updated text is fine with me as it at least accurately documents the fact that using that feature is fraught with peril and reads more of a documentation of the behavior than anything else. So thanks for the final (I hope?) update Nick, and sorry this thread got all ragey. On the plus side this spurred me into trying to put the thoughts I’ve had for a newer Wheel format down into a PEP and I think it’s turning out well so far :] - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On 31 Jan 2014 02:48, Donald Stufft don...@stufft.io wrote: How am i supposed to manage that using pip and virtual envs in production? The same way you'd use them in development? Hell I believe you can even do: $ virtualenv my_virtualenv $ my_virtualenv/bin/pip install path/to/wheelhouse/* The point is it's just installed libraries, asking this question doesn't really make much sense, it's like asking how do you use apt-get or yum in production. Most Java apps don't typically use those either, they just drop jar files in a directory on CLASSPATH :P (Hence why I only *tolerate* that approach for wheels for advanced use cases that can't easily be handled any other way, while strongly discouraging it in general) Cheers, Nick. Thanks, Eugene - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
Thanks for pointing that out Noah - I was planning to come back and check if that wording was considered more acceptable. On 31 Jan 2014 07:12, Donald Stufft don...@stufft.io wrote: The updated text is fine with me as it at least accurately documents the fact that using that feature is fraught with peril and reads more of a documentation of the behavior than anything else. Aye, clearly explaining yes, you *can*, but that doesn't mean you *should* was always my primary goal with the new FAQ entry. Until you commented on the original update, I didn't realise there might be a perception that this egg-replacement use case didn't need to be taken into account in designing future iterations of the wheel format, and it was only after I went back and watched the PyCon 2013 panel video that I realised we had likely run afoul of the conference curse where a shared offline understanding hadn't been properly captured in the online materials (it's also possible Daniel and I discussed it privately back before the wheel spec became a PEP, but I didn't dig back that far into my email archives). So thanks for the final (I hope?) update Nick, and sorry this thread got all ragey. Hah, if you want to see ragey, look up the epic PEP 343 threads between me and PJE before Guido broke the deadlock by proposing to just remove the cause of the argument entirely (the implicit dynamic context manager support that was in the originally accepted with statement design turned out to be thoroughly confusing because it made it too hard to come up with consistent terminology for the new concepts involved). I know all too well how disconcerting it can be to find that someone else has a significantly different interpretation of a spec that appeared to be crystal clear (especially a case like this where the written spec *was* clear, but missing a critical detail). I'm also nowhere near hypocritical enough to berate anyone *else* for passionately objecting to a course of action that appears to lead places we don't want to go :) On the plus side this spurred me into trying to put the thoughts I've had for a newer Wheel format down into a PEP and I think it's turning out well so far :] Huzzah! Turning caremad into PEPs can be surprisingly cathartic (cf. PEP 462). Cheers, Nick. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))
On 30 Jan 2014 23:26, Paul Moore p.f.mo...@gmail.com wrote: On 30 January 2014 12:29, Nick Coghlan ncogh...@gmail.com wrote: I actually think this is a useful thing to experiment with, I'm just not sure distlib is the best place for that experiment. With appropriately secure tempfile handling and the right sys.path (and module __path__) manipulation it's not obviously *impossible* to handle C extensions at arbitrary positions in the module namespace this way, just difficult. zipimport itself is a bad place to experiment though, since not only is it currently a complex ball of C code, but adding such a feature without clear evidence of robust support in a third party project would be irresponsible. I just sent a long message that essentially gave a chunk of history and suggested a similar thing - an enhanced zipimport module to experiment with solutions in this space. With importlib available, an experimental implementation shouldn't even be too hard. The only difference is that I see very little reason why such a solution can't apply to all zipfiles, and not just wheels. The advantage of wheels over plain zipfiles for this use case is the structured metadata. distlib.mount doesn't try to guess the package structure for the extensions, you have to provide an EXTENSIONS file in the metadata that explains what C extensions are present and how they should map into the module namespace. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))
On 30 January 2014 22:38, Nick Coghlan ncogh...@gmail.com wrote: The advantage of wheels over plain zipfiles for this use case is the structured metadata. distlib.mount doesn't try to guess the package structure for the extensions, you have to provide an EXTENSIONS file in the metadata that explains what C extensions are present and how they should map into the module namespace. OK, I think I get the idea now. I'm still not comfortable with the temp directory clutter that unpacking leaves (in particular on Windows where deletion isn't even possible in an atexit routine) but I'll survive. I *would* like to see the various technical issues and implications in the API documentation, though. The implications and limitations, and in particular the manual cache management requirements, need to be made explicit. (I thought I'd seen docs somewhere, but they definitely aren't in the API reference for the distlib.wheel module). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))
On Thu, 30/1/14, Paul Moore p.f.mo...@gmail.com wrote: I'm still not comfortable with the temp directory clutter that unpacking leaves (in particular on Windows where deletion isn't even possible in an atexit routine) but I'll survive. It's up to the using application to do cache management of this type. It's not as if the cache will grow unbounded in most realistic scenarios, and perfectly possible to do cleanup during startup. It may not be ideal, but seems acceptable given the limitations of the underlying platform. I *would* like to see the various technical issues and implications in the API documentation, though. The implications and limitations, and in particular the manual cache management requirements, need to be made explicit. (I thought I'd seen docs somewhere, but they definitely aren't in the API reference for the distlib.wheel module). There's some mention in the distlib.resources documentation (the cache would also be used for things like cacert.pem), but I'll certainly update the wheel documentation to cover this area, and update the relevant parts of the distlib.resources and distlib.util documentation, too. BTW, I raised the whole issue of extracting C extensions for import from zips on python-dev in March last year: https://mail.python.org/pipermail/python-dev/2013-March/124970.html The feedback I got indicated (to me) that while people felt there were some problem areas (e.g. cleanup was mentioned), there were no show-stoppers. No ringing endorsements, but no red flags were raised either. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
