Re: [Distutils] How to handle launcher script importability?
Nick Coghlan ncoghlan at gmail.com writes: I previously thought distlib was going to be the repository for the agreed, stable, this is going to happen stuff. It's OK that I was wrong - I think you're right that somewhere is needed as an experimental location to show some of the *possibilities* of the new metadata, and to seed ideas for making it into the eventual standard base that people can assume is readily available. It's not just about completely new, experimental stuff. For example, the resources functionality isn't completely new territory. The PyPI interfacing is (IMO) a saner API than the one in distutils2. A better Windows story (for when launcher support when py.exe can't be used) is also not rocket science. What that means though, is we need *something else* that indicates the common core that people can assume will always be available. It's this If that something else you're thinking of is something that is supposed to live in the stdlib, then I see no reason why a subset of distlib couldn't be that something else, since stdlib changes are not on the table for 3.4. I certainly have never envisaged that distlib would be adopted wholesale into the stdlib (if at all) without peer review and any changes coming out of that. common core which pip will need to factor out to remove their dependency on setuptools, rather than adopting distlib wholesale, experimental features and all. I honestly think you're making a bit too much of the experimental label here, even though it is a label that I use myself. For me, that label is most appropriate for the extended metadata that I collect from PyPI and which is the basis for distlib's smarter dependency resolution. If your concerns are about instability due to experimental features (and I quite understand the importance of stability in packaging), then there's nothing stopping anyone doing a technical review of distlib to see what any actual risks are. Indeed, I've invited such review from day one. Currently, pip doesn't expose a programmatic API. I suggested to Donald that it may make sense to start exposing one as piplib. The I think this would be a mistake, and it seems a little early to make this sort of decision. You've given me to understand that pip could at some future point use (some subset of) distlib under the covers, with compatibility maintained at the CLI level. If that is still the case, then I don't see much value in having two lib layers. Like setuptools, pip has done sterling service, but it's not a codebase I'd like to see become the basis for our long-term packaging infrastructure. I don't mean to offend anybody by saying that - it's just software that has grown organically over time and IMO there will be technical debt to repay if we go down the route of exposing bits of it as Python APIs. It certainly feels like you're side-lining distlib, or planning to, whether or not that's the message you're intending to send. No matter :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Am 20.08.2013 19:39, schrieb PJ Eby: I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? To give a definite answer, after thinking it over: It works, for quite some extensions. The main problem is this: If it does NOT work (process crashes) there is no way to find out why. It is nearly impossible to debug because you end up with all the machine code from the extensions/dlls mapped into the process and the debugger has no info about it. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Disclaimer: everything I say below about pip is ultimately up to the pip devs. I'm just pointing out what I think makes sense, and my reading of Donald's comments means that I expect he would feel the same way. On 22 August 2013 17:22, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Nick Coghlan ncoghlan at gmail.com writes: What that means though, is we need *something else* that indicates the common core that people can assume will always be available. It's this If that something else you're thinking of is something that is supposed to live in the stdlib, then I see no reason why a subset of distlib couldn't be that something else, since stdlib changes are not on the table for 3.4. I certainly have never envisaged that distlib would be adopted wholesale into the stdlib (if at all) without peer review and any changes coming out of that. I *can't* tell people we're going to migrate to distlib as the reference packaging infrastructure implementation when I really mean we're going to migrate to some as yet undefined subset of distlib, so 'import distlib' won't be the long term answer. Incorporating only a subset of an existing published API into the standard library is a mistake - the PyXML debacle shows us that. If the API is different (even if that means a strict subset), then it needs a different name. common core which pip will need to factor out to remove their dependency on setuptools, rather than adopting distlib wholesale, experimental features and all. I honestly think you're making a bit too much of the experimental label here, even though it is a label that I use myself. For me, that label is most appropriate for the extended metadata that I collect from PyPI and which is the basis for distlib's smarter dependency resolution. If your concerns are about instability due to experimental features (and I quite understand the importance of stability in packaging), then there's nothing stopping anyone doing a technical review of distlib to see what any actual risks are. Indeed, I've invited such review from day one. It has nothing to do with code quality, and everything to do with being able to explain the migration plan to people. I *can* say to them pip is going to cherry pick parts of distlib and potentially other libraries and make them available as 'piplib', which will be installed automatically when you install pip. At the moment, I no longer feel I can say distlib will become the reference implementation. Note that there's also the bootstrapping issue with having pip depend on an external library: having the core library *in pip* makes that problem go away. Currently, pip doesn't expose a programmatic API. I suggested to Donald that it may make sense to start exposing one as piplib. The I think this would be a mistake, and it seems a little early to make this sort of decision. You've given me to understand that pip could at some future point use (some subset of) distlib under the covers, with compatibility maintained at the CLI level. If that is still the case, then I don't see much value in having two lib layers. When I made that suggestion, I misunderstood your plans for distlib. If pip are only adopting a subset of it, they can't use the same name, or people will get confused. I now think it makes more sense for pip to migrate to a more tightly constrained public library API, that doesn't go beyond the documented metadata standards (except for legacy compatibility reasons). Like setuptools, pip has done sterling service, but it's not a codebase I'd like to see become the basis for our long-term packaging infrastructure. I don't mean to offend anybody by saying that - it's just software that has grown organically over time and IMO there will be technical debt to repay if we go down the route of exposing bits of it as Python APIs. I don't expect the contents of piplib to match the contents of the existing pip module. This is about enabling a gradual refactoring over to a cleaner core library with a public API, rather than a big bang migration to an alternative solution like distlib. It certainly feels like you're side-lining distlib, or planning to, whether or not that's the message you're intending to send. No matter :-) I don't currently believe your plans for distlib and my plans for the standard library software distribution support (whether directly in the standard library or via the pip bootstrapping) are compatible. If I am correct, then distlib remains extemely valuable in that scenario, but the nature of its role changes. I want a completely barebones absolutely no features that aren't covered by an Accepted PEP library as a candidate for future inclusion. Such a library *cannot* be particularly useful at this point in time, because the metadata 2.0 PEPs are nowhere near complete enough for that. By contrast, you understandably wish for distlib to be useful *now*, which means running ahead of the standardisation process, filling in missing
Re: [Distutils] How to handle launcher script importability?
Nick Coghlan ncoghlan at gmail.com writes: standard library is a mistake - the PyXML debacle shows us that. If the API is different (even if that means a strict subset), then it needs a different name. I'm not really hung up about a specific name - what's in a name? It has nothing to do with code quality, and everything to do with being able to explain the migration plan to people. I *can* say to Code quality is pertinent when it's the subtext behind experimental. them pip is going to cherry pick parts of distlib and potentially other libraries and make them available as 'piplib', which will be installed automatically when you install pip. At the moment, I no longer feel I can say distlib will become the reference implementation. Note that there's also the bootstrapping issue with having pip depend on an external library: having the core library *in pip* makes that problem go away. When I made that suggestion, I misunderstood your plans for distlib. If pip are only adopting a subset of it, they can't use the same name, or people will get confused. I can certainly see that there are ways to avoid confusion. But never mind, I see that you've made your decision. I would have hoped for a more transparent decision process, but that's probably due to my slowness of uptake. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 23 August 2013 00:19, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Nick Coghlan ncoghlan at gmail.com writes: When I made that suggestion, I misunderstood your plans for distlib. If pip are only adopting a subset of it, they can't use the same name, or people will get confused. I can certainly see that there are ways to avoid confusion. But never mind, I see that you've made your decision. I would have hoped for a more transparent decision process, but that's probably due to my slowness of uptake. The only decision I've made is to stop saying distlib is the future, since that is now in doubt, and I certainly don't have the time available or expertise needed to review all the APIs that have been added to it (I thought it was just the four distutils2 interfaces that almost made it into Python 3.3 and that all your experimental interfaces were in distil, not distlib. While there was plenty of evidence to indicate I was wrong in that belief, I wasn't paying proper attention to it and it didn't properly register until it came up in this thread). The next step is up to the pip folks - if they think adopting distlib wholesale makes sense for them, fine, I have no direct say in that. If they decide to make a piplib instead, to expose a public API for an updated version of pip's own infrastructure (perhaps derived from distlib), that's fine by me, too. The only absolute in this space relates to the default installation toolchain: it *will* be pip. Unlike setutptools as a build system, I consider easy_install irredeemable as an installer (from a social perspective), and there's no way I would ever inflict yet another change of recommended client on the community. Given that, any future changes to the core infrastructure will be heavily influenced by the technical choices of the pip developers. Other tools will exist around that, since packaging is a complex topic where one size really doesn't fit all, but pip will be the centrepiece. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 22 August 2013 16:04, Nick Coghlan ncogh...@gmail.com wrote: The next step is up to the pip folks - if they think adopting distlib wholesale makes sense for them, fine, I have no direct say in that. If they decide to make a piplib instead, to expose a public API for an updated version of pip's own infrastructure (perhaps derived from distlib), that's fine by me, too. For what it's worth, we currently have a vendored copy of distlib bundled into pip but (a) it's pretty out of date now and (b) we only make minimal use of it - in particular we do not use it for any of the wheel support at the moment. I don't have any feel for what we might do going forward - I suspect we'll wait until the dust settles a bit on the whole issue in distutils-sig before trying to make a decision. For virtualenv, I have a longer-term plan to switch to bundling pip and setuptools wheels instead of sdists, but again I don't plan on rushing into a decision on how I'll do that. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Donald Stufft donald at stufft.io writes: Mounting Wheels seems like a bad idea, it was one of the things Daniel explicitly removed (since Wheels are basically cleaned up eggs). Adding it back in ex post facto seems like it's an idea that's going down the wrong track. Like I said, the sky will fall. Isn't importing from zips what's being discussed in this part of the thread? Unless something is expressly verboten, I can't see a reason why particular implementations of a PEP can't provide additional functionality, as long as they implement the contents of the standard. And in Python's consenting-adults world, I can't recall seeing any such express proscriptions. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 07:36, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: Mounting Wheels seems like a bad idea, it was one of the things Daniel explicitly removed (since Wheels are basically cleaned up eggs). Adding it back in ex post facto seems like it's an idea that's going down the wrong track. Like I said, the sky will fall. Isn't importing from zips what's being discussed in this part of the thread? Unless something is expressly verboten, I can't see a reason why particular implementations of a PEP can't provide additional functionality, as long as they implement the contents of the standard. And in Python's consenting-adults world, I can't recall seeing any such express proscriptions. I'm concerned that you need extra metadata (not described in the wheel spec) to do this. It means that there are in effect two subtly different types of wheel. To be specific, if I create a wheel for (say) pyzmq using distil, and mount it, everything works. But if I create the same wheel with bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages the credibility of wheel as a standardised format. Can I suggest that if you need to add features like this, you need to get the wheel spec updated to mandate them, so that *all* wheels will follow the same spec. Essentially, I am -1 on any feature that uses information that is not documented in the wheel spec. Pip in particular resisted adding support for wheels until they were standardised in a PEP. It's frustrating if that PEP *still* doesn't mean that the wheel format is the same for all tools. (Note that another area where this is an issue is script wrappers, as the spec is silent about the fact that they are specified using entry-points.txt in metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel for his consideration). Paul PS Variances like this make me want to go back to the original idea that wheel support functions should be implemented in the stdlib, rather than having competing implementations in the wild. Or at least, that one implementation should be considered the reference implementation that all others need to be compatible with :-( ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Paul Moore p.f.moore at gmail.com writes: I'm concerned that you need extra metadata (not described in the wheel spec) to do this. It means that there are in effect two subtly different types of wheel. To be specific, if I create a wheel for (say) pyzmq using distil, and mount it, everything works. But if I create the same wheel with bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages the credibility of wheel as a standardised format. If the additional metadata isn't there, then distlib just doesn't do anything additional - it just makes the Python modules importable (by adding the wheel to sys.path, which AFAIK is uncontroversial). Can I suggest that if you need to add features like this, you need to get the wheel spec updated to mandate them, so that *all* wheels will follow the same spec. Essentially, I am -1 on any feature that uses information that is not documented in the wheel spec. Pip in particular resisted adding support for wheels until they were standardised in a PEP. It's frustrating if that PEP *still* doesn't mean that the wheel format is the same for all tools. (Note that another area where this is an issue is script wrappers, as the spec is silent about the fact that they are specified using entry-points.txt in metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel for his consideration). Well, you don't really want to stifle innovation, do you? ;-) As far as I can tell, Daniel's wheel implementation allows files that are not specifically mentioned in the PEP to be installed into a distribution's .dist-info. This is also allowed in distlib - ISTM this is one way in which different packaging tools can add features which are special to them, and hold state relevant to distributions they build and/or install. If you accept that multiple competing implementations if a PEP are a good thing, then they can't all be functionally identical, though they must all implement a common set of functions described in the PEP they're implementing. As far as the advocacy for C-extension import support in wheel mounting goes, I see opposition to the idea on the basis that some people have shot themselves in the foot with a similar feature in pkg_resources. However, I've not seen any analysis that indicates (with specifics) why the feature is inherently bad (if there are problems with a specific implementation of the idea, then those could perhaps be resolved, but you can't really argue against you're going to have a bad time, and you should feel bad). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 3:32 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: I'm concerned that you need extra metadata (not described in the wheel spec) to do this. It means that there are in effect two subtly different types of wheel. To be specific, if I create a wheel for (say) pyzmq using distil, and mount it, everything works. But if I create the same wheel with bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages the credibility of wheel as a standardised format. If the additional metadata isn't there, then distlib just doesn't do anything additional - it just makes the Python modules importable (by adding the wheel to sys.path, which AFAIK is uncontroversial). Can I suggest that if you need to add features like this, you need to get the wheel spec updated to mandate them, so that *all* wheels will follow the same spec. Essentially, I am -1 on any feature that uses information that is not documented in the wheel spec. Pip in particular resisted adding support for wheels until they were standardised in a PEP. It's frustrating if that PEP *still* doesn't mean that the wheel format is the same for all tools. (Note that another area where this is an issue is script wrappers, as the spec is silent about the fact that they are specified using entry-points.txt in metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel for his consideration). Well, you don't really want to stifle innovation, do you? ;-) As far as I can tell, Daniel's wheel implementation allows files that are not specifically mentioned in the PEP to be installed into a distribution's .dist-info. This is also allowed in distlib - ISTM this is one way in which different packaging tools can add features which are special to them, and hold state relevant to distributions they build and/or install. If you accept that multiple competing implementations if a PEP are a good thing, then they can't all be functionally identical, though they must all implement a common set of functions described in the PEP they're implementing. I was one of the advocates for extension support in the new metadata, I want tools to be able to try things out and innovate. However what I don't really want is to be using someones personal testbed for features they think is cool. There's nothing *wrong* with you trying new ideas out in distlib, it just means that distib isn't the library I want to build tooling around. My basic problem is if the library we're pointing at to be the reference implementation of all of these things is adding new features it's confusing what is standard and what are just distlib's extensions. So basically I want people to innovate, that's something I feel very strongly is a good thing, I just don't want innovations to happen in the reference library. Maybe we need a smaller reference library which is strictly the PEPs to allow distlib to experiment. If it's experimentations turns out to be good and useful we can make PEPs for them and add them to the reference library. As far as the advocacy for C-extension import support in wheel mounting goes, I see opposition to the idea on the basis that some people have shot themselves in the foot with a similar feature in pkg_resources. However, I've not seen any analysis that indicates (with specifics) why the feature is inherently bad (if there are problems with a specific implementation of the idea, then those could perhaps be resolved, but you can't really argue against you're going to have a bad time, and you should feel bad). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 08:45, Donald Stufft don...@stufft.io wrote: My basic problem is if the library we're pointing at to be the reference implementation of all of these things is adding new features it's confusing what is standard and what are just distlib's extensions. So basically I want people to innovate, that's something I feel very strongly is a good thing, I just don't want innovations to happen in the reference library. Maybe we need a smaller reference library which is strictly the PEPs to allow distlib to experiment. If it's experimentations turns out to be good and useful we can make PEPs for them and add them to the reference library. +1 My problem is that as someone who wants to implement code that uses the new features like wheels, I want a usable reference implementation that covers the (agreed) standards. I don't particularly want my application to incorporate support for extensions to the standard, nor do I want to have to implement my own support all the time. In particular, at the present time there are two tools that can generate wheels (bdist_wheel and distlib) and 3 that can install them (wheel, distlib and pip). They have subtly different behaviours outside of the standard definitions, which means that they are not completely interoperable. I am not happy at all about that - and if that counts as being against innovation, then I'm afraid that yes, I am... (I don't think it does, by the way, but you may differ). At the moment the wheel PEP is lagging a little behind some of the ongoing discussions, in particular in terms of script generation. That's fine, it's a work in progress. I hope it will be updated soon so that the spec matches what's been agreed. But I think we have a reasonable consensus on how scripts should work, and I think that should be reflected in the spec and the various tools be brought up to date with the spec before we move onto other areas and forget to tidy up around this one. Pip resisted including wheel support until we had a standard. I'm pretty unhappy that now we do have a standard, we still have situations where a wheel generated by one tool can have problems when installed with another - try pip install wheel --use-wheel on Windows to see what I mean, the exe wrappers are missing (this uses the wheel from PyPI, not a home-built one). OK, so here's a concrete question for distutils-sig. If I want to use wheels in my app (built them, install them, whatever) what should I use as my reference implementation. I don't want to implement the code myself, I just want to produce lowest-common-denominator wheels that can be used anywhere, and consume wheels that conform to the spec correctly. This is not a hypothetical question - in the first instance I'm looking to add support for loading setuptools/pip from wheels in virtualenv, and I need to know what code to bundle to make that happen. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 4:07 AM, Paul Moore p.f.mo...@gmail.com wrote: OK, so here's a concrete question for distutils-sig. If I want to use wheels in my app (built them, install them, whatever) what should I use as my reference implementation. I don't want to implement the code myself, I just want to produce lowest-common-denominator wheels that can be used anywhere, and consume wheels that conform to the spec correctly. This is not a hypothetical question - in the first instance I'm looking to add support for loading setuptools/pip from wheels in virtualenv, and I need to know what code to bundle to make that happen. Probably Wheel at this point. There's just the problem with the scripts which we need to actually get into the PEP and implemented. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Donald Stufft donald at stufft.io writes: However what I don't really want is to be using someones personal testbed for features they think is cool. There's nothing *wrong* with you trying new ideas out in distlib, it just means that distib isn't the library I want to build tooling around. Someone's personal test-bed? How do you think all open source software starts out? What innovation isn't based on a feature someone thinks is cool? From our interactions, I don't get a feeling that you're particularly interested in building tooling around distlib anyway, for reasons best known to you. This specific feature could be easily pulled, and distlib is still only at version 0.1.2. Nick has indicated he doesn't think it's appropriate to consider distlib ready for endorsement in a 3.4 time-frame, so there's plenty of time to make changes which are deemed necessary. How on earth do you expect people to try things out, see where the problems are and fix them, if you don't release such features? It would certainly help if you describe specific problems with this feature, rather than just I don't like it. My basic problem is if the library we're pointing at to be the reference implementation of all of these things is adding new features it's confusing what is standard and what are just distlib's extensions. There are already features in distlib which aren't mentioned in any PEP - e.g. the ability to calculate dependency graphs without downloading any archives, or the ability to install scripts as single executables on Windows, or the ability to give better feedback when uninstalling. So basically I want people to innovate, that's something I feel very strongly is a good thing, I just don't want innovations to happen in the reference library. Maybe we need a smaller reference library which is ISTM distlib is not yet that reference library - it's just another library for most people, judging from the low level of feedback I've had overall. There's plenty of time to try things out and make changes, so I find your approach a bit of an over-reaction, and feel that it doesn't square with what you're saying about innovation. You know that feeling you mentioned that you get, where you think everyone is just out to block you and stop you getting stuff done? Are you trying to spread that feeling around? ;-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 09:09, Donald Stufft don...@stufft.io wrote: On Aug 21, 2013, at 4:07 AM, Paul Moore p.f.mo...@gmail.com wrote: OK, so here's a concrete question for distutils-sig. If I want to use wheels in my app (built them, install them, whatever) what should I use as my reference implementation. I don't want to implement the code myself, I just want to produce lowest-common-denominator wheels that can be used anywhere, and consume wheels that conform to the spec correctly. This is not a hypothetical question - in the first instance I'm looking to add support for loading setuptools/pip from wheels in virtualenv, and I need to know what code to bundle to make that happen. Probably Wheel at this point. There's just the problem with the scripts which we need to actually get into the PEP and implemented. I thought someone would say that. Wheel implies a dependency on setuptools (pkg_resources) which is viable for the virtualenv use case, but makes me somewhat sad in the more general case (because depending on setuptools at runtime feels wrong to me and there's no standalone pkg_resources). But nevertheless I think you're right. Thanks. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 4:23 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: However what I don't really want is to be using someones personal testbed for features they think is cool. There's nothing *wrong* with you trying new ideas out in distlib, it just means that distib isn't the library I want to build tooling around. Someone's personal test-bed? How do you think all open source software starts out? What innovation isn't based on a feature someone thinks is cool? From our interactions, I don't get a feeling that you're particularly interested in building tooling around distlib anyway, for reasons best known to you. This specific feature could be easily pulled, and distlib is still only at version 0.1.2. Nick has indicated he doesn't think it's appropriate to consider distlib ready for endorsement in a 3.4 time-frame, so there's plenty of time to make changes which are deemed necessary. How on earth do you expect people to try things out, see where the problems are and fix them, if you don't release such features? It would certainly help if you describe specific problems with this feature, rather than just I don't like it. I don't particularly care about this feature. I care about having a simple reference library. That is what I thought distlib was supposed to be (and I'm not alone). However if you want to innovate and experiment inside it then great, that's fine. I just won't tell people that distlib is the reference library. My basic problem is if the library we're pointing at to be the reference implementation of all of these things is adding new features it's confusing what is standard and what are just distlib's extensions. There are already features in distlib which aren't mentioned in any PEP - e.g. the ability to calculate dependency graphs without downloading any archives, or the ability to install scripts as single executables on Windows, or the ability to give better feedback when uninstalling. As mentioned above, I'm not trying to argue against this particular feature I'm just trying to make sure we have a simple reference library for people to be pointed at. So basically I want people to innovate, that's something I feel very strongly is a good thing, I just don't want innovations to happen in the reference library. Maybe we need a smaller reference library which is ISTM distlib is not yet that reference library - it's just another library for most people, judging from the low level of feedback I've had overall. That's totally fine. We just need to be clear that it's not the reference library and is instead one implementation. There's plenty of time to try things out and make changes, so I find your approach a bit of an over-reaction, and feel that it doesn't square with what you're saying about innovation. You know that feeling you mentioned that you get, where you think everyone is just out to block you and stop you getting stuff done? Are you trying to spread that feeling around? ;-) I think the way you view distlib and the way other are viewing distlib are different (and that's ok). We just need to know what distlib is so we can have reasonable expectations of it. What i'm getting from you is that, at least right now, distlib isn't what I thought it was and I (and others) should stop treating it as *the* reference implementation of the new standards. I'm not trying to stop you from innovating, I'm just trying to make sure everyone has reasonable expectations all around. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 09:56, Donald Stufft don...@stufft.io wrote: ISTM distlib is not yet that reference library - it's just another library for most people, judging from the low level of feedback I've had overall. That's totally fine. We just need to be clear that it's not the reference library and is instead one implementation. Yep, there's certainly been a perception that distlib is the reference implementation. Apologies if I perpetuated that. We do have a slightly different issue then, in that there *isn't* a reference implementation for a lot of this stuff... (I guess wheel counts as the reference implementation for wheel, doh, so that part's covered). People are starting to write code to use these new facilities, so having an actual reference implementation is important (IMO, that's one area where packaging/distutils2 got in a mess, so I'm concerned we don't fall into the same trap). We need people using the new stuff to help us identify potential issues. Paul PS Apologies if I appear to be a little irritable on this subject. I have a series of scripts that maintain a local cache of wheels for various projects. I'm starting to hit cases where the wheels aren't usable because of subtle differences in how the spec's being implemented, and it feels like I'm trying to hit a moving target, which is what I thought having the wheel 1.0 PEP accepted was designed to avoid. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 5:27 AM, Paul Moore p.f.mo...@gmail.com wrote: On 21 August 2013 09:56, Donald Stufft don...@stufft.io wrote: ISTM distlib is not yet that reference library - it's just another library for most people, judging from the low level of feedback I've had overall. That's totally fine. We just need to be clear that it's not the reference library and is instead one implementation. Yep, there's certainly been a perception that distlib is the reference implementation. Apologies if I perpetuated that. We do have a slightly different issue then, in that there *isn't* a reference implementation for a lot of this stuff... (I guess wheel counts as the reference implementation for wheel, doh, so that part's covered). People are starting to write code to use these new facilities, so having an actual reference implementation is important (IMO, that's one area where packaging/distutils2 got in a mess, so I'm concerned we don't fall into the same trap). We need people using the new stuff to help us identify potential issues. I might see about making a stripped down library which only implements the PEPs and nothing else. Paul PS Apologies if I appear to be a little irritable on this subject. I have a series of scripts that maintain a local cache of wheels for various projects. I'm starting to hit cases where the wheels aren't usable because of subtle differences in how the spec's being implemented, and it feels like I'm trying to hit a moving target, which is what I thought having the wheel 1.0 PEP accepted was designed to avoid. Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 10:29, Donald Stufft don...@stufft.io wrote: Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal). The scripts one is the key one (and yes, that needs to be fixed by updating the spec to confirm the consensus and then updating the tools to match). Another one IIRC was that distlib didn't put entry-points.txt in the .dist-info directory in the wheel (which breaks entry points). I think that's fixed now (and again, the Wheel spec is silent on what is correct behaviour here). I'll try to dig out the others, but they weren't anywhere near as major (some were also script-related such as inconsistent rewriting of shebang lines, which are superseded by the current state of play on scripts anyway). Generally, though, they were all areas where the 1.0 spec is silent (or just a bit vague) on details, and I was hitting implementation-defined differences. (For example, I was surprised to find that nowhere in the Wheel 1.0 spec does it actually state the metadata version that wheels should contain internally, just that it should be stored in a file called METADATA and that it should follow PKG-INFO which sort of implies that it's a 1.X format). Without an implementation that matches the spec precisely, though, it's often hard to know where to report bugs. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 5:46 AM, Paul Moore p.f.mo...@gmail.com wrote: On 21 August 2013 10:29, Donald Stufft don...@stufft.io wrote: Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal). The scripts one is the key one (and yes, that needs to be fixed by updating the spec to confirm the consensus and then updating the tools to match). +1 Another one IIRC was that distlib didn't put entry-points.txt in the .dist-info directory in the wheel (which breaks entry points). I think that's fixed now (and again, the Wheel spec is silent on what is correct behaviour here). Ah yes, This one is a bit harder because it's a non standard file added by setuptools, but yea it probably should be adding that file. I'll try to dig out the others, but they weren't anywhere near as major (some were also script-related such as inconsistent rewriting of shebang lines, which are superseded by the current state of play on scripts anyway). Generally, though, they were all areas where the 1.0 spec is silent (or just a bit vague) on details, and I was hitting implementation-defined differences. (For example, I was surprised to find that nowhere in the Wheel 1.0 spec does it actually state the metadata version that wheels should contain internally, just that it should be stored in a file called METADATA and that it should follow PKG-INFO which sort of implies that it's a 1.X format). Without an implementation that matches the spec precisely, though, it's often hard to know where to report bugs. Paul. I think Wheel files are (and should be) independent of the particular metadata version used. That file should contain the required information in order to know what version of the metadata is included with the Wheel. This means that as metadata evolves Wheels can just start using the new meta data version without requiring an update to the spec. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 10:48, Donald Stufft don...@stufft.io wrote: I think Wheel files are (and should be) independent of the particular metadata version used. That file should contain the required information in order to know what version of the metadata is included with the Wheel. This means that as metadata evolves Wheels can just start using the new meta data version without requiring an update to the spec. That implies that any wheel reference implementation needs to expose APIs for reading and writing the metadata to/from the wheel. I don't have a problem with that, but I don't think the existing implementations do[1]... (And it could be a bit of a beast to design such an interface in a sufficiently future-proof manner, unless we also standardise a metadata object type...) Specifically, if I have a wheel and want to introspect it to find out the author email address, how do I do this? (Doing so is not supported is a valid answer, of course, but should also be documented...) Paul [1] Just checked. Distlib doesn't. Wheel doesn't (and wheel doesn't even have a fully documented API). Pip's wheel support is purely internal so doesn't count. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 6:17 AM, Paul Moore p.f.mo...@gmail.com wrote:
On 21 August 2013 10:48, Donald Stufft don...@stufft.io wrote:
I think Wheel files are (and should be) independent of the particular
metadata version used. That file should contain the required information in
order to know what version of the metadata is included with the Wheel. This
means that as metadata evolves Wheels can just start using the new meta data
version without requiring an update to the spec.
That implies that any wheel reference implementation needs to expose APIs for
reading and writing the metadata to/from the wheel. I don't have a problem
with that, but I don't think the existing implementations do[1]... (And it
could be a bit of a beast to design such an interface in a sufficiently
future-proof manner, unless we also standardise a metadata object type...)
Specifically, if I have a wheel and want to introspect it to find out the
author email address, how do I do this? (Doing so is not supported is a
valid answer, of course, but should also be documented...)
Paul
[1] Just checked. Distlib doesn't. Wheel doesn't (and wheel doesn't even have
a fully documented API). Pip's wheel support is purely internal so doesn't
count.
Yes I believe there should be an API for reading/writing to the dist-info
directory and that's how a Wheel API should handle exposing that. It means you
can compose api's so you have 1 API for reading/writing metadata which can be
used for Wheels, Sdist 2.0, The on disk installed database format etc.
Quick off the cuff design of an API for introspecting the author email address
(Please realize this is totally off the top of my head and has not been thought
through or played with or explored or anything that would make it reasonable to
actually expect it to be sane for more than this simple example).
from ref import Metadata
from ref import Wheel, Sdist2
whl = Wheel.from_file(something-something.whl)
print(whl.dist_info)
{METADATA: …, entry_points.txt: …}
meta = Metadata.from_mapping(whl.dist_info)
print(meta[summary])
A Something or Other Library
sdist2 = Sdist2.from_file(something-something.sdist2)
print(sdist2.dist_info)
{pydist.json: …, README: …}
meta = Metadata.from_mapping(sdist2.dist_info)
print(meta[summary])
A something or Other Library
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 6:29 AM, Donald Stufft don...@stufft.io wrote: introspecting the author email address Of course I wrote that and then did summary because the location of the author email address changed between Metadata 1.x and 2.x and I didn't feel like looking up the exact difference. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 11:30, Donald Stufft don...@stufft.io wrote: On Aug 21, 2013, at 6:29 AM, Donald Stufft don...@stufft.io wrote: introspecting the author email address Of course I wrote that and then did summary because the location of the author email address changed between Metadata 1.x and 2.x and I didn't feel like looking up the exact difference. :-) So would you see the API for author being if meta.version (2, 0): author = meta[whatever_it_is_in_1.x] else: author = meta[whatever_it_is_in_2.x] (i.e., the metadata version would be available)? Sounds like a sensible approach. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 6:36 AM, Paul Moore p.f.mo...@gmail.com wrote: On 21 August 2013 11:30, Donald Stufft don...@stufft.io wrote: On Aug 21, 2013, at 6:29 AM, Donald Stufft don...@stufft.io wrote: introspecting the author email address Of course I wrote that and then did summary because the location of the author email address changed between Metadata 1.x and 2.x and I didn't feel like looking up the exact difference. :-) So would you see the API for author being if meta.version (2, 0): author = meta[whatever_it_is_in_1.x] else: author = meta[whatever_it_is_in_2.x] (i.e., the metadata version would be available)? Sounds like a sensible approach. Paul. Yes, and possibly some sort of compat shim could be made to smooth over the differences if it was felt to be warranted. But in general I'm very much in favor of composed APIs instead of having things like WheelMetadata, Sdist2Metadata, InstalledDBMetadata etc. Less code and easier to switch out pieces (e.g. in your own code you wouldn't need to change the metadata handling based on if it was a Wheel or a Sdist2, just load the appropriate file type and pass the mapping into the Metadata api). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Paul Moore p.f.moore at gmail.com writes: My problem is that as someone who wants to implement code that uses the new features like wheels, I want a usable reference implementation that covers the (agreed) standards. I don't particularly want my application to incorporate support for extensions to the standard, nor do I want to have to implement my own support all the time. The wheel implementation in distlib conforms to the existing PEPs, as far as I know. It covers the agreed standards and, as far as I know, in a reasonable way. Why should you care if there are additional methods in a Wheel class, if you never call on them? It seems a step too far for a spec of this type to dictate precisely what specific methods should or should not be present in a class. If you find that distlib's behaviour somehow violates the agreed standards, please tell me how and I'll fix it. I think I've been pretty responsive on issues raised, but please don't raise vague fears if they have no basis in fact. In particular, at the present time there are two tools that can generate wheels (bdist_wheel and distlib) and 3 that can install them (wheel, distlib and pip). They have subtly different behaviours outside of the standard definitions, which means that they are not completely interoperable. I am not happy at all about that - and if that counts as being against innovation, then I'm afraid that yes, I am... (I don't think it does, by the way, but you may differ). As I understand it, any interoperability issues are an effect of everything being a work in progress and some implementations lagging behind others in terms of PEP compliance. I can't speak for Daniel, but I've certainly made efforts to ensure that distlib can work with wheel-produced wheels, even where it means using old METADATA and entry_points.txt formats. I find it always helps to focus on specifics - since the implementations can't be identical, what specifically are the differences that make you unhappy, and why do they make you unhappy? If some of those can be laid at distlib's door, fine - I think I'm generally quite responsive to specific issues raised. Also, note that some features in an implementation might be useful to some people, even if not enshrined in a PEP. For example, distlib's wheel installation provides an option to install only the site-packages parts - no scripts or headers - a feature which you specifically requested, and which I readily added (as I can see its usefulness) without playing a it's not in the PEP card. Is that a harmless innovation, or a dangerous deviation from the standard? At the moment the wheel PEP is lagging a little behind some of the ongoing discussions, in particular in terms of script generation. That's fine, it's a work in progress. I hope it will be updated soon so that the spec matches what's been agreed. Well, let's not forget that distlib is a work in progress, too. But I think we have a reasonable consensus on how scripts should work, Do we? Someone might tell me tomorrow that the cool one-executable solution I've recently implemented as an option on Windows (append shebang / archive to a stock executable) is an offence against all that's holy ;-) had a standard. I'm pretty unhappy that now we do have a standard, we still have situations where a wheel generated by one tool can have problems when installed with another - try pip install wheel --use-wheel on Windows to see what I mean, the exe wrappers are missing (this uses the wheel from PyPI, not a home-built one). If distlib is proving to be a problem either as a producer or consumer of wheels, please raise issues. OK, so here's a concrete question for distutils-sig. If I want to use wheels in my app (built them, install them, whatever) what should I use as my reference implementation. I don't want to implement the code myself, I just want to produce lowest-common-denominator wheels that can be used anywhere, and consume wheels that conform to the spec correctly. This is not a hypothetical question - in the first instance I'm looking to add support for loading setuptools/pip from wheels in virtualenv, and I need to know what code to bundle to make that happen. Some will say wheel, even though it doesn't fully implement the spec, apparently because the wheel mount code (which,naturally, doesn't do anything unless it's called) is offensive to the point that some sort of fork is warranted ;-) Seriously - I already install setuptools and pip from wheels into venvs using distil (which of course uses distlib) - have you tried it? You're welcome to try with wheels built using any project, and if you run into any issues I will do my best to fix them. Have you had a better offer? :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 6:56 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: My problem is that as someone who wants to implement code that uses the new features like wheels, I want a usable reference implementation that covers the (agreed) standards. I don't particularly want my application to incorporate support for extensions to the standard, nor do I want to have to implement my own support all the time. The wheel implementation in distlib conforms to the existing PEPs, as far as I know. It covers the agreed standards and, as far as I know, in a reasonable way. Why should you care if there are additional methods in a Wheel class, if you never call on them? It seems a step too far for a spec of this type to dictate precisely what specific methods should or should not be present in a class. If you find that distlib's behaviour somehow violates the agreed standards, please tell me how and I'll fix it. I think I've been pretty responsive on issues raised, but please don't raise vague fears if they have no basis in fact. The spec does not and will not dictate what particular feature any one implementation has or does not have other than the base set outlined in the spec. In particular, at the present time there are two tools that can generate wheels (bdist_wheel and distlib) and 3 that can install them (wheel, distlib and pip). They have subtly different behaviours outside of the standard definitions, which means that they are not completely interoperable. I am not happy at all about that - and if that counts as being against innovation, then I'm afraid that yes, I am... (I don't think it does, by the way, but you may differ). As I understand it, any interoperability issues are an effect of everything being a work in progress and some implementations lagging behind others in terms of PEP compliance. I can't speak for Daniel, but I've certainly made efforts to ensure that distlib can work with wheel-produced wheels, even where it means using old METADATA and entry_points.txt formats. I find it always helps to focus on specifics - since the implementations can't be identical, what specifically are the differences that make you unhappy, and why do they make you unhappy? If some of those can be laid at distlib's door, fine - I think I'm generally quite responsive to specific issues raised. Also, note that some features in an implementation might be useful to some people, even if not enshrined in a PEP. For example, distlib's wheel installation provides an option to install only the site-packages parts - no scripts or headers - a feature which you specifically requested, and which I readily added (as I can see its usefulness) without playing a it's not in the PEP card. Is that a harmless innovation, or a dangerous deviation from the standard? Nobody is saying that the only useful features are the ones enshrined in a PEP. At the moment the wheel PEP is lagging a little behind some of the ongoing discussions, in particular in terms of script generation. That's fine, it's a work in progress. I hope it will be updated soon so that the spec matches what's been agreed. Well, let's not forget that distlib is a work in progress, too. But I think we have a reasonable consensus on how scripts should work, Do we? Someone might tell me tomorrow that the cool one-executable solution I've recently implemented as an option on Windows (append shebang / archive to a stock executable) is an offence against all that's holy ;-) had a standard. I'm pretty unhappy that now we do have a standard, we still have situations where a wheel generated by one tool can have problems when installed with another - try pip install wheel --use-wheel on Windows to see what I mean, the exe wrappers are missing (this uses the wheel from PyPI, not a home-built one). If distlib is proving to be a problem either as a producer or consumer of wheels, please raise issues. OK, so here's a concrete question for distutils-sig. If I want to use wheels in my app (built them, install them, whatever) what should I use as my reference implementation. I don't want to implement the code myself, I just want to produce lowest-common-denominator wheels that can be used anywhere, and consume wheels that conform to the spec correctly. This is not a hypothetical question - in the first instance I'm looking to add support for loading setuptools/pip from wheels in virtualenv, and I need to know what code to bundle to make that happen. Some will say wheel, even though it doesn't fully implement the spec, apparently because the wheel mount code (which,naturally, doesn't do anything unless it's called) is offensive to the point that some sort of fork is warranted ;-) It has nothing to do with the Wheel mount code in particular. A reference implantation is extremely useful but a reference implementation cannot contain random features that
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 11:56, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: But I think we have a reasonable consensus on how scripts should work, Do we? To the level of wheels builders should write metadata that defines the scripts and wheel installers should generate the necessary wrappers then yes. We may or may not (depending on your viewpoint) have agreement on the format of that metadata. Personally, I think that Metadata 2.0 where the wheel uses that format, and entry-points.txt for older versions, is the only realistic option here. I would like to see some clarity on the status of the wheel 1.0 spec. Where there are areas like this where the spec is either silent or missing sufficient detail to allow us to implement a common approach, should we be updating the 1.0 spec or should we be creating a 1.1 spec (I'd prefer the former)? Who does those updates? Must they go through Daniel as the author (who's been quite quiet, possibly he's hiding somewhere rather than being drawn into the fray :-))? Someone might tell me tomorrow that the cool one-executable solution I've recently implemented as an option on Windows (append shebang / archive to a stock executable) is an offence against all that's holy ;-) That won't be me. I'm more interested in discussions about interoperability issues than about how the functionality gets implemented. had a standard. I'm pretty unhappy that now we do have a standard, we still have situations where a wheel generated by one tool can have problems when installed with another - try pip install wheel --use-wheel on Windows to see what I mean, the exe wrappers are missing (this uses the wheel from PyPI, not a home-built one). If distlib is proving to be a problem either as a producer or consumer of wheels, please raise issues. I don't believe that distlib creates scripts based on entry-points.txt for pre-2.0 metadata. I have raised an issue for that. That's the only significant issue I have with distlib from the POV of what it can solve. There may be other interoperability issues where other tools can't consume things that distlib produces - I haven't tested anything like all of the combinations. We currently have 2 producers and 3 consumers. That's already 6 interactions to test. And if there *is* an issue, it's not always at all clear who to report the problem to. I'm not complaining about *any* of the implementations here. And I apologise if you in particular feel picked on (there's probably a distutils-sig victim of the week rota somewhere for this ;-)) My real concern is that we're drifting away from standards-based design towards implementation-based. And I think that updating the wheel PEP to be tighter on some of the details is what's really needed (hey, look, it's Daniel's turn to be picked on! :-)) I'd be willing to author an updated version of the wheel spec if all of the interested parties are OK with that (in particular, Nick and Daniel). If nothing else, that means that everyone can pick on me so I wouldn't feel quite so much like a troublemaker :-) Paul PS I really, really don't want anyone here to feel like anything they do is not valued. This is about tidying up the documentation, not about stifling anyone's enthusiasm or blocking any experimentation. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Paul Moore p.f.moore at gmail.com writes: Another one IIRC was that distlib didn't put entry-points.txt in the .dist-info directory in the wheel (which breaks entry points). I think that's fixed now (and again, the Wheel spec is silent on what is correct behaviour here). Right. The recent PEP 426 updates with the commands key supersedes the entry points stuff, so I'm not sure what exactly should be done here. The distlib code uses the latest PEP information. Since wheel is moving to pydist.json, I assume that (when it gets around to it) it will have the relevant scripts info in pydist.json, so I haven't implemented using scripts declared in entry_points.txt in distlib for that reason. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Donald Stufft donald at stufft.io writes: I think the way you view distlib and the way other are viewing distlib are different (and that's ok). We just need to know what distlib is so we can have reasonable expectations of it. What i'm getting from you is that, at least right now, distlib isn't what I thought it was and I (and others) should stop treating it as *the* reference implementation of the new standards. I'm not trying to stop you from innovating, I'm just trying to make sure everyone has reasonable expectations all around. I don't see how anyone is treating it as a reference implementation, other than just talking about it being such. I have had very good feedback from one or two people who are using it, but for the most part I see no evidence that people on this list are using it to the extent they would if they really thought this might be a good candidate for a reference implementation - I see it's early days, and there's work to do, but it seems usable, so let me check it out and see that my use cases are covered, and if anything's been overlooked. In my view, nothing deserves to be a considered a reference implementation other than through merit, or perhaps by being number one in a field of one. Merit isn't earned unless the software is used and refined based on real-world experience. The time to try distlib is now (not in production environments, obviously), to allow any problems with it to be identified early in its life. That would be more helpful than any sniping from the sidelines about whether it does more than what it needs to for PEP conformance. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 21, 2013 at 8:01 AM, Paul Moore p.f.mo...@gmail.com wrote: On 21 August 2013 12:22, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: Another one IIRC was that distlib didn't put entry-points.txt in the .dist-info directory in the wheel (which breaks entry points). I think that's fixed now (and again, the Wheel spec is silent on what is correct behaviour here). Right. The recent PEP 426 updates with the commands key supersedes the entry points stuff, so I'm not sure what exactly should be done here. The distlib code uses the latest PEP information. Since wheel is moving to pydist.json, I assume that (when it gets around to it) it will have the relevant scripts info in pydist.json, so I haven't implemented using scripts declared in entry_points.txt in distlib for that reason. OK, I see what you're saying here. But the Wheel 1.0 spec says that metadata is in the METADATA file (and comes from PKG-INFO). So my reading of that means that Metadata 1.x will remain valid for the foreseeable future (sure, Metadata 2.0 may become acceptable *as well* but the point of having a Wheel 1.0 spec is that we won't stop supporting it for some time yet). So you need to have a pre-2.0 solution in place, and while entry-points.txt isn't explicitly stated in the wheel PEP, it's the obvious equivalent. If you want to say distlib won't support pre-Metadata 2.0 specifications of script metadata, then that's your choice - it's not contrary to the standards but I'd view it as a quality of implementation choice. I view the underspecification in the Wheel 1.0 spec as similarly a quality of detail issue, and I'd expect to fix it in either an update to Wheel 1.0, or a Wheel 1.1 which does not make the jump to pure Metadata 2.0. Paul The wheel spec should probably say that its .dist-info directory should conform to a .dist-info PEP rather than saying it contains any particular files like METADATA. The only files that belong to wheel itself are the manifest and the WHEEL file that contains the version of the wheel format itself. Before the (useful) scripts wrapper feature, the wheel installer did not need to look at the PEP 426 or setuptools metadata at all. You can install them with unzip. The wheel command line tool lately has a wheel install-scripts [package] command that uses setuptools to rewrite entry-points script wrappers for an installed package. It is a proof of concept but it comes in handy. When pip is updated to be able to generate script wrappers at install time then a simultaneously released bdist_wheel will be able to update its default behavior to defer script generation and produce the newer metadata. In the meantime scripts are imperfect. Egg's controversial features like being a valid sys.path entry were not really left out of wheel. The format could have been a .tar file after all. Instead, wheel's documentation emphasizes what turned out to be the least problematic use pattern as an installation format. Egg got there from the other direction starting as a plugin container. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 9:02 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: I think the way you view distlib and the way other are viewing distlib are different (and that's ok). We just need to know what distlib is so we can have reasonable expectations of it. What i'm getting from you is that, at least right now, distlib isn't what I thought it was and I (and others) should stop treating it as *the* reference implementation of the new standards. I'm not trying to stop you from innovating, I'm just trying to make sure everyone has reasonable expectations all around. I don't see how anyone is treating it as a reference implementation, other than just talking about it being such. I have had very good feedback from one or two people who are using it, but for the most part I see no evidence that people on this list are using it to the extent they would if they really thought this might be a good candidate for a reference implementation - I see it's early days, and there's work to do, but it seems usable, so let me check it out and see that my use cases are covered, and if anything's been overlooked. In my view, nothing deserves to be a considered a reference implementation other than through merit, or perhaps by being number one in a field of one. Merit isn't earned unless the software is used and refined based on real-world experience. The time to try distlib is now (not in production environments, obviously), to allow any problems with it to be identified early in its life. That would be more helpful than any sniping from the sidelines about whether it does more than what it needs to for PEP conformance. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig I think you're using a completely different definition of reference implementation than I've ever seen used. A reference implementation by definition cannot contain customizations or additions or extensions from the spec. The entire *point* of a reference implementation is to act as programatic reference to the spec. Something being the reference implementation does not speak to the quality of the implementation and as such it may not be the *best* implementation. It becomes extremely useful however when you want to test conformance against the spec because it gives you a baseline with which to test against. Instead of needing to test against N different implementations people wanting to work with Wheel would only need to test against the reference implementation, and if that works then they can assume that their code will work against any other implementation of Wheel that properly implements the standard. If you don't have a reference implementation then people need to interpret the standards on their own and hopefully get it right. An example is the wsgiref from the standard library. Very few projects actively use wsgiref for much at all if they use it at all. However it's existence means that web servers like gunicorn, mod_wsgi etc can simply test against it instead of needing to test against every implementation of WSGI. Which implementation is used (and ultimately possibly enshrined in the standard library) is decided through merit. Which implementation is used as the reference implementation is typically decided by the standards body (in this case, distutils-sig or Nick or whoever). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Donald Stufft donald at stufft.io writes: I think you're using a completely different definition of reference implementation than I've ever seen used. A reference implementation Quite possibly, but I feel justified in this case ... I'll say why below. by definition cannot contain customizations or additions or extensions from the spec. The entire *point* of a reference implementation is to act as programatic reference to the spec. Something being the reference implementation does not speak to the quality of the implementation and as such it may not be the *best* implementation. The packaging PEPs don't do down to programmatic detail in their specification. In the stricter definition of reference implementation (RI) you're talking about, a spec comes with a reference implementation but also a test suite (provided by the spec author) which conforming implementations must pass. While this is de rigueur in the Java world, it's not common in the Python world. Without such a test conformance suite, I think it's reasonable to use the looser definition of RI that I did. An example is the wsgiref from the standard library. Very few projects actively use wsgiref for much at all if they use it at all. However it's existence means that web servers like gunicorn, mod_wsgi etc can simply test against it instead of needing to test against every implementation of WSGI. While you can certainly use wsgiref for interoperability testing (much as I use wheel's wheel implementation for interoperability testing with distlib) I don't think WSGI implementations run a suite of tests provided in wsgiref before claiming conformance. I'm happy to be corrected if I'm wrong about that :-) In my experience of implementing PEPs (282, 391, 397, 405) all implementations have had test suites, but none have had the tests implemented by any central authority. In discussions the relevant implementations have been referred to as RIs and that's the way I'm using it now. Which implementation is used (and ultimately possibly enshrined in the standard library) is decided through merit. Which implementation is used as the reference implementation is typically decided by the standards body (in this case, distutils-sig or Nick or whoever). If Nick or whoever is planning to write a test suite which all implementations must pass, great. But that would (I suppose) mean tying things down to a specific Python interface at the module, class and function level - not something I've seen imposed externally before. But in the absence of such, I don't see any problem with my interpretation of RI, since the stricter interpretation only makes sense in the presence of accompanying tests generated by the spec originators. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 11:30 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: I think you're using a completely different definition of reference implementation than I've ever seen used. A reference implementation Quite possibly, but I feel justified in this case ... I'll say why below. by definition cannot contain customizations or additions or extensions from the spec. The entire *point* of a reference implementation is to act as programatic reference to the spec. Something being the reference implementation does not speak to the quality of the implementation and as such it may not be the *best* implementation. The packaging PEPs don't do down to programmatic detail in their specification. In the stricter definition of reference implementation (RI) you're talking about, a spec comes with a reference implementation but also a test suite (provided by the spec author) which conforming implementations must pass. While this is de rigueur in the Java world, it's not common in the Python world. Without such a test conformance suite, I think it's reasonable to use the looser definition of RI that I did. An example is the wsgiref from the standard library. Very few projects actively use wsgiref for much at all if they use it at all. However it's existence means that web servers like gunicorn, mod_wsgi etc can simply test against it instead of needing to test against every implementation of WSGI. While you can certainly use wsgiref for interoperability testing (much as I use wheel's wheel implementation for interoperability testing with distlib) I don't think WSGI implementations run a suite of tests provided in wsgiref before claiming conformance. I'm happy to be corrected if I'm wrong about that :-) I have no idea what the WSGI servers actually do, it was just an example. In my experience of implementing PEPs (282, 391, 397, 405) all implementations have had test suites, but none have had the tests implemented by any central authority. In discussions the relevant implementations have been referred to as RIs and that's the way I'm using it now. None of those peps are defining a standard with a primary goal of being able to be completely replaced by a different module and still work. They are for adding a single implementation to the standard library. So it would make sense for those not to have a reference implementation because there are not any expected other implementations besides the one that got added to the stdlib. Which implementation is used (and ultimately possibly enshrined in the standard library) is decided through merit. Which implementation is used as the reference implementation is typically decided by the standards body (in this case, distutils-sig or Nick or whoever). If Nick or whoever is planning to write a test suite which all implementations must pass, great. But that would (I suppose) mean tying things down to a specific Python interface at the module, class and function level - not something I've seen imposed externally before. But in the absence of such, I don't see any problem with my interpretation of RI, since the stricter interpretation only makes sense in the presence of accompanying tests generated by the spec originators. I would expect that the reference implementation would include acceptance tests that test the various specs yes. For Wheel this wouldn't (in my mind) be at the Python level (so it wouldn't encode a particular python level API) other than a very minimal interface (which may simply be a CLI script?) needed to actually power the testing but the primary use of the acceptance tests would be to verify that the implementation can create and install Wheel files to the spec. It's purpose would not be to set a specific Pythonic API as the spec because the spec is defining a file format not a Python API. Not having behavior defined by an implementation is very important to me, hence why I'm being such a stickler for wanting something that other tools can use to test against. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 08/21/2013 03:29 AM, Donald Stufft wrote: Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal). Since you asked, I'll mention the two that I've hit (though I think you're also aware of these already): 1) Wheel's conversion of - to _ in version strings embedded in filenames, which breaks with setuptools precedent; see https://github.com/pypa/pip/issues/1150 and https://bitbucket.org/dholth/wheel/issue/78/wheel-rewrites-versions-preventing 2) Wheel's decision to follow distutils' documentation rather than distutils' behavior when it comes to the location for installing data_files with relative paths; see https://bitbucket.org/dholth/wheel/issue/80/wheel-does-not-install-data_files-in-site Carl signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 21, 2013 at 9:24 AM, Donald Stufft don...@stufft.io wrote: An example is the wsgiref from the standard library. It's an example, alright, but not for your side. ;-) The wsgiref library doesn't just implement the spec, it implements a ton of utility classes for use with the spec. The validator was almost an afterthought grafted on later, borrowed from another project. It implements a framework with all sorts of features that are not technically part of the spec, but are just useful if you want to implement the spec. Very few of the classes, methods, etc. in the entire package are specified by the spec, except in the sense that many of them match a calling signature defined in the PEP. (The PEP doesn't specify any method names, except for things like read() on file-like objects.) IOW, wsgiref is a collection of generally useful tools for anybody doing things with the spec, as an combination of examples of how to do this and ready-to-use code for working with the spec. Personally, I'm very happy to see Vinay's extensions, because they are IMO important validations of whether the new specs are likely to be useful for replacing all of setuptools' functionality. There are people who need to mount eggs and have their extensions run, so if it wasn't possible to build tools that support them under the new specs (whether that support is required by the spec or not), that would still be a reason to use setuptools -- meaning, IMO, that the new spec effort is failing to create a unified packaging world. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 12:31 PM, PJ Eby p...@telecommunity.com wrote: Personally, I'm very happy to see Vinay's extensions, because they are IMO important validations of whether the new specs are likely to be useful for replacing all of setuptools' functionality. There are people who need to mount eggs and have their extensions run, so if it wasn't possible to build tools that support them under the new specs (whether that support is required by the spec or not), that would still be a reason to use setuptools -- meaning, IMO, that the new spec effort is failing to create a unified packaging world. I'm perfectly happy that he was able (and did) write those extensions, I was one of the people who wanted metadata 2.0 to be extensible for exactly that purpose. None of my arguments were against any particular feature. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 08/21/2013 10:32 AM, Daniel Holth wrote: 2) Wheel's decision to follow distutils' documentation rather than distutils' behavior when it comes to the location for installing data_files with relative paths; see https://bitbucket.org/dholth/wheel/issue/80/wheel-does-not-install-data_files-in-site Django has fixed it by using package_data appropriately: https://code.djangoproject.com/ticket/19252 . The problem isn't unique to wheel, the same data_files mishap happens with bdist_wininst. Regardless, comment 5 is correct that we jump through way too many hoops in our setup.py in order to try to trick distutils into handling data_files as if they were package_data, and that is the root cause of this bug. Instead we should just use package_data and solve the problem properly. Yup, that's my comment you're quoting :-) I do think from the packager end using package_data is the right solution. But given the existence of distributions using data_files this way (and the likelihood that not all of them will be fixed), is there a good argument for wheel to not maintain compatibility with sdists and python setup.py install? Are there distributions out there relying on the bdist_wininst behavior? Carl signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 21, 2013 at 11:52 AM, Carl Meyer c...@oddbird.net wrote: On 08/21/2013 03:29 AM, Donald Stufft wrote: Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal). Since you asked, I'll mention the two that I've hit (though I think you're also aware of these already): 1) Wheel's conversion of - to _ in version strings embedded in filenames, which breaks with setuptools precedent; see https://github.com/pypa/pip/issues/1150 and https://bitbucket.org/dholth/wheel/issue/78/wheel-rewrites-versions-preventing No good solution to this one just yet. 2) Wheel's decision to follow distutils' documentation rather than distutils' behavior when it comes to the location for installing data_files with relative paths; see https://bitbucket.org/dholth/wheel/issue/80/wheel-does-not-install-data_files-in-site Django has fixed it by using package_data appropriately: https://code.djangoproject.com/ticket/19252 . The problem isn't unique to wheel, the same data_files mishap happens with bdist_wininst. Regardless, comment 5 is correct that we jump through way too many hoops in our setup.py in order to try to trick distutils into handling data_files as if they were package_data, and that is the root cause of this bug. Instead we should just use package_data and solve the problem properly. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
1) Wheel's conversion of - to _ in version strings embedded in filenames, which breaks with setuptools precedent; see https://github.com/pypa/pip/issues/1150 and https://bitbucket.org/dholth/wheel/issue/78/wheel-rewrites-versions-preventing No good solution to this one just yet. not a great solution, but wheel can just declare it doesn't support _ in versions, and not build wheels in that case. then installers know they can assume - was meant. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Paul Moore p.f.moore at gmail.com writes: That implies that any wheel reference implementation needs to expose APIs for reading and writing the metadata to/from the wheel. Not necessarily. For example, distlib's approach side-steps the need for such a write API: you tell Wheel.build which directories contain purelib, platlib, scripts, headers etc. and it puts all the stuff that it finds in those directories into appropriate locations in the wheel, then updates RECORD, WHEEL etc. This way, it doesn't need to worry about what custom files installers put in .dist-info, and reduces the coupling between the wheel code and users of it. There's no need for a special read API either, since wheels are just zip files and you can use the zipfile API to look at anything inside a wheel. As a convenience, distlib's Wheel instances have a metadata property, which returns as a dict the pydist.json from the wheel's .dist-info. Why make it more complicated than that? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Nick Coghlan ncoghlan at gmail.com writes: Right. I wasn't really aware Vinay was adding experimental ideas to distlib, I thought it was just the proven stable core from distutils2, plus support for the draft PEPs, with the experimental stuff entirely in distil rather than in distlib. I've been up front about what's in distlib all along - check the overview page in the distlib docs. Above all, I want the stuff I do to be *useful*, rather than tick boxes here and there. For example, there's no PEP covering distlib's functionality whereby dependency resolution happens *without* downloading/unpacking archives and running egg_info. We seem to take this sort of dependency resolution for granted in Linux distros, but for some reason Python packaging has to make do with a clunkier approach? Initially my work in this area was experimental, but it seems to work well (though some areas still need more work), and at some point I would expect to propose a PEP to cover it. However, lots of things are in flux and not in my direct control, so directing effort there now would likely involve rework when e.g. existing PEPs change, so it wouldn't be productive to do it yet. If Vinay wants to do experimental extensions, they either need to happen somewhere other than distlib, or else we need a new library which is just the candidate for standard library inclusion with *nothing* that hasn't been discussed and agreed to through the PEP process. Distlib aims to conform to the PEPs as they evolve, and anyone can raise an issue if they find non-conformance. In my view, one of the failures of distutils2 was that it did not include functionality that was actually useful to people and used by them, such as exports and package resources. I implemented exports in distlib before you added them to PEP 426, but that doesn't mean that I'm some kind of heretic for doing so. I've implemented package resources in distlib too, and there's no PEP yet covering it. You've specifically told me that you don't see distlib as a candidate for inclusion in Python in the 3.4 time frame, so what's the big hurry with getting the pitchforks out now? As I said, I *thought* distlib was that library, but it appears Vinay doesn't currently see it that way :( Nick, you haven't discussed this with me at all, and I don't see how you can come up with that interpretation from anything I've said in my posts here (or anywhere else). As I see it, you've explicitly told me that there's a very long time to go before distlib is even considered as a possible candidate for standardisation, and I can't see any valid reason why I can't keep adding useful features to it for now, because the experience gained from using them will inform any future PEPs relating to those features. When the time to consider standardisation is nearer, decisions can be made about what might need to come out and what can stay in, etc. and PEPs written to propose any things that aren't already covered. I've written distlib in a modular fashion and I don't expect such a process to be painful, and I would certainly expect to produce PEPs where needed. Right. distlib can be a candidate for stdlib inclusion, or it can be a vehicle for distributing experimental features not covered by any PEP, it can't be both at the same time. How is it that you're ready to call time on it now, already, even though you've told me standardisation is not something to be even considered until Python 3.5? Of the two options quoted above, you had already decided that it can't be the former in the short to medium term, and now you're saying you don't want it to be the latter either? I'm confused. I'd certainly like to keep making distlib and distil more useful. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 22 August 2013 08:12, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: I've been up front about what's in distlib all along - check the overview page in the distlib docs. Above all, I want the stuff I do to be *useful*, rather than tick boxes here and there. Right, you didn't do anything wrong, I just wasn't really paying attention because promoting distlib adoption has been firmly in the later bucket for me, after the setuptools rehabilitation, pip bootstrapping, etc. I can't fault you for not realising I believe something I had never really written down anywhere :) Nick Coghlan ncoghlan at gmail.com writes: Right. distlib can be a candidate for stdlib inclusion, or it can be a vehicle for distributing experimental features not covered by any PEP, it can't be both at the same time. How is it that you're ready to call time on it now, already, even though you've told me standardisation is not something to be even considered until Python 3.5? Of the two options quoted above, you had already decided that it can't be the former in the short to medium term, and now you're saying you don't want it to be the latter either? I'm confused. I'd certainly like to keep making distlib and distil more useful. I previously thought distlib was going to be the repository for the agreed, stable, this is going to happen stuff. It's OK that I was wrong - I think you're right that somewhere is needed as an experimental location to show some of the *possibilities* of the new metadata, and to seed ideas for making it into the eventual standard base that people can assume is readily available. What that means though, is we need *something else* that indicates the common core that people can assume will always be available. It's this common core which pip will need to factor out to remove their dependency on setuptools, rather than adopting distlib wholesale, experimental features and all. I'm actually OK with that, since it means we can aim for a self-contained pip that serves as the basis for the rest of the ecosystem, and is upgraded on the *pip* update cycle, rather than CPython's. I'd been wondering how we'd avoid the stale standard library support issue we ran into with distutils, and I think the pip bootstrapping proposals give us the answer: we don't make the core distribution infrastructure part of the standard library, we make it part of *pip*. One of the key concepts of the bootstrapping idea is that CPython maintenance releases will bundle newer versions of pip, and also that pip will be able to upgrade itself in place, so if people need newer distribution infrastructure upgrade pip is a much lower risk proposition than upgrade to a newer version of Python. Currently, pip doesn't expose a programmatic API. I suggested to Donald that it may make sense to start exposing one as piplib. The bootstrapping would then provide the pip CLI and the common utilities in piplib, and that would be the more conservative core, leaving distlib and distil free to push the boundaries with experimental features that go beyond what the agreed standards currently support. As ideas from distlib make their way into the relevant PEPs and hence into piplib, then they will become available by default, but in the meantime, people would be able to do a build dependency on distlib to get the experimental features. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Back from holidays, I read this very interesting discussion... Am 14.08.2013 16:33, schrieb Nick Coghlan: Aside from the lack of embedded C extension support (which could likely be fixed if zipimport was migrated to Python code for 3.5), ...but I don't understand what you mean by this. Can you please explain? Thanks, Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 20 Aug 2013 04:22, Thomas Heller thel...@ctypes.org wrote: Back from holidays, I read this very interesting discussion... Am 14.08.2013 16:33, schrieb Nick Coghlan: Aside from the lack of embedded C extension support (which could likely be fixed if zipimport was migrated to Python code for 3.5), ...but I don't understand what you mean by this. Can you please explain? Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. Cheers, Nick. Thanks, Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Am 20.08.2013 15:42, schrieb Nick Coghlan: Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. So what - zipimport is a builtin module (on Windows at least). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 20 Aug 2013 09:18, Thomas Heller thel...@ctypes.org wrote: Am 20.08.2013 15:42, schrieb Nick Coghlan: Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. So what - zipimport is a builtin module (on Windows at least). Huh? That's irrelevant to the fact that doing the tempdir creation, file extraction and subsequent import entirely in C code would be painfully tedious. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Am 20.08.2013 18:22, schrieb Nick Coghlan: On 20 Aug 2013 09:18, Thomas Heller thel...@ctypes.org mailto:thel...@ctypes.org wrote: Am 20.08.2013 15:42, schrieb Nick Coghlan: Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. So what - zipimport is a builtin module (on Windows at least). Huh? That's irrelevant to the fact that doing the tempdir creation, file extraction and subsequent import entirely in C code would be painfully tedious. Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): def __load(): import imp, os path = os.path.join(__loader__.archive, --EXTENSIONS--, 'bz2.pyd') data = __loader__.get_data(path) dstpath = os.path.join(TEMPDIR, 'bz2.pyd') with open(dstpath, wb) as dll: dll.write(data) imp.load_dynamic(__name__, dstpath) __load() del __load (py2exe for Python 3, which is work in progress, uses this approach) Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 20, 2013 at 12:39 PM, Thomas Heller thel...@ctypes.org wrote: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): ... (py2exe for Python 3, which is work in progress, uses this approach) Setuptools has also done this since the egg format was developed, but it has some well-known problems, which unfortunately your example has worse versions of. ;-) Setuptools takes the approach of keeping a per-user cache directory (so that cleanup isn't required, and so there are no security issues where somebody can replace a tempfile between you writing it and importing it), and it uses a unique subdirectory per egg so that different (say) bz2.pyd files can't conflict with each other. Even with these adjustments, Unix users frequently run into issues where the user a process is running as doesn't have access to a suitable cache directory, and so it's a common complaint about the use of zipped eggs. I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? (ISTR that there was some sort of licensing issue, too.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Am 20.08.2013 19:39, schrieb PJ Eby: On Tue, Aug 20, 2013 at 12:39 PM, Thomas Heller thel...@ctypes.org wrote: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): ... (py2exe for Python 3, which is work in progress, uses this approach) Setuptools has also done this since the egg format was developed, but it has some well-known problems, which unfortunately your example has worse versions of. ;-) The code I posted was some 'pseudocode' how to extract and import an extension from a zip-file without coding it in C ;-). For example, TEMPDIR is not the usual TEMP directory, instead py2exe will use a per-process temp directory and cleanup after process exit. So, at least some of the problems you list below should be solved or solvable. Setuptools takes the approach of keeping a per-user cache directory (so that cleanup isn't required, and so there are no security issues where somebody can replace a tempfile between you writing it and importing it), and it uses a unique subdirectory per egg so that different (say) bz2.pyd files can't conflict with each other. Even with these adjustments, Unix users frequently run into issues where the user a process is running as doesn't have access to a suitable cache directory, and so it's a common complaint about the use of zipped eggs. I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? (ISTR that there was some sort of licensing issue, too.) Yes, that was me. It worked out so-so, fine for extensions from wx-python and numpy, for example, not so good for others. Until recently it did only work for win32, not win64, but this will be fixed. And, of course, it is windows only! The code it is based on is MPL2.0 licensed: https://github.com/fancycode/MemoryModule Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Thomas Heller theller at ctypes.org writes: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): In distlib, I've built on top of the zipfile support to allow C extensions to be available. Ordinary zipfiles contain no metadata indicating the extensions available, but wheels built with distil do. The wheel handling code in distlib takes advantage of this. The Wheel.mount() API [1] takes care of this (adding the wheel to sys.path, extracting the extensions to a user-specific directory and using an import hook to call imp.load_dynamic when required, so that both Python modules and extensions are available for import). It seems to work, though when I introduced the Wheel.mount() API I was told that it was very dangerous, and the sky would fall, or something :-) Regards, Vinay Sajip [1] http://distlib.readthedocs.org/en/latest/tutorial.html#mounting-wheels ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 20, 2013, at 8:57 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Thomas Heller theller at ctypes.org writes: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): In distlib, I've built on top of the zipfile support to allow C extensions to be available. Ordinary zipfiles contain no metadata indicating the extensions available, but wheels built with distil do. The wheel handling code in distlib takes advantage of this. The Wheel.mount() API [1] takes care of this (adding the wheel to sys.path, extracting the extensions to a user-specific directory and using an import hook to call imp.load_dynamic when required, so that both Python modules and extensions are available for import). It seems to work, though when I introduced the Wheel.mount() API I was told that it was very dangerous, and the sky would fall, or something :-) Regards, Vinay Sajip [1] http://distlib.readthedocs.org/en/latest/tutorial.html#mounting-wheels ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Mounting Wheels seems like a bad idea, it was one of the things Daniel explicitly removed (since Wheels are basically cleaned up eggs). Adding it back in ex post facto seems like it's an idea that's going down the wrong track. If Wheels are to be importable it should be enshrined in the PEP not an adhoc feature of one possible implementation. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
I know I'm bike-shedding here, but my preference for extension would be '.pye' as it indicates something to execute, but without indicating exactly how (i.e. that it's via a separate launcher executable). .pyx that said, both .pye and .pyx are superior to .pyl (is that last character an ell, an eye or a one?) -matt ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
.pyx that said, both .pye and .pyx are superior to .pyl (is that last character an ell, an eye or a one?) I agree that .pyl is less readable/more confusable. We can't use .pyx, though, as that is already used in the Python ecosystem for Pyrex files (a forerunner of Cython). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 23:30, Chris Barker - NOAA Federal chris.bar...@noaa.govwrote: On Tue, Aug 13, 2013 at 2:27 PM, Paul Moore p.f.mo...@gmail.com wrote: 3) I'd rather not have to mess with PATHEXT, and I particularly don't want to have to tell my students to do it -- environment variables are a pain, and somehow PATHEXT has been fragile for me (and I don't use Cygwin) Nobody is suggesting that end users mess with PATHEXT. The proposal is that the Python installer does this ouch! I don't think I'd want PATHEXT set for *.py files -- I'd rather they get opened by an editor by default than run...or is point+click behavior different than command line -- shows you how well I know Windows. It's OK, PATHEXT has nothing to do with double clicks - that's the file associations (and run the script is the default action set by the installer, and has been for many, many versions). (indeed, that's been done for Python 3.4, I don't know if the installer for the standalone launcher has been updated in the same way yet). There are those of us still in the 2.7 world -- and I suspect for a good while. I understand this. It's the biggest issue here, that any changes have to not forget users of older Pythons. Personally, I tend to err towards a view that we improve things for current Pythons (3.3+) and make sure we don't ruin things for older versions, possibly by providing workarounds the user needs to apply. I view install the standalone launcher to be in this category of workaround. There are others who are more conservative, so don't worry your views are well represented! I'm suggesting that we collect specifics on any fragility (can you provide details of what has gone wrong for you?) well, for PATHEXT, the env variable has to be set right, and Windows kind of hides all that from you -- it's really a pain to edit them by hand, so if the installer doesn't do it, or someone re-builds their registry or profile, or what have you, then it'll break. Oh and the cygwin (and who know what other shell alternatives) issue. At least we can pretty much count on an exe running if the shell can find it.. Thanks. I'd class most of that as relatively non-specific or fixed. Users shouldn't have to set PATHEXT if the installer does it, and in any case I'd like to clearly understand what sort of users are writing command line Python scripts that they want to run transparently as if they were executables, who still wouldn't know how to set an environment variable (or be able to understand whatever documentation we produce on how to do it). The cygwin issue has been mentioned/addressed elsewhere, but thanks for that. It's certainly something that needs to be weighed in the balance. Fragiity for the exe approach -- all I know is that the setuptools binary was proken a while back -- but setuptools itself was in a bit of a void of not-quite-sure-if-its maintained, and not-sure-even-how-to-report-a-bug state. It seems that setuptools (or whatever this will be part of) has now been adopted by the core Python community, so we can expect good support -- let's hope so. Anyway, whether the exe approach was more or less fragile than anyting else, is sure was harder to debug/fix for anyone that doesn't know Windows and non-python development well. My personal experience with the exes: 1. If the shebang line in the script gets corrupted, the resulting error is misleading and extremely hard to debug. 2. The existence of the exe in an installer makes it platform-specific, which is a problem for otherwise portable scripts/packages. 3. To use the exes, you really need to create them via setuptools, so they are not practical for standalone scripts - and having 2 distinct ways of setting scripts up to be runnable is less than ideal. 4. If you are upgrading/reinstalling the script, you cannot do so without workarounds if the exe is currently running (mostly a nice case for pip upgrading itself). Also, not directly related to the exes, but definitely to setuptools wrappers (see point 3 above): 1. The wrappers need pkg_resources installed *at runtime* not just at build time. I cant help thinking a more elegant solution exists, but maybe not. Personally, I believe that executable .py scripts (or maybe a dedicated .pys/.pye/.pya/whatever extension) *is* a more elegant solution - but equally, I concede, maybe not... well, I guess that's the way Windows does things -- *nix has the executable bit, Windows uses extensions, so I suppose that is the way to get an executable script -- but it really SHOULDN'T be *.py! Agreed, we need two concepts Python module (.py) and Python script (the new extension). I've lost track of what needs to be tried -- can't we just associate an extension with py and give it a go? (got to get that Windos VM working) Yes, essentially. I can confidently report that the solution is flawless, and should be adopted at once :-) More seriously, we need
Re: [Distutils] How to handle launcher script importability?
Jason R. Coombs jaraco at jaraco.com writes: This means that instead of installing, for example: Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest Just to muddy the waters a little, I'd like to suggest an alternative approach which doesn't appear to have been tried: 1. The installer just installs foo.exe on Windows for script 'foo', where the foo.exe contains the actual script 'foo' as a resource. 2. The launcher, before looking for foo-script.py, examines its resources. If a script resource is found, it is extracted and written to foo-script.py in the same directory as foo.exe. If such a resource isn't found, it continues to the next step. 3. The launcher looks for 'foo-script.py' in its directory, invokes it and waits for it to complete. 4. If a 'foo-script.py' was written in step 2, it is deleted. The launcher comes with an embedded manifest, so no external manifest is needed. The insertion of a script as a resource is easy to do using ctypes: I've tested it using a suitably modified version of the current distlib launcher and it seems to work as expected. Advantages of this approach: 1. No additional file clutter: Just one installed file per script. 2. No need to worry about PATHEXT, or whether the PEP 397 launcher is installed. 3. Since the script is foo-script.py, no import clashes will occur. Disadvantages of this approach: 1. The scripts are hard to inspect because they're in the .exe. OTOH, (a) we don't really want people to mess with the scripts, and (b) they should be stock wrappers for other code which does the real work. (There are developer tools which allow inspection of resources, if really needed.) 2. Requires ctypes in the installer, so perhaps problematic for versions of Python 2.5. 3. Scripts take up more space than a .py-with-PEP 397-launcher solution, but that approach can still give rise to the import issue, as well as requiring PATHEXT setup. Thoughts? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 20:58, Paul Moore p.f.mo...@gmail.com wrote: On 13 August 2013 18:08, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 13 August 2013 17:33, Paul Moore p.f.mo...@gmail.com wrote: On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way. So say I have a ~/bin directory where I put my scripts that I want to be generally available. I install something with python setup.py install --install-scripts=~/bin so that the scripts/script-wrappers go in there because I want to be able to always access that program under that name. Don't be fooled by the unixy tilde: I'm running ordinary Windows Python in that command in git-bash, not Cygwin. Now if that folder is on PATH while I am in Cygwin I can run the program with the same name if an .exe wrapper was added. I can't run it with the same name if it's a .py/,bat file because Cygwin doesn't have the implicit strip-the-extension PATHEXT feature and can't run .bat files. Ah, OK, thanks for the clarification. In that case I can see why you'd prefer exe wrappers (or maybe cygwin bash shell wrappers, or shell aliases...). Maybe an option to still use exe wrappers is worth it - but honestly, I'd say that in that context you probably have enough expertise to understand the issue and make your own solution relatively easily. Yes, but I'd like it if pip install some_cmd would just work. What about having in your .bashrc: for prog in ls ~/bin/*.py; do alias $(basename $prog .py)=$prog done (Excuse me if I got the precise details wrong there). OK, you need to rerun .bashrc if you add new scripts. It's not perfect. But it's not a showstopper either. There are ways to make it work for every different environment where I would type the command. Really though it's a pain to have to set these things up everywhere. Also this still doesn't work with subprocess(..., shell=False). There are a huge range of programs that can invoke subprocesses of a given name and I want them all to work with commands that I install from pypi. There are good reasons to use shell=False: the subprocess documentation contains no less than 5 warning boxes about shell=True! This is not peculiar to Python's subprocess module: it is the underlying Windows API calls regardless of which language the parent process is implemented in. Here's a demo of what happens with Robert Kern's kernprof.py script that doesn't have an .exe wrapper (on my system; it's possible that I didn't install it with pip). $ python Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. import subprocess subprocess.call(['kernprof.py'], shell=True) # Uses file-association Usage: kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ... 2 import os os.environ['PATHEXT'] '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PY;.PYC;.PSC1;.RB;.RBW' subprocess.call(['kernprof'], shell=True) # Uses PATHEXT Usage: kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ... 2 subprocess.call(['kernprof'], shell=False) # Needs an .exe wrapper! Traceback (most recent call last): File stdin, line 1, in module File q:\tools\Python27\lib\subprocess.py, line 524, in call return Popen(*popenargs, **kwargs).wait() File q:\tools\Python27\lib\subprocess.py, line 711, in __init__ errread, errwrite) File q:\tools\Python27\lib\subprocess.py, line 948, in _execute_child startupinfo) WindowsError: [Error 2] The system cannot find the file specified subprocess.call(['kernprof.py'], shell=False) # Needs an .exe wrapper! Traceback (most recent call last): File stdin, line 1, in module File q:\tools\Python27\lib\subprocess.py, line 524, in call return Popen(*popenargs, **kwargs).wait() File q:\tools\Python27\lib\subprocess.py, line 711, in __init__ errread, errwrite) File q:\tools\Python27\lib\subprocess.py, line 948, in _execute_child startupinfo) WindowsError: [Error 193] %1 is not a valid Win32 application Here's what happens if I put kernprof.bat next to kernprof.py (the .bat file just @echos running kernprof): import subprocess subprocess.call(['kernprof'], shell=True) # PATHEXT in action running kernprof 0 subprocess.call(['kernprof'], shell=False) # No PATHEXT Traceback (most recent call last): File stdin, line 1, in module File q:\tools\Python27\lib\subprocess.py, line 524, in call return Popen(*popenargs, **kwargs).wait() File q:\tools\Python27\lib\subprocess.py, line 711, in __init__ errread, errwrite) File q:\tools\Python27\lib\subprocess.py, line 948, in _execute_child startupinfo) WindowsError: [Error 2] The system cannot find the file specified subprocess.call(['kernprof.bat'], shell=False) # Works but
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 7:34 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Jason R. Coombs jaraco at jaraco.com writes: This means that instead of installing, for example: Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest Just to muddy the waters a little, I'd like to suggest an alternative approach which doesn't appear to have been tried: 1. The installer just installs foo.exe on Windows for script 'foo', where the foo.exe contains the actual script 'foo' as a resource. 2. The launcher, before looking for foo-script.py, examines its resources. If a script resource is found, it is extracted and written to foo-script.py in the same directory as foo.exe. If such a resource isn't found, it continues to the next step. 3. The launcher looks for 'foo-script.py' in its directory, invokes it and waits for it to complete. 4. If a 'foo-script.py' was written in step 2, it is deleted. The launcher comes with an embedded manifest, so no external manifest is needed. Better suggestion: just append a PEP 441 .pyz to the .exe, and no extraction is necessary; the .exe just reads out the #! part. For Python 2.6 and up, the .exe can simply pass itself as argv[1] to the interpreter. (For older Pythons, a little -c and PYTHONPATH munging is required, but should be doable.) For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-) Thoughts? Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. In any case, it'd probably be a good idea to offer some command line tools for manipulating such .exes, to e.g. show/change what Python it's set to use, extract/dump/replace the zip, etc. (As for ctypes, if that's needed for this approach (which I somewhat doubt), there are official Windows binaries available for 2.3 and 2.4.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 12:42, Oscar Benjamin oscar.j.benja...@gmail.com wrote: I do think, as I said before, that this needs some sort of policy-type PEP on the standard approach for wrapping scripts, with all the pros and cons of the various approaches documented and reviewed. There have been so many emails on this list that I can't immediately find it but somewhere Steve Dower of Microsoft said something like: ''' Don't worry. Exe wrappers are *definitely* the best solution for this. ''' (quote is approximate and emphasis added by me). My experience has lead me to the same conclusion as Steve. It may be worth documenting the reasons why but make no mistake about it: .exe wrappers of some form are the way to go. I would like it documented, with all the reasons, so that we don't keep rehashing this whole thing over and over. Also, if we do recommend exes, I'd like to see something (in the stdlib ultimately) that makes it trivially easy to wrap an existing .py script (a standalone one, not one with an associated setup.py) into an exe. Having two ways to write a command using Python is icky. Also, I'd like a single-file solution. More times than I care to remember I have put a program on a USB key or in my dropbox/skydrive and forgotten the support stuff (whether that's required DLLs, resource files, or whatever - it's not a Python-specific problem). But I do see your point regarding things like subprocess. It's a shame, but anything other than exes do seem to be second class citizens on Windows. BTW, you mention bat files - it bugs me endlessly that bat files seem to have a more privileged status than other script formats whether that's .py or .ps1 or whatever. I've never managed to 100% convince myself that they are special in a way that you can't replicate with suitable settings (PATHEXT, etc, etc). I think it's that .bat is hard-coded in the OS search algorithm or something, though. The docs are not easy to locate on the various aspects of matter. (If bat files didn't have their horrible nesting and ctrl-C handling behaviours, they'd be a viable solution...) Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 13:57, PJ Eby p...@telecommunity.com wrote: Thoughts? Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? Good point. No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. Again, agreed - we have executable zipfiles for Python, and a combined exe/zipfile is a perfectly viable format (it's used by most self-extracting zip formats, as well as wininst formats). In any case, it'd probably be a good idea to offer some command line tools for manipulating such .exes, to e.g. show/change what Python it's set to use, extract/dump/replace the zip, etc. I'd say tools supporting the format are essential. exe/zip formats will never be as user friendly as a pure text file script - we need to make the extra effort as minimal as possible. In particular, see my other post - I don't want to have one format (exe) for installed commands packaged with setuptools, and a separate format for one-file scripts I write myself. Actually, this sounds like a very good solution. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Better suggestion: just append a PEP 441 .pyz to the .exe, and no IIUC PEP 441 is about tooling to create archives; don't we just need a Python-compatible .zip (i.e. with a __main__.py)? For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-) This is less helpful; one might have N scripts per project, no need to stick the whole project in with each one, or am I misunderstanding? Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good True. idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? That could probably be taken care of with a bit of footwork. No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. How would such an offset be used? Are you saying the -c scriptlet would use that offset to extract the script? Or do you mean something else? (As for ctypes, if that's needed for this approach (which I somewhat doubt), there are official Windows binaries available for 2.3 and 2.4.) It's needed only if you use the specific approach I used - which was to use the Windows UpdateResource API to embed the script as a pukka Windows resource. Of course, if you're just appending something to the .exe, you don't need ctypes. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 09:58, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Better suggestion: just append a PEP 441 .pyz to the .exe, and no IIUC PEP 441 is about tooling to create archives; don't we just need a Python-compatible .zip (i.e. with a __main__.py)? For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-) This is less helpful; one might have N scripts per project, no need to stick the whole project in with each one, or am I misunderstanding? I believe PJE's suggestion is to expand the scope of PEP 441 a bit to include the ability to generate a valid Windows executable with the archive appended, instead of just the *nix/Python launcher shebang line. If you prepend the shebang line, you get a .pya file (Note: I suggested to Daniel that the extension be changed to pya for Python application, but the PEP hasn't been updated yet), if you prepend the executable, you get an actual Windows executable. That would also let us avoid the need for a separate .pyaw extension - you would just prepend a Windows GUI executable to handle that case, with .pya only handling console applications. The script wrapper case would then just be a particular use of the PEP 441 executable generation features: prepend the console executable for console scripts and the GUI executable for GUI wrappers, with the wrapper itself being a __main__.py file in the attached archive. Given that Vinay already gave the Python launcher the ability to do this (when built with different options), this sounds quite feasible to me. Aside from the lack of embedded C extension support (which could likely be fixed if zipimport was migrated to Python code for 3.5), you'd have the essentials of py2exe right there in the standard library :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 14:48, Paul Moore p.f.mo...@gmail.com wrote: But I do see your point regarding things like subprocess. It's a shame, but anything other than exes do seem to be second class citizens on Windows. BTW, you mention bat files - it bugs me endlessly that bat files seem to have a more privileged status than other script formats whether that's .py or .ps1 or whatever. I've never managed to 100% convince myself that they are special in a way that you can't replicate with suitable settings (PATHEXT, etc, etc). I think it's that .bat is hard-coded in the OS search algorithm or something, though. I think it is hard-coded into CreateProcess (at least on some versions of Windows). It certainly isn't a documented feature, but as demonstrated in my previous post it does work on XP. The docs are not easy to locate on the various aspects of matter. I just tried to find documentation but all I found was this (with dead-links to MS): http://blog.kalmbachnet.de/?postid=34 (If bat files didn't have their horrible nesting and ctrl-C handling behaviours, they'd be a viable solution...) You were right to cry about these previously. To give an example of where these subprocess issues might matter. sphinx auto-generates Makefiles that call 'sphinx-build' with no extension. The sphinx-build command has a setuptools .exe wrapper so that it will be picked up. I wouldn't confidently assume that for all combinations of Windows version and 'make' implementation that 'make' would know how to find sphinx-build for anything other than an .exe. A quick experiment shows that my own make handles shebangs if present and then falls back to just calling CreateProcess which handles .exe files and (via the undocumented hack above) .bat files . It does not respect PATHEXT and the error when the extension is provided but no shebang is given clearly shows it using the same sys-call as used by Python's subprocess module: Q:\tmpshow main 'show' is not recognized as an internal or external command, operable program or batch file. Q:\tmptype Makefile all: mycmd.py Q:\tmptype mycmd.py print 'hello' Q:\tmpmake mycmd.py process_begin: CreateProcess(Q:\tmp\mycmd.py, mycmd.py, ...) failed. make (e=193): Error 193 make: *** [all] Error 193 Q:\tmpmycmd.py hello Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 9:58 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
IIUC PEP 441 is about tooling to create archives; don't we just need a
Python-compatible .zip (i.e. with a __main__.py)?
I meant that it has a #! line before the zip part, so that the
launcher knows what Python to invoke.
There are also some challenges for older Pythons to invoke __main__,
since the normal Python import machinery frowns on reloading __main__.
I expect the zip would need an extra __main.py stub to bootstrap the
loading of __main__, and then invoke python with something like '-c
__import__('sys').path[0:0]=['/path/to','path/to/exe''];
__import__('__main').go()'.
(It can't have the import run the app as a side effect, because
otherwise the import lock will be held, leading to Bad Things in
multi-threaded apps.)
This is less helpful; one might have N scripts per project, no need to stick
the whole project in with each one, or am I misunderstanding?
I just meant that for cases where there's only one script, or where
you are doing a custom-built application. This also becomes The One
Obvious Way to do py2exe-like things.
How would such an offset be used? Are you saying the -c scriptlet would use
that offset to extract the script? Or do you mean something else?
Extract the script by seeking to the offset and reading it. It's far
from ideal, though; the .zip is much better since everything back to
2.3 can support it in some fashion.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 16:49, Oscar Benjamin oscar.j.benja...@gmail.com wrote: To give an example of where these subprocess issues might matter. sphinx auto-generates Makefiles that call 'sphinx-build' with no extension. The sphinx-build command has a setuptools .exe wrapper so that it will be picked up. I wouldn't confidently assume that for all combinations of Windows version and 'make' implementation that 'make' would know how to find sphinx-build for anything other than an .exe. OK, that's a pretty solid use case, and pretty clearly demonstrates that there will be issues with anything other than an exe. So we come full circle again - I'm pretty sure the last time this came up a month or so ago, someone came up with a scenario that convinced me to give up on executable script files. I definitely will at some point write up *some* sort of document on best practices for wrapping Python code (scripts, apps, whatever) as OS commands, in a cross-platform way. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 18:46, PJ Eby p...@telecommunity.com wrote: Shouldn't naming the file .pyw already work today for that case? Certainly, the .pyw extension is already suitable for manually creating GUI scripts in a text editor. Unless there's something special about how the 'pythonw' executable processes the command line, it should work just as well for a zipped archive. .pyw files can be imported as modules, just like .py, so you hit the issue of scripts named the same as modules that they import. Naming a zipped archive .pyw is no better or worse than naming it .py - both work most of the time, but are disconcerting to users who think the filetype implies text and both have the problems associated with being both executable and importable. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 2:14 PM, Paul Moore p.f.mo...@gmail.com wrote: .pyw files can be imported as modules, just like .py, Darn. Okay, so another extension *is* needed if you want to be able to make non-console apps runnable-but-not-importable. IIUC it should by '.pywa' rather than '.pya', though, because of the issue with only the first three characters of an extension working in PowerShell, which means it would be executed by the wrong PyLauncher under some circumstances. (Honestly, though, I'm not sure whether anybody cares about PATH/PATHEXT in relation to GUI apps; ISTM they'd mostly be invoked by shortcuts, and there's also a registry mechanism that's supposed to be used for these things nowadays, rather than PATH... but I think it only works for .exe's, so there we go again back to the land of .exe's just plain Work Better On Windows.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 01:01, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Just a thought -- is there any need in this day and age for extensions to be limited to 3 characters? There's a bug affecting PowerShell, which Microsoft have pretty much confirmed that they won't fix, which means that longer extensions aren't handled properly when used in PATHEXT. Longer extensions *can* be used, but not for the purposes we want them :-( Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 20:55 +, Vinay Sajip wrote: Donald Stufft donald at stufft.io writes: Hopefully this all will solve this problem, as it is right now if you use setuptools entry points then Wheels erroneously pretend to be platform agnostic. That's not unreasonable, as long as they don't contain executables. With the current version of distil, built wheels don't contain executables for scripts. The executable launchers are determined / written at wheel installation time. While distlib currently uses bespoke launchers, I plan to update it before the next release to use the PEP 397 launcher compiled with SCRIPT_WRAPPER. FWIW, i think that's the way to go. I also understood from a discussion with Daniel Holth that this matches his plans. cheers, holger signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Vinay Sajip vinay_sajip at yahoo.co.uk writes: While distlib currently uses bespoke launchers, I plan to update it before the next release to use the PEP 397 launcher compiled with SCRIPT_WRAPPER. One more data point - the launcher currently used by distlib is found at [1]. Since it doesn't have all the bells and whistles that the PEP 397 launcher has (e.g. no searching for Pythons in the registry - the shebang is assumed to point at the Python to launch; no customised command handling; no configuration options; no support for passing parameters to Python itself), it is a lot simpler and has smaller executables as a result (around 64K, as opposed to 100-150K for the PEP 397 launcher). This might be an issue for some, when there would potentially be lots of copies of these executables. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/simple_launcher/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
-Original Message-
From: Steve Dower [mailto:steve.do...@microsoft.com]
Sent: Monday, 12 August, 2013 15:03
Jason R. Coombs wrote:
6. Two to three files to do the job of one. In fact, the job isn't
much more than to invoke code elsewhere, so it seems ugly to require
as many as three files to do the job. Then multiply that by the
Python-specific version and you have up to six files for a single script.
While I can understand this from the POV of the implementer/maintainer,
I've never heard a single Windows user mention it. And with an embedded
manifest, it's no more than one .py and one .exe-per-Python-version.
Thanks Steve (for this and other comments).
I do want to remind that silence is not consent. The multiple files per script
has always bugged me (as a user, not implementer), but up until now, I've
always considered it a necessary evil. And now that it's not necessary, it's
just evil. Over the past day, I've realized/recalled even more problems
stemming from side-by-side executables (some of which also apply to other
non-executable side-by-side solutions such as markers and manifests):
1. Renames, deletes, and other actions must be synchronized. There's an
implicit connection between the files, but it's implicit. And while it's
relatively easy to imagine how one can manage the synchronicity, in practice,
it's harder. For example, to delete a script, one has to be careful to delete
{script_name}-script.py and {script_name}.exe. If one wants to rename, two
renames have to occur. What would otherwise be a simple, intuitive operation
now has a semantic and technical burden.
2. Discoverability is diminished. Imagine for example that you want to delete
all scripts that reference a particular package (as one is wont to do when
that package is removed). If there's any side-by-side content, it's not
sufficient to grep the files and delete the matching files, but one must
instead write a routine or otherwise resolve the matches to their side-by-side
equivalents and perform the same operation on them.
3. A directory listing is distracting and unnecessarily twice as long, as
there's two files per script. This necessarily means more scrolling and actual
human time spent parsing and organizing that structure. And since there's not
exactly two files per script (only those with a launcher), the pairing isn't
consistent. With one file per script, the listing matches the essence of the
directory's content.
4. Updates to the launcher won't apply to existing scripts. If the launcher is
updated, the side-by-side versions will remain out-of-date until their scripts
are re-installed. If the launcher is associated with the file type, then the
state of the launcher can be managed independently of the scripts.
5. Files in use can't be replaced. Because a Windows executable that's in use
is not allowed to be overwritten, it's not possible to use a script to update
itself. For example, running 'easy_install -U setuptools' will result in an
error because the easy_install.exe is in use.
6. There are potentially privilege issues and security aspects to the separate
files that haven't yet been uncovered. On a multi-user system, there are
considerations about ownership and permissions on the files. When there are
multiple files per script, it's less obvious how permissions should be
assigned. It's not obvious to me this poses a significant issue, but it sure
seems easier to manage these issues on a single file rather than multiple.
All of these issues except for (6) have impacted my ability to work
effectively with the side-by-side launchers currently on the system. It's one
of the things I love about working on the Unix systems. Now that I feel like
we're so close to a similarly-elegant solution on Windows, I want to see it
employed (even if it's only opt-in, although I would prefer we work toward a
solution that ultimately defaults to a single file mode for scripts).
smime.p7s
Description: S/MIME cryptographic signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 13, 2013 at 8:54 AM, Jason R. Coombs jar...@jaraco.com wrote:
1. Renames, deletes, and other actions must be synchronized.
Why are you manually deleting or altering executables? Why are you
renaming them at all?
I've been using .exe wrappers since they were written, and have never
had a single one of the issues you mention, because I never do any of
the things you mention by hand. IMO that's what tools are for.
Doesn't pip uninstall scripts?
I may be slightly biased in my preference for .exe, because files with
other extensions don't work with Cygwin (which doesn't support
PATHEXT), but I work primarily with Windows Python rather than Cygwin
Python. So, if there *has* to be a single file, I would greatly
prefer an .exe with the script embedded, rather than a non-.exe file.
It's a bit less discoverable, but at least it'll discourage anybody
from editing the contents. (Because nobody should be editing
generated scripts anyway.)
(Also relevant: not every situation where wrapper scripts are used is
going to be one where a PyLauncher install is possible. For example,
portable deployment of an app to USB stick with a bundled Python can't
assume PATHEXT and a globally-installed PyLauncher.)
4. Updates to the launcher won't apply to existing scripts. If the launcher is
updated, the side-by-side versions will remain out-of-date until their scripts
are re-installed.
This is kind of a bogus point; *any* update to how scripts are
generated isn't automatically applied to existing scripts; the format
in which they're written is of no relevance.
5. Files in use can't be replaced. Because a Windows executable that's in use
is not allowed to be overwritten,
But they can be renamed, and deleted afterwards. For example, when
updating, you can do the simple dance of:
1. delete scriptname.exe.deleteme if it exists
2. rename scriptname.exe to scriptname.exe.deleteme
3. replace scriptname.exe
4. try to delete the .deleteme file created in step 2, ignoring errors.
And since this only needs to be done for the wrappers on installation
tools themselves (pip, easy_install, etc.), it's not like a lot of
people are going to have to write this code.
It can also be further enhanced, by having the .exe wrapper check (as
it exits) whether it was renamed, and if so, spin off a 'python -c
import os, time; time.sleep(0.1); os.unlink('path to .deleteme')'
and immediately exit. (Or use one of the other tricks from
http://www.catch22.net/tuts/self-deleting-executables -- but I think
this one is the simplest and best for our purposes, since the wrapper
already knows at this point it can invoke Python using the path it
previously found, and it's not doing anything questionable with
process invocations that might raise red flags with security tools.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 16:58, PJ Eby p...@telecommunity.com wrote: 5. Files in use can't be replaced. Because a Windows executable that's in use is not allowed to be overwritten, But they can be renamed, and deleted afterwards. For example, when updating, you can do the simple dance of: 1. delete scriptname.exe.deleteme if it exists 2. rename scriptname.exe to scriptname.exe.deleteme 3. replace scriptname.exe 4. try to delete the .deleteme file created in step 2, ignoring errors. And since this only needs to be done for the wrappers on installation tools themselves (pip, easy_install, etc.), it's not like a lot of people are going to have to write this code. This works, but is an ugly, fragile workaround. It's *not* a huge problem, it's just how executables work on Windows, and all installers have to deal with this dance (it's why a lot of things need a reboot to complete installation - the delete on next reboot API). But it's not *nice*. I would never use exe wrappers for my own personal scripts - I *always* write them as .py files and rely on PATHEXT. I only use exe wrappers for commands installed as part of a Python package (pip.exe, nosetests.exe, etc). That says something about how friendly they are as a general tool. On the other hand, it also acts as a reminder that when used in a suitably managed situation (stuff installed by pip/easy_install) the ugliness of exe wrappers is hidden well enough to be a non-issue. So while Jason may be persuaded to retain exe wrappers for setuptools, I doubt he'd want to use them for his personal scripts (if I read his posts correctly). I know I won't. On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 17:33, Paul Moore p.f.mo...@gmail.com wrote: On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way. So say I have a ~/bin directory where I put my scripts that I want to be generally available. I install something with python setup.py install --install-scripts=~/bin so that the scripts/script-wrappers go in there because I want to be able to always access that program under that name. Don't be fooled by the unixy tilde: I'm running ordinary Windows Python in that command in git-bash, not Cygwin. Now if that folder is on PATH while I am in Cygwin I can run the program with the same name if an .exe wrapper was added. I can't run it with the same name if it's a .py/,bat file because Cygwin doesn't have the implicit strip-the-extension PATHEXT feature and can't run .bat files. Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 13, 2013 at 12:33 PM, Paul Moore p.f.mo...@gmail.com wrote: This works, but is an ugly, fragile workaround. It's *not* a huge problem, it's just how executables work on Windows, and all installers have to deal with this dance (it's why a lot of things need a reboot to complete installation - the delete on next reboot API). But it's not *nice*. I would never use exe wrappers for my own personal scripts - I *always* write them as .py files and rely on PATHEXT. I only use exe wrappers for commands installed as part of a Python package (pip.exe, nosetests.exe, etc). That says something about how friendly they are as a general tool. In an ironic reversal, I use them for any command I plan to use frequently. In other words, if I use it often enough to care about how easy it is to use, I take the trouble to wrap it in a project and then use setup.py develop to create the script wrappers. From then on, I can edit the *source* scripts, and the wrappers run the right thing. (I don't edit the -script.py's directly, since they're not where the real code is.) On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way. I said I'm using *Windows* Python from the Cygwin shell. I often test my projects with Cygwin Python, to ensure coverage of Unixisms, but I only write dedicated Cygwin Python scripts if I need to use Cygwin paths or APIs, which is relatively infrequent. In any case, the use of .exe means that my invocation patterns are unchanged between commands I've implemented in Cygwin Python vs. Windows Python. If the Windows Python versions used a different extension, then I'd have to remember whether which language a specific command was written in in order to invoke it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 18:08, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 13 August 2013 17:33, Paul Moore p.f.mo...@gmail.com wrote: On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way. So say I have a ~/bin directory where I put my scripts that I want to be generally available. I install something with python setup.py install --install-scripts=~/bin so that the scripts/script-wrappers go in there because I want to be able to always access that program under that name. Don't be fooled by the unixy tilde: I'm running ordinary Windows Python in that command in git-bash, not Cygwin. Now if that folder is on PATH while I am in Cygwin I can run the program with the same name if an .exe wrapper was added. I can't run it with the same name if it's a .py/,bat file because Cygwin doesn't have the implicit strip-the-extension PATHEXT feature and can't run .bat files. Ah, OK, thanks for the clarification. In that case I can see why you'd prefer exe wrappers (or maybe cygwin bash shell wrappers, or shell aliases...). Maybe an option to still use exe wrappers is worth it - but honestly, I'd say that in that context you probably have enough expertise to understand the issue and make your own solution relatively easily. What about having in your .bashrc: for prog in ls ~/bin/*.py; do alias $(basename $prog .py)=$prog done (Excuse me if I got the precise details wrong there). OK, you need to rerun .bashrc if you add new scripts. It's not perfect. But it's not a showstopper either. I do think, as I said before, that this needs some sort of policy-type PEP on the standard approach for wrapping scripts, with all the pros and cons of the various approaches documented and reviewed. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Just $0.02 from a user... I'm primarily an OS-X user these days, but have to do Windows once in a while, and help others do Windows (including as an intro to Python instructor) Once I discovered setuptools develop mode, I never looked bak -- it is simpl;y THE way to develop code, particularly if you are working on a lib you want to use from other projects and/or actively develping utility scripts. I really like that I can add some scripts to setup.py, and use either develop mode or regular old install and get nice commands, and it works the same way on Windows and *nix (with the hood closed0 I did all this for a good while before I even noticed that exe launchers -- it just worked. In fact, the only time I noticed the launchers was a couple years ago when a beta version of setuptools released a broken version -- very frustration -- it would fire up another command window that would then close when the script was done -- not very helpful for nosetests and the like... When I did discover how it all worked, I did think it was a little weird, but Windows simply hasn't been built for command line stuff, so you do what you have to do. Conclusions: 1) an extra bunch of files is a on-issue for most users -- we just need something that works. 2) the exe launcher is a bit fragile and hard to maintain (and even harder to debug) -- but there are smart people working on this. 3) I'd rather not have to mess with PATHEXT, and I particularly don't want to have to tell my students to do it -- environment variables are a pain, and somehow PATHEXT has been fragile for me (and I don't use Cygwin) I cant help thinking a more elegant solution exists, but maybe not. Thanks to everyone hashing this out! -Chris On Tue, Aug 13, 2013 at 12:58 PM, Paul Moore p.f.mo...@gmail.com wrote: On 13 August 2013 18:08, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 13 August 2013 17:33, Paul Moore p.f.mo...@gmail.com wrote: On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way. So say I have a ~/bin directory where I put my scripts that I want to be generally available. I install something with python setup.py install --install-scripts=~/bin so that the scripts/script-wrappers go in there because I want to be able to always access that program under that name. Don't be fooled by the unixy tilde: I'm running ordinary Windows Python in that command in git-bash, not Cygwin. Now if that folder is on PATH while I am in Cygwin I can run the program with the same name if an .exe wrapper was added. I can't run it with the same name if it's a .py/,bat file because Cygwin doesn't have the implicit strip-the-extension PATHEXT feature and can't run .bat files. Ah, OK, thanks for the clarification. In that case I can see why you'd prefer exe wrappers (or maybe cygwin bash shell wrappers, or shell aliases...). Maybe an option to still use exe wrappers is worth it - but honestly, I'd say that in that context you probably have enough expertise to understand the issue and make your own solution relatively easily. What about having in your .bashrc: for prog in ls ~/bin/*.py; do alias $(basename $prog .py)=$prog done (Excuse me if I got the precise details wrong there). OK, you need to rerun .bashrc if you add new scripts. It's not perfect. But it's not a showstopper either. I do think, as I said before, that this needs some sort of policy-type PEP on the standard approach for wrapping scripts, with all the pros and cons of the various approaches documented and reviewed. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 21:20, Chris Barker - NOAA Federal chris.bar...@noaa.govwrote: Conclusions: 1) an extra bunch of files is a on-issue for most users -- we just need something that works. Agreed - the extra files clutter is a relatively small issue. 2) the exe launcher is a bit fragile and hard to maintain (and even harder to debug) -- but there are smart people working on this. Nobody is really working on the launcher itself AIUI. The code is pretty much static, except when it breaks (for example, the whole UAC/manifest issue). 3) I'd rather not have to mess with PATHEXT, and I particularly don't want to have to tell my students to do it -- environment variables are a pain, and somehow PATHEXT has been fragile for me (and I don't use Cygwin) Nobody is suggesting that end users mess with PATHEXT. The proposal is that the Python installer does this (indeed, that's been done for Python 3.4, I don't know if the installer for the standalone launcher has been updated in the same way yet). I'm suggesting that we collect specifics on any fragility (can you provide details of what has gone wrong for you?) so that we can document and address any genuine issues. But without specifics, we're currently faced with nothing more than a two-pronged I think it might not work/why change it if it works at the moment argument that has nothing we can actually address... (Not criticising anyone here, it is often hard to be specific). I cant help thinking a more elegant solution exists, but maybe not. Personally, I believe that executable .py scripts (or maybe a dedicated .pys/.pye/.pya/whatever extension) *is* a more elegant solution - but equally, I concede, maybe not... I think it's worth trying, though. Also, you mention develop mode. I don't use develop mode, most of my standalone scripts are single-file scripts, not anything that I'd bother packaging up with a setup.py, etc. Or I run them via python -m, or I install a supporting package and have a (standalone, as previously) driver script. And many of the issues I've had with the exe wrappers are particular to built installers (wheels, wininsts) and *not* develop mode. Those also need to be classified precisely and reviewed - I don't claim they are any more important than the issues you mention with pure scripts - but I'd prefer that we understand the trade-offs and make an informed decision. It may even be that different solutions are better depending on how you work (develop mode vs installing). In the interests of getting more concrete data, can I suggest that setuptools add an off-by-default option, which can be set globally using a config file or environment variable or something (so that it can be used transparently by tools like pip) to use scripts rather than exe wrappers? People can then try it and see whether they hit issues with it. Otherwise, I think we're going to remain forever stuck with theorising and guesswork. Paul. PS I still think that long-term a policy PEP on the recommended way of making executable scripts is worthwhile. As I've said, if no-one else wants to pick it up, ask me again in October or so... ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco@python.org] On Behalf Of Paul Moore Sent: Tuesday, 13 August, 2013 17:28 In the interests of getting more concrete data, can I suggest that setuptools add an off-by-default option, which can be set globally using a config file or environment variable or something (so that it can be used transparently by tools like pip) to use scripts rather than exe wrappers? People can then try it and see whether they hit issues with it. Otherwise, I think we're going to remain forever stuck with theorising and guesswork. That's exactly what setuptools 1.0 does, but I want to get the first draft right, so I'm seeking comment on the technique (using env var to enable), the proper extension to use (.py has problems), and any other suggestions. I'm glad you mentioned the issues on powershell with 4 characters. I'm going to explore that issue to confirm that something like .pygs isn't viable (I use powershell and PATHEXT quite a bit). smime.p7s Description: S/MIME cryptographic signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 13, 2013 at 2:27 PM, Paul Moore p.f.mo...@gmail.com wrote: 3) I'd rather not have to mess with PATHEXT, and I particularly don't want to have to tell my students to do it -- environment variables are a pain, and somehow PATHEXT has been fragile for me (and I don't use Cygwin) Nobody is suggesting that end users mess with PATHEXT. The proposal is that the Python installer does this ouch! I don't think I'd want PATHEXT set for *.py files -- I'd rather they get opened by an editor by default than run...or is point+click behavior different than command line -- shows you how well I know Windows. (indeed, that's been done for Python 3.4, I don't know if the installer for the standalone launcher has been updated in the same way yet). There are those of us still in the 2.7 world -- and I suspect for a good while. I'm suggesting that we collect specifics on any fragility (can you provide details of what has gone wrong for you?) well, for PATHEXT, the env variable has to be set right, and Windows kind of hides all that from you -- it's really a pain to edit them by hand, so if the installer doesn't do it, or someone re-builds their registry or profile, or what have you, then it'll break. Oh and the cygwin (and who know what other shell alternatives) issue. At least we can pretty much count on an exe running if the shell can find it... Fragiity for the exe approach -- all I know is that the setuptools binary was proken a while back -- but setuptools itself was in a bit of a void of not-quite-sure-if-its maintained, and not-sure-even-how-to-report-a-bug state. It seems that setuptools (or whatever this will be part of) has now been adopted by the core Python community, so we can expect good support -- let's hope so. Anyway, whether the exe approach was more or less fragile than anyting else, is sure was harder to debug/fix for anyone that doesn't know Windows and non-python development well. I cant help thinking a more elegant solution exists, but maybe not. Personally, I believe that executable .py scripts (or maybe a dedicated .pys/.pye/.pya/whatever extension) *is* a more elegant solution - but equally, I concede, maybe not... well, I guess that's the way Windows does things -- *nix has the executable bit, Windows uses extensions, so I suppose that is the way to get an executable script -- but it really SHOULDN'T be *.py! I think it's worth trying, though. agreed. I've lost track of what needs to be tried -- can't we just associate an extension with py and give it a go? (got to get that Windos VM working) Also, you mention develop mode. I don't use develop mode, most of my standalone scripts are single-file scripts, not anything that I'd bother packaging up with a setup.py, etc. Good point -- it seem most of my scripts are part of a larger package, either a set of scripts, or a couple scripts that all rely on a particular package, so the whole setup.py thing works well. but for a single stand-alone script, the PATHEXT and py launcher approach seems really natural. And many of the issues I've had with the exe wrappers are particular to built installers (wheels, wininsts) and *not* develop mode. in theory, wheels and develop mode should do the same thing -- not sure about wininsts -- previously, the launcher thing was setuptools, not plain distutils, not sure how that was handled. but it would be best if they all did it the same way. PS I still think that long-term a policy PEP on the recommended way of making executable scripts is worthwhile. Probably, yes: There should be one-- and preferably only one --obvious way to do it. I'm leaning toward the PATHEXT approach -- it seems more si milar to the *nix way, and perhaps easeir for lay folks to debug and fix. But I'm saying tha tthe exe method worked fine for many of us, too. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 11 Aug 2013 21:37, PJ Eby p...@telecommunity.com wrote: On Sun, Aug 11, 2013 at 7:31 PM, Jason R. Coombs jar...@jaraco.com wrote: -Original Message- From: Nick Coghlan [mailto:ncogh...@gmail.com] Sent: Sunday, 11 August, 2013 17:14 We actually have a proposal on import-sig to allow module specific import path manipulation (including the ability to say don't import this module from this directory, even though it looks like it is here). I'd favour that mechanism over a new not importable file extension. I don't believe this mechanism would suffice. My previous example was over-simplified to the general problem, which is that any script could potentially be imported as a module of the same name. So if I were to launch easy_install.py, it would set sys.path[0] to Scripts\ and if it were then to import cython (which it does), it would import Scripts/cython.py as cython, unless there were some way to globally declare all installed scripts somewhere so they're excluded from import. Indeed. It really *does* need to be a don't import this extension, though it doesn't much matter what that extension is. Except on Windows, of course, where it has to be something associated with Python that also still works as a console app and is listed in PATHEXT. (As you surmised earlier, my choice of '-script.py' was indeed chosen to prevent accidental importing, as the '-' ensures it's not a valid module name.) Having an empty cython.ref file (extension TBC) would tell Python to skip that directory for import cython regardless of the presence of something that otherwise would be considered. The actual downside is that won't land until 3.4 at the earliest, while PyLauncher can associate additional extensions with itself (and modify PATHEXT) earlier than that. Having pys and pyz for executable, but not importable (source and zip archive forms) could be quite clean. In effect, the pys extension would bring windows to parity with *nix, where no extension at all has traditionally served the purpose of making it impossible to import a script. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 7:33 AM, Nick Coghlan ncogh...@gmail.com wrote: Having pys and pyz for executable, but not importable (source and zip archive forms) could be quite clean. In effect, the pys extension would bring windows to parity with *nix, where no extension at all has traditionally served the purpose of making it impossible to import a script. Oh, that reminds me: IIUC, it's not necessary to *actually* zip a .pyz. Remember, Python doesn't use the extension to determine the contents, it sniffs for a zip trailer. Likewise, there was IIRC no plan for pylauncher to inspect zip contents -- it just reads the #! line and runs python on the file. This means that you can actually write source as a .pyz or .pwz file on Windows, and it would Just Work -- *without any sys.path modification*. For *all versions of Python*, provided PyLauncher is installed. (And setuptools could check the environment and registry for that, if you opt into the non-.exe scripts, and error out if you don't have them set up correctly.) IOW, implementing PEP 441 in PyLauncher gives us the executable, not importable format for free. (Granted, I can see reasons for not wanting to use the same extension for source and zipped versions, mostly in the area of tools other than pylauncher, but if you do have different extensions then there have to be *four*, due to console vs. windowed and source vs. zipped.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 12 August 2013 14:01, PJ Eby p...@telecommunity.com wrote: This means that you can actually write source as a .pyz or .pwz file on Windows, and it would Just Work -- *without any sys.path modification*. Conversely, you can right now rename a zipped file as xxx.py and it will be run happily as a Python standalone zip file. It confuses the heck out of things like text editors and zip file managers, of course :-) We currently have 2 extensions on Windows - .py and .pyw. These get a default run action of the launcher (py.exe and pyw.exe versions respectively) and both are treated as marking Python modules (I don't know why anyone would create a Python module with a .pyw extension - sounds like an unnecessary attempt to apply consistency to me). Having two more that had the launcher as default action but did *not* mark Python modules might make sense. Having further extensions is fundamentally a documentation-only exercise - they will not be treated any differently by any code shipped by Python. The problem is that documentation (and user expectation) is important - nobody expects a .py file to be a binary zip file (and they'd get a shock if they opened it in a text editor). On the other hand, grabbing a huge host of file extensions (.py, .pyw, .pyo, .pyc, .pyz, .pwz, .pys, .pws) is not very friendly, as well as adding the burden of clearly documenting what all these various extensions *mean*. My view would be: 1. We can't touch .py/.pyw behaviour for backward compatibility reasons 2. A new suffix that is associated with the launcher but which does *not* mark importable modules would be good. (Call it .pys for now). 3. I'd rather not see further extensions added, six is plenty. As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content. Apologies if I've missed an obvious flaw - I'm way behind on sleep at the moment... Paul. PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws extension, if people prefer to reduce the number of extensions claimed still further. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore p.f.mo...@gmail.com wrote: On 12 August 2013 14:01, PJ Eby p...@telecommunity.com wrote: As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content. Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue. I think the issue of too many extensions vs. source/binary confusion is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP. If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, hey, run this as a console app via PyLauncher, not that it's a script (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile. PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws extension, if people prefer to reduce the number of extensions claimed still further. But those would only be available for future Python versions. A file extension would solve the problem upon installing PyLauncher and PATHEXT, at least for those OSes and shells that recognize PATHEXT. Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it. (The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 12 August 2013 16:35, Nick Coghlan ncogh...@gmail.com wrote: Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it. (The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.) This sounds like a plausible approach, especially if we add the bootstrapping being considered for 3.4+ to PyLauncher for earlier versions. (Donald has a draft PEP for that, he's just making a few tweaks before publishing it for broader comment) Do you want the time machine keys back? :-) http://bugs.python.org/issue18491 Committed by Vinay in http://hg.python.org/cpython/rev/4123e002a1af The wrapper source can be built that way if SCRIPT_WRAPPER is defined, but the build infrastructure does not currently define that. See the patch and issue log for details. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 12 August 2013 11:21, PJ Eby p...@telecommunity.com wrote: On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore p.f.mo...@gmail.com wrote: On 12 August 2013 14:01, PJ Eby p...@telecommunity.com wrote: As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content. Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue. I think the issue of too many extensions vs. source/binary confusion is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP. If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, hey, run this as a console app via PyLauncher, not that it's a script (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile. I like this idea. PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws extension, if people prefer to reduce the number of extensions claimed still further. But those would only be available for future Python versions. A file extension would solve the problem upon installing PyLauncher and PATHEXT, at least for those OSes and shells that recognize PATHEXT. Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it. (The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.) This sounds like a plausible approach, especially if we add the bootstrapping being considered for 3.4+ to PyLauncher for earlier versions. (Donald has a draft PEP for that, he's just making a few tweaks before publishing it for broader comment) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
-Original Message- From: Distutils-SIG [mailto:distutils-sig- bounces+jaraco=jaraco@python.org] On Behalf Of PJ Eby Sent: Monday, 12 August, 2013 11:22 On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore p.f.mo...@gmail.com wrote: On 12 August 2013 14:01, PJ Eby p...@telecommunity.com wrote: As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content. Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue. My preference is to reject the idea of the side-by-side executable launcher. There are several downsides that I'm trying to avoid by moving away from the executable: 1. Disparity with Unix. Better parity means cleaner code, easier documentation, and less confusion moving from platform to platform. 2. Executables that look like installers. If a launcher executable is used and Windows detects that it looks like an installer and it's a 32-bit executable and it doesn't have a manifest to disable the functionality, Windows will execute the file in a separate UAC context (often in a separate Window). 3. Additional files are needed. In particular, due to (2), a manifest must be provided for 32-bit executables. 4. Word size accounting. It's not clear to me what word size is needed. 32-bit may be sufficient, though 64-bit seem to have some advantages: a manifest is not needed, and it can match the word size of the installed Python executable (for consistency). Setuptools currently provides both (and installs the one that matches the Python executable). 5. Platform support. We're fortunate that Windows is one of the most stable binary platforms out there. Nevertheless, Setuptools recently got support for AMD binaries in the launcher. By relying on an external launcher, the launcher becomes responsible for platform support. 6. Two to three files to do the job of one. In fact, the job isn't much more than to invoke code elsewhere, so it seems ugly to require as many as three files to do the job. Then multiply that by the Python-specific version and you have up to six files for a single script. 7. Obfuscation of purpose. A single script pretty directly communicates its purpose. When there are multiple files, it's not obvious why they exist or what their purpose is. Indeed, I went years without realizing we had an open issue in Distribute due to a missing manifest (which was fixed in Setuptools), all because I used the 64-bit executable. While it may take some time for the community to learn what a '.pyl' is, it's easily documented and simple to grasp, unlike the subtle and sometimes implicit nuances (and fragility) of a side-by-side executable. 8. Unwanted content. Some Unix users have complained about finding Windows executables in their Linux packages, so now Setuptools has special handling to omit the launchers when installed on Unix systems. This is far from beautiful. I think the issue of too many extensions vs. source/binary confusion is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP. If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, hey, run this as a console app via PyLauncher, not that it's a script (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile. If it's '.py*', I don't see why it's not reasonable to allow omission of the shebang, and assume the default python. After encountering and now understanding the subtle import semantics, I'm hoping that this new extension can also be used in my personal 'scripts' collection to serve the same purpose it does for setuptools console entry points. I guess one could require #!/usr/bin/python in each, but that seems superfluous on Windows. I don't feel at all strongly on this point. PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws
Re: [Distutils] How to handle launcher script importability?
I know I'm bike-shedding here, but my preference for extension would be '.pye' as it indicates something to execute, but without indicating exactly how (i.e. that it's via a separate launcher executable). +1 (I spent the whole time reading this thread thinking I'd prefer pye, or maybe pyx (pee-why-ex-ecutable)) I don't think we're voting on it yet, but also +1 for putting .PYE in PATHEXT instead of .PY. As I've mentioned before, I'll be upset if one day typing pip install ... opens my editor instead of running pip... And any reason we need a separate extension for pyw.exe? Can't that be specified in the shebang? Cheers, Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Jason R. Coombs wrote: My preference is to reject the idea of the side-by-side executable launcher. There are several downsides that I'm trying to avoid by moving away from the executable: [SNIP] 2. Executables that look like installers. If a launcher executable is used and Windows detects that it looks like an installer and it's a 32-bit executable and it doesn't have a manifest to disable the functionality, Windows will execute the file in a separate UAC context (often in a separate Window). The problematic part of detection is the filename. The Installer Detection section of http://msdn.microsoft.com/en-us/library/bb530410.aspx has more details, but isn't 100% precise. I'm sure there's a precise description somewhere, but I don't know where it is. 3. Additional files are needed. In particular, due to (2), a manifest must be provided for 32-bit executables. 4. Word size accounting. It's not clear to me what word size is needed. 32-bit may be sufficient, though 64-bit seem to have some advantages: a manifest is not needed, and it can match the word size of the installed Python executable (for consistency). Setuptools currently provides both (and installs the one that matches the Python executable). 32-bit is sufficient, and the manifest can be embedded. Since all the executable is doing is looking for a matching/similarly-named file in its directory and launching it, there's no need for a 64-bit version, or for any differences to exist between the executables. They could all be copied from the same source whenever one is needed. (A 64-bit binary is only required when loading 64-bit DLLs. It is not required to launch a 64-bit process.) [SNIP] 6. Two to three files to do the job of one. In fact, the job isn't much more than to invoke code elsewhere, so it seems ugly to require as many as three files to do the job. Then multiply that by the Python-specific version and you have up to six files for a single script. While I can understand this from the POV of the implementer/maintainer, I've never heard a single Windows user mention it. And with an embedded manifest, it's no more than one .py and one .exe-per-Python-version. [SNIP] 8. Unwanted content. Some Unix users have complained about finding Windows executables in their Linux packages, so now Setuptools has special handling to omit the launchers when installed on Unix systems. This is far from beautiful. Do us Windows users get .sh and .DSStore files filtered out too? :) (More seriously, why isn't the onus on package developers to specify platform-specific files?) Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it. (The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.) I'm warming up to this idea a bit, especially how it supports the most elegant approach but degrades gracefully. py.exe is a great file to use as a launcher, but it would also be easy to make a more specific one. Some questions that arise: Where would Setuptools expect to find these launchers? Would it expect them to be present on the system? I don't think that's unreasonable. Perhaps they should always install into Python's path (C:\Python##\py.exe) so they are discoverable? Not everybody can install into C:\Windows\System32 (or SysWOW64) Would it symlink or hardlink them or simply copy? Copying is most reliable and the only way to handle installation onto different drive partitions. Some people will worry about the size, but I don't know that much can be done about that. A smaller executable could be made with no icons and relying on the OS to find py.exe wherever it's been installed. You can hardlink the version-specific executables safely. Is py.exe subject to the 'looks like installer' behavior, such that it would need a manifest? The one that's in Python 3.4 Alpha already has an embedded manifest, but I don't believe it would trigger the installer heuristics anyway. I still feel like this approach would require substantial special-casing, but since it provides a transition to the simple, elegant approach, I'm not opposed. Shouldn't require that much special casing, except that you probably wouldn't be able to use it with an adjacent py.py
Re: [Distutils] How to handle launcher script importability?
On 12 August 2013 19:35, Steve Dower steve.do...@microsoft.com wrote: And any reason we need a separate extension for pyw.exe? Can't that be specified in the shebang? The association specifies the exe to run, that exe then relaunches whatever the shebang specifies. But it's the *initial* executable's console/GUI flag that specifies the behaviour, so we need two launchers and hence two extensions. You could put pythonw in the shebang of a .py file, but that would open a console window when double clicked. Similarly, putting python in the shebang of a .pyw file will detach the script from the current console when run from the command line. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 12 August 2013 19:14, Jason R. Coombs jar...@jaraco.com wrote:
My preference is to reject the idea of the side-by-side executable
launcher.
There are several downsides that I'm trying to avoid by moving away from
the
executable:
Using a dedicated filetype and associated systemwide launcher is the ideal
solution in many ways, but the behaviour is subtle, shell-dependent, and
under-documented. I have been adding .PY to PATHEXT for ages now, and
running .py scripts as commands with no issues - but people have raised
objections to the proposal that we do this for exe wrappers, and I don't
have convincing answers in all cases (mostly because I've never seen that
case is often my response...). I can't recall details, but look back over
recent threads on distutils-sig and python-dev that I've been involved in,
for details.
Also, a dedicated filetype won't be registered for users of older Pythons -
unless they install the latest launcher. Whether that is an issue to be
concerned about needs to be agreed. (Backward compatibility vs making
progress, I guess...)
If it's '.py*', I don't see why it's not reasonable to allow omission of
the
shebang, and assume the default python. After encountering and now
understanding the subtle import semantics, I'm hoping that this new
extension
can also be used in my personal 'scripts' collection to serve the same
purpose
it does for setuptools console entry points. I guess one could require
#!/usr/bin/python in each, but that seems superfluous on Windows. I don't
feel
at all strongly on this point.
IIRC, the shebang can be omitted, and in that case you get the launcher's
configured default Python (from either py.ini, or built in). I tend to add
/usr/bin/python{2,3} (or /usr/bin/env python if I want to respect PATH) for
Unix compatibility even though I never expect the scripts to be used on
Unix.
Also, in my mind, this approach is most directly addressing the fundamental
challenge (distinguishing a (executable) script from a module) in much the
way
Unix has previously enjoyed.
Agreed - and I think that's a more important point than the merely
technical issue of whether it manipulates sys.path the way we want.
I'm warming up to this idea a bit, especially how it supports the most
elegant
approach but degrades gracefully. Some questions that arise:
Where would Setuptools expect to find these launchers? Would it expect
them to
be present on the system? Would it symlink or hardlink them or simply
copy? Is
py.exe subject to the 'looks like installer' behavior, such that it would
need
a manifest?
I still feel like this approach would require substantial special-casing,
but
since it provides a transition to the simple, elegant approach, I'm not
opposed.
You ask important questions (and I don't know the answers :-)). I suggested
(in the issue where I implemented the wrapper functionality) that we add a
stdlib module that knows where the wrapper is and can wrap a script for
you (something like distlib's distlib.script module). But that felt a bit
over-engineered and the idea was abandoned almost immediately.
In practical terms, of course, there are many, many ways of writing
wrappers, depending on what you want them to do, and how manageable you
want to make the job of manipulating them. The current setuptools wrappers
make manipulating them easy (it's just an exe plus a script, with related
names) at the cost of needing multiple files. You could append the script
to the exe, which is a one-file solution but harder to edit the script. You
could append a zipfile to the exe, which has the advantage that you can
prepend arbitrary content to a zipfile, so there are no mixed content
issues to worry about, in theory. The exe would be coded slightly
differently in each case, of course, so there's a maintenance cost...
Also, of course, as was mentioned elsewhere in the thread, you need
separate wrapper exes for every architecture/platform you intend to
support. And unless the wrappers are added at install time, wrapped scripts
change a platform-neutral script into a platform-specific exe.
I would still like to see the standard be registered .pye (I'm happy with a
bikeshed of this colour) and .pwe extensions which are added to PATHEXT and
associated with the launcher. But someone would need to collect and
document the issues which have been raised with this approach, confirm
which ones are real and which are merely potential, offer solutions for the
real ones and confirm that the potential ones don't actually happen (or
they do!). I won't have the time to do this in the near future - but I
could consider it in a month or two if someone reminds me. It should
probably be a PEP, although I'm not 100% sure what that PEP would propose
(what changes in Python and wording in a PEP would be needed for you to
change setuptools to abandon wrappers, for instance? And why would you need
a PEP?)
Paul
___
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 2:14 PM, Jason R. Coombs jar...@jaraco.com wrote: -Original Message- From: Distutils-SIG [mailto:distutils-sig- bounces+jaraco=jaraco@python.org] On Behalf Of PJ Eby Sent: Monday, 12 August, 2013 11:22 On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore p.f.mo...@gmail.com wrote: On 12 August 2013 14:01, PJ Eby p...@telecommunity.com wrote: As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content. Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue. My preference is to reject the idea of the side-by-side executable launcher. There are several downsides that I'm trying to avoid by moving away from the executable: 1. Disparity with Unix. Better parity means cleaner code, easier documentation, and less confusion moving from platform to platform. 2. Executables that look like installers. If a launcher executable is used and Windows detects that it looks like an installer and it's a 32-bit executable and it doesn't have a manifest to disable the functionality, Windows will execute the file in a separate UAC context (often in a separate Window). 3. Additional files are needed. In particular, due to (2), a manifest must be provided for 32-bit executables. 4. Word size accounting. It's not clear to me what word size is needed. 32-bit may be sufficient, though 64-bit seem to have some advantages: a manifest is not needed, and it can match the word size of the installed Python executable (for consistency). Setuptools currently provides both (and installs the one that matches the Python executable). 5. Platform support. We're fortunate that Windows is one of the most stable binary platforms out there. Nevertheless, Setuptools recently got support for AMD binaries in the launcher. By relying on an external launcher, the launcher becomes responsible for platform support. 6. Two to three files to do the job of one. In fact, the job isn't much more than to invoke code elsewhere, so it seems ugly to require as many as three files to do the job. Then multiply that by the Python-specific version and you have up to six files for a single script. 7. Obfuscation of purpose. A single script pretty directly communicates its purpose. When there are multiple files, it's not obvious why they exist or what their purpose is. Indeed, I went years without realizing we had an open issue in Distribute due to a missing manifest (which was fixed in Setuptools), all because I used the 64-bit executable. While it may take some time for the community to learn what a '.pyl' is, it's easily documented and simple to grasp, unlike the subtle and sometimes implicit nuances (and fragility) of a side-by-side executable. 8. Unwanted content. Some Unix users have complained about finding Windows executables in their Linux packages, so now Setuptools has special handling to omit the launchers when installed on Unix systems. This is far from beautiful. I think the issue of too many extensions vs. source/binary confusion is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP. If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, hey, run this as a console app via PyLauncher, not that it's a script (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile. If it's '.py*', I don't see why it's not reasonable to allow omission of the shebang, and assume the default python. After encountering and now understanding the subtle import semantics, I'm hoping that this new extension can also be used in my personal 'scripts' collection to serve the same purpose it does for setuptools console entry points. I guess one could require #!/usr/bin/python in each, but that seems superfluous on Windows. I don't feel at all strongly on this point. PS Either the ref file marker approach, or a new Python command line
Re: [Distutils] How to handle launcher script importability?
Jason R. Coombs jaraco at jaraco.com writes: My preference is to reject the idea of the side-by-side executable launcher. I agree that it's not ideal to have side-by-side executables, but how do you propose to address PJE's point about older Python versions? You can't force people to install the standalone launcher where they have only older Python versions installed. Are you proposing that an installer looks for py.exe at installation time, and does one thing (install as foo.pye) if it is found, and another thing (install as foo-script.py + foo.exe) if it isn't? This could cause breakage if a user subsequently uninstalled the launcher (perhaps unknowingly, by uninstalling the version of Python it came with) - are we OK with that? They might seem ugly and redundant (multiple copies of identical executables - yecch), but the years of service given by setuptools executable launchers suggests that they are not a problem for end users, whereas the user experience with the alternatives might be in problematic some scenarios. While it's good to look at alternatives, it seems like this area is a solved problem even if the solution isn't especially elegant, and I would guess that are probably other issues which should have a higher priority. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 4:04 PM, Paul Moore p.f.mo...@gmail.com wrote: I would still like to see the standard be registered .pye (I'm happy with a bikeshed of this colour) and .pwe extensions which are added to PATHEXT and As long as we're discussing bikeshed colors, I'd like to counterpropose .pya and .pwa, to be registered in the Windows class registry as Python Console Application and Python Windowed Application, respectively. Since the *only* reason we need these extensions is for Windows (other OSes do fine at making things executable without an extension), and Windows calls things like these Applications or Apps in Explorer normally, I think it's better to call them what Windows calls them. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 12, 2013, at 4:04 PM, Paul Moore p.f.mo...@gmail.com wrote: Also, of course, as was mentioned elsewhere in the thread, you need separate wrapper exes for every architecture/platform you intend to support. And unless the wrappers are added at install time, wrapped scripts change a platform-neutral script into a platform-specific exe. Hopefully this all will solve this problem, as it is right now if you use setuptools entry points then Wheels erroneously pretend to be platform agnostic. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 4:29 PM, Donald Stufft don...@stufft.io wrote: Hopefully this all will solve this problem, as it is right now if you use setuptools entry points then Wheels erroneously pretend to be platform agnostic. IMO it's okay to give up having ready-to-use scripts in a platform-agnostic wheel; scripts are sadly not a platform-agnostic thing. (If only MS-DOS had been a little bit more Unix, and a little less CP/M...) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
PJ Eby wrote: (Granted, I can see reasons for not wanting to use the same extension for source and zipped versions, mostly in the area of tools other than pylauncher, but if you do have different extensions then there have to be *four*, due to console vs. windowed and source vs. zipped.) Just a thought -- is there any need in this day and age for extensions to be limited to 3 characters? Going beyond that would give us more room to define a consistent scheme for the various combinations. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 10:38 AM, Jason R. Coombs jar...@jaraco.com wrote: In Setuptools 1.0 (currently in beta), I've added an experimental, opt-in feature to install pure Python launcher scripts on Windows instead of installing a launcher executable for each script, with the intention that these scripts will be launched by pylauncher or Python directly, eventually obviating the need for a launcher executable in setuptools at all. This means that instead of installing, for example: Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest Instead Setuptools just installs: Scripts\my-command.py This technique is much like the scripts that get installed to bin/ on Unix, except that due to the nature of launching commands on Windows, the .py extension is essentially required. One problem with this technique is that if the script is also a valid module name, it can be imported, and because Python puts the script's directory at the top of sys.path, it _will_ be imported if that name is imported. This happens, for example, after installing Cython. Cython provides a command, 'cython', and a (extension) module called 'cython'. If one launches cython using the script launcher, the 'cython' module will not be importable (because import cython will import the launcher script). Presumably, this is why '-script' was added to the launcher scripts in previous versions. This is a rather unfortunate situation, and I'd like to solicit comments for a way to avoid this situation. I see a few options: 1. Have the setuptools-generated launcher scripts del sys.path[0] before launching. 2. Accept the implementation and file bugs with the offending projects like Cython to have them either rename their script or rename their internal module. 3. Continue to generate the script names with '-script.py' appended, requiring invocation to always include -script.py on Windows. 4. Continue to generate executables, duplicating the effort of pylauncher, and dealing with the maintenance burden of that functionality. I don't see (2), (3), or (4) as really viable, so my proposal is to move forward with (1) if there aren't any better suggestions. If we move forward with (1), there are a few concerns that come to mind. Here's another problem with #1: you will break single-directory standalone portable app installs, where you use easy_install -mad somedir to install all of an app's dependencies to a single directory that the app can then be run from (assuming Python is available). In order to work around this issue, you'd need to hardcode sys.path entries for the dependencies, or do something else more complicated in order to ensure that dependency resolution will pick up the adjacent distributions before searching anything else on sys.path. Third, is it possible some users are depending on the presence of sys.path[0] Absolutely. It's a documented feature of Python that the script directory is always first on sys.path, so that you can provide modules and packages adjacent to it. That's how portable app installs work with easy_install. May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non-console apps) as a zipped Python application. .pyz files aren't importable, but *are* executable. That's basically all that's needed to prevent importing -- a file extension that's launchable but not importable. (There's also an option 6, which is to use import system hooks to prevent the script modules from being found in the sys.path[0] entry, but that's rather hairier.) Using option 5 means the feature can only work with versions of Python on Windows that install the necessary PATHEXT support to allow that extension to work, but you're kind of limited to that anyway, because by default .py files aren't findable via PATH on Windows. Your post doesn't make it clear whether you're aware of that, btw: IIUC, on most Windows setups, executing a .py file via PATH doesn't work unless you've set up PATHEXT to include .py. So your feature's going to break until that's fixed, and AFAIK there is *no* Windows Python that fixes this, with the possible exception of 3.4 alpha, possibly a future alpha that hasn't been released yet, because last I saw on Python-Dev it was still being discussed *how* to update PATHEXT safely from the installer. In short: dropping .exe wrappers is not supportable on *any* current version of Python for Windows, in the sense that anybody who uses it will not yet be able to execute the scripts if they are currently doing so via PATH (and haven't manually fixed their PATHEXT). (This was one of the main reasons for using .exe wrappers in the first place.) The .pyz approach of course has the same drawback, but at least it should be viable for future Python versions, and doesn't have the sys.path[0] problems. I think you are going to have to keep .exe wrappers the default for all Python versions 3.4.
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 12:17 PM, PJ Eby p...@telecommunity.com wrote: May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non-console apps) as a zipped Python application. .pyz files aren't importable, but *are* executable. That's basically all that's needed to prevent importing -- a file extension that's launchable but not importable. (Details I forgot to mention: the script would be in __main__.py inside the zipped application file, and it would need to change sys.path[0], because sys.path[0] will be the .pyz file itself; it should replace it with the directory containing the .pyz file before doing anything else. That would be the correct way to simulate the existing .exe approach.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
-Original Message- From: PJ Eby [mailto:p...@telecommunity.com] Sent: Sunday, 11 August, 2013 12:17 Here's another problem with #1: you will break single-directory standalone portable app installs, where you use easy_install -mad somedir to install all of an app's dependencies to a single directory that the app can then be run from (assuming Python is available). In order to work around this issue, you'd need to hardcode sys.path entries for the dependencies, or do something else more complicated in order to ensure that dependency resolution will pick up the adjacent distributions before searching anything else on sys.path. Third, is it possible some users are depending on the presence of sys.path[0] Absolutely. It's a documented feature of Python that the script directory is always first on sys.path, so that you can provide modules and packages adjacent to it. That's how portable app installs work with easy_install. All good points to be considered. Thanks. May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non- console apps) as a zipped Python application. .pyz files aren't importable, but *are* executable. That's basically all that's needed to prevent importing -- a file extension that's launchable but not importable. This sounds like a suitable idea, but as you mention in a subsequent message, this format has issues with sys.path assumptions as well. In this case, I'm inclined to suggest yet another option (7) - create another extension to specifically represent executable but not importable scripts, perhaps .pys/.pysw (or .pycs/.pygs to more closely match console script and gui script). It sounds as if there is a fundamental need for Python to define an extension that distinguishes a script from a module. (There's also an option 6, which is to use import system hooks to prevent the script modules from being found in the sys.path[0] entry, but that's rather hairier.) Agreed, this sounds like it has its own subtle challenges. Using option 5 means the feature can only work with versions of Python on Windows that install the necessary PATHEXT support to allow that extension to work, but you're kind of limited to that anyway, because by default .py files aren't findable via PATH on Windows. Your post doesn't make it clear whether you're aware of that, btw: IIUC, on most Windows setups, executing a .py file via PATH doesn't work unless you've set up PATHEXT to include .py. So your feature's going to break until that's fixed, and AFAIK there is *no* Windows Python that fixes this, with the possible exception of 3.4 alpha, possibly a future alpha that hasn't been released yet, because last I saw on Python-Dev it was still being discussed *how* to update PATHEXT safely from the installer. In short: dropping .exe wrappers is not supportable on *any* current version of Python for Windows, in the sense that anybody who uses it will not yet be able to execute the scripts if they are currently doing so via PATH (and haven't manually fixed their PATHEXT). (This was one of the main reasons for using .exe wrappers in the first place.) The .pyz approach of course has the same drawback, but at least it should be viable for future Python versions, and doesn't have the sys.path[0] problems. I think you are going to have to keep .exe wrappers the default for all Python versions 3.4. I am aware of the PATHEXT factor. I personally add .py and .pyw to the PATHEXT (for all users) on my systems, so I was unsure if Python 3.3 did add those or if pylauncher would add them (if installed separately). I was _hoping_ that was the case, but it sounds like it is not. I did include in the documentation notes about this requirement (https://bitbucket.org/pypa/setuptools/src/1.0b1/docs/easy_install.txt#cl-98 ). I do want to explore the possibility of setuptools facilitating this configuration such that it's easy for a Windows user to enable these settings even if Python does not. The nice thing about (7) is it would define an extension that's specifically meant to be executable, so would be an obvious (and potentially less controversial) choice to include in PATHEXT. smime.p7s Description: S/MIME cryptographic signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 1:58 PM, Jason R. Coombs jar...@jaraco.com wrote: This sounds like a suitable idea, but as you mention in a subsequent message, this format has issues with sys.path assumptions as well. Meh. It's basically, a one-line fix in the __main__.py, i.e.: import sys,os.path; sys.path[0] = os.path.dirname(sys.path[0]) Not exactly rocket science. ;-) In this case, I'm inclined to suggest yet another option (7) - create another extension to specifically represent executable but not importable scripts, perhaps .pys/.pysw (or .pycs/.pygs to more closely match console script and gui script). Probably .pys/.pyws or .pws would be needed, due to issues with some Windows shells using extensions longer than three characters. (This came up in PEP 441 discussions on Python-Dev.) It sounds as if there is a fundamental need for Python to define an extension that distinguishes a script from a module. Yep. I am aware of the PATHEXT factor. I personally add .py and .pyw to the PATHEXT (for all users) on my systems, so I was unsure if Python 3.3 did add those or if pylauncher would add them (if installed separately). I was _hoping_ that was the case, but it sounds like it is not. I did include in the documentation notes about this requirement (https://bitbucket.org/pypa/setuptools/src/1.0b1/docs/easy_install.txt#cl-98 ). I do want to explore the possibility of setuptools facilitating this configuration such that it's easy for a Windows user to enable these settings even if Python does not. It would definitely make sense to have an installer that sets this up, but it would need to be a Windows installer, I think, not a Python program. That is, I don't think setuptools can really do anything about it directly. Personally, I'm not sure I see the point of pushing for early elimination of .exe's - they don't depend on the registry or environment variables or anything else, which makes them great for standalone applications, and they work across all Python versions. Meanwhile, the experimental nature of your change -- and its inability to be the default on versions below 3.4 -- means you're going to be maintaining two sets of code for a very long time. OTOH, implementing a way to deploy an app as a .pyz/.pwz file is a useful feature in its own right, so it might not be doubling up as much. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
