Re: [Distutils] distutils.util.get_platform() - Linux vs Windows
> -Original Message- > From: Chris Barker - NOAA Federal [mailto:chris.bar...@noaa.gov] > Sent: Tuesday, August 20, 2013 5:47 PM > To: Ferencik, Samuel: Markets (PRG) > Cc: distutils-sig@python.org > Subject: Re: [Distutils] distutils.util.get_platform() - Linux vs Windows > > On Mon, Aug 19, 2013 at 11:15 PM, wrote: > >> What does your 'uname -m' return? > > x86_64 > >> Is it possible you're really running a 32-bit >> Python on a *32-bit* OS X kernel? [http://superuser.com/q/161195] > > nope -- I am quite deliberately running a 32 bit Python on my 64 bit > OS (I have some custom code C++ I"m using that is not yet 64 bit > safe). That's strange. I'm on Python 3.3.1, and it seems to me that get_platform() derives the value from uname for OS X, similar to Linux. (osname, host, release, version, machine) = os.uname() ... elif osname[:6] == "darwin": import _osx_support, distutils.sysconfig osname, release, machine = _osx_support.get_platform_osx( distutils.sysconfig.get_config_vars(), osname, release, machine) return "%s-%s-%s" % (osname, release, machine) so I would expect "uname -m" to be in line with get_plaform(). But maybe I'm misreading that... Also, I don't have access to the _osx_support source code. > >> return value is wrong on Linux and correct on >> Windows, right? > > no -- I'm saying that it's right on Windows (and OS-X), but wrong on Linux. I think you have misread my sentence, and we actually agree here. What's the next action? Report a Python bug? (That's a cultural question; I'm new to Python.) Regards, Sam > >> That get_platform() should return "32-bit" for a 32-bit process >> running on a 64-bit system. > > yes, it should. > >> TBH, I was expecting the opposite; to me, "platform" >> means the OS, which would mean that Linux does well to derive the return >> value >> from the OS's architecture. > > except what would be the utility of that? this is a call made within > python, and it's part of distutils, so what the caller wants to know > is the platform that this particular python was build for, NOT the > platform is happens to be running on. i.e. what platform do I want to > build binary extensions for, and/or what platform do I want to > download binary wheels for. > > So I'm pretty sure that currently Windows and OS-X have it right, and > Linux is broken. I'm guessing running 32 bit python on a 64 bit LInux > is not that common, however. (and it's less common to download > binaries...) > > To add complexity, if I run the Apple-supplied python2.7.1 (which is > 32_64 bit universal, but runs 64 bit on my machine), I get: > > >>> distutils.util.get_platform() > 'macosx-10.7-intel' > > Which is more useful than it may look at first -- "intel" means "both > intel platforms", i.e. i386 and x86_64. and 10.7 means -- built for > OS-X 10.7 and above. > > so I think it's doing the right thing. > > -Chris ___ This message is for information purposes only, it is not a recommendation, advice, offer or solicitation to buy or sell a product or service nor an official confirmation of any transaction. It is directed at persons who are professionals and is not intended for retail customer use. Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer. For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding market commentary from Barclays Sales and/or Trading, who are active market participants; and in respect of Barclays Research, including disclosures relating to specific issuers, please see http://publicresearch.barclays.com. ___ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
On 22 August 2013 08:20, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> Um, the current wheel spec uses PEP 345 + setuptools metadata only. If > distlib is expecting PEP 426 metadata in wheel files, it is not compliant with > the spec. > > I can certainly rectify that - I was possibly confused by the fact that the > latest wheel implementation writes pydist.json to the wheel (though the Wheel- > Version in WHEEL is still 1.0). Yeah, they were certainly coupled together originally - that's why PEPs 425, 426 and 427 all happened around the same time. However, Daniel tweaked the wheel format spec in PEP 427 to remove the dependency on the new metadata spec once he realised that most of the features that wheels really needed already existed in the setuptools metadata, and the metadata spec was going to take a *lot* longer to stabilise than he originally thought. However, the pydist.json that wheel currently writes is in the category of "arbitrary additional metadata in the dist-info directory", since the metadata 2.0 spec is still far from stable. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 22 August 2013 08:12, Vinay Sajip wrote: > I've been up front about what's in distlib all along - check the overview > page in the distlib docs. Above all, I want the stuff I do to be *useful*, > rather than tick boxes here and there. Right, you didn't do anything wrong, I just wasn't really paying attention because promoting distlib adoption has been firmly in the "later" bucket for me, after the setuptools rehabilitation, pip bootstrapping, etc. I can't fault you for not realising I believe something I had never really written down anywhere :) > Nick Coghlan gmail.com> writes: >> Right. distlib can be a candidate for stdlib inclusion, or it can be a >> vehicle for distributing experimental features not covered by any PEP, it >> can't be both at the same time. > > How is it that you're ready to call time on it now, already, even though > you've told me standardisation is not something to be even considered until > Python 3.5? Of the two options quoted above, you had already decided that it > can't be the former in the short to medium term, and now you're saying you > don't want it to be the latter either? I'm confused. I'd certainly like to > keep making distlib and distil more useful. I previously thought distlib was going to be the repository for the agreed, stable, "this is going to happen" stuff. It's OK that I was wrong - I think you're right that somewhere is needed as an experimental location to show some of the *possibilities* of the new metadata, and to seed ideas for making it into the eventual standard base that people can assume is readily available. What that means though, is we need *something else* that indicates the common core that people can assume will always be available. It's this common core which pip will need to factor out to remove their dependency on setuptools, rather than adopting distlib wholesale, experimental features and all. I'm actually OK with that, since it means we can aim for a self-contained pip that serves as the basis for the rest of the ecosystem, and is upgraded on the *pip* update cycle, rather than CPython's. I'd been wondering how we'd avoid the "stale standard library support" issue we ran into with distutils, and I think the pip bootstrapping proposals give us the answer: we don't make the core distribution infrastructure part of the standard library, we make it part of *pip*. One of the key concepts of the bootstrapping idea is that CPython maintenance releases will bundle newer versions of pip, and also that pip will be able to upgrade itself in place, so if people need newer distribution infrastructure "upgrade pip" is a much lower risk proposition than "upgrade to a newer version of Python". Currently, pip doesn't expose a programmatic API. I suggested to Donald that it may make sense to start exposing one as "piplib". The bootstrapping would then provide the pip CLI and the common utilities in piplib, and that would be the more conservative core, leaving distlib and distil free to push the boundaries with experimental features that go beyond what the agreed standards currently support. As ideas from distlib make their way into the relevant PEPs and hence into piplib, then they will become available by default, but in the meantime, people would be able to do a build dependency on distlib to get the experimental features. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
Nick Coghlan gmail.com> writes: > Um, the current wheel spec uses PEP 345 + setuptools metadata only. If distlib is expecting PEP 426 metadata in wheel files, it is not compliant with the spec. I can certainly rectify that - I was possibly confused by the fact that the latest wheel implementation writes pydist.json to the wheel (though the Wheel- Version in WHEEL is still 1.0). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Nick Coghlan gmail.com> writes: > Right. I wasn't really aware Vinay was adding experimental ideas to > distlib, I thought it was just the proven stable core from distutils2, > plus support for the draft PEPs, with the experimental stuff entirely in > distil rather than in distlib. I've been up front about what's in distlib all along - check the overview page in the distlib docs. Above all, I want the stuff I do to be *useful*, rather than tick boxes here and there. For example, there's no PEP covering distlib's functionality whereby dependency resolution happens *without* downloading/unpacking archives and running egg_info. We seem to take this sort of dependency resolution for granted in Linux distros, but for some reason Python packaging has to make do with a clunkier approach? Initially my work in this area was experimental, but it seems to work well (though some areas still need more work), and at some point I would expect to propose a PEP to cover it. However, lots of things are in flux and not in my direct control, so directing effort there now would likely involve rework when e.g. existing PEPs change, so it wouldn't be productive to do it yet. > If Vinay wants to do experimental extensions, they either need to happen > somewhere other than distlib, or else we need a new library which is just > the candidate for standard library inclusion with *nothing* that hasn't been > discussed and agreed to through the PEP process. Distlib aims to conform to the PEPs as they evolve, and anyone can raise an issue if they find non-conformance. In my view, one of the failures of distutils2 was that it did not include functionality that was actually useful to people and used by them, such as exports and package resources. I implemented exports in distlib before you added them to PEP 426, but that doesn't mean that I'm some kind of heretic for doing so. I've implemented package resources in distlib too, and there's no PEP yet covering it. You've specifically told me that you don't see distlib as a candidate for inclusion in Python in the 3.4 time frame, so what's the big hurry with getting the pitchforks out now? > As I said, I *thought* distlib was that library, but it appears Vinay doesn't > currently see it that way :( Nick, you haven't discussed this with me at all, and I don't see how you can come up with that interpretation from anything I've said in my posts here (or anywhere else). As I see it, you've explicitly told me that there's a very long time to go before distlib is even considered as a possible candidate for standardisation, and I can't see any valid reason why I can't keep adding useful features to it for now, because the experience gained from using them will inform any future PEPs relating to those features. When the time to consider standardisation is nearer, decisions can be made about what might need to come out and what can stay in, etc. and PEPs written to propose any things that aren't already covered. I've written distlib in a modular fashion and I don't expect such a process to be painful, and I would certainly expect to produce PEPs where needed. > Right. distlib can be a candidate for stdlib inclusion, or it can be a > vehicle for distributing experimental features not covered by any PEP, it > can't be both at the same time. How is it that you're ready to call time on it now, already, even though you've told me standardisation is not something to be even considered until Python 3.5? Of the two options quoted above, you had already decided that it can't be the former in the short to medium term, and now you're saying you don't want it to be the latter either? I'm confused. I'd certainly like to keep making distlib and distil more useful. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 21 August 2013 22:13, Nick Coghlan wrote: > Wheel is a suitable replacement for bdist_wininst (although anything that > needs install hooks will have to wait for wheel 1.1, which will support > metadata 2.0). It's just not a replacement for what hashdist and conda let > you do when you care more about reproducibility than you do about security > updates. OK, that's a good statement - wheels as a better bdist_wininst is all I want to be able to promote (and yes, if you need post-install hooks, wait for wheel 1.1). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 22 Aug 2013 06:57, "Paul Moore" wrote: > > On 21 August 2013 21:35, Nick Coghlan wrote: >> >> I'm reasonably confident the wheel format *doesn't* meet the scientific community's needs in the general case, and can't be made to do so without a lot of additional complexity. That's why I explicitly support the hashdist/conda approach which abandons some of the goals of pip and wheel (notably, easier handling of security updates and easier conversion to Linux distro packages) in order to better handle complex binary dependencies. > > > While "the general case" may include some specialised situations, in my view, if the wheel format isn't a suitable replacement for bdist_wininst (and by implication, cannot be used by the numpy, scipy and similar projects to deliver Windows binary distributions) - maybe not for specialised use, but certainly for casual users like myself - then it will be essentially a failure. Sure, it can replace bdist_wininst. What it *can't* do is tag an arbitrarily complex dependency tree the way hashdist can. So you're either going to need fat wheels which figure out the right binary extensions to use in a post-install hook, or else you're going to have to bless a default configuration that is published as a wheel. Since wheel 1.0 has no support for post-install hooks, that option is not yet available, and I don't expect the idea of blessing a default binary version to be popular either. > > The only reason I am interested in wheels *at all* is as a format that allows me to "pip install" all those projects that currently provide bdist_wininst installers. In the first instance, via "wheel convert", but ultimately by the projects themselves switching from wininst format to wheel (or via some form of build farm mechanism, it doesn't matter to me, as long as the wheels are on PyPI). > > Note that "wheel convert" is proof that this is the case right now, so this is not setting the bar unreasonably high. Nor am I saying that there's even a problem here at the moment. But if distutils-sig is sending a message that they don't think wheels are a suitable distribution format for (say) numpy or scipy, then I will be very disappointed. Wheel is a suitable replacement for bdist_wininst (although anything that needs install hooks will have to wait for wheel 1.1, which will support metadata 2.0). It's just not a replacement for what hashdist and conda let you do when you care more about reproducibility than you do about security updates. Cheers, Nick. > > Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 21 August 2013 21:35, Nick Coghlan wrote: > I'm reasonably confident the wheel format *doesn't* meet the scientific > community's needs in the general case, and can't be made to do so without a > lot of additional complexity. That's why I explicitly support the > hashdist/conda approach which abandons some of the goals of pip and wheel > (notably, easier handling of security updates and easier conversion to > Linux distro packages) in order to better handle complex binary > dependencies. While "the general case" may include some specialised situations, in my view, if the wheel format isn't a suitable replacement for bdist_wininst (and by implication, cannot be used by the numpy, scipy and similar projects to deliver Windows binary distributions) - maybe not for specialised use, but certainly for casual users like myself - then it will be essentially a failure. The only reason I am interested in wheels *at all* is as a format that allows me to "pip install" all those projects that currently provide bdist_wininst installers. In the first instance, via "wheel convert", but ultimately by the projects themselves switching from wininst format to wheel (or via some form of build farm mechanism, it doesn't matter to me, as long as the wheels are on PyPI). Note that "wheel convert" is proof that this is the case right now, so this is not setting the bar unreasonably high. Nor am I saying that there's even a problem here at the moment. But if distutils-sig is sending a message that they don't think wheels are a suitable distribution format for (say) numpy or scipy, then I will be very disappointed. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 Aug 2013 17:46, "Donald Stufft" wrote: > > > On Aug 21, 2013, at 3:32 AM, Vinay Sajip wrote: > > > Paul Moore gmail.com> writes: > > > >> I'm concerned that you need extra metadata (not described in the wheel > > spec) to do this. It means that there are in effect two subtly different > > types of wheel. To be specific, if I create a wheel for (say) pyzmq using > > distil, and mount it, everything works. But if I create the same wheel with > > bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages > > the credibility of wheel as a standardised format. > > > > If the additional metadata isn't there, then distlib just doesn't do > > anything additional - it just makes the Python modules importable (by adding > > the wheel to sys.path, which AFAIK is uncontroversial). > > > >> Can I suggest that if you need to add features like this, you need to get > > the wheel spec updated to mandate them, so that *all* wheels will follow the > > same spec. > >> Essentially, I am -1 on any feature that uses information that is not > > documented in the wheel spec. Pip in particular resisted adding support for > > wheels until they were standardised in a PEP. It's frustrating if that PEP > > *still* doesn't mean that the wheel format is the same for all tools. (Note > > that another area where this is an issue is script wrappers, as the spec is > > silent about the fact that they are specified using entry-points.txt in > > metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel > > for his consideration). > > > > Well, you don't really want to stifle innovation, do you? ;-) > > > > As far as I can tell, Daniel's wheel implementation allows files that are > > not specifically mentioned in the PEP to be installed into a distribution's > > .dist-info. This is also allowed in distlib - ISTM this is one way in which > > different packaging tools can add features which are special to them, and > > hold state relevant to distributions they build and/or install. If you > > accept that multiple competing implementations if a PEP are a good thing, > > then they can't all be functionally identical, though they must all > > implement a common set of functions described in the PEP they're implementing. > > I was one of the advocates for extension support in the new metadata, I want > tools to be able to try things out and innovate. > > However what I don't really want is to be using someones personal testbed > for features they think is cool. There's nothing *wrong* with you trying new > ideas out in distlib, it just means that distib isn't the library I want to build > tooling around. Right. I wasn't really aware Vinay was adding experimental ideas to distlib, I thought it was just the proven stable core from distutils2, plus support for the draft PEPs, with the experimental stuff entirely in distil rather than in distlib. If Vinay wants to do experimental extensions, they either need to happen somewhere other than distlib, or else we need a new library which is just the candidate for standard library inclusion with *nothing* that hasn't been discussed and agreed to through the PEP process. As I said, I *thought* distlib was that library, but it appears Vinay doesn't currently see it that way :( > > My basic problem is if the library we're pointing at to be the reference > implementation of all of these things is adding new features it's confusing what > is standard and what are just distlib's extensions. > > So basically I want people to innovate, that's something I feel very strongly > is a good thing, I just don't want innovations to happen in the reference > library. Maybe we need a smaller reference library which is strictly the PEPs > to allow distlib to experiment. If it's experimentations turns out to be good and > useful we can make PEPs for them and add them to the reference library. Right. distlib can be a candidate for stdlib inclusion, or it can be a vehicle for distributing experimental features not covered by any PEP, it can't be both at the same time. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
On 21 August 2013 21:40, Nick Coghlan wrote: > Um, the current wheel spec uses PEP 345 + setuptools metadata only. If > distlib is expecting PEP 426 metadata in wheel files, it is not compliant > with the spec. > > There won't be a new version of the wheel spec until after PEP 426 is > done, and that's likely months away. > OK, thanks. That's pretty clear. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
On 21 Aug 2013 22:42, "Vinay Sajip" wrote: > > Paul Moore gmail.com> writes: > > > BUT, this means that there is no spec of the current behaviour, and no > implementation of the Wheel 1.0 spec anywhere. > [snip] > > or the wheel spec needs a review reasonably soon. > > I think it's this. I'm not sure to what extent wheels are being used in > anger out there, but it would make sense to review the spec in light of PEP > 426 developments and release a 1.1. Um, the current wheel spec uses PEP 345 + setuptools metadata only. If distlib is expecting PEP 426 metadata in wheel files, it is not compliant with the spec. There won't be a new version of the wheel spec until after PEP 426 is done, and that's likely months away. Cheers, Nick. > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 21 Aug 2013 20:40, "Paul Moore" wrote: > > On 21 August 2013 11:29, Oscar Benjamin wrote: >> >> I may have misunderstood it but looking at this >> https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147 >> I think that the installer ships variants for each architecture and >> decides at install time which to place on the target system. If that's >> the case then would it be possible for a wheel to ship all variants so >> that a post-install script could sort it out (rename/delete) after the >> wheel is installed? > > > Wheel 1.0 does not have the ability to bundle multiple versions (and I don't think tags are fine-grained enough to cover the differences numpy need, which are at the "do you have the SSE instruction set?" level AIUI). Multi-version wheels are a possible future extension, but I don't know if anyone has thought about fine-grained tags. > > This is precisely the sort of input that the numpy people could provide to make sure that the wheel design covers their needs. I'm reasonably confident the wheel format *doesn't* meet the scientific community's needs in the general case, and can't be made to do so without a lot of additional complexity. That's why I explicitly support the hashdist/conda approach which abandons some of the goals of pip and wheel (notably, easier handling of security updates and easier conversion to Linux distro packages) in order to better handle complex binary dependencies. >From the pip/wheel point of view, the conda ecosystem then becomes just another downstream "system" package manager to interoperate with. Cheers, Nick. > Paul > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Paul Moore gmail.com> writes: > That implies that any wheel reference implementation needs to expose APIs > for reading and writing the metadata to/from the wheel. Not necessarily. For example, distlib's approach side-steps the need for such a write API: you tell Wheel.build which directories contain purelib, platlib, scripts, headers etc. and it puts all the stuff that it finds in those directories into appropriate locations in the wheel, then updates RECORD, WHEEL etc. This way, it doesn't need to worry about what custom files installers put in .dist-info, and reduces the coupling between the wheel code and users of it. There's no need for a special read API either, since wheels are just zip files and you can use the zipfile API to look at anything inside a wheel. As a convenience, distlib's Wheel instances have a metadata property, which returns as a dict the pydist.json from the wheel's .dist-info. Why make it more complicated than that? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
> > > > 1) Wheel's conversion of - to _ in version strings embedded in > > filenames, which breaks with setuptools precedent; see > > https://github.com/pypa/pip/issues/1150 and > > > https://bitbucket.org/dholth/wheel/issue/78/wheel-rewrites-versions-preventing > > No good solution to this one just yet. > not a great solution, but wheel can just declare it doesn't support "_" in versions, and not build wheels in that case. then installers know they can assume "-" was meant. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 21, 2013 at 11:52 AM, Carl Meyer wrote: > On 08/21/2013 03:29 AM, Donald Stufft wrote: >> Can you send me a list (or post them here) of what issues you've hit? >> The biggest one i'm aware of is the scripts problem which is a >> fundamental problem with the 1.0 Wheel (or rather that any library with >> console entry points cannot be universal). > > Since you asked, I'll mention the two that I've hit (though I think > you're also aware of these already): > > 1) Wheel's conversion of - to _ in version strings embedded in > filenames, which breaks with setuptools precedent; see > https://github.com/pypa/pip/issues/1150 and > https://bitbucket.org/dholth/wheel/issue/78/wheel-rewrites-versions-preventing No good solution to this one just yet. > 2) Wheel's decision to follow distutils' documentation rather than > distutils' behavior when it comes to the location for installing > data_files with relative paths; see > https://bitbucket.org/dholth/wheel/issue/80/wheel-does-not-install-data_files-in-site Django has fixed it by using package_data appropriately: https://code.djangoproject.com/ticket/19252 . The problem isn't unique to wheel, the same data_files mishap happens with bdist_wininst. "Regardless, comment 5 is correct that we jump through way too many hoops in our setup.py in order to try to trick distutils into handling data_files as if they were package_data, and that is the root cause of this bug. Instead we should just use package_data and solve the problem properly." ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 08/21/2013 10:32 AM, Daniel Holth wrote: >> 2) Wheel's decision to follow distutils' documentation rather than >> distutils' behavior when it comes to the location for installing >> data_files with relative paths; see >> https://bitbucket.org/dholth/wheel/issue/80/wheel-does-not-install-data_files-in-site > > Django has fixed it by using package_data appropriately: > https://code.djangoproject.com/ticket/19252 . The problem isn't unique > to wheel, the same data_files mishap happens with bdist_wininst. > > "Regardless, comment 5 is correct that we jump through way too many > hoops in our setup.py in order to try to trick distutils into handling > data_files as if they were package_data, and that is the root cause of > this bug. Instead we should just use package_data and solve the > problem properly." Yup, that's my comment you're quoting :-) I do think from the packager end using package_data is the right solution. But given the existence of distributions using data_files this way (and the likelihood that not all of them will be fixed), is there a good argument for wheel to not maintain compatibility with sdists and python setup.py install? Are there distributions out there relying on the bdist_wininst behavior? Carl signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing from a wheel
On 21 August 2013 17:21, Oscar Benjamin wrote: > Okay I've just tried that and that's definitely the way I want to use it. > > So basically: > $ python setup.py bdist_wheel # Makes wheels > With pip and wheel installed pip wheel . will also build a wheel from the current directory (to wheelhouse\proj-ver.whl). You can use -w to put the wheel somewhere else if you prefer. Or if you want to build a wheel from a sdist on PyPI, something like "pip wheel projectname" will download it and build a wheel. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing from a wheel
On Wed, Aug 21, 2013 at 12:21 PM, Oscar Benjamin wrote: > On 21 August 2013 15:57, Daniel Holth wrote: >> >> A fresh virtualenv would have been the humane way to get a working >> 'pip install wheel'. > > Good point. I think I learned an important point going through that > upgrade mess though: uninstall/reinstall is safer than upgrade. > >> Wheel's built in installer isn't intended to replace or be better than >> pip in any way. It's just for reference or bootstrapping. > > Fair enough. Can I suggest that it have a --version option (since it > is traditional)? There is a nearly-done PR in wheel's https://bitbucket.org/dholth/wheel >> FYI if you point pip directly at the .whl file you can omit --use-wheel. > > Okay I've just tried that and that's definitely the way I want to use it. > > So basically: > $ python setup.py bdist_wheel # Makes wheels > and > $ pip install foo.whl # Installs wheels > > If someone wants to import the bdist_wheel command and use it outside > of setuptools setup() (in the way that numpy does) where should they > import it from? I'm thinking of something like this: > https://github.com/numpy/numpy/blob/master/numpy/distutils/command/bdist_rpm.py > > Is the following appropriate? > > from wheel.bdist_wheel import bdist_wheel > > class mybdist_wheel(bdist_wheel): > ... > > (the wheel API docs don't describe using bdist_wheel from Python code.) It should be about as appropriate as any distutils subclassing or extension exercise... a lot of debugging work and probably a bad idea... but if you must, that's how you would do it. > > Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 12:31 PM, PJ Eby wrote: > Personally, I'm very happy to see Vinay's extensions, because they are > IMO important validations of whether the new specs are likely to be > useful for replacing all of setuptools' functionality. There are > people who need to mount eggs and have their extensions run, so if it > wasn't possible to build tools that support them under the new specs > (whether that support is required by the spec or not), that would > still be a reason to use setuptools -- meaning, IMO, that the new spec > effort is failing to create a unified packaging world. I'm perfectly happy that he was able (and did) write those extensions, I was one of the people who wanted metadata 2.0 to be extensible for exactly that purpose. None of my arguments were against any particular feature. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 21, 2013 at 9:24 AM, Donald Stufft wrote: > An example is the wsgiref from the standard library. It's an example, alright, but not for your side. ;-) The wsgiref library doesn't just implement the spec, it implements a ton of utility classes for use with the spec. The validator was almost an afterthought grafted on later, borrowed from another project. It implements a framework with all sorts of features that are not technically part of the spec, but are just useful if you want to implement the spec. Very few of the classes, methods, etc. in the entire package are specified by the spec, except in the sense that many of them match a calling signature defined in the PEP. (The PEP doesn't specify any method names, except for things like read() on file-like objects.) IOW, wsgiref is a collection of generally useful tools for anybody doing things with the spec, as an combination of "examples of how to do this" and "ready-to-use code for working with the spec". Personally, I'm very happy to see Vinay's extensions, because they are IMO important validations of whether the new specs are likely to be useful for replacing all of setuptools' functionality. There are people who need to mount eggs and have their extensions run, so if it wasn't possible to build tools that support them under the new specs (whether that support is required by the spec or not), that would still be a reason to use setuptools -- meaning, IMO, that the new spec effort is failing to create a unified packaging world. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing from a wheel
On 21 August 2013 15:57, Paul Moore wrote: > On 21 August 2013 15:48, Oscar Benjamin wrote: >> >> Is it perhaps safer to suggest the following? >> a) uninstall pip/setuptools/distribute >> b) run ez_setup.py >> c) run get-pip.py > > It probably is. I've heard concerns that people want to avoid suggesting > manual uninstalls and having to download the setup scripts. But it seems > simple enough to me. (What would I know, I just run virtualenv and leave it > at that :-)) I walked right into that one: I definitely could have used a virtualenv for this. However I couldn't have used a virtualenv to update my system pip so it would support wheel installation. > Glad it worked in the end, anyway, and sorry if my instructions made it > harder than it needed to be. No they didn't. There was no point when I didn't know how to revert everything with 'rm -r'. > As regards distribute, I suspect that the reason you hit issues is that if > you have a setuptools that's older than 0.7 (or whatever the first merged > version was) then an upgrade can end up jumping through some hoops and going > through a "dummy" distribute version that's there to handle the > fork/re-merge somehow. I honestly don't know how it all works, I'm just > going off what I saw on some of the discussions on pypa-dev at the time. It > all sounded very clever to me, but a bit fragile. I'm a simple soul, and > prefer to just wipe it out and reinstall, so I zoned out after a while:-) I > doubt the details matter to you now, though... No they don't. But I'm with you on the uninstall/reinstall thing. That would be my recommendation to anyone who needs to upgrade. Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing from a wheel
On 21 August 2013 15:57, Daniel Holth wrote: > > A fresh virtualenv would have been the humane way to get a working > 'pip install wheel'. Good point. I think I learned an important point going through that upgrade mess though: uninstall/reinstall is safer than upgrade. > Wheel's built in installer isn't intended to replace or be better than > pip in any way. It's just for reference or bootstrapping. Fair enough. Can I suggest that it have a --version option (since it is traditional)? > FYI if you point pip directly at the .whl file you can omit --use-wheel. Okay I've just tried that and that's definitely the way I want to use it. So basically: $ python setup.py bdist_wheel # Makes wheels and $ pip install foo.whl # Installs wheels If someone wants to import the bdist_wheel command and use it outside of setuptools setup() (in the way that numpy does) where should they import it from? I'm thinking of something like this: https://github.com/numpy/numpy/blob/master/numpy/distutils/command/bdist_rpm.py Is the following appropriate? from wheel.bdist_wheel import bdist_wheel class mybdist_wheel(bdist_wheel): ... (the wheel API docs don't describe using bdist_wheel from Python code.) Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 08/21/2013 03:29 AM, Donald Stufft wrote: > Can you send me a list (or post them here) of what issues you've hit? > The biggest one i'm aware of is the scripts problem which is a > fundamental problem with the 1.0 Wheel (or rather that any library with > console entry points cannot be universal). Since you asked, I'll mention the two that I've hit (though I think you're also aware of these already): 1) Wheel's conversion of - to _ in version strings embedded in filenames, which breaks with setuptools precedent; see https://github.com/pypa/pip/issues/1150 and https://bitbucket.org/dholth/wheel/issue/78/wheel-rewrites-versions-preventing 2) Wheel's decision to follow distutils' documentation rather than distutils' behavior when it comes to the location for installing data_files with relative paths; see https://bitbucket.org/dholth/wheel/issue/80/wheel-does-not-install-data_files-in-site Carl signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 11:30 AM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> I think you're using a completely different definition of "reference >> implementation" than I've ever seen used. A reference implementation > > Quite possibly, but I feel justified in this case ... I'll say why below. > >> by definition cannot contain customizations or additions or extensions >> from the spec. The entire *point* of a reference implementation is to >> act as programatic reference to the spec. Something being the reference >> implementation does not speak to the quality of the implementation and >> as such it may not be the *best* implementation. > > The packaging PEPs don't do down to programmatic detail in their > specification. In the stricter definition of "reference implementation" (RI) > you're talking about, a spec comes with a reference implementation but also a > test suite (provided by the spec author) which conforming > implementations must pass. While this is de rigueur in the Java world, it's > not common in the Python world. Without such a test conformance suite, I > think it's reasonable to use the looser definition of RI that I did. > >> An example is the wsgiref from the standard library. Very few projects >> actively use wsgiref for much at all if they use it at all. However it's >> existence means that web servers like gunicorn, mod_wsgi etc can simply >> test against it instead of needing to test against every implementation >> of WSGI. > > While you can certainly use wsgiref for interoperability testing (much as I > use wheel's wheel implementation for interoperability testing with distlib) I > don't think WSGI implementations run a suite of tests provided in wsgiref > before claiming conformance. I'm happy to be corrected > if I'm wrong about that :-) I have no idea what the WSGI servers actually do, it was just an example. > > In my experience of implementing PEPs (282, 391, 397, 405) all > implementations have had test suites, but none have had the tests implemented > by any central authority. In discussions the relevant implementations have > been referred to as RIs and that's the way I'm using it now. None of those peps are defining a standard with a primary goal of being able to be completely replaced by a different module and still work. They are for adding a single implementation to the standard library. So it would make sense for those not to have a reference implementation because there are not any expected other implementations besides the one that got added to the stdlib. > >> Which implementation is used (and ultimately possibly enshrined in the >> standard library) is decided through merit. Which implementation is >> used as the reference implementation is typically decided by the standards >> body (in this case, distutils-sig or Nick or whoever). > > If Nick or whoever is planning to write a test suite which all > implementations must pass, great. But that would (I suppose) mean tying > things down to a specific Python interface at the module, class and function > level - not something I've seen imposed externally before. But in > the absence of such, I don't see any problem with my interpretation of RI, > since the stricter interpretation only makes sense in the presence of > accompanying tests generated by the spec originators. I would expect that the reference implementation would include acceptance tests that test the various specs yes. For Wheel this wouldn't (in my mind) be at the Python level (so it wouldn't encode a particular python level API) other than a very minimal interface (which may simply be a CLI script?) needed to actually power the testing but the primary use of the acceptance tests would be to verify that the implementation can create and install Wheel files to the spec. It's purpose would not be to set a specific Pythonic API as the spec because the spec is defining a file format not a Python API. Not having behavior defined by an implementation is very important to me, hence why I'm being such a stickler for wanting something that other tools can use to test against. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Donald Stufft stufft.io> writes: > I think you're using a completely different definition of "reference > implementation" than I've ever seen used. A reference implementation Quite possibly, but I feel justified in this case ... I'll say why below. > by definition cannot contain customizations or additions or extensions > from the spec. The entire *point* of a reference implementation is to > act as programatic reference to the spec. Something being the reference > implementation does not speak to the quality of the implementation and > as such it may not be the *best* implementation. The packaging PEPs don't do down to programmatic detail in their specification. In the stricter definition of "reference implementation" (RI) you're talking about, a spec comes with a reference implementation but also a test suite (provided by the spec author) which conforming implementations must pass. While this is de rigueur in the Java world, it's not common in the Python world. Without such a test conformance suite, I think it's reasonable to use the looser definition of RI that I did. > An example is the wsgiref from the standard library. Very few projects > actively use wsgiref for much at all if they use it at all. However it's > existence means that web servers like gunicorn, mod_wsgi etc can simply > test against it instead of needing to test against every implementation > of WSGI. While you can certainly use wsgiref for interoperability testing (much as I use wheel's wheel implementation for interoperability testing with distlib) I don't think WSGI implementations run a suite of tests provided in wsgiref before claiming conformance. I'm happy to be corrected if I'm wrong about that :-) In my experience of implementing PEPs (282, 391, 397, 405) all implementations have had test suites, but none have had the tests implemented by any central authority. In discussions the relevant implementations have been referred to as RIs and that's the way I'm using it now. > Which implementation is used (and ultimately possibly enshrined in the > standard library) is decided through merit. Which implementation is > used as the reference implementation is typically decided by the standards > body (in this case, distutils-sig or Nick or whoever). If Nick or whoever is planning to write a test suite which all implementations must pass, great. But that would (I suppose) mean tying things down to a specific Python interface at the module, class and function level - not something I've seen imposed externally before. But in the absence of such, I don't see any problem with my interpretation of RI, since the stricter interpretation only makes sense in the presence of accompanying tests generated by the spec originators. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing from a wheel
On 21 August 2013 15:48, Oscar Benjamin wrote: > > For people watching at home, upgrading pip really isn't this scary :-) > I'm > > just making it sound scary (a) because I don't know the precise upgrade > > instructions for setuptools and (b) because you need to do setuptools and > > pip separately, and use python -m pip on Windows to upgrade pip (neither > of > > which are immediately obvious). > > Is it perhaps safer to suggest the following? > a) uninstall pip/setuptools/distribute > b) run ez_setup.py > c) run get-pip.py > > That's what I just did and it worked fine. Perhaps there could be one > script that does steps a), b) and c) to smooth the path for anyone > upgrading? It probably is. I've heard concerns that people want to avoid suggesting manual uninstalls and having to download the setup scripts. But it seems simple enough to me. (What would I know, I just run virtualenv and leave it at that :-)) Glad it worked in the end, anyway, and sorry if my instructions made it harder than it needed to be. As regards distribute, I suspect that the reason you hit issues is that if you have a setuptools that's older than 0.7 (or whatever the first merged version was) then an upgrade can end up jumping through some hoops and going through a "dummy" distribute version that's there to handle the fork/re-merge somehow. I honestly don't know how it all works, I'm just going off what I saw on some of the discussions on pypa-dev at the time. It all sounded very clever to me, but a bit fragile. I'm a simple soul, and prefer to just wipe it out and reinstall, so I zoned out after a while:-) I doubt the details matter to you now, though... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing from a wheel
A fresh virtualenv would have been the humane way to get a working
'pip install wheel'.
Wheel's built in installer isn't intended to replace or be better than
pip in any way. It's just for reference or bootstrapping.
FYI if you point pip directly at the .whl file you can omit --use-wheel.
PS I didn't mean to release the wheel 0.21.0 .whl sans scripts. Sorry
Paul if that's caused confusion. It is just a bug in the released
archive.
On Wed, Aug 21, 2013 at 10:48 AM, Oscar Benjamin
wrote:
> On 21 August 2013 14:56, Paul Moore wrote:
>> On 21 August 2013 14:28, Oscar Benjamin wrote:
>>>
>>> So I tried updating everything e.g.:
>>>
>>> $ pip install -U wheel pip setuptools
>>
>> [lots omitted for brevity]
>>
>> Some thoughts.
>>
>> pip 1.3.1 predates pip's wheel support so you wouldn't have had pip install
>> --use-wheel there.
>>
>> The upgrade error may have been because pip install -U pip tries to install
>> a new pip.exe while pip.exe is in use. The error might not be too bad
>> (pip.exe doesn't actually need to change).
>
> Maybe, although the path
> c:\\docume~1\\enojb\\locals~1\\temp\\pip-6echt4-uninstall\\tools\\python27\\scripts\\pip.exe
> is not the location of the pip I used and is not on PATH.
>
>> For safety, "python -m pip install -U pip --force-reinstall" might be worth
>> doing.
>
> Okay done. Seems to work fine.
>
>> You quite probably shouldn't have upgraded setuptools like you did. It looks
>> like you had a pre-merge version, and upgrading across the distribute merge
>> appears to be fun (I have personally never encountered that particular
>> flavour of fun, but that's what I'm led to believe).
>
> This is not an old Python installation. I installed this as a clean
> installation to test the patches I uploaded for issue12641 2-3 months
> ago. I wouldn't have deliberately installed distribute (I know it's
> obsoleted) so I don't know how it got there.
>
>> For safety you should
>> check your site-packages for setuptools and distribute installations. Maybe
>> manually remove distribute if present,
>
> I got this far:
> $ rm -r /q/tools/Python27/Lib/site-packages/distribute-0.6.40-py2.7.egg/
>
> and then
> $ pip
> Traceback (most recent call last):
> File "q:\tools\Python27\Scripts\pip-script.py", line 5, in
> from pkg_resources import load_entry_point
> ImportError: No module named pkg_resources
>
>> and then "python -m pip install -U setuptools --force-reinstall"
>
> Alas, this one doesn't work any more either:
> $ python -m pip
> q:\tools\Python27\python.exe: No module named pkg_resources; 'pip' is
> a package and cannot be directly executed
>
>> (don't do a combined run of pip and setuptools
>> together, that's one of the scary failure modes IIRC).
>
> Okay so I manually deleted everything for
> pip/setuptools/distribute/easy_install from Scripts and site-packages
> and started again. Following the instructions for install pip and
> setuptools for Windows I downloaded and ran ez_setup.py followed by
> get-pip.py. Then I got this error:
>
> $ py -2.7 get-pip.py
> Downloading/unpacking pip
>
> pip can't proceed with requirement 'pip' due to a pre-existing build
> directory.
> location: c:\docume~1\enojb\locals~1\temp\pip-build-enojb\pip
> This is likely due to a previous installation that failed.
> pip is being responsible and not assuming it can delete this.
> Please delete it and try again.
>
> Cleaning up...
> Exception:
> Traceback (most recent call last):
> File
> "c:\docume~1\enojb\locals~1\temp\unpacker-c3e81q-scratchdir\pip\basecommand.py",
> line 134, in main
> status = self.run(options, args)
> File
> "c:\docume~1\enojb\locals~1\temp\unpacker-c3e81q-scratchdir\pip\commands\install.py",
> line 236, in run
> requirement_set.prepare_files(finder,
> force_root_egg_info=self.bundle, bundle=self.bundle)
> File
> "c:\docume~1\enojb\locals~1\temp\unpacker-c3e81q-scratchdir\pip\req.py",
> line 1071, in prepare_files
> raise e
> PreviousBuildDirError:
> pip can't proceed with requirement 'pip' due to a pre-existing build
> directory.
> location: c:\docume~1\enojb\locals~1\temp\pip-build-enojb\pip
> This is likely due to a previous installation that failed.
> pip is being responsible and not assuming it can delete this.
> Please delete it and try again.
>
> The path it refers to doesn't exist but deleting a similar directory
> gets it working:
>
> $ rm -r ~/Local\ Settings/temp/pip-6echt4-uninstall/
> enojb@ENM-OB:/q$ py -2.7 get-pip.py
> Downloading/unpacking pip
> Downloading pip-1.4.1.tar.gz (445kB): 445kB downloaded
> Running setup.py egg_info for package pip
>...
>
> Okay so now I'm back in business ('pip list' works etc.).
>
>> pip 1.4.1 should be able to pip uninstall a distribution installed from a
>> wheel (but TBH, I would have expected 1.3.1 to be able to, as well. The
>> installed data looked OK).
>
> Yes it can:
>
> $ pip uninstall spam
> Uninstalling spam:
> q:\tools\python27\lib\site-packages\spam-1.0.dist-info\description.rst
>
Re: [Distutils] Installing from a wheel
On 21 August 2013 14:56, Paul Moore wrote:
> On 21 August 2013 14:28, Oscar Benjamin wrote:
>>
>> So I tried updating everything e.g.:
>>
>> $ pip install -U wheel pip setuptools
>
> [lots omitted for brevity]
>
> Some thoughts.
>
> pip 1.3.1 predates pip's wheel support so you wouldn't have had pip install
> --use-wheel there.
>
> The upgrade error may have been because pip install -U pip tries to install
> a new pip.exe while pip.exe is in use. The error might not be too bad
> (pip.exe doesn't actually need to change).
Maybe, although the path
c:\\docume~1\\enojb\\locals~1\\temp\\pip-6echt4-uninstall\\tools\\python27\\scripts\\pip.exe
is not the location of the pip I used and is not on PATH.
> For safety, "python -m pip install -U pip --force-reinstall" might be worth
> doing.
Okay done. Seems to work fine.
> You quite probably shouldn't have upgraded setuptools like you did. It looks
> like you had a pre-merge version, and upgrading across the distribute merge
> appears to be fun (I have personally never encountered that particular
> flavour of fun, but that's what I'm led to believe).
This is not an old Python installation. I installed this as a clean
installation to test the patches I uploaded for issue12641 2-3 months
ago. I wouldn't have deliberately installed distribute (I know it's
obsoleted) so I don't know how it got there.
> For safety you should
> check your site-packages for setuptools and distribute installations. Maybe
> manually remove distribute if present,
I got this far:
$ rm -r /q/tools/Python27/Lib/site-packages/distribute-0.6.40-py2.7.egg/
and then
$ pip
Traceback (most recent call last):
File "q:\tools\Python27\Scripts\pip-script.py", line 5, in
from pkg_resources import load_entry_point
ImportError: No module named pkg_resources
> and then "python -m pip install -U setuptools --force-reinstall"
Alas, this one doesn't work any more either:
$ python -m pip
q:\tools\Python27\python.exe: No module named pkg_resources; 'pip' is
a package and cannot be directly executed
> (don't do a combined run of pip and setuptools
> together, that's one of the scary failure modes IIRC).
Okay so I manually deleted everything for
pip/setuptools/distribute/easy_install from Scripts and site-packages
and started again. Following the instructions for install pip and
setuptools for Windows I downloaded and ran ez_setup.py followed by
get-pip.py. Then I got this error:
$ py -2.7 get-pip.py
Downloading/unpacking pip
pip can't proceed with requirement 'pip' due to a pre-existing build directory.
location: c:\docume~1\enojb\locals~1\temp\pip-build-enojb\pip
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
Please delete it and try again.
Cleaning up...
Exception:
Traceback (most recent call last):
File
"c:\docume~1\enojb\locals~1\temp\unpacker-c3e81q-scratchdir\pip\basecommand.py",
line 134, in main
status = self.run(options, args)
File
"c:\docume~1\enojb\locals~1\temp\unpacker-c3e81q-scratchdir\pip\commands\install.py",
line 236, in run
requirement_set.prepare_files(finder,
force_root_egg_info=self.bundle, bundle=self.bundle)
File "c:\docume~1\enojb\locals~1\temp\unpacker-c3e81q-scratchdir\pip\req.py",
line 1071, in prepare_files
raise e
PreviousBuildDirError:
pip can't proceed with requirement 'pip' due to a pre-existing build directory.
location: c:\docume~1\enojb\locals~1\temp\pip-build-enojb\pip
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
Please delete it and try again.
The path it refers to doesn't exist but deleting a similar directory
gets it working:
$ rm -r ~/Local\ Settings/temp/pip-6echt4-uninstall/
enojb@ENM-OB:/q$ py -2.7 get-pip.py
Downloading/unpacking pip
Downloading pip-1.4.1.tar.gz (445kB): 445kB downloaded
Running setup.py egg_info for package pip
...
Okay so now I'm back in business ('pip list' works etc.).
> pip 1.4.1 should be able to pip uninstall a distribution installed from a
> wheel (but TBH, I would have expected 1.3.1 to be able to, as well. The
> installed data looked OK).
Yes it can:
$ pip uninstall spam
Uninstalling spam:
q:\tools\python27\lib\site-packages\spam-1.0.dist-info\description.rst
q:\tools\python27\lib\site-packages\spam-1.0.dist-info\metadata
q:\tools\python27\lib\site-packages\spam-1.0.dist-info\pydist.json
q:\tools\python27\lib\site-packages\spam-1.0.dist-info\record
q:\tools\python27\lib\site-packages\spam-1.0.dist-info\top_level.txt
q:\tools\python27\lib\site-packages\spam-1.0.dist-info\wheel
q:\tools\python27\lib\site-packages\spam.py
Proceed (y/n)? y
Successfully uninstalled spam
Also 'pip install --use-wheel' has a more intuitive interface than wheel:
$ pip install --use-wheel dist/spam-1.0-py27-none-any.whl
Unpacking .\dist\spam-1.0-py27-none-any.whl
Installing collected packages: spam
Successfully installed spam
Cleaning up...
> For pe
Re: [Distutils] Installing from a wheel
On 21 August 2013 14:28, Oscar Benjamin wrote: > So I tried updating everything e.g.: > > $ pip install -U wheel pip setuptools > [lots omitted for brevity] Some thoughts. pip 1.3.1 predates pip's wheel support so you wouldn't have had pip install --use-wheel there. The upgrade error may have been because pip install -U pip tries to install a new pip.exe while pip.exe is in use. The error might not be too bad (pip.exe doesn't actually need to change). For safety, "python -m pip install -U pip --force-reinstall" might be worth doing. You quite probably shouldn't have upgraded setuptools like you did. It looks like you had a pre-merge version, and upgrading across the distribute merge appears to be fun (I have personally never encountered that particular flavour of fun, but that's what I'm led to believe). For safety you should check your site-packages for setuptools and distribute installations. Maybe manually remove distribute if present, and then "python -m pip install -U setuptools --force-reinstall" (don't do a combined run of pip and setuptools together, that's one of the scary failure modes IIRC). pip 1.4.1 should be able to pip uninstall a distribution installed from a wheel (but TBH, I would have expected 1.3.1 to be able to, as well. The installed data looked OK). For people watching at home, upgrading pip really isn't this scary :-) I'm just making it sound scary (a) because I don't know the precise upgrade instructions for setuptools and (b) because you need to do setuptools and pip separately, and use python -m pip on Windows to upgrade pip (neither of which are immediately obvious). Let me know how you get on. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing from a wheel
On 21 August 2013 14:08, Paul Moore wrote:
> On 21 August 2013 13:59, Oscar Benjamin wrote:
>>
>> $ cat spam.py
>> # spam.py
>> print('running spam from:', __file__)
[snip]
>
> Looks good. You might want to add the (undocumented) universal flag to
> setup.cfg, as your wheel is Python only and works for Python 2 and 3, and so
> not version-specific.
Really I need to import print_function for that universality to be
true but we'll overlook that :)
> setup.cfg:
>
> [wheel]
> universal=1
Okay so I need setup.cfg as well as setup.py.
[snip]
>
> Looks good. I thought wheel install gave some progress output, but it's a
> long time since I used it and I may be misremembering. You can also use pip
> install --use-wheel if you prefer (assuming you have pip 1.4+)
Okay, that's good. I'd rather just use the pip command than use wheel directly.
>> So now how do I uninstall it?
>>
>> $ pip uninstall spam
>> Can't uninstall 'spam'. No files were found to uninstall.
>>
>> The wheel command doesn't seem to have an uninstall option either
>
> Odd. pip uninstall should work. Can you confirm your version of pip and
> wheel? And can you list the contents of the spam-1.0.dist-info directory in
> your site-packages?
$ pip --version
pip 1.3.1 from q:\tools\python27\lib\site-packages\pip-1.3.1-py2.7.egg
(python 2.7)
$ wheel --version # This gives a usage message
>>> import wheel
>>> wheel.__version__
'0.21.0'
$ ls /q/tools/Python27/Lib/site-packages/spam-1.0.dist-info/
DESCRIPTION.rst METADATA RECORD WHEEL pydist.json top_level.txt
$ cat /q/tools/Python27/Lib/site-packages/spam-1.0.dist-info/*
UNKNOWN
Metadata-Version: 2.0
Name: spam
Version: 1.0
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
UNKNOWN
spam-1.0.dist-info\DESCRIPTION.rst,sha256=OCTuuN6LcWulhHS3d5rfjdsQtW22n7HENFRh6jC6ego,10
spam-1.0.dist-info\METADATA,sha256=N7NDv-twCNGywvm1HXdz67MoFL4xIUoT5p39--tGGB8,179
spam-1.0.dist-info\WHEEL,sha256=ceN1GNMAiWCEADx3_5pdpmZwt4A_AtSxSxYSCyHhhPw,98
spam-1.0.dist-info\pydist.json,sha256=rptnmxTtRo0YZfBQZbIxMdHWDAg48f0UhCDmdymzHbk,174
spam-1.0.dist-info\top_level.txt,sha256=KE4wKczjrl7gsFhmEA4wAEY1n1OuTHf-azTAWqenLO4,5
spam.py,sha256=_5V9b8A2xHt-590km2JzJniHeWIiXbdU_wVHONhTzms,48
spam-1.0.dist-info/RECORD,,
Wheel-Version: 1.0
Generator: bdist_wheel (0.21.0)
Root-Is-Purelib: true
Tag: py27-none-any
{"document_names": {"description": "DESCRIPTION.rst"}, "name": "spam",
"metadata_version": "2.0", "generator": "bdist_wheel (0.21.0)",
"summary": "UNKNOWN", "version": "1.0"}spam
So I tried updating everything e.g.:
$ pip install -U wheel pip setuptools
Requirement already up-to-date: wheel in q:\tools\python27\lib\site-packages
Downloading/unpacking pip from
https://pypi.python.org/packages/source/p/pip/pip-1.4.1.tar.gz#md5=6afbb46aeb48abac658d4df742bff714
Downloading pip-1.4.1.tar.gz (445kB): 445kB downloaded
Running setup.py egg_info for package pip
warning: no files found matching '*.html' under directory 'docs'
warning: no previously-included files matching '*.rst' found under
directory 'docs\_build'
no previously-included directories found matching 'docs\_build\_sources'
Downloading/unpacking distribute from
https://pypi.python.org/packages/source/d/distribute/distribute-0.7.3.zip#md5=c6c59594a7b180af57af8a0cc0cf5b4a
Downloading distribute-0.7.3.zip (145kB): 145kB downloaded
Running setup.py egg_info for package distribute
Downloading/unpacking setuptools>=0.7 from
https://pypi.python.org/packages/source/s/setuptools/setuptools-1.0.tar.gz#md5=3d196ffb6e5e4425daddbb4fe42a4a74
(from distribute)
Downloading setuptools-1.0.tar.gz (679kB): 679kB downloaded
Running setup.py egg_info for package setuptools
Installing collected packages: pip, distribute, setuptools
Found existing installation: pip 1.3.1
Uninstalling pip:
Successfully uninstalled pip
Running setup.py install for pip
warning: no files found matching '*.html' under directory 'docs'
warning: no previously-included files matching '*.rst' found under
directory 'docs\_build'
no previously-included directories found matching 'docs\_build\_sources'
Installing pip-script.py script to q:\tools\Python27\Scripts
Installing pip.exe script to q:\tools\Python27\Scripts
Installing pip.exe.manifest script to q:\tools\Python27\Scripts
Installing pip-2.7-script.py script to q:\tools\Python27\Scripts
Installing pip-2.7.exe script to q:\tools\Python27\Scripts
Installing pip-2.7.exe.manifest script to q:\tools\Python27\Scripts
Exception:
Traceback (most recent call last):
File
"q:\tools\Python27\lib\site-packages\pip-1.3.1-py2.7.egg\pip\basecommand.py",
line 139, in main
File
"q:\tools\Python27\lib\site-packages\pip-1.3.1-py2.7.egg\pip\commands\install.py",
line 271, in run
File "q:\tools\Python27\lib\site-packages\pip-1.3.1-py2.7.egg\pip\req.py",
line 1193, in install
File "q:\tools\Python27\lib\site-packages\pip-1.3.1-py2.7.e
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 9:02 AM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> I think the way you view distlib and the way other are viewing distlib are >> different (and that's ok). We just need to know what distlib is so we can >> have reasonable expectations of it. What i'm getting from you is that, at >> least right now, distlib isn't what I thought it was and I (and others) >> should >> stop treating it as *the* reference implementation of the new standards. >> >> I'm not trying to stop you from innovating, I'm just trying to make sure > everyone >> has reasonable expectations all around. > > I don't see how anyone is treating it as a reference implementation, other > than just talking about it being such. I have had very good feedback from > one or two people who are using it, but for the most part I see no evidence > that people on this list are using it to the extent they would if they > really thought "this might be a good candidate for a reference > implementation - I see it's early days, and there's work to do, but it seems > usable, so let me check it out and see that my use cases are covered, and if > anything's been overlooked". > > In my view, nothing deserves to be a considered a "reference implementation" > other than through merit, or perhaps by "being number one in a field of > one". Merit isn't earned unless the software is used and refined based on > real-world experience. The time to try distlib is now (not in production > environments, obviously), to allow any problems with it to be identified > early in its life. That would be more helpful than any sniping from the > sidelines about whether it does more than what it needs to for PEP > conformance. > > Regards, > > Vinay Sajip > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig I think you're using a completely different definition of "reference implementation" than I've ever seen used. A reference implementation by definition cannot contain customizations or additions or extensions from the spec. The entire *point* of a reference implementation is to act as programatic reference to the spec. Something being the reference implementation does not speak to the quality of the implementation and as such it may not be the *best* implementation. It becomes extremely useful however when you want to test conformance against the spec because it gives you a baseline with which to test against. Instead of needing to test against N different implementations people wanting to work with Wheel would only need to test against the reference implementation, and if that works then they can assume that their code will work against any other implementation of Wheel that properly implements the standard. If you don't have a reference implementation then people need to interpret the standards on their own and hopefully get it right. An example is the wsgiref from the standard library. Very few projects actively use wsgiref for much at all if they use it at all. However it's existence means that web servers like gunicorn, mod_wsgi etc can simply test against it instead of needing to test against every implementation of WSGI. Which implementation is used (and ultimately possibly enshrined in the standard library) is decided through merit. Which implementation is used as the reference implementation is typically decided by the standards body (in this case, distutils-sig or Nick or whoever). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 21, 2013 at 8:01 AM, Paul Moore wrote: > On 21 August 2013 12:22, Vinay Sajip wrote: >> >> Paul Moore gmail.com> writes: >> >> > Another one IIRC was that distlib didn't put entry-points.txt in the >> > .dist-info directory in the wheel (which breaks entry points). I think >> > that's fixed now (and again, the Wheel spec is silent on what is correct >> > behaviour here). >> >> Right. The recent PEP 426 updates with the "commands" key supersedes the >> entry points stuff, so I'm not sure what exactly should be done here. The >> distlib code uses the latest PEP information. Since wheel is moving to >> pydist.json, I assume that (when it gets around to it) it will have the >> relevant scripts info in pydist.json, so I haven't implemented using >> scripts >> declared in entry_points.txt in distlib for that reason. > > > OK, I see what you're saying here. But the Wheel 1.0 spec says that metadata > is in the METADATA file (and comes from PKG-INFO). So my reading of that > means that Metadata 1.x will remain valid for the foreseeable future (sure, > Metadata 2.0 may become acceptable *as well* but the point of having a Wheel > 1.0 spec is that we won't stop supporting it for some time yet). So you need > to have a pre-2.0 solution in place, and while entry-points.txt isn't > explicitly stated in the wheel PEP, it's the obvious equivalent. > > If you want to say distlib won't support pre-Metadata 2.0 specifications of > script metadata, then that's your choice - it's not contrary to the > standards but I'd view it as a quality of implementation choice. I view the > underspecification in the Wheel 1.0 spec as similarly a quality of detail > issue, and I'd expect to fix it in either an update to Wheel 1.0, or a Wheel > 1.1 which does not make the jump to pure Metadata 2.0. > > Paul The wheel spec should probably say that its .dist-info directory should conform to a .dist-info PEP rather than saying it contains any particular files like METADATA. The only files that belong to wheel itself are the manifest and the WHEEL file that contains the version of the wheel format itself. Before the (useful) scripts wrapper feature, the wheel installer did not need to look at the PEP 426 or setuptools metadata at all. You can install them with unzip. The wheel command line tool lately has a "wheel install-scripts [package]" command that uses setuptools to rewrite entry-points script wrappers for an installed package. It is a proof of concept but it comes in handy. When pip is updated to be able to generate script wrappers at install time then a simultaneously released bdist_wheel will be able to update its default behavior to defer script generation and produce the newer metadata. In the meantime scripts are imperfect. Egg's controversial features like being a valid sys.path entry were not really left out of wheel. The format could have been a .tar file after all. Instead, wheel's documentation emphasizes what turned out to be the least problematic use pattern as an installation format. Egg got there from the other direction starting as a plugin container. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing from a wheel
On 21 August 2013 13:59, Oscar Benjamin wrote:
> This is the first time that I've tested using wheels and I have a
> couple of questions.
>
> Here's what I did (is this right?):
>
> $ cat spam.py
> # spam.py
> print('running spam from:', __file__)
> $ cat setup.py
> from setuptools import setup
>
> setup(name='spam',
> version='1.0',
> py_modules=['spam'])
>
> $ python setup.py bdist_wheel
> running bdist_wheel
> ...
> creating build\bdist.win32\wheel\spam-1.0.dist-info\WHEEL
> $ ls
> build dist setup.py spam.egg-info spam.py
> $ ls dist/
> spam-1.0-py27-none-any.whl
>
> Okay, so far so good. I have the wheel and everything makes sense.
Looks good. You might want to add the (undocumented) universal flag to
setup.cfg, as your wheel is Python only and works for Python 2 and 3, and
so not version-specific.
setup.cfg:
[wheel]
universal=1
> Now
> I want to test installing it:
>
> $ wheel install --wheel-dir=./dist/ spam
>
> The line above gives no output. I expect something like 'installing
> spam... installed.'. It also ran so quickly that I thought that
> nothing had happened.
>
> A quick check reveals that the module was installed:
>
> $ cd ~
> $ python -m spam
> ('running spam from:', 'q:\\tools\\Python27\\lib\\site-packages\\spam.py')
> $ pip list | grep spam
> spam (1.0)
>
Looks good. I thought wheel install gave some progress output, but it's a
long time since I used it and I may be misremembering. You can also use pip
install --use-wheel if you prefer (assuming you have pip 1.4+)
So now how do I uninstall it?
>
> $ pip uninstall spam
> Can't uninstall 'spam'. No files were found to uninstall.
>
> The wheel command doesn't seem to have an uninstall option either
>
Odd. pip uninstall should work. Can you confirm your version of pip and
wheel? And can you list the contents of the spam-1.0.dist-info directory in
your site-packages?
Paul
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Donald Stufft stufft.io> writes: > I think the way you view distlib and the way other are viewing distlib are > different (and that's ok). We just need to know what distlib is so we can > have reasonable expectations of it. What i'm getting from you is that, at > least right now, distlib isn't what I thought it was and I (and others) should > stop treating it as *the* reference implementation of the new standards. > > I'm not trying to stop you from innovating, I'm just trying to make sure everyone > has reasonable expectations all around. I don't see how anyone is treating it as a reference implementation, other than just talking about it being such. I have had very good feedback from one or two people who are using it, but for the most part I see no evidence that people on this list are using it to the extent they would if they really thought "this might be a good candidate for a reference implementation - I see it's early days, and there's work to do, but it seems usable, so let me check it out and see that my use cases are covered, and if anything's been overlooked". In my view, nothing deserves to be a considered a "reference implementation" other than through merit, or perhaps by "being number one in a field of one". Merit isn't earned unless the software is used and refined based on real-world experience. The time to try distlib is now (not in production environments, obviously), to allow any problems with it to be identified early in its life. That would be more helpful than any sniping from the sidelines about whether it does more than what it needs to for PEP conformance. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Installing from a wheel
This is the first time that I've tested using wheels and I have a
couple of questions.
Here's what I did (is this right?):
$ cat spam.py
# spam.py
print('running spam from:', __file__)
$ cat setup.py
from setuptools import setup
setup(name='spam',
version='1.0',
py_modules=['spam'])
$ python setup.py bdist_wheel
running bdist_wheel
...
creating build\bdist.win32\wheel\spam-1.0.dist-info\WHEEL
$ ls
build dist setup.py spam.egg-info spam.py
$ ls dist/
spam-1.0-py27-none-any.whl
Okay, so far so good. I have the wheel and everything makes sense. Now
I want to test installing it:
$ wheel install --wheel-dir=./dist/ spam
The line above gives no output. I expect something like 'installing
spam... installed.'. It also ran so quickly that I thought that
nothing had happened.
A quick check reveals that the module was installed:
$ cd ~
$ python -m spam
('running spam from:', 'q:\\tools\\Python27\\lib\\site-packages\\spam.py')
$ pip list | grep spam
spam (1.0)
So now how do I uninstall it?
$ pip uninstall spam
Can't uninstall 'spam'. No files were found to uninstall.
The wheel command doesn't seem to have an uninstall option either.
Oscar
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
Paul Moore gmail.com> writes: > BUT, this means that there is no spec of the current behaviour, and no implementation of the Wheel 1.0 spec anywhere. [snip] > or the wheel spec needs a review reasonably soon. I think it's this. I'm not sure to what extent wheels are being used in anger out there, but it would make sense to review the spec in light of PEP 426 developments and release a 1.1. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
On 21 August 2013 13:01, Paul Moore wrote: > If you want to say distlib won't support pre-Metadata 2.0 specifications > of script metadata, then that's your choice - it's not contrary to the > standards but I'd view it as a quality of implementation choice. I view the > underspecification in the Wheel 1.0 spec as similarly a quality of detail > issue, and I'd expect to fix it in either an update to Wheel 1.0, or a > Wheel 1.1 which does not make the jump to pure Metadata 2.0. Ignore this (to an extent). I've just seen your comments on the distlib issue. For those not following along, both wheel and distlib currently process a pydist.json file in the wheel dist-info directory. The wheel project doesn't put entry-points.txt data in there yet, but distlib expects to find script metadata there. So the incompatibility here is merely one of two implementations being out of sync - no big deal. BUT, this means that there is no spec of the current behaviour, and no implementation of the Wheel 1.0 spec anywhere. Either there is movement behind the scenes that I'm not aware of, and I'm jumping the gun in trying to make sense of the current state of play, or the wheel spec needs a review reasonably soon. Whichever is the case, I don't think that my current investigations are adding anything productive here. So I'm going to take a step back from all of this for a week or two and see what develops. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 12:22, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > > > Another one IIRC was that distlib didn't put entry-points.txt in the > > .dist-info directory in the wheel (which breaks entry points). I think > > that's fixed now (and again, the Wheel spec is silent on what is correct > > behaviour here). > > Right. The recent PEP 426 updates with the "commands" key supersedes the > entry points stuff, so I'm not sure what exactly should be done here. The > distlib code uses the latest PEP information. Since wheel is moving to > pydist.json, I assume that (when it gets around to it) it will have the > relevant scripts info in pydist.json, so I haven't implemented using > scripts > declared in entry_points.txt in distlib for that reason. OK, I see what you're saying here. But the Wheel 1.0 spec says that metadata is in the METADATA file (and comes from PKG-INFO). So my reading of that means that Metadata 1.x will remain valid for the foreseeable future (sure, Metadata 2.0 may become acceptable *as well* but the point of having a Wheel 1.0 spec is that we won't stop supporting it for some time yet). So you need to have a pre-2.0 solution in place, and while entry-points.txt isn't explicitly stated in the wheel PEP, it's the obvious equivalent. If you want to say distlib won't support pre-Metadata 2.0 specifications of script metadata, then that's your choice - it's not contrary to the standards but I'd view it as a quality of implementation choice. I view the underspecification in the Wheel 1.0 spec as similarly a quality of detail issue, and I'd expect to fix it in either an update to Wheel 1.0, or a Wheel 1.1 which does not make the jump to pure Metadata 2.0. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Paul Moore gmail.com> writes: > Another one IIRC was that distlib didn't put entry-points.txt in the > .dist-info directory in the wheel (which breaks entry points). I think > that's fixed now (and again, the Wheel spec is silent on what is correct > behaviour here). Right. The recent PEP 426 updates with the "commands" key supersedes the entry points stuff, so I'm not sure what exactly should be done here. The distlib code uses the latest PEP information. Since wheel is moving to pydist.json, I assume that (when it gets around to it) it will have the relevant scripts info in pydist.json, so I haven't implemented using scripts declared in entry_points.txt in distlib for that reason. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 11:56, Vinay Sajip wrote: > > But I think we have a reasonable consensus on how scripts should work, > > Do we? To the level of "wheels builders should write metadata that defines the scripts and wheel installers should generate the necessary wrappers" then yes. We may or may not (depending on your viewpoint) have agreement on the format of that metadata. Personally, I think that Metadata 2.0 where the wheel uses that format, and entry-points.txt for older versions, is the only realistic option here. I would like to see some clarity on the status of the wheel 1.0 spec. Where there are areas like this where the spec is either silent or missing sufficient detail to allow us to implement a common approach, should we be updating the 1.0 spec or should we be creating a 1.1 spec (I'd prefer the former)? Who does those updates? Must they go through Daniel as the author (who's been quite quiet, possibly he's hiding somewhere rather than being drawn into the fray :-))? > Someone might tell me tomorrow that the "cool" one-executable > solution I've recently implemented as an option on Windows (append shebang > / > archive to a stock executable) is an offence against all that's holy ;-) > That won't be me. I'm more interested in discussions about interoperability issues than about how the functionality gets implemented. > had a standard. I'm pretty unhappy that now we do have a standard, we > > still have situations where a wheel generated by one tool can have > > problems when installed with another - try "pip install wheel > > --use-wheel" on Windows to see what I mean, the exe wrappers are missing > > (this uses the wheel from PyPI, not a home-built one). > > If distlib is proving to be a problem either as a producer or consumer of > wheels, please raise issues. > I don't believe that distlib creates scripts based on entry-points.txt for pre-2.0 metadata. I have raised an issue for that. That's the only significant issue I have with distlib from the POV of what it can solve. There may be other interoperability issues where other tools can't consume things that distlib produces - I haven't tested anything like all of the combinations. We currently have 2 producers and 3 consumers. That's already 6 interactions to test. And if there *is* an issue, it's not always at all clear who to report the problem to. I'm not complaining about *any* of the implementations here. And I apologise if you in particular feel "picked on" (there's probably a distutils-sig "victim of the week" rota somewhere for this ;-)) My real concern is that we're drifting away from standards-based design towards implementation-based. And I think that updating the wheel PEP to be tighter on some of the details is what's really needed (hey, look, it's Daniel's turn to be picked on! :-)) I'd be willing to author an updated version of the wheel spec if all of the interested parties are OK with that (in particular, Nick and Daniel). If nothing else, that means that everyone can pick on me so I wouldn't feel quite so much like a troublemaker :-) Paul PS I really, really don't want anyone here to feel like anything they do is not valued. This is about tidying up the documentation, not about stifling anyone's enthusiasm or blocking any experimentation. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 6:56 AM, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> My problem is that as someone who wants to implement code that uses the >> new features like wheels, I want a usable reference implementation that >> covers the (agreed) standards. I don't particularly want my application >> to incorporate support for extensions to the standard, nor do I want to >> have to implement my own support all the time. > > The wheel implementation in distlib conforms to the existing PEPs, as far as > I know. It covers the agreed standards and, as far as I know, in a > reasonable way. Why should you care if there are additional methods in a > Wheel class, if you never call on them? It seems a step too far for a spec > of this type to dictate precisely what specific methods should or should not > be present in a class. If you find that distlib's behaviour somehow violates > the agreed standards, please tell me how and I'll fix it. I think I've been > pretty responsive on issues raised, but please don't raise vague fears if > they have no basis in fact. The spec does not and will not dictate what particular feature any one implementation has or does not have other than the base set outlined in the spec. > >> In particular, at the present time there are two tools that can generate >> wheels (bdist_wheel and distlib) and 3 that can install them (wheel, >> distlib and pip). They have subtly different behaviours outside of the >> standard definitions, which means that they are not completely >> interoperable. I am not happy at all about that - and if that counts as >> "being against innovation", then I'm afraid that yes, I am... (I don't >> think it does, by the way, but you may differ). > > As I understand it, any interoperability issues are an effect of everything > being a work in progress and some implementations lagging behind others in > terms of PEP compliance. I can't speak for Daniel, but I've certainly made > efforts to ensure that distlib can work with wheel-produced wheels, even > where it means using old METADATA and entry_points.txt formats. I find it > always helps to focus on specifics - since the implementations can't be > identical, what specifically are the differences that make you unhappy, and > why do they make you unhappy? If some of those can be laid at distlib's > door, fine - I think I'm generally quite responsive to specific issues raised. > > Also, note that some features in an implementation might be useful to some > people, even if not enshrined in a PEP. For example, distlib's wheel > installation provides an option to install only the site-packages parts - no > scripts or headers - a feature which you specifically requested, and which I > readily added (as I can see its usefulness) without playing a "it's not in > the PEP" card. Is that a "harmless innovation", or a "dangerous deviation > from the standard"? Nobody is saying that the only useful features are the ones enshrined in a PEP. > >> At the moment the wheel PEP is lagging a little behind some of the >> ongoing discussions, in particular in terms of script generation. That's >> fine, it's a work in progress. I hope it will be updated soon so that the >> spec matches what's been agreed. > > Well, let's not forget that distlib is a work in progress, too. > >> But I think we have a reasonable consensus on how scripts should work, > > Do we? Someone might tell me tomorrow that the "cool" one-executable > solution I've recently implemented as an option on Windows (append shebang / > archive to a stock executable) is an offence against all that's holy ;-) > >> had a standard. I'm pretty unhappy that now we do have a standard, we >> still have situations where a wheel generated by one tool can have >> problems when installed with another - try "pip install wheel >> --use-wheel" on Windows to see what I mean, the exe wrappers are missing >> (this uses the wheel from PyPI, not a home-built one). > > If distlib is proving to be a problem either as a producer or consumer of > wheels, please raise issues. > >> OK, so here's a concrete question for distutils-sig. If I want to use >> wheels in my app (built them, install them, whatever) what should I use >> as my "reference implementation". I don't want to implement the code >> myself, I just want to produce lowest-common-denominator wheels that can >> be used anywhere, and consume wheels that conform to the spec correctly. >> This is not a hypothetical question - in the first instance I'm looking >> to add support for loading setuptools/pip from wheels in virtualenv, and >> I need to know what code to bundle to make that happen. > > Some will say "wheel", even though it doesn't fully implement the spec, > apparently because the wheel mount code (which,naturally, doesn't do > anything unless it's called) is offensive to the point that some sort of > fork is warranted ;-) It has nothing to do with the Wheel mount code in particular. A reference implantation i
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 21 August 2013 11:47, Donald Stufft wrote: > Metadata 2.0 includes the ability to have a post install script, but Wheel > is not yet using Metadata 2.0 (and it's not yet finalized). But when Metadata 2.0 support is available, what you (Oscar) suggest does sound like a reasonable approach. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Paul Moore gmail.com> writes: > My problem is that as someone who wants to implement code that uses the > new features like wheels, I want a usable reference implementation that > covers the (agreed) standards. I don't particularly want my application > to incorporate support for extensions to the standard, nor do I want to > have to implement my own support all the time. The wheel implementation in distlib conforms to the existing PEPs, as far as I know. It covers the agreed standards and, as far as I know, in a reasonable way. Why should you care if there are additional methods in a Wheel class, if you never call on them? It seems a step too far for a spec of this type to dictate precisely what specific methods should or should not be present in a class. If you find that distlib's behaviour somehow violates the agreed standards, please tell me how and I'll fix it. I think I've been pretty responsive on issues raised, but please don't raise vague fears if they have no basis in fact. > In particular, at the present time there are two tools that can generate > wheels (bdist_wheel and distlib) and 3 that can install them (wheel, > distlib and pip). They have subtly different behaviours outside of the > standard definitions, which means that they are not completely > interoperable. I am not happy at all about that - and if that counts as > "being against innovation", then I'm afraid that yes, I am... (I don't > think it does, by the way, but you may differ). As I understand it, any interoperability issues are an effect of everything being a work in progress and some implementations lagging behind others in terms of PEP compliance. I can't speak for Daniel, but I've certainly made efforts to ensure that distlib can work with wheel-produced wheels, even where it means using old METADATA and entry_points.txt formats. I find it always helps to focus on specifics - since the implementations can't be identical, what specifically are the differences that make you unhappy, and why do they make you unhappy? If some of those can be laid at distlib's door, fine - I think I'm generally quite responsive to specific issues raised. Also, note that some features in an implementation might be useful to some people, even if not enshrined in a PEP. For example, distlib's wheel installation provides an option to install only the site-packages parts - no scripts or headers - a feature which you specifically requested, and which I readily added (as I can see its usefulness) without playing a "it's not in the PEP" card. Is that a "harmless innovation", or a "dangerous deviation from the standard"? > At the moment the wheel PEP is lagging a little behind some of the > ongoing discussions, in particular in terms of script generation. That's > fine, it's a work in progress. I hope it will be updated soon so that the > spec matches what's been agreed. Well, let's not forget that distlib is a work in progress, too. > But I think we have a reasonable consensus on how scripts should work, Do we? Someone might tell me tomorrow that the "cool" one-executable solution I've recently implemented as an option on Windows (append shebang / archive to a stock executable) is an offence against all that's holy ;-) > had a standard. I'm pretty unhappy that now we do have a standard, we > still have situations where a wheel generated by one tool can have > problems when installed with another - try "pip install wheel > --use-wheel" on Windows to see what I mean, the exe wrappers are missing > (this uses the wheel from PyPI, not a home-built one). If distlib is proving to be a problem either as a producer or consumer of wheels, please raise issues. > OK, so here's a concrete question for distutils-sig. If I want to use > wheels in my app (built them, install them, whatever) what should I use > as my "reference implementation". I don't want to implement the code > myself, I just want to produce lowest-common-denominator wheels that can > be used anywhere, and consume wheels that conform to the spec correctly. > This is not a hypothetical question - in the first instance I'm looking > to add support for loading setuptools/pip from wheels in virtualenv, and > I need to know what code to bundle to make that happen. Some will say "wheel", even though it doesn't fully implement the spec, apparently because the wheel mount code (which,naturally, doesn't do anything unless it's called) is offensive to the point that some sort of fork is warranted ;-) Seriously - I already install setuptools and pip from wheels into venvs using distil (which of course uses distlib) - have you tried it? You're welcome to try with wheels built using any project, and if you run into any issues I will do my best to fix them. Have you had a better offer? :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On Aug 21, 2013, at 6:46 AM, Oscar Benjamin wrote: > On 21 August 2013 11:39, Paul Moore wrote: >> On 21 August 2013 11:29, Oscar Benjamin wrote: >>> >>> I may have misunderstood it but looking at this >>> >>> https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147 >>> I think that the installer ships variants for each architecture and >>> decides at install time which to place on the target system. If that's >>> the case then would it be possible for a wheel to ship all variants so >>> that a post-install script could sort it out (rename/delete) after the >>> wheel is installed? >> >> Wheel 1.0 does not have the ability to bundle multiple versions (and I don't >> think tags are fine-grained enough to cover the differences numpy need, >> which are at the "do you have the SSE instruction set?" level AIUI). >> Multi-version wheels are a possible future extension, but I don't know if >> anyone has thought about fine-grained tags. > > No, but the wheel could do like the current numpy installer and ship >_numpy.pyd.nosse >_numpy.pyd.sse1 >_numpy.pyd.sse2 >_numpy.pyd.sse3 > as platlib files and then a post-install script can check for SSE > support, rename the appropriate file to _numpy.pyd and delete the > other _numpy.pyd.* files. > >> This is precisely the sort of input that the numpy people could provide to >> make sure that the wheel design covers their needs. > > I'm I right in guessing (since the question keeps being evaded :) ) > that a post-install script is not possible with pip+wheel+PyPI?. > > > Oscar > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Metadata 2.0 includes the ability to have a post install script, but Wheel is not yet using Metadata 2.0 (and it's not yet finalized). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 21 August 2013 11:39, Paul Moore wrote: > On 21 August 2013 11:29, Oscar Benjamin wrote: >> >> I may have misunderstood it but looking at this >> >> https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147 >> I think that the installer ships variants for each architecture and >> decides at install time which to place on the target system. If that's >> the case then would it be possible for a wheel to ship all variants so >> that a post-install script could sort it out (rename/delete) after the >> wheel is installed? > > Wheel 1.0 does not have the ability to bundle multiple versions (and I don't > think tags are fine-grained enough to cover the differences numpy need, > which are at the "do you have the SSE instruction set?" level AIUI). > Multi-version wheels are a possible future extension, but I don't know if > anyone has thought about fine-grained tags. No, but the wheel could do like the current numpy installer and ship _numpy.pyd.nosse _numpy.pyd.sse1 _numpy.pyd.sse2 _numpy.pyd.sse3 as platlib files and then a post-install script can check for SSE support, rename the appropriate file to _numpy.pyd and delete the other _numpy.pyd.* files. > This is precisely the sort of input that the numpy people could provide to > make sure that the wheel design covers their needs. I'm I right in guessing (since the question keeps being evaded :) ) that a post-install script is not possible with pip+wheel+PyPI?. Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 6:36 AM, Paul Moore wrote: > On 21 August 2013 11:30, Donald Stufft wrote: > On Aug 21, 2013, at 6:29 AM, Donald Stufft wrote: > >> introspecting the author email address > > Of course I wrote that and then did summary because the location of the > author email address changed between Metadata 1.x and 2.x and I didn't feel > like looking up the exact difference. > > :-) > > So would you see the API for author being > > if meta.version < (2, 0): > author = meta["whatever_it_is_in_1.x"] > else: > author = meta["whatever_it_is_in_2.x"] > > (i.e., the metadata version would be available)? > > Sounds like a sensible approach. > > Paul. Yes, and possibly some sort of compat shim could be made to smooth over the differences if it was felt to be warranted. But in general I'm very much in favor of composed APIs instead of having things like WheelMetadata, Sdist2Metadata, InstalledDBMetadata etc. Less code and easier to switch out pieces (e.g. in your own code you wouldn't need to change the metadata handling based on if it was a Wheel or a Sdist2, just load the appropriate file type and pass the mapping into the Metadata api). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 21 August 2013 11:29, Oscar Benjamin wrote: > I may have misunderstood it but looking at this > > https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147 > I think that the installer ships variants for each architecture and > decides at install time which to place on the target system. If that's > the case then would it be possible for a wheel to ship all variants so > that a post-install script could sort it out (rename/delete) after the > wheel is installed? > Wheel 1.0 does not have the ability to bundle multiple versions (and I don't think tags are fine-grained enough to cover the differences numpy need, which are at the "do you have the SSE instruction set?" level AIUI). Multi-version wheels are a possible future extension, but I don't know if anyone has thought about fine-grained tags. This is precisely the sort of input that the numpy people could provide to make sure that the wheel design covers their needs. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 11:30, Donald Stufft wrote: > On Aug 21, 2013, at 6:29 AM, Donald Stufft wrote: > > introspecting the author email address > > > Of course I wrote that and then did summary because the location of the > author email address changed between Metadata 1.x and 2.x and I didn't feel > like looking up the exact difference. > :-) So would you see the API for author being if meta.version < (2, 0): author = meta["whatever_it_is_in_1.x"] else: author = meta["whatever_it_is_in_2.x"] (i.e., the metadata version would be available)? Sounds like a sensible approach. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 6:29 AM, Donald Stufft wrote: > introspecting the author email address Of course I wrote that and then did summary because the location of the author email address changed between Metadata 1.x and 2.x and I didn't feel like looking up the exact difference. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 6:17 AM, Paul Moore wrote:
> On 21 August 2013 10:48, Donald Stufft wrote:
> I think Wheel files are (and should be) independent of the particular
> metadata version used. That file should contain the required information in
> order to know what version of the metadata is included with the Wheel. This
> means that as metadata evolves Wheels can just start using the new meta data
> version without requiring an update to the spec.
>
> That implies that any wheel reference implementation needs to expose APIs for
> reading and writing the metadata to/from the wheel. I don't have a problem
> with that, but I don't think the existing implementations do[1]... (And it
> could be a bit of a beast to design such an interface in a sufficiently
> future-proof manner, unless we also standardise a metadata object type...)
> Specifically, if I have a wheel and want to introspect it to find out the
> author email address, how do I do this? ("Doing so is not supported" is a
> valid answer, of course, but should also be documented...)
>
> Paul
>
> [1] Just checked. Distlib doesn't. Wheel doesn't (and wheel doesn't even have
> a fully documented API). Pip's wheel support is purely internal so doesn't
> count.
Yes I believe there should be an API for reading/writing to the dist-info
directory and that's how a Wheel API should handle exposing that. It means you
can compose api's so you have 1 API for reading/writing metadata which can be
used for Wheels, Sdist 2.0, The on disk installed database format etc.
Quick off the cuff design of an API for introspecting the author email address
(Please realize this is totally off the top of my head and has not been thought
through or played with or explored or anything that would make it reasonable to
actually expect it to be sane for more than this simple example).
>>> from ref import Metadata
>>> from ref import Wheel, Sdist2
>>> whl = Wheel.from_file("something-something.whl")
>>> print(whl.dist_info)
{"METADATA": "…", "entry_points.txt": "…"}
>>> meta = Metadata.from_mapping(whl.dist_info)
>>> print(meta["summary"])
"A Something or Other Library"
>>> sdist2 = Sdist2.from_file("something-something.sdist2")
>>> print(sdist2.dist_info)
{"pydist.json": "…", "README": "…"}
>>> meta = Metadata.from_mapping(sdist2.dist_info)
>>> print(meta["summary"])
"A something or Other Library"
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 21 August 2013 08:04, Vinay Sajip wrote: > Oscar Benjamin gmail.com> writes: > >> I think that they are responsible for installing the f2py script in >> each of my Scripts directories. I never use this script and I don't >> know what numpy wants with it (my understanding is that the Fortran >> parts of numpy were all shifted over to scipy). > > IIUC, if a third-party extension wants to use Fortran, the build process > converts it using f2py in to a Python-importable extension. It may be a > feature for distributions that use numpy, even if numpy doesn't use Fortran > itself. Okay, that makes sense. I'm sure that's not a big problem. It won't work very well on Windows (the case where wheels are really needed) anyway since it doesn't have a wrapper script and won't get picked up by make etc. >> > 2. Tags (not in general, but AIUI numpy distribute a fancy installer that >> > decides what compiled code to use depending on whether you have certain CPU >> > features - they may want to retain that, and to do so may prefer to have >> > more fine-grained tags, which in turn may or may not be possible to >> > support). I don't think that's a critical issue though. >> >> I guess this is what you mean: >> https://github.com/numpy/numpy/blob/master/tools/win32build/cpuid/test.c >> >> Is there no way for them to run a post-install script when pip >> installing wheels from PyPI? > > I'm not sure that would be enough. The numpy installation checks for various > features available at build time, and then writes numpy source code which is > then installed. When building and installing on the same machine, perhaps no > problem - but there could be problems when installation happens on a > different machine, since the sources written to the wheel at build time > would encode information about the build environment which may not be valid > in the installation environment. > > ISTM for numpy to work with wheels, all of this logic would need to move > from build time to run time, but I don't know how pervasive the > source-writing approach is and how much work would be entailed in switching > over to run-time adaptation to the environment. I may have misunderstood it but looking at this https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147 I think that the installer ships variants for each architecture and decides at install time which to place on the target system. If that's the case then would it be possible for a wheel to ship all variants so that a post-install script could sort it out (rename/delete) after the wheel is installed? Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 10:48, Donald Stufft wrote:
> I think Wheel files are (and should be) independent of the particular
> metadata version used. That file should contain the required information in
> order to know what version of the metadata is included with the Wheel. This
> means that as metadata evolves Wheels can just start using the new meta
> data version without requiring an update to the spec.
That implies that any wheel reference implementation needs to expose APIs
for reading and writing the metadata to/from the wheel. I don't have a
problem with that, but I don't think the existing implementations do[1]...
(And it could be a bit of a beast to design such an interface in a
sufficiently future-proof manner, unless we also standardise a metadata
object type...) Specifically, if I have a wheel and want to introspect it
to find out the author email address, how do I do this? ("Doing so is not
supported" is a valid answer, of course, but should also be documented...)
Paul
[1] Just checked. Distlib doesn't. Wheel doesn't (and wheel doesn't even
have a fully documented API). Pip's wheel support is purely internal so
doesn't count.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 5:46 AM, Paul Moore wrote: > On 21 August 2013 10:29, Donald Stufft wrote: > Can you send me a list (or post them here) of what issues you've hit? The > biggest one i'm aware of is the scripts problem which is a fundamental > problem with the 1.0 Wheel (or rather that any library with console entry > points cannot be universal). > > The scripts one is the key one (and yes, that needs to be fixed by updating > the spec to confirm the consensus and then updating the tools to match). +1 > > Another one IIRC was that distlib didn't put entry-points.txt in the > .dist-info directory in the wheel (which breaks entry points). I think that's > fixed now (and again, the Wheel spec is silent on what is correct behaviour > here). Ah yes, This one is a bit harder because it's a non standard file added by setuptools, but yea it probably should be adding that file. > > I'll try to dig out the others, but they weren't anywhere near as major (some > were also script-related such as inconsistent rewriting of shebang lines, > which are superseded by the current state of play on scripts anyway). > Generally, though, they were all areas where the 1.0 spec is silent (or just > a bit vague) on details, and I was hitting implementation-defined > differences. (For example, I was surprised to find that nowhere in the Wheel > 1.0 spec does it actually state the metadata version that wheels should > contain internally, just that it should be stored in a file called METADATA > and that it should follow PKG-INFO which sort of implies that it's a 1.X > format). Without an implementation that matches the spec precisely, though, > it's often hard to know where to report bugs. > > Paul. I think Wheel files are (and should be) independent of the particular metadata version used. That file should contain the required information in order to know what version of the metadata is included with the Wheel. This means that as metadata evolves Wheels can just start using the new meta data version without requiring an update to the spec. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 10:29, Donald Stufft wrote: > Can you send me a list (or post them here) of what issues you've hit? The > biggest one i'm aware of is the scripts problem which is a fundamental > problem with the 1.0 Wheel (or rather that any library with console entry > points cannot be universal). The scripts one is the key one (and yes, that needs to be fixed by updating the spec to confirm the consensus and then updating the tools to match). Another one IIRC was that distlib didn't put entry-points.txt in the .dist-info directory in the wheel (which breaks entry points). I think that's fixed now (and again, the Wheel spec is silent on what is correct behaviour here). I'll try to dig out the others, but they weren't anywhere near as major (some were also script-related such as inconsistent rewriting of shebang lines, which are superseded by the current state of play on scripts anyway). Generally, though, they were all areas where the 1.0 spec is silent (or just a bit vague) on details, and I was hitting implementation-defined differences. (For example, I was surprised to find that nowhere in the Wheel 1.0 spec does it actually state the metadata version that wheels should contain internally, just that it should be stored in a file called METADATA and that it should follow PKG-INFO which sort of implies that it's a 1.X format). Without an implementation that matches the spec precisely, though, it's often hard to know where to report bugs. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 5:27 AM, Paul Moore wrote: > On 21 August 2013 09:56, Donald Stufft wrote: > > ISTM distlib is not yet that reference library - it's just another library > > for most people, judging from the low level of feedback I've had overall. > > That's totally fine. We just need to be clear that it's not the reference > library and is instead one implementation. > > Yep, there's certainly been a perception that distlib is the reference > implementation. Apologies if I perpetuated that. > > We do have a slightly different issue then, in that there *isn't* a reference > implementation for a lot of this stuff... (I guess wheel counts as the > reference implementation for wheel, doh, so that part's covered). People are > starting to write code to use these new facilities, so having an actual > reference implementation is important (IMO, that's one area where > packaging/distutils2 got in a mess, so I'm concerned we don't fall into the > same trap). We need people using the new stuff to help us identify potential > issues. I might see about making a stripped down library which only implements the PEPs and nothing else. > > Paul > > PS Apologies if I appear to be a little irritable on this subject. I have a > series of scripts that maintain a local cache of wheels for various projects. > I'm starting to hit cases where the wheels aren't usable because of subtle > differences in how the spec's being implemented, and it feels like I'm trying > to hit a moving target, which is what I thought having the wheel 1.0 PEP > accepted was designed to avoid. Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 09:56, Donald Stufft wrote: > > ISTM distlib is not yet that reference library - it's just another > library > > for most people, judging from the low level of feedback I've had overall. > > That's totally fine. We just need to be clear that it's not the reference > library and is instead one implementation. Yep, there's certainly been a perception that distlib is the reference implementation. Apologies if I perpetuated that. We do have a slightly different issue then, in that there *isn't* a reference implementation for a lot of this stuff... (I guess wheel counts as the reference implementation for wheel, doh, so that part's covered). People are starting to write code to use these new facilities, so having an actual reference implementation is important (IMO, that's one area where packaging/distutils2 got in a mess, so I'm concerned we don't fall into the same trap). We need people using the new stuff to help us identify potential issues. Paul PS Apologies if I appear to be a little irritable on this subject. I have a series of scripts that maintain a local cache of wheels for various projects. I'm starting to hit cases where the wheels aren't usable because of subtle differences in how the spec's being implemented, and it feels like I'm trying to hit a moving target, which is what I thought having the wheel 1.0 PEP accepted was designed to avoid. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 4:23 AM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> However what I don't really want is to be using someones personal testbed >> for features they think is cool. There's nothing *wrong* with you trying >> new ideas out in distlib, it just means that distib isn't the library I >> want to build tooling around. > > "Someone's personal test-bed"? How do you think all open source software > starts out? What innovation isn't based on a feature someone thinks "is cool"? > > From our interactions, I don't get a feeling that you're particularly > interested in building tooling around distlib anyway, for reasons best known > to you. This specific feature could be easily pulled, and distlib is still > only at version 0.1.2. Nick has indicated he doesn't think it's appropriate > to consider distlib ready for endorsement in a 3.4 time-frame, so there's > plenty of time to make changes which are deemed necessary. How on earth do > you expect people to try things out, see where the problems are and fix > them, if you don't release such features? It would certainly help if you > describe specific problems with this feature, rather than just "I don't like > it". I don't particularly care about this feature. I care about having a simple reference library. That is what I thought distlib was supposed to be (and I'm not alone). However if you want to innovate and experiment inside it then great, that's fine. I just won't tell people that distlib is the reference library. > >> My basic problem is if the library we're pointing at to be the reference >> implementation of all of these things is adding new features it's >> confusing what is standard and what are just distlib's extensions. > > There are already features in distlib which aren't mentioned in any PEP - > e.g. the ability to calculate dependency graphs without downloading any > archives, or the ability to install scripts as single executables on > Windows, or the ability to give better feedback when uninstalling. As mentioned above, I'm not trying to argue against this particular feature I'm just trying to make sure we have a simple reference library for people to be pointed at. > >> So basically I want people to innovate, that's something I feel very >> strongly is a good thing, I just don't want innovations to happen in the >> reference library. Maybe we need a smaller reference library which is > > ISTM distlib is not yet that reference library - it's just another library > for most people, judging from the low level of feedback I've had overall. That's totally fine. We just need to be clear that it's not the reference library and is instead one implementation. > > There's plenty of time to try things out and make changes, so I find your > approach a bit of an over-reaction, and feel that it doesn't square with > what you're saying about innovation. You know that feeling you mentioned > that you get, where you think everyone is just out to block you and stop you > getting stuff done? Are you trying to spread that feeling around? ;-) I think the way you view distlib and the way other are viewing distlib are different (and that's ok). We just need to know what distlib is so we can have reasonable expectations of it. What i'm getting from you is that, at least right now, distlib isn't what I thought it was and I (and others) should stop treating it as *the* reference implementation of the new standards. I'm not trying to stop you from innovating, I'm just trying to make sure everyone has reasonable expectations all around. > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 09:09, Donald Stufft wrote: > > On Aug 21, 2013, at 4:07 AM, Paul Moore wrote: > > OK, so here's a concrete question for distutils-sig. If I want to use > wheels in my app (built them, install them, whatever) what should I use as > my "reference implementation". I don't want to implement the code myself, I > just want to produce lowest-common-denominator wheels that can be used > anywhere, and consume wheels that conform to the spec correctly. This is > not a hypothetical question - in the first instance I'm looking to add > support for loading setuptools/pip from wheels in virtualenv, and I need to > know what code to bundle to make that happen. > > > Probably Wheel at this point. There's just the problem with the scripts > which we need to actually get into the PEP and implemented. > I thought someone would say that. Wheel implies a dependency on setuptools (pkg_resources) which is viable for the virtualenv use case, but makes me somewhat sad in the more general case (because depending on setuptools at runtime feels wrong to me and there's no standalone pkg_resources). But nevertheless I think you're right. Thanks. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Donald Stufft stufft.io> writes: > However what I don't really want is to be using someones personal testbed > for features they think is cool. There's nothing *wrong* with you trying > new ideas out in distlib, it just means that distib isn't the library I > want to build tooling around. "Someone's personal test-bed"? How do you think all open source software starts out? What innovation isn't based on a feature someone thinks "is cool"? >From our interactions, I don't get a feeling that you're particularly interested in building tooling around distlib anyway, for reasons best known to you. This specific feature could be easily pulled, and distlib is still only at version 0.1.2. Nick has indicated he doesn't think it's appropriate to consider distlib ready for endorsement in a 3.4 time-frame, so there's plenty of time to make changes which are deemed necessary. How on earth do you expect people to try things out, see where the problems are and fix them, if you don't release such features? It would certainly help if you describe specific problems with this feature, rather than just "I don't like it". > My basic problem is if the library we're pointing at to be the reference > implementation of all of these things is adding new features it's > confusing what is standard and what are just distlib's extensions. There are already features in distlib which aren't mentioned in any PEP - e.g. the ability to calculate dependency graphs without downloading any archives, or the ability to install scripts as single executables on Windows, or the ability to give better feedback when uninstalling. > So basically I want people to innovate, that's something I feel very > strongly is a good thing, I just don't want innovations to happen in the > reference library. Maybe we need a smaller reference library which is ISTM distlib is not yet that reference library - it's just another library for most people, judging from the low level of feedback I've had overall. There's plenty of time to try things out and make changes, so I find your approach a bit of an over-reaction, and feel that it doesn't square with what you're saying about innovation. You know that feeling you mentioned that you get, where you think everyone is just out to block you and stop you getting stuff done? Are you trying to spread that feeling around? ;-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 4:07 AM, Paul Moore wrote: > OK, so here's a concrete question for distutils-sig. If I want to use wheels > in my app (built them, install them, whatever) what should I use as my > "reference implementation". I don't want to implement the code myself, I just > want to produce lowest-common-denominator wheels that can be used anywhere, > and consume wheels that conform to the spec correctly. This is not a > hypothetical question - in the first instance I'm looking to add support for > loading setuptools/pip from wheels in virtualenv, and I need to know what > code to bundle to make that happen. Probably Wheel at this point. There's just the problem with the scripts which we need to actually get into the PEP and implemented. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 21 August 2013 08:45, Donald Stufft wrote: > My basic problem is if the library we're pointing at to be the reference > implementation of all of these things is adding new features it's > confusing what > is standard and what are just distlib's extensions. > > So basically I want people to innovate, that's something I feel very > strongly > is a good thing, I just don't want innovations to happen in the reference > library. Maybe we need a smaller reference library which is strictly the > PEPs > to allow distlib to experiment. If it's experimentations turns out to be > good and > useful we can make PEPs for them and add them to the reference library. > +1 My problem is that as someone who wants to implement code that uses the new features like wheels, I want a usable reference implementation that covers the (agreed) standards. I don't particularly want my application to incorporate support for extensions to the standard, nor do I want to have to implement my own support all the time. In particular, at the present time there are two tools that can generate wheels (bdist_wheel and distlib) and 3 that can install them (wheel, distlib and pip). They have subtly different behaviours outside of the standard definitions, which means that they are not completely interoperable. I am not happy at all about that - and if that counts as "being against innovation", then I'm afraid that yes, I am... (I don't think it does, by the way, but you may differ). At the moment the wheel PEP is lagging a little behind some of the ongoing discussions, in particular in terms of script generation. That's fine, it's a work in progress. I hope it will be updated soon so that the spec matches what's been agreed. But I think we have a reasonable consensus on how scripts should work, and I think that should be reflected in the spec and the various tools be brought up to date with the spec before we move onto other areas and forget to tidy up around this one. Pip resisted including wheel support until we had a standard. I'm pretty unhappy that now we do have a standard, we still have situations where a wheel generated by one tool can have problems when installed with another - try "pip install wheel --use-wheel" on Windows to see what I mean, the exe wrappers are missing (this uses the wheel from PyPI, not a home-built one). OK, so here's a concrete question for distutils-sig. If I want to use wheels in my app (built them, install them, whatever) what should I use as my "reference implementation". I don't want to implement the code myself, I just want to produce lowest-common-denominator wheels that can be used anywhere, and consume wheels that conform to the spec correctly. This is not a hypothetical question - in the first instance I'm looking to add support for loading setuptools/pip from wheels in virtualenv, and I need to know what code to bundle to make that happen. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 21, 2013, at 3:32 AM, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> I'm concerned that you need extra metadata (not described in the wheel > spec) to do this. It means that there are in effect two subtly different > types of wheel. To be specific, if I create a wheel for (say) pyzmq using > distil, and mount it, everything works. But if I create the same wheel with > bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages > the credibility of wheel as a standardised format. > > If the additional metadata isn't there, then distlib just doesn't do > anything additional - it just makes the Python modules importable (by adding > the wheel to sys.path, which AFAIK is uncontroversial). > >> Can I suggest that if you need to add features like this, you need to get > the wheel spec updated to mandate them, so that *all* wheels will follow the > same spec. >> Essentially, I am -1 on any feature that uses information that is not > documented in the wheel spec. Pip in particular resisted adding support for > wheels until they were standardised in a PEP. It's frustrating if that PEP > *still* doesn't mean that the wheel format is the same for all tools. (Note > that another area where this is an issue is script wrappers, as the spec is > silent about the fact that they are specified using entry-points.txt in > metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel > for his consideration). > > Well, you don't really want to stifle innovation, do you? ;-) > > As far as I can tell, Daniel's wheel implementation allows files that are > not specifically mentioned in the PEP to be installed into a distribution's > .dist-info. This is also allowed in distlib - ISTM this is one way in which > different packaging tools can add features which are special to them, and > hold state relevant to distributions they build and/or install. If you > accept that multiple competing implementations if a PEP are a good thing, > then they can't all be functionally identical, though they must all > implement a common set of functions described in the PEP they're implementing. I was one of the advocates for extension support in the new metadata, I want tools to be able to try things out and innovate. However what I don't really want is to be using someones personal testbed for features they think is cool. There's nothing *wrong* with you trying new ideas out in distlib, it just means that distib isn't the library I want to build tooling around. My basic problem is if the library we're pointing at to be the reference implementation of all of these things is adding new features it's confusing what is standard and what are just distlib's extensions. So basically I want people to innovate, that's something I feel very strongly is a good thing, I just don't want innovations to happen in the reference library. Maybe we need a smaller reference library which is strictly the PEPs to allow distlib to experiment. If it's experimentations turns out to be good and useful we can make PEPs for them and add them to the reference library. > > As far as the advocacy for C-extension import support in wheel mounting > goes, I see opposition to the idea on the basis that some people have shot > themselves in the foot with a similar feature in pkg_resources. However, > I've not seen any analysis that indicates (with specifics) why the feature > is inherently bad (if there are problems with a specific implementation of > the idea, then those could perhaps be resolved, but you can't really argue > against "you're going to have a bad time, and you should feel bad"). > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Paul Moore gmail.com> writes: > I'm concerned that you need extra metadata (not described in the wheel spec) to do this. It means that there are in effect two subtly different types of wheel. To be specific, if I create a wheel for (say) pyzmq using distil, and mount it, everything works. But if I create the same wheel with bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages the credibility of wheel as a standardised format. If the additional metadata isn't there, then distlib just doesn't do anything additional - it just makes the Python modules importable (by adding the wheel to sys.path, which AFAIK is uncontroversial). > Can I suggest that if you need to add features like this, you need to get the wheel spec updated to mandate them, so that *all* wheels will follow the same spec. > Essentially, I am -1 on any feature that uses information that is not documented in the wheel spec. Pip in particular resisted adding support for wheels until they were standardised in a PEP. It's frustrating if that PEP *still* doesn't mean that the wheel format is the same for all tools. (Note that another area where this is an issue is script wrappers, as the spec is silent about the fact that they are specified using entry-points.txt in metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel for his consideration). Well, you don't really want to stifle innovation, do you? ;-) As far as I can tell, Daniel's wheel implementation allows files that are not specifically mentioned in the PEP to be installed into a distribution's .dist-info. This is also allowed in distlib - ISTM this is one way in which different packaging tools can add features which are special to them, and hold state relevant to distributions they build and/or install. If you accept that multiple competing implementations if a PEP are a good thing, then they can't all be functionally identical, though they must all implement a common set of functions described in the PEP they're implementing. As far as the advocacy for C-extension import support in wheel mounting goes, I see opposition to the idea on the basis that some people have shot themselves in the foot with a similar feature in pkg_resources. However, I've not seen any analysis that indicates (with specifics) why the feature is inherently bad (if there are problems with a specific implementation of the idea, then those could perhaps be resolved, but you can't really argue against "you're going to have a bad time, and you should feel bad"). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
Oscar Benjamin gmail.com> writes: > I think that they are responsible for installing the f2py script in > each of my Scripts directories. I never use this script and I don't > know what numpy wants with it (my understanding is that the Fortran > parts of numpy were all shifted over to scipy). IIUC, if a third-party extension wants to use Fortran, the build process converts it using f2py in to a Python-importable extension. It may be a feature for distributions that use numpy, even if numpy doesn't use Fortran itself. > > 2. Tags (not in general, but AIUI numpy distribute a fancy installer that > > decides what compiled code to use depending on whether you have certain CPU > > features - they may want to retain that, and to do so may prefer to have > > more fine-grained tags, which in turn may or may not be possible to > > support). I don't think that's a critical issue though. > > I guess this is what you mean: > https://github.com/numpy/numpy/blob/master/tools/win32build/cpuid/test.c > > Is there no way for them to run a post-install script when pip > installing wheels from PyPI? I'm not sure that would be enough. The numpy installation checks for various features available at build time, and then writes numpy source code which is then installed. When building and installing on the same machine, perhaps no problem - but there could be problems when installation happens on a different machine, since the sources written to the wheel at build time would encode information about the build environment which may not be valid in the installation environment. ISTM for numpy to work with wheels, all of this logic would need to move from build time to run time, but I don't know how pervasive the source-writing approach is and how much work would be entailed in switching over to run-time adaptation to the environment. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
