Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On Jul 13, 2013, at 1:31 AM, Nick Coghlan ncogh...@gmail.com wrote: I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. We could simply check it into the site-packages inside the CPython source tree could we not? *Not* providing a bootstrap script and merely checking it into the default site-packages means it's available for everyone. No matter how python installed. If Linux packagers really don't want it installed by default they could simply just remove it and either install it along with Python, or continue to keep it how it is today as a separate package? There are a number of things that have to be taken into account when downloading pip from the internet that are completely side stepped when we well don't download it from the internet :) And bundling it as a pre-installed python module and not in the standard library solves basically all of the problems I have with putting it inside of Python. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On Jul 13, 2013, at 1:57 AM, Noah Kantrowitz n...@coderanger.net wrote: I would absolutely advocate to packagers that installing the main python package results in a working pip install, regardless of how that is accomplished. +1 to this as well. Ideally, if we go down this route, installing python just comes with pip preinstalled. However that takes place :) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
In article 55b209b3-9576-4cf0-b58c-2a1e692af...@stufft.io, Donald Stufft don...@stufft.io wrote: On Jul 13, 2013, at 1:31 AM, Nick Coghlan ncogh...@gmail.com wrote: I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. Off the top of my head, including a copy of pip as a pre-installed global site-package seems like a very reasonable suggestion. For the python.org OS X installer, it should be no problem to implement. It would be equally easy to implement for future 2.7 and 3.3 maintenance releases. We could simply check it into the site-packages inside the CPython source tree could we not? *Not* providing a bootstrap script and merely checking it into the default site-packages means it's available for everyone. No matter how python installed. If Linux packagers really don't want it installed by default they could simply just remove it and either install it along with Python, or continue to keep it how it is today as a separate package? This sounds an unnecessary complication. I suspect that there is a small minority of users who actually build Python from source. And they should know what they are doing. I believe most users either use a distribution-provided Python (via their OS) or a third-party package provider (including python.org binary installers and their derivatives). The OS distributors are going to do what they currently do; the only change needed is to persuade them to include their pip package as a mandatory dependency. Trying to hack the Python source build process to include a copy of pip is just not worth the effort. -- Ned Deily, n...@acm.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On 13 July 2013 05:25, Brett Cannon br...@python.org wrote: On Fri, Jul 12, 2013 at 2:16 PM, Donald Stufft don...@stufft.io wrote: On Jul 12, 2013, at 2:00 PM, Brett Cannon br...@python.org wrote: Speaking with my python-dev hat on which has a badge from when I led the stdlib cleanup for Python 3, I would say anything that has a PEP should probably have a module in the stdlib for it. That way standard management of whatever is specified in the PEP will be uniform and expected to be maintained and work. Beyond that code will exist outside the stdlib. This is basically the exact opposite of what Nick has said the intent has been (Ecosystem first). Not at all as no module will go in immediately until after a PEP has landed and been vetted as needed. Adding packaging tools beyond bootstrapping pip at this point in the game is IMO a huge mistake. If what Nick has said and PEPs are not appropriate for things that don't have a module in the standard lib well that's fine I guess. You misunderstand what I mean. I'm just saying that *if* anything were to go into the stdlib it would only be to have code which implements a PEP in the stdlib to prevent everyone from re-implementing a standard. What Brett is saying is closer to what I was thinking when we were discussing this at the language summit. However, I'm no longer sure distlib will be quite baked enough to suggest bundling it in 3.4, in which case it will only be a pip install distlib away (that's the entire point of PEP 439). I just won't worry about trying to write PEPs :) No, the PEPs are important to prevent version skew and make sure everyone is on the same page. And that's also what a module in the stdlib would do; make sure everyone is on the same page in terms of semantics by using a single code base. I mean I wouldn't expect anything more than maybe code parsing the JSON metadata that does some validation and parsing version numbers that can support comparisons and verifying platform requirements; that's it. Stuff that every installation tool will need to do in order to follow the PEPs properly. And it wouldn't go in until everyone was very happy with the PEPs and ready to commit to code enshrining it in the stdlib. Otherwise I hope distlib moves into PyPA and everyone who develops installation tools, etc. uses that library. Vinay already moved both distlib and pylauncher over to the PyPA account on BitBucket: https://bitbucket.org/pypa/ PEP 439 is the critical piece, since that's the one that takes the pressure off getting the other components (including distlib and pkg_resources) into the base installers. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On 13 July 2013 16:05, Donald Stufft don...@stufft.io wrote: On Jul 13, 2013, at 1:31 AM, Nick Coghlan ncogh...@gmail.com wrote: I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. We could simply check it into the site-packages inside the CPython source tree could we not? *Not* providing a bootstrap script and merely checking it into the default site-packages means it's available for everyone. No matter how python installed. Source code that isn't maintained through bugs.python.org isn't getting checked into the CPython repo - see PEP 360. Getting the latest released version of something from PyPI is a different story, though. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On Sat, Jul 13, 2013 at 8:07 AM, Donald Stufft don...@stufft.io wrote: +1 to this as well. Ideally, if we go down this route, installing python just comes with pip preinstalled. However that takes place :) Well, I don't want a however that takes place that causes more packaging problems in the future, so I'm -1 on that. :-) I think including it in binary installers in such a way that pip can upgrade itself is definitely a good idea. Including it in the source code distributions seems less of a benefit. In fact, I don't even think I want it as things stand right now. Story time: This week a lot of people had problems with zc.buildout, because of the new releases of setuptools and distribute. Essentially they tried to upgrade themselves and failed for various reasons, such as distribute or setuptools being installed in the system python and not being upgradeable, or not being upgradeable for other reasons, like permissions. Mostly, I had no problems, because the Python installs I use have only one package installed: Virtualenv. That means that for most buildouts I used, buildout had istelf installed setuptools or distribute in its own isolated way. I only got errors where an overly complex build tool insisted on making virtualenvs, installing distribute in them and then running buildout with those virtualenvs. In those cases I had to run bin/pip -U distribute, so it was also an easy fix. So by default I prefer to have no packages except virtualenv for my source-built Pythons. This isn't a big issue, and I'm not gonna throw a hissyfit or anything, but I think I'll -1 the idea of including pip in a source install. Having pyvenv install pip by default in the new environment is +lots from me though. Also, having pyvenv be able to install itself in a non-empty directory would be great. ;-) //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On 13 July 2013 01:02, Donald Stufft don...@stufft.io wrote: The installer side of things the purist side of me doesn't like adding it to the standard library for all the same reasons but the pragmatic side of me wants it there because it enables fetching the other bits that are needed for pip install X to be a reasonable official response to these kind of questions. But I pushed for and still believe that if a prerequisite for doing that involves locking in pip or any of it's dependencies by adding them to the standard library then I am vehemently against doing it. Nobody seems to be suggesting doing that, though. I was (trying?) to explain that my belief doesn't extend to only distlib here and instead to the entire toolchain. (The above quote may not be the best point to comment on, but I wanted to avoid quoting the whole text just to make a general point on this subject) In my view packaging (specifically install) tools are a bit different from other things, because generally packaging tools with dependencies suck. Look at pip's reliance on setuptools, for example. Managing setuptools with pip is a pain, and bootstrapping pip involves getting setuptools installed without already having pip available. I'm +1 on having basic infrastructure in the stdlib, because that way people can concentrate on innovating in more important areas of packaging rather than endlessly reinventing basic stuff. The trick is knowing what counts as basic infrastructure. Things I have *regularly* come up against, over a long period of writing little tools: 1. Version parsing and ordering (I usually use distutils.LooseVersion, just because it's available and close enough :-() 2. Reading metadata from distributions (not just parsing it, but getting it out of dist-info files or sdists and the like as well) 3. Installing wheels 4. Locating distributions on PyPI or local indexes At the moment, my choices are to write my own code (usually meaning far more code than the actual functionality I want to write!) require distlib (not good in a tool for building zero-dependency venvs from the ground up, for example) or vendoring in distlib (impractical in a one-file script). Having something in the stdlib (even if it's only able to bootstrap distlib or an alternative) solves all of these issues. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On 13 July 2013 06:31, Nick Coghlan ncogh...@gmail.com wrote: * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the bundled application model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. Personally, I have never upgraded pip using itself, because I only ever install pip in virtualenvs, which don't have a lifespan as long as a pip release cycle :-) It would be easy to imagine a new pip release resulting in a *lot* of bugs raised against Python (rather than pip) saying that the upgrade fails. And of course if an upgrade fails, we can't just release a new version of pip that fixes the issue, because it's the *old* version that is installed and has to do the upgrade. So there's manual fiddling to do. Not a good experience for Python users. My current workflow is to have absolutely nothing installed in the system Python and use virtualenvs for everything. This is a bit extreme, but the issues I've hit in the past when package management has gone wrong have made me very cautious. If the pip upgrade process is rock-solid, this isn't an issue, but I'm not sure that it is, myself. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
From: Donald Stufft don...@stufft.io As I said in my email, because it's more or less standalone and it has the greatest utility outside of installers/builders/archivers/indexes. Even if that were true, it doesn't mean that it's the *only* thing that's worth considering. I've looked at many other languages where they had widely successful packaging tools that weren't added to the standard lib until they were ubiquitous and stable. Something the new tools for Python are not. So I don't think adding it to the standard library is required. As I said earlier, I'm not arguing for *premature* inclusion of distlib or anything else in the stdlib. I'm only saying that there's less likelihood that any one approach outside the stdlib will get univerally adopted, leading to balkanisation. to reuse some it's functionality. So pointing towards setuptools just exposes the fact that improving it in the standard library was hard enough that it was done externally. It seems like it wasn't for technical reasons that this approach was taken, just as Distribute wasn't forked from setuptools for technical reasons. Well I am of the mind that the standard library is where software goes to die, and No kidding? :-) want to use my software at all. A huge thing i've been trying to push for is decoupling packaging from a specific implementation so that we have a protocol (ala HTTP) and not a tool (ala distutils). However the allure of working to the implementation and not the standard is fairly high when there is a singular blessed implementation. I'm not aware of this - have you published any protocols around the work you're doing on warehouse, which Nick said was going to be the next-generation PyPI? It's funny you picked and example where improvements *couldn't* take place and the entire system had to be thrown out and a new one written. getopt had to become a new module named opt parse, which had to become a new module named argparse I picked that example specifically to show that even if things go wrong, it's not the end of the world. You can gain interoperability in a few ways. One way is to just pick an implementation If that were done, it wouldn't make any difference whether the thing picked were in the stdlib or not. But people have a tendency to roll their own stuff, whether there's a good technical reason or not. and make that the standard. Another is to define *actual* standards. The second one is harder, requires more thought and work. But it means that completely different software can work together. It means that something written in Ruby can easily work with a python package without shelling out to Python or without That's exactly why there are all these packaging PEPs around, isn't it? And that's fine for a certain class of problems. It's not that useful for something where you want interoperability outside of that tool. How terrible would it be if HTTP was well whatever Apache does, that's what HTTP is. That wouldn't have been so terrible if you replace Apache with W3C, since you would have a reference implementation by the creators of the standard. A singular blessed tool in the standard library incentivizes the standard becoming and implementation detail. I *want* there to be multiple implementations written by different people working on different slices of the problem. That incentivizes doing the extra work on PEPs and other documents so that we maintain a highly documented standard. It's true that adding something to the standard library doesn't rule that out but it provides an incentive against properly doing standards because it's easier and simpler to just change it in the implementation. Are you planning to produce any standards relating to PyPI-like functionality? This is important for the dependency resolution slice, amongst others. The flip side of this coin is, talking in the abstract without any working code is sub-optimal. It's reasonable for standards and implementations of them to grow together, because each informs the other, at least in the early stages. Most standards PEPs are accepted with a reference implementation in place. It's not blessed and a particular packaging project should use it if it fits their needs and they want to use it. Or they shouldn't use it if they don't want. Standards exist for a reason. So you can have multiple implementations that all work together. That's true independent of whether one particular implementation of the standard is blessed in some way. I didn't make any claims as to it's stability or the amount of testing that went into it. My ability to be convinced of that stems primarily from the fact that it's sort of a side piece of the whole packaging infrastructure and toolchain and it's also a piece that is most likely to be useful on it's own. But the arguments about agility and stability apply to any software - version-handling doesn't get a special pass. Proper version handling
Re: [Distutils] PEP 439 and pip bootstrap updated
On 13 July 2013 13:12, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: I'll just finish by re-iterating that I think there should be some stdlib underpinning for packaging in general, and that there should be some focus on exactly what that underpinning should be, and that I'm by no means saying that distlib is it. I consider distlib as still in its early days but showing some promise (and deserving of more peer review than it has received to date). +1 to all of this Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On 13 July 2013 13:25, Nick Coghlan ncogh...@gmail.com wrote: I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. Agreed, up to a point. What I've worked on in the past is things like automated wheel/sdist installers for systems with no internet access at all (distributions copied onto the PC via portable disk) or behind broken proxies (Internet Explorer works to download files from the net, but nothing else does). For those environments, the key to me is that I *only* use stuff that is available in a stock Python build. Something like pyton -m getpip *will not work*, so I have to roll my own means of bootstrapping. Of course I could copy a pip sdist to the machine, unpack it and run python setup.py install. More likely, I write bootstrap code in my script to do that automatically. It's a very specialised use case, certainly, and there are plenty of ways around the issues, but I have seen give up and just use VBScript./perl used as the fallback, as well, and that makes me sad... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On Sat, Jul 13, 2013 at 1:31 AM, Nick Coghlan ncogh...@gmail.com wrote: In addition to the long thread based on Richard's latest set of updates, I've also received a few off-list comments on the current state of the proposal. So, I figured I'd start a new thread summarising my current point of view and see where we want to go from there. 1. However we end up solving the bootstrapping problem, I'm *definitely* a fan of us updating pyvenv in 3.4 to ensure that pip is available by default in new virtual environments created with that tool. I also have an idea for a related import system feature that I'll be sending to import-sig this afternoon (it's a variant on *.pth and *.egg-link files that should be able to address a variety of existing problems, including the one of *selectively* making system and user packages available in a virtual environment in a cross-platform way without needing to copy them) 2. While I was originally a fan of the implicit bootstrapping on demand design, I no longer like that notion. While Richard's bootstrap script is a very nice piece of work, the edge cases and neat tricks have built up to the point where they trip my if the implementation is hard to explain, it's a bad idea filter. Accordingly, I no longer think the implicit bootstrapping is a viable option. 3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned): * an explicit bootstrapping script * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the bundled application model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. For Linux, if you're using the system Python on a Debian or Fedora derivative, then sudo apt-get python-pip and sudo yum install python-pip are both straightforward, and if you're using something else, then it's unlikely getting pip bootstrapped using the bootstrap script is a task that will bother you :) The python -m getpip command is still something we will want to provide, as it is useful to people that build their own copy of Python from source. But is it going to make a difference? If we shift to using included copies of pip in binary installers over a bootstrap I say leave out the bootstrap as anyone building from source should know how to get pip installed on their machine or venv. The only reason I see it worth considering is if pyvenv starts bootstrapping pip and we want to support the case of pip not being installed. But if we are including it in the binary installer and are going to assume it's available through OS distros, then there isn't a need to as pip can then install pip for us into the venv and skip any initial pip bootstrap. If pip isn't found we can simply either point to the docs in the failure message or print out the one-liner it takes to install pip (and obviously there can be a --no-pip flag to skip this for people who want to install it manually like me who build from source). IOW I think taking the worldview in Python 3.4 that pip will come installed with Python unless you build from source negates the need for the bootstrap script beyond just saying ``curl https://pypi.python.org/get-pip.py | python`` if pip isn't found. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On Sat, Jul 13, 2013 at 9:38 AM, Paul Moore p.f.mo...@gmail.com wrote: On 13 July 2013 14:31, Brett Cannon br...@python.org wrote: +1 on the inversion. I don't know what that will do to pip, it makes sense to have the installer self-contained and the packaging/building libraries be something that you grab using the installer. Having to grab the packaging infrastructure to get an installer is the more painful route. TBH, I don't understand what the inversion implies. If it means pip taking all of the distlib/setuptools code that it currently uses, and making it part of pip and maintained within pip (essentially as a fork while the inversion is going on) then I'm not keen on that. Personally, I don't want to have to maintain that code myself - I guess if Vinay and Jason were pip maintainers and looked after that code, then that's an option. If it means pip vendoring distlib and setuptools, then OK (we do that for distlib already) The point is you shouldn't have to grab a packaging tool just to install stuff if you never need the packaging tool. Since pip is supposed to be *the* first thing you install for Python you don't want that to have its own dependencies, muddying up the installation process. but I don't see the benefit - no-ione should be doing from pip.vendor.distlib.version import Version. That's just asking for trouble if someone did that (plus if you did that it would be pip._vendor to get the privacy point across). I'd need to know better what it means for pip, I guess... I suspect we all do. =) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
Nick Coghlan ncoghlan at gmail.com writes: 1. However we end up solving the bootstrapping problem, I'm *definitely* a fan of us updating pyvenv in 3.4 to ensure that pip is available by default in new virtual environments created with that tool. Will that need green-lighting on python-dev? As events have shown, that script has needed updating between Python releases. OTOH, I'm not sure anyone could have predicted at 3.3 release time that setuptools and Distribute would kiss and make up :-) We should probably ensure that the pip and setuptools URLs used in that script (pyvenvex.py, to be renamed to pyvenv.py) are formally agreed with the relevant maintainers. Or, since those URLs just fetch the latest releases, perhaps some different URLs should be used which refer to more stable releases (for some definition of more stable) - perhaps those should be python.org URLs, rather than BitBucket and GitHub as they are at present. That way, changing the resources which those URLs reference would have to be an active decision by someone, rather than just following the latest developments on setuptools and pip. There are pluses and minuses either way, of course. 2. While I was originally a fan of the implicit bootstrapping on demand design, I no longer like that notion. While Richard's bootstrap script is a very nice piece of work, the edge cases and neat tricks have built up to the point where they trip my if the implementation is hard to explain, it's a bad idea filter. Accordingly, I no longer think the implicit bootstrapping is a viable option. But if your reservation stems from one specific implementation of the idea, then might not an alternative implementation fit the bill? Consider: the pyvenvex.script merely runs bootstrapping scripts from setuptools and pip in the venv - there's no magic. I couldn't see Richard's script referenced in the PEP, which just referred to the PIP issue tracker which had no obvious link to any commit or script. So I don't know what the edge cases and neat tricks are that you're referring to. I adapted the pyvenvex.py script into a getpip.py script, available here: https://gist.github.com/vsajip/5990837 82 lines all told - what cases does it not cover? It installs setuptools and pip into the system site-packages for the invoking Python, if not already present. It can, of course, be refined to e.g. install even if the packages are already present, which is tantamount to upgrading. I smoke-tested the script on vanilla Python 3.3 installations on Windows and OS X. On OS X, the pip script was written to the appropriate Frameworks folder, but not to /usr/local/bin - I assume it would be simple enough to arrange that? On Windows, the pip script (including Windows native launcher) were written to c:\Python33\Scripts. The bundling idea will obviously need to be discussed with the installer builders, and on python-dev in general If python-dev agrees to the updated pyvenv.py script, then this type of addition should be uncontentious, as it basically does the same thing. It seems a whole lot less work than bundling, to me. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Expectations on how pip needs to change for Python 3.4
This issue has been skirted round for some time now, and I think it needs explicit discussion, as I am not at all sure everyone has the same expectations. We're talking about Python 3.4 installations having pip as the default package manager - whether by bundling, having a bootstrap process or whatever. Regardless of the means, pip will be *the* installer for Python 3.4+. And yet, I don't think pip 1.4 currently does what people want the Python 3.4 pip to do in some ways - and we need to make sure that any work on the pip side is understood, agreed to, and planned to match the Python 3.4 timescales. So, here's my initial list of things that I think people might be expecting to happen. This is just my impressions, and I don't necessarily have a view on the individual items. And if anyone else can think of other things to add to the list, please do so! 1. Install to user-packages by default. 2. Not depend on setuptools (??? - Nick's inversion idea) 3. Possibly change the wrapper command name from pip to pip3 on Unix. 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get stuck on the Python-supplied version. I'm sure I've seen people say other things that have made me think are you expecting the pip maintainers to make that change? in the various threads, so I doubt this list is definitive. Comments anyone? Is this discussion premature? The pip maintainers team is not huge, so we'll need time (or assistance!) to plan in and make changes like this, if they are needed... At a minimum, can we get the key items logged on the pip issue tracker with a milestone of Python 3.4? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On 13 July 2013 13:25, Nick Coghlan ncogh...@gmail.com wrote: I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. If we do this, I think people will start to expect to be able to code scripts to the pip API. (We've had people ask this on the pip tracker already). If we don't want pip to end up like distutils (with people depending on all sorts of random bits of the internals, because there's no documented API) as a backward-compatibility nightmare, we need to consider how to handle this. Of course, saying explicitly only the python -m pip command line interface is stable and supported may well be enough. But didn't we just say that setuptools and distlib depend on the pip API? So either they have special privileges (presumably because they are under the umbrella of the PyPA) or we can't avoid documenting/supporting some API... I don't believe that pip is currently in a state to offer a solid documented internal API. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
Also (see my reply to Nick's inversion proposal) we can add: 5. Provide a stable documented programming interface. Paul On 13 July 2013 15:54, Paul Moore p.f.mo...@gmail.com wrote: This issue has been skirted round for some time now, and I think it needs explicit discussion, as I am not at all sure everyone has the same expectations. We're talking about Python 3.4 installations having pip as the default package manager - whether by bundling, having a bootstrap process or whatever. Regardless of the means, pip will be *the* installer for Python 3.4+. And yet, I don't think pip 1.4 currently does what people want the Python 3.4 pip to do in some ways - and we need to make sure that any work on the pip side is understood, agreed to, and planned to match the Python 3.4 timescales. So, here's my initial list of things that I think people might be expecting to happen. This is just my impressions, and I don't necessarily have a view on the individual items. And if anyone else can think of other things to add to the list, please do so! 1. Install to user-packages by default. 2. Not depend on setuptools (??? - Nick's inversion idea) 3. Possibly change the wrapper command name from pip to pip3 on Unix. 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get stuck on the Python-supplied version. I'm sure I've seen people say other things that have made me think are you expecting the pip maintainers to make that change? in the various threads, so I doubt this list is definitive. Comments anyone? Is this discussion premature? The pip maintainers team is not huge, so we'll need time (or assistance!) to plan in and make changes like this, if they are needed... At a minimum, can we get the key items logged on the pip issue tracker with a milestone of Python 3.4? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 10:54 AM, Paul Moore p.f.mo...@gmail.com wrote: This issue has been skirted round for some time now, and I think it needs explicit discussion, as I am not at all sure everyone has the same expectations. We're talking about Python 3.4 installations having pip as the default package manager - whether by bundling, having a bootstrap process or whatever. Regardless of the means, pip will be *the* installer for Python 3.4+. And yet, I don't think pip 1.4 currently does what people want the Python 3.4 pip to do in some ways - and we need to make sure that any work on the pip side is understood, agreed to, and planned to match the Python 3.4 timescales. So, here's my initial list of things that I think people might be expecting to happen. This is just my impressions, and I don't necessarily have a view on the individual items. And if anyone else can think of other things to add to the list, please do so! 1. Install to user-packages by default. Do people really want this? I hadn't seen it (other than if pip was installed to user by default). I think it's a bad idea to switch this on people. I doubt the user-packages is going to be in people's default PATH so they'll easily get into cases where things are installed but they don't know where it was installed too. 2. Not depend on setuptools (??? - Nick's inversion idea) I wanted to do this anyways. It will still depend on it, but it will just bundle setuptools itself like its other dependencies. For pip dependencies are an implementation detail not an actual thing it can/should have. 3. Possibly change the wrapper command name from pip to pip3 on Unix. Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and pip all available and not install the less specific ones if they already exist but that might be too hard? 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get stuck on the Python-supplied version. I've always used pip to upgrade pip. The only time i've had problems is when setuptools messes up (which would be prevented if bundled). I'm sure I've seen people say other things that have made me think are you expecting the pip maintainers to make that change? in the various threads, so I doubt this list is definitive. Comments anyone? Is this discussion premature? The pip maintainers team is not huge, so we'll need time (or assistance!) to plan in and make changes like this, if they are needed... At a minimum, can we get the key items logged on the pip issue tracker with a milestone of Python 3.4? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/13/2013 08:25 AM, Nick Coghlan wrote: I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. - -1. That would effectively mean inlining the bulk of setuptools' code into pip (which is just a UI / policy shim over it). You might as well just have your bootstrapper install both pip and setuptools and be done Unless distlib (or something like it) lands in the stdlib with enough features to support a setuptools-less pip, of course. At-which-point-the-State-will-wither-away'ly, Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHhbLkACgkQ+gerLs4ltQ51fQCfZrmZN5mJKrtoGFTk0YqQrBHd F/IAnRp6XjoU4SpXZ4v3Uz6iOBrCZZZn =gSA5 -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On Jul 13, 2013, at 11:00 AM, Paul Moore p.f.mo...@gmail.com wrote: Of course, saying explicitly only the python -m pip command line interface is stable and supported may well be enough. But didn't we just say that setuptools and distlib depend on the pip API? So either they have special privileges (presumably because they are under the umbrella of the PyPA) or we can't avoid documenting/supporting some API... I don't think we need this any more than if pip was not pre-installed. So still nice to have for things like chef but not a requirement. setuptools and distlib won't depend on pip, pip will just bundle them (like it already does for distlib). The idea should be that pip itself has no dependencies because a package manager with dependencies is kind of strange and can easily lead to issues where the package manager breaks and is unifiable (e.g. setuptools breaks and pip can't be used to fix it). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On Jul 13, 2013, at 11:05 AM, Tres Seaver tsea...@palladion.com wrote: Signed PGP part On 07/13/2013 08:25 AM, Nick Coghlan wrote: I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. - -1. That would effectively mean inlining the bulk of setuptools' code into pip (which is just a UI / policy shim over it). You might as well just have your bootstrapper install both pip and setuptools and be done Unless distlib (or something like it) lands in the stdlib with enough features to support a setuptools-less pip, of course. At-which-point-the-State-will-wither-away'ly, Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig I was planning on doing this to pip anyways. pip should not have any dependencies setuptools or otherwise. So regardless of what happens with this PEP I want pip to be inlining setuptools and providing the code to make that transparent during install. This would mean that for people who are _just_ installing packages they don't need setuptools installed. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 11:01 AM, Paul Moore p.f.mo...@gmail.com wrote: 5. Provide a stable documented programming interface. As I said in the other thread I don't think this is required any more than it does normally. I do think we need to have testing infrastructure in pip that tests against the development branch of CPython though. If pip is going to be included in the releases we need to make sure it works prior to it being released. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 13 July 2013 16:03, Donald Stufft don...@stufft.io wrote: 1. Install to user-packages by default. Do people really want this? I hadn't seen it (other than if pip was installed to user by default). I think it's a bad idea to switch this on people. I doubt the user-packages is going to be in people's default PATH so they'll easily get into cases where things are installed but they don't know where it was installed too. I believe Nick wants to make user-packages the default. I know at least some of the pip maintainers (yourself included) have reservations. Personally, I've never used user-packages, so I don't know what issues might arise. But I hope to try it out sometime when I get the chance, just to get some specific information. 2. Not depend on setuptools (??? - Nick's inversion idea) I wanted to do this anyways. It will still depend on it, but it will just bundle setuptools itself like its other dependencies. For pip dependencies are an implementation detail not an actual thing it can/should have. Bundling is not the same as Nick's suggestion. I personally have no problem with bundling, but pip install with a bundled setuptools might not work because the setup subprocess won't see the bundled setuptools when it imports it in setup.py. But either way, it's doable, I just want to know if it's on the critical path... 3. Possibly change the wrapper command name from pip to pip3 on Unix. Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and pip all available and not install the less specific ones if they already exist but that might be too hard? 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get stuck on the Python-supplied version. I've always used pip to upgrade pip. The only time i've had problems is when setuptools messes up (which would be prevented if bundled). I've never tried myself, but I'm on Windows and I expect in-place stuff like this to fail. Maybe I'm paranoid :-) Again I need to check. Thanks for the comments. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Jul 13, 2013, at 8:12 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: I'm not aware of this - have you published any protocols around the work you're doing on warehouse, which Nick said was going to be the next-generation PyPI? I think we're talking past each other at this point but I wanted to respond to this point. Warehouse will evolve by publishing standards yes. Currently its not making API changes and is primarily working on taking the existing APIs and porting them to a modern framework, adding tests, etc. I do have some changes I want to make to the API and I've started a PEP to propose it that once it's done will be published for discussion here at distutils-sig. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout/setuptools/distribute unhelpful error message (0.7.x issue?)
Hi Chris, It looks like something is trying to install Setuptools 0.7.2, possibly with a temporary version of distribute or one that's not visible by default in your Python environment. When you get that error message, I suggest you upgrade away from distribute. The easiest way to do this if you have distribute installed is to 'easy_install -U distribute', which will grab distribute 0.7.3 and install setuptools=0.7. If this doesn't work (as it may not if you in fact don't have Distribute), you may be able to pro-actively avoid the problem by installing the latest Setuptools (0.9 at the time of this writing) using the published installation instructions: https://pypi.python.org/pypi/setuptools/0.9 I hope that helps. Please report back if that doesn't get you going. Regards, Jason -Original Message- From: Distutils-SIG [mailto:distutils-sig- bounces+jaraco=jaraco@python.org] On Behalf Of Chris Withers Sent: Sunday, 07 July, 2013 03:09 To: distutils sig Subject: [Distutils] buildout/setuptools/distribute unhelpful error message (0.7.x issue?) Hi All, What is this exception trying to tell me? Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools- 0.7.2.tar.gz Extracting in /tmp/tmpJNVsOY Now working in /tmp/tmpJNVsOY/setuptools-0.7.2 Building a Setuptools egg in /tmp/tmpBLZGeg /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg Traceback (most recent call last): File bootstrap.py, line 91, in module pkg_resources.working_set.add_entry(path) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 451, in add_entry self.add(dist, entry, False) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 542, in add self._added_new(dist) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 705, in _added_new callback(dist) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 2727, in lambda add_activation_listener(lambda dist: dist.activate()) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 2227, in activate self.insert_on(path) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 2328, in insert_on with distribute. Found one at %s % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg I don't see any distribute in there, and I don't know where it might be... $ python2.6 Python 2.6.8 (unknown, Jan 29 2013, 10:05:44) [GCC 4.7.2] on linux2 Type help, copyright, credits or license for more information. import setuptools Traceback (most recent call last): File stdin, line 1, in module ImportError: No module named setuptools cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig smime.p7s Description: S/MIME cryptographic signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Sat, Jul 13, 2013 at 11:15 AM, Paul Moore p.f.mo...@gmail.com wrote: On 13 July 2013 16:03, Donald Stufft don...@stufft.io wrote: 1. Install to user-packages by default. Do people really want this? I hadn't seen it (other than if pip was installed to user by default). I think it's a bad idea to switch this on people. I doubt the user-packages is going to be in people's default PATH so they'll easily get into cases where things are installed but they don't know where it was installed too. I believe Nick wants to make user-packages the default. I know at least some of the pip maintainers (yourself included) have reservations. Personally, I've never used user-packages, so I don't know what issues might arise. But I hope to try it out sometime when I get the chance, just to get some specific information. I would assume the executable script was installed next to the python binary but the library parts went into user-packages. That way -m would work for all binaries of the same version. 2. Not depend on setuptools (??? - Nick's inversion idea) I wanted to do this anyways. It will still depend on it, but it will just bundle setuptools itself like its other dependencies. For pip dependencies are an implementation detail not an actual thing it can/should have. Bundling is not the same as Nick's suggestion. I personally have no problem with bundling, but pip install with a bundled setuptools might not work because the setup subprocess won't see the bundled setuptools when it imports it in setup.py. But either way, it's doable, I just want to know if it's on the critical path... 3. Possibly change the wrapper command name from pip to pip3 on Unix. Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and pip all available and not install the less specific ones if they already exist but that might be too hard? Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
1. Install to user-packages by default. there was a thread a few weeks back on this. everyone seemed to agree at the end that just better error messages were enough. changing the default install location is a huge leap. http://mail.python.org/pipermail/distutils-sig/2013-May/020673.html 2. Not depend on setuptools (??? - Nick's inversion idea) with the bootstrap installing setuptools, it's not necessary, but I plan on considering/helping/working on one or multiple of these for pip v1.5 anyway: 1) bundling setuptools (Donald's idea). it might not work, but interesting to try. lotta pros to doing this 2) replacing pkg_resources with distlib (vinay posted a PR for this) 3) if not #1, pip installing setuptools on-demand when building is needed (this was the old plan I think for PEP439 until the recent changes, and get's us closer to the MEBs model) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
2) replacing pkg_resources with distlib (vinay posted a PR for this) 3) if not #1, pip installing setuptools on-demand when building is needed (this was the old plan I think for PEP439 until the recent changes, and get's us closer to the MEBs model) to be clearer for everyone, #3 depends on #2, so that pip could install setuptools from wheel (without needing setuptools) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 12:59 PM, Brett Cannon br...@python.org wrote: Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. I find the -m interface ugly as a primary cli api. It's ok for bonus functionality (ala json.tool) and debugging utilities (ala SimpleServer) but as a developer of user facing tools I don't think I'd ever want to tell them that they should use ``python -m`` to execute my tool. It's also a massive change in functionality from the existing pip interface. ``pip install`` is what everyone uses. The point is more or less moot though unless you're advocating not including an executable script at all. Because pip is already able to be executed with ``python -m pip`` however I don't believe i've seen anyone use that in practice. It also provides the pip and pip-X.Y commands which should probably be normalized to pip, pipX, and pipX.Y. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 9:59 AM, Brett Cannon wrote: On Sat, Jul 13, 2013 at 11:15 AM, Paul Moore p.f.mo...@gmail.com wrote: On 13 July 2013 16:03, Donald Stufft don...@stufft.io wrote: 1. Install to user-packages by default. Do people really want this? I hadn't seen it (other than if pip was installed to user by default). I think it's a bad idea to switch this on people. I doubt the user-packages is going to be in people's default PATH so they'll easily get into cases where things are installed but they don't know where it was installed too. I believe Nick wants to make user-packages the default. I know at least some of the pip maintainers (yourself included) have reservations. Personally, I've never used user-packages, so I don't know what issues might arise. But I hope to try it out sometime when I get the chance, just to get some specific information. I would assume the executable script was installed next to the python binary but the library parts went into user-packages. That way -m would work for all binaries of the same version. 2. Not depend on setuptools (??? - Nick's inversion idea) I wanted to do this anyways. It will still depend on it, but it will just bundle setuptools itself like its other dependencies. For pip dependencies are an implementation detail not an actual thing it can/should have. Bundling is not the same as Nick's suggestion. I personally have no problem with bundling, but pip install with a bundled setuptools might not work because the setup subprocess won't see the bundled setuptools when it imports it in setup.py. But either way, it's doable, I just want to know if it's on the critical path... 3. Possibly change the wrapper command name from pip to pip3 on Unix. Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and pip all available and not install the less specific ones if they already exist but that might be too hard? Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. No, this is not how any user ever will expect unix programs to work. I know that python -m is very cute, and I use it myself for some debug and helper functionality at times, but it can never replace normal scripts. This is a user experience expectation, and we will have to meet it. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Current status of PEP 439 (pip boostrapping)
On Jul 13, 2013, at 6:46 AM, Brett Cannon wrote: On Sat, Jul 13, 2013 at 1:31 AM, Nick Coghlan ncogh...@gmail.com wrote: In addition to the long thread based on Richard's latest set of updates, I've also received a few off-list comments on the current state of the proposal. So, I figured I'd start a new thread summarising my current point of view and see where we want to go from there. 1. However we end up solving the bootstrapping problem, I'm *definitely* a fan of us updating pyvenv in 3.4 to ensure that pip is available by default in new virtual environments created with that tool. I also have an idea for a related import system feature that I'll be sending to import-sig this afternoon (it's a variant on *.pth and *.egg-link files that should be able to address a variety of existing problems, including the one of *selectively* making system and user packages available in a virtual environment in a cross-platform way without needing to copy them) 2. While I was originally a fan of the implicit bootstrapping on demand design, I no longer like that notion. While Richard's bootstrap script is a very nice piece of work, the edge cases and neat tricks have built up to the point where they trip my if the implementation is hard to explain, it's a bad idea filter. Accordingly, I no longer think the implicit bootstrapping is a viable option. 3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned): * an explicit bootstrapping script * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the bundled application model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. For Linux, if you're using the system Python on a Debian or Fedora derivative, then sudo apt-get python-pip and sudo yum install python-pip are both straightforward, and if you're using something else, then it's unlikely getting pip bootstrapped using the bootstrap script is a task that will bother you :) The python -m getpip command is still something we will want to provide, as it is useful to people that build their own copy of Python from source. But is it going to make a difference? If we shift to using included copies of pip in binary installers over a bootstrap I say leave out the bootstrap as anyone building from source should know how to get pip installed on their machine or venv. The only reason I see it worth considering is if pyvenv starts bootstrapping pip and we want to support the case of pip not being installed. But if we are including it in the binary installer and are going to assume it's available through OS distros, then there isn't a need to as pip can then install pip for us into the venv and skip any initial pip bootstrap. If pip isn't found we can simply either point to the docs in the failure message or print out the one-liner it takes to install pip (and obviously there can be a --no-pip flag to skip this for people who want to install it manually like me who build from source). IOW I think taking the worldview in Python 3.4 that pip will come installed with Python unless you build from source negates the need for the bootstrap script beyond just saying ``curl https://pypi.python.org/get-pip.py | python`` if pip isn't found. This is highly unhelpful for dealing with systems automation. For the foreseeable future, the bulk of Python 3.4 installations will either be source installs, or homegrown packages based on source installs. The bundled pip doesn't need to be included with, say, an hg clone that you then build and install, but it does have to come with an install from an official release source tarball. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 12:59 PM, Brett Cannon br...@python.org wrote: Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. Also looking at what already ships with Python. idle, idle2, idle2.7 smtpd.py, smtpd2.py, smptd2.7.py pydoc, pydoc2, pydoc2.7 2to3, 2to3-2, 2to3-2.7 This is also the convention anywhere someone does versioned scripts in a Python package in the ecosystem. PEP439 is there to streamline the process so that python dependencies are much easier to install and there's a smaller barrier to entry so that projects like Django can give simple instructions for dependencies instead of needing to opt not to have dependencies or have to give instructions on how to install the installer. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 13 July 2013 18:30, Donald Stufft don...@stufft.io wrote: On Jul 13, 2013, at 12:59 PM, Brett Cannon br...@python.org wrote: Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. I find the -m interface ugly as a primary cli api. It's ok for bonus functionality (ala json.tool) and debugging utilities (ala SimpleServer) but as a developer of user facing tools I don't think I'd ever want to tell them that they should use ``python -m`` to execute my tool. It's also a massive change in functionality from the existing pip interface. ``pip install`` is what everyone uses. The point is more or less moot though unless you're advocating not including an executable script at all. Because pip is already able to be executed with ``python -m pip`` however I don't believe i've seen anyone use that in practice. It also provides the pip and pip-X.Y commands which should probably be normalized to pip, pipX, and pipX.Y. That's the point of aggressively promote. We'd advocate python -m pip as the primary means of running pip. I agree it's less convenient for users than having a simple pip command, but there are a number of downsides to pip being the primary interface (note that you can always alias pip to python -m pip in your shell - it's no harder than managing PATH, which is what many people need to do at the moment). 1. It's not *actually* the case that the command is always pip. Maybe it's pip3 if your system makes the default Python be python 2, but you want to use python 3. Maybe you're creating a virtualenv and you haven't activated it yet. In that case a plain pip will quietly do the wrong thing (at the moment, I don't install pip in my system python precisely to avoid this issue). 2. On Windows, ...\Python34\Scripts is not on PATH by default. Even if python is (as python.exe is in a different directory to the one distutils installs executables in). Again, you can change your own PATH. 3. There's a lot of clutter. On Windows, you have 3 executables (pip.exe, pip3.exe and pip3.4.exe) and 3 scripts alongside them. For one command. Apart from the first of these, the issues are all Windows ones, and it's reasonable to say well, fix the Windows setup, then, it's silly. I have some sympathy with that view, but backward compatibility and many, many years of history will make that extremely difficult. Also, some of it may simply not be fixable because people won't agree on the solution (that's been the case in the past). I assume, perhaps naively, that improving the experience on Windows is just as key as improving it on Unix. In my view, the key initial userbase for the new packaging tools will be Windows users wanting access to binary wheels. Paul. PS I actually *do* prefer just having a pip command. It's just that I doubt I'll get that on Windows, no matter *what* approach people take - I'll have to roll my own solution. Obviously, aliasing pip to python -m pip is the easiest, and probably the one I'd choose, but then again, I know what I'm doing :-) A normal Windows user will just say these instructions are c**p, there's no pip command and either have to resort to Google (probably with a question like why doesn't Python's packaging work? which isn't good PR for us), or give up. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 13 July 2013 18:51, Donald Stufft don...@stufft.io wrote: On Jul 13, 2013, at 12:59 PM, Brett Cannon br...@python.org wrote: Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. Also looking at what already ships with Python. idle, idle2, idle2.7 smtpd.py, smtpd2.py, smptd2.7.py pydoc, pydoc2, pydoc2.7 2to3, 2to3-2, 2to3-2.7 This is also the convention anywhere someone does versioned scripts in a Python package in the ecosystem. PEP439 is there to streamline the process so that python dependencies are much easier to install and there's a smaller barrier to entry so that projects like Django can give simple instructions for dependencies instead of needing to opt not to have dependencies or have to give instructions on how to install the installer. None of these commands work at the command line in a base Python install on Windows. They are all obscure enough that nobody cares (or they are GUI apps that have a start menu entry provided, i.e. Idle) - that won't be the case for pip. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 1:55 PM, Paul Moore p.f.mo...@gmail.com wrote: 1. It's not *actually* the case that the command is always pip. Maybe it's pip3 if your system makes the default Python be python 2, but you want to use python 3. Maybe you're creating a virtualenv and you haven't activated it yet. In that case a plain pip will quietly do the wrong thing (at the moment, I don't install pip in my system python precisely to avoid this issue). 2. On Windows, ...\Python34\Scripts is not on PATH by default. Even if python is (as python.exe is in a different directory to the one distutils installs executables in). Again, you can change your own PATH. 3. There's a lot of clutter. On Windows, you have 3 executables (pip.exe, pip3.exe and pip3.4.exe) and 3 scripts alongside them. For one command. Apart from the first of these, the issues are all Windows ones, and it's reasonable to say well, fix the Windows setup, then, it's silly. I have some sympathy with that view, but backward compatibility and many, many years of history will make that extremely difficult. Also, some of it may simply not be fixable because people won't agree on the solution (that's been the case in the past). 1. There's no good way to make it so you don't have to modify your command depending on what python you want to install into. In the case of both ``pip`` and ``python -m pip`` the changes a person would need to make is equivalent. They need to add a version number. The virtualenv case I don't see how that's relevant at all because if pip is preinstalled then pip will be available in both the virtualenv and the system environment. So both ``python -m pip`` and ``pip`` will be operating on the system python if you don't have it activated. 2. This sounds like something that needs fixed on Windows. Even if you say ``-m`` for pip then things are still broken by default for any other package on PyPI that installs a script. So this feels like something wrong with Python on windows not wrong with the script approach. 3. I don't really get the clutter argument. Does it *hurt* to have extra files there? I don't think i've ever looked at the directories on my $PATH and gone wow I wish there was _less_ things in here. Is this an actual problem people have? Even if it was I think user experience trumps this case. I'm not sure what Brett is exactly advocating for. If he just wants to document it as ``python -m pip`` well whatever. I have absolutely zero faith that method of invocation will ever become popular. Every single piece of documentation that i've ever seen out there for installing things with pip tells people to use ``pip install``. Every developer that I've ever seen out there is using ``pip install``. An explicit command is shorter, easier to type, and already has basically all of the mindshare behind it. People gravitate towards what's easiest and in my opinion ``-m`` is easier only for the folks implementing this, not for the end users. Even if ``python -m pip`` is documented we still need to handle the CLI case, and I think that following the convention used by most other programs on *Nix and by windows itself of making the commands, pip, pipX, and pipX.Y makes the most sense. If Brett is advocating we _remove_ the command line options and expose only the ``python -m pip`` command that I am vehemently against that and in my opinion that makes for a far worst experience than users have now. The very first thing I would do, if it did happen this way, is create a package pip-sanity on PyPI that did nothing but restored the commands and then we end up with a similar situation we have now. That people need to run some bullshit before they can start using pip in the way they want to. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] flip the pip dependencies (was Current status of PEP 439)
I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. not sure about the flip, but let me break some things down a bit for those who don't know: what pip has internally already (i.e. literally in it's package namespace): - pypi crawling/downloading - wheel installing (does not require the pypi wheel project; only building wheels requires that) what pip has bundled' already: - distlib (in 'pip.vendor'; currently only used for some --pre version logic) what pip still needs to be self-sufficient to do wheel installs: - something bundled or internal that does what pkg_resources does theoretical options: 1) bundle setuptools/pkg_resources 2) use the bundled distlib to replace our use of pkg_resources 3) internalize pkg_resources as pip.pkg_resources (i.e. fork off pkg_resources) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] flip the pip dependencies (was Current status of PEP 439)
On Jul 13, 2013, at 2:30 PM, Marcus Smith qwc...@gmail.com wrote: I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. not sure about the flip, but let me break some things down a bit for those who don't know: what pip has internally already (i.e. literally in it's package namespace): - pypi crawling/downloading - wheel installing (does not require the pypi wheel project; only building wheels requires that) what pip has bundled' already: - distlib (in 'pip.vendor'; currently only used for some --pre version logic) what pip still needs to be self-sufficient to do wheel installs: - something bundled or internal that does what pkg_resources does theoretical options: 1) bundle setuptools/pkg_resources 2) use the bundled distlib to replace our use of pkg_resources 3) internalize pkg_resources as pip.pkg_resources (i.e. fork off pkg_resources) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to break pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] flip the pip dependencies (was Current status of PEP 439)
As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to break pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip. I'm a fan of bundling too (if it works), but the dynamic install of setuptools idea also offers what you mention, although admittedly with more fragility. If a user uninstalled setuptools, it would be installed again when needed, and users only need pip to get started, and don't have to think about the setuptools dependency themselves. The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) 2) users can't upgrade themselves (for use in pip) 3) more tedium in our release process. 4) feels odd to bundle it knowing we'd likely drop it later, if we do the MEBs thing. Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] flip the pip dependencies (was Current status of PEP 439)
On Jul 13, 2013, at 3:35 PM, Marcus Smith qwc...@gmail.com wrote: As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to break pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip. I'm a fan of bundling too (if it works), but the dynamic install of setuptools idea also offers what you mention, although admittedly with more fragility. If a user uninstalled setuptools, it would be installed again when needed, and users only need pip to get started, and don't have to think about the setuptools dependency themselves. The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) 2) users can't upgrade themselves (for use in pip) 3) more tedium in our release process. 4) feels odd to bundle it knowing we'd likely drop it later, if we do the MEBs thing. Marcus 1) That's kinda FUD-y yea ;) But I'd say it's equally as likely to have weird bugs/side effects due to people using different combinations of pip/setuptools with pip than we've tested. 2) This much is true, the question then becomes how important is that? If there's a major regression in setuptools that needs fixed I'd think we'd release an updated pip. If there's new functionality I would guess we'd need to expose that in pip anyways. 3) I think this isn't as big of a deal as it sounds. Especially given we can write tooling to make it simpler :) 4) Even if MEBs were here *right now* we'd still have nearly 150k source dists that required setuptools. So either in the MEB system we'd be grabbing setuptools *a lot* or we could just bundle it to provide a better UX for people using the large corpus of existing software. I think it will be a long time once the MEBs exist before they gain enough traction that even the bulk of installs are using that system. MEBs depend on sdist 2.0 which hasn't even been started yet ;) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 13 July 2013 19:24, Donald Stufft don...@stufft.io wrote: 2. This sounds like something that needs fixed on Windows. Even if you say ``-m`` for pip then things are still broken by default for any other package on PyPI that installs a script. So this feels like something wrong with Python on windows not wrong with the script approach. It is, and it should be fixed. But in many years, nobody has managed to come up with an acceptable solution. The debates seem to be largely around what happens if you install multiple versions of Python and then remove some of them, and how badly your system PATH gets messed up by this. I don't know how many people actually do things like that, but nevertheless it's never been sorted out. (Not all of the arguments are trivial, either, there are some genuinely difficult issues to resolve, IIRC). Ultimately, I guess there are a few options: * Accept that Windows is a problem in this regard, but don't worry about it - install executable wrappers/scripts and let the user deal with path issues. * Promote python -m pip as a least common denominator approach, and mildly irritate people who don't use Windows (they can still use the commands, but the docs look odd to them). * Only provide python -m pip and seriously annoy people who don't use Windows. * Document the difference, which implies either a certain level of repetitious pip install (or py -m pip install on Windows) type of thing, or a high level For Windows, the pip command is not available directly, you should use ``python -m pip`` in its place (or wrap this in the shell if you prefer) which people may miss. It would be nice to get feedback from normal users on this. I suspect that the scientific community would make a good cross-section (AIUI there's quite a lot of Windows use, and for many people in the community Python is very much a tool, rather than a way of life :-)). Does anyone have links into the scipy groups? I lurk on the IPython lists, so I could ask there, at a pinch... Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] flip the pip dependencies (was Current status of PEP 439)
On 13 July 2013 20:35, Marcus Smith qwc...@gmail.com wrote: The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) One possible issue is with the install from sdist code, which runs Python in a subprocess with the import setuptools, etc etc incantation to force always using setuptools. That may break (or at least need changing) for a bundled setuptools, which won't be visible from the top level by default. Worth checking, anyway. Of course, this code path becomes less important as we move towards installing from wheels, but it's going to be a while before that's the norm. Paul PS Apologies if I already said this. I'm losing track of what I've replied to and what I've just thought about on this thread :-( ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 3:59 PM, Paul Moore p.f.mo...@gmail.com wrote: On 13 July 2013 19:24, Donald Stufft don...@stufft.io wrote: 2. This sounds like something that needs fixed on Windows. Even if you say ``-m`` for pip then things are still broken by default for any other package on PyPI that installs a script. So this feels like something wrong with Python on windows not wrong with the script approach. It is, and it should be fixed. But in many years, nobody has managed to come up with an acceptable solution. The debates seem to be largely around what happens if you install multiple versions of Python and then remove some of them, and how badly your system PATH gets messed up by this. I don't know how many people actually do things like that, but nevertheless it's never been sorted out. (Not all of the arguments are trivial, either, there are some genuinely difficult issues to resolve, IIRC). Ultimately, I guess there are a few options: * Accept that Windows is a problem in this regard, but don't worry about it - install executable wrappers/scripts and let the user deal with path issues. Ultimately I think this is what the community is going to do regardless of what happens here unless we remove the command line tools all together. * Promote python -m pip as a least common denominator approach, and mildly irritate people who don't use Windows (they can still use the commands, but the docs look odd to them). This also has the problem where the existing documentation (project READMEs etc) are pointing to ``pip``. So it fractures the documentation about what the command should be. * Only provide python -m pip and seriously annoy people who don't use Windows. Also invalidate all the existing documentation :) * Document the difference, which implies either a certain level of repetitious pip install (or py -m pip install on Windows) type of thing, or a high level For Windows, the pip command is not available directly, you should use ``python -m pip`` in its place (or wrap this in the shell if you prefer) which people may miss. This is probably the most realistic approach, at least in my eyes. If the Scripts directory isn't available on windows people are going to need to know to either add it or execute it with python -m pip and that's going to include documentation outside our control. So given that there s a lot of existing documentation around ``pip`` and people are likely to continue that practice windows users will need to know that when random projects say to do ``pip install foo`` they need to translate that too ``python -m pip foo``. It would be nice to get feedback from normal users on this. I suspect that the scientific community would make a good cross-section (AIUI there's quite a lot of Windows use, and for many people in the community Python is very much a tool, rather than a way of life :-)). Does anyone have links into the scipy groups? I lurk on the IPython lists, so I could ask there, at a pinch… I don't know any windows users off hand except for you ;) (And you already said you use ``pip`` and not ``python -m pip`` which already works with pip :) Paul. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] flip the pip dependencies (was Current status of PEP 439)
yea, all those comebacks make sense to me. we should try the bundle and see if it works. we already do some fancy footwork when working with setup.py https://github.com/pypa/pip/blob/develop/pip/req.py#L602 https://github.com/pypa/pip/blob/develop/pip/req.py#L687 https://github.com/pypa/pip/blob/develop/pip/req.py#L269 https://github.com/pypa/pip/blob/develop/pip/wheel.py#L291 I guess we'd be doing some additional override work in sys.modules. Marcus On Sat, Jul 13, 2013 at 12:50 PM, Donald Stufft don...@stufft.io wrote: On Jul 13, 2013, at 3:35 PM, Marcus Smith qwc...@gmail.com wrote: As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to break pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip. I'm a fan of bundling too (if it works), but the dynamic install of setuptools idea also offers what you mention, although admittedly with more fragility. If a user uninstalled setuptools, it would be installed again when needed, and users only need pip to get started, and don't have to think about the setuptools dependency themselves. The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) 2) users can't upgrade themselves (for use in pip) 3) more tedium in our release process. 4) feels odd to bundle it knowing we'd likely drop it later, if we do the MEBs thing. Marcus 1) That's kinda FUD-y yea ;) But I'd say it's equally as likely to have weird bugs/side effects due to people using different combinations of pip/setuptools with pip than we've tested. 2) This much is true, the question then becomes how important is that? If there's a major regression in setuptools that needs fixed I'd think we'd release an updated pip. If there's new functionality I would guess we'd need to expose that in pip anyways. 3) I think this isn't as big of a deal as it sounds. Especially given we can write tooling to make it simpler :) 4) Even if MEBs were here *right now* we'd still have nearly 150k source dists that required setuptools. So either in the MEB system we'd be grabbing setuptools *a lot* or we could just bundle it to provide a better UX for people using the large corpus of existing software. I think it will be a long time once the MEBs exist before they gain enough traction that even the bulk of installs are using that system. MEBs depend on sdist 2.0 which hasn't even been started yet ;) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 13 July 2013 21:14, Donald Stufft don...@stufft.io wrote: I don't know any windows users off hand except for you ;) (And you already said you use ``pip`` and not ``python -m pip`` which already works with pip :) You caught me :-) My problem is that I'm pretty sure I'm seriously atypical in never installing anything into my system Python. (And so I only use the pip command in activated virtualenvs, which *do* add the scripts directory to PATH). Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 6:44 PM, Steve Dower steve.do...@microsoft.com wrote: Because of the issues around compilation on Windows, we believe that most users avoid pip in favor of precompiled installers. The model of download an executable that matches my Python version and run it is more familiar than a command line tool, and unlikely to go away anytime soon. Luckily for them the upcoming pip 1.4 includes support for compiled packages called Wheels ;) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
Because of the issues around compilation on Windows, we believe that most users avoid pip in favor of precompiled installers. The model of download an executable that matches my Python version and run it is more familiar than a command line tool, and unlikely to go away anytime soon. Those who use pip are going to be quite capable of managing their PATH variable to ensure the correct one is used (or they'll do what I do and use a full path). There are also GUI apps, but I have no idea how widely used they are. Sent from my Windows Phone From: Paul Mooremailto:p.f.mo...@gmail.com Sent: 7/13/2013 14:36 To: Donald Stufftmailto:don...@stufft.io Cc: Distutilsmailto:distutils-sig@python.org Subject: Re: [Distutils] Expectations on how pip needs to change for Python 3.4 On 13 July 2013 21:14, Donald Stufft don...@stufft.iomailto:don...@stufft.io wrote: I don't know any windows users off hand except for you ;) (And you already said you use ``pip`` and not ``python -m pip`` which already works with pip :) You caught me :-) My problem is that I'm pretty sure I'm seriously atypical in never installing anything into my system Python. (And so I only use the pip command in activated virtualenvs, which *do* add the scripts directory to PATH). Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
Paul Moore p.f.moore at gmail.com writes: 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get stuck on the Python-supplied version. Perhaps one could add to your list, the ability to downgrade to the previous version should there be a problem with a newly-upgraded version. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip and virtualenv release candidates
pip-1.4rc4 and virtualenv-1.10rc6 are now available the changes from the previous RCs: - virtualenv now contains setuptools v0.9 (which enables indexes to use md5, sha1, or one of the sha2 variants in their urls) - the new pip install --pre option now applies to all packages installed in the command, not just top-level requirements - pip support for building and installing pybundles is now noted as deprecated (our plan is to remove it in v1.5) here's the RC install instructions again: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.10rc6.tar.gzhttps://github.com/pypa/virtualenv/archive/1.10rc4.tar.gz $ tar zxf 1.10rc6.tar.gz $ python virtualenv-1.10rc6/virtualenv.py myVE $ myVE/bin/pip --version pip 1.4rc4 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 14 July 2013 00:54, Paul Moore p.f.mo...@gmail.com wrote:
This issue has been skirted round for some time now, and I think it needs
explicit discussion, as I am not at all sure everyone has the same
expectations.
We're talking about Python 3.4 installations having pip as the default
package manager - whether by bundling, having a bootstrap process or
whatever. Regardless of the means, pip will be *the* installer for Python
3.4+. And yet, I don't think pip 1.4 currently does what people want the
Python 3.4 pip to do in some ways - and we need to make sure that any work
on the pip side is understood, agreed to, and planned to match the Python
3.4 timescales.
Good point. We also need to start to articulate the relevant questions for
the core development side of the fence - Richard, it would be good if PEP
439 could be the vehicle for this, even though it does mean I'm shifting
the goal posts on you fairly substantially.
Then we can turn it into a set of tracker issues for pip and CPython. As a
reminder, here's the current deadlines as per PEP 429 (the 3.4 release
schedule):
Changes to CPython: November 23, 2013 (3.4 beta 1)
Changes to pip: January 18, 2014 (3.4 rc 1)
(allowing changes to bundled applications up until the first RC is what I
think we *should* do, but that will require agreement from Larry Hastings
as release manager)
So, here's my initial list of things that I think people might be expecting
to happen. This is just my impressions, and I don't necessarily have a view
on the individual items. And if anyone else can think of other things to
add to the list, please do so!
1. Install to user-packages by default.
I made this suggestion at one point, but Marcus and others convinced me it
was a bad idea. Issue for better error message filed as
https://github.com/pypa/pip/issues/1048 (I don't have the power to set
milestones)
2. Not depend on setuptools (??? - Nick's inversion idea)
I think Donald is right that bundling a vendor'ed copy of setuptools is the
most sensible near-term option - there's too much risk of upgrade/downgrade
issues when allowing arbitrary combinations of pip with setuptools on
target systems for source based installs.
3. Possibly change the wrapper command name from pip to pip3 on Unix.
I think there's a bit more to it than that. Really, what we want to try to
ensure is that the following commands are available across Windows, Mac OS
X and *nix (ignoring, for the moment, the behaviour of vendor provided
installations for Mac OS X and Linux):
* python, python3, python3.4
* pip, pip3, pip3.4
That is, the version qualifier on the pip executable would relate to the
*default Python version* associated with that executable/script, rather
than the version of pip. We would take whatever steps were needed in our
Windows and Mac OS X installers to ensure all these wrappers were provided.
The reason why I think we still want to offer python -m getpip is because
I think that sends a clearer message to repackagers that we *do* consider
pip a part of Python now, but we keep the source control and issue
management for the two projects separate for pragmatic reasons (notably,
the different update lifecycles). It's still going to map to two separate
source tarballs (and hence SRPMs) for
While I acknowledge you *can* invoke the Python launcher directly on
Windows, I think it is better to leave that in the background as a tool for
advanced users, as well the engine that lets us base shebang line
processing on Windows file associations. If/when we start offering a py
style launcher on POSIX systems as well, then we can revisit that question.
4. Ensure that pip upgrading itself in-place is sufficiently robust and
reliable that users don't get stuck on the Python-supplied version.
As Vinay noted, we also need to ensure downgrades work. However, must of
these have been related to depending on an external setuptools, so
eliminating that should help a lot.
I'm sure I've seen people say other things that have made me think are
you expecting the pip maintainers to make that change? in the various
threads, so I doubt this list is definitive.
The other big one is the one you noted about pip *not* offering a stable
API, *but* exposing an apparently stable API to introspection.
Introspection currently tells me that pip exports *at least* 32 public
names (and this is without checking for public submodules that aren't
implicitly imported by pip/__init__.py):
import pip; public = set(k for k, v in pip.__dict__.items() if not
k.startswith('_') and (not hasattr(v, __name__) or hasattr(v,
__module__) or v.__name__.startswith(pip.))); print(len(public))
32
If pip really has no stable public API, then it should properly indicate
this under introspection (if it already uses relative imports correctly,
then the easiest ways to achieve that are to just shove everything under a
pip._impl subpackage or shuffle it sideways into a _pip package).
Comments anyone?
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 10:20 PM, Nick Coghlan ncogh...@gmail.com wrote: On 14 July 2013 00:54, Paul Moore p.f.mo...@gmail.com wrote: This issue has been skirted round for some time now, and I think it needs explicit discussion, as I am not at all sure everyone has the same expectations. We're talking about Python 3.4 installations having pip as the default package manager - whether by bundling, having a bootstrap process or whatever. Regardless of the means, pip will be *the* installer for Python 3.4+. And yet, I don't think pip 1.4 currently does what people want the Python 3.4 pip to do in some ways - and we need to make sure that any work on the pip side is understood, agreed to, and planned to match the Python 3.4 timescales. Good point. We also need to start to articulate the relevant questions for the core development side of the fence - Richard, it would be good if PEP 439 could be the vehicle for this, even though it does mean I'm shifting the goal posts on you fairly substantially. Then we can turn it into a set of tracker issues for pip and CPython. As a reminder, here's the current deadlines as per PEP 429 (the 3.4 release schedule): Changes to CPython: November 23, 2013 (3.4 beta 1) Changes to pip: January 18, 2014 (3.4 rc 1) Good dates to have! (allowing changes to bundled applications up until the first RC is what I think we *should* do, but that will require agreement from Larry Hastings as release manager) So, here's my initial list of things that I think people might be expecting to happen. This is just my impressions, and I don't necessarily have a view on the individual items. And if anyone else can think of other things to add to the list, please do so! 1. Install to user-packages by default. I made this suggestion at one point, but Marcus and others convinced me it was a bad idea. Issue for better error message filed as https://github.com/pypa/pip/issues/1048 (I don't have the power to set milestones) Added this to the 1.5 milestone and mentioned my agreement to the implementation on the ticket. 2. Not depend on setuptools (??? - Nick's inversion idea) I think Donald is right that bundling a vendor'ed copy of setuptools is the most sensible near-term option - there's too much risk of upgrade/downgrade issues when allowing arbitrary combinations of pip with setuptools on target systems for source based installs. https://github.com/pypa/pip/issues/1049 3. Possibly change the wrapper command name from pip to pip3 on Unix. I think there's a bit more to it than that. Really, what we want to try to ensure is that the following commands are available across Windows, Mac OS X and *nix (ignoring, for the moment, the behaviour of vendor provided installations for Mac OS X and Linux): * python, python3, python3.4 * pip, pip3, pip3.4 That is, the version qualifier on the pip executable would relate to the *default Python version* associated with that executable/script, rather than the version of pip. We would take whatever steps were needed in our Windows and Mac OS X installers to ensure all these wrappers were provided. https://github.com/pypa/pip/issues/1050 The reason why I think we still want to offer python -m getpip is because I think that sends a clearer message to repackagers that we *do* consider pip a part of Python now, but we keep the source control and issue management for the two projects separate for pragmatic reasons (notably, the different update lifecycles). It's still going to map to two separate source tarballs (and hence SRPMs) for I don't care if getpip is available especially if that's the command that is actually executed to pre-install pip for the CPython releases. (To be clear, I agree with Noah that pip should be pre-installed for every type of official release Python makes. However It does not need to be there from a hg.python.org checkout). While I acknowledge you *can* invoke the Python launcher directly on Windows, I think it is better to leave that in the background as a tool for advanced users, as well the engine that lets us base shebang line processing on Windows file associations. If/when we start offering a py style launcher on POSIX systems as well, then we can revisit that question. 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get stuck on the Python-supplied version. As Vinay noted, we also need to ensure downgrades work. However, must of these have been related to depending on an external setuptools, so eliminating that should help a lot. In the years of using pip the only time i've ever had any issue upgrading or downgrading pip was related to setuptools screw ups. But possibly we do want to have some explicit testing around this? I'm sure I've seen people say other things that have made me think are you expecting the pip maintainers
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
It is easy to forget that pip only needs the package database part of setuptools (pkg_resources.py) to install things. With the small catch that the rest of setuptools is required to install anything besides wheels. MEBS is just about implementing build requirements properly and giving pip a consistent interface to build traditional sdists *or* any new (sdist 2.0, distil, bento) kinds of packages that may come along. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip and virtualenv release candidates
pip-1.4rc4 and virtualenv-1.10rc7 are now available the changes from the previous RCs: - virtualenv now contains setuptools v0.9.1 here's the RC install instructions again: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.10rc7.tar.gz $ tar zxf 1.10rc7.tar.gz $ python virtualenv-1.10rc7/virtualenv.py myVE $ myVE/bin/pip --version pip 1.4rc4 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 14 July 2013 12:46, Donald Stufft don...@stufft.io wrote:
I'm sure I've seen people say other things that have made me think are
you expecting the pip maintainers to make that change? in the various
threads, so I doubt this list is definitive.
The other big one is the one you noted about pip *not* offering a stable
API, *but* exposing an apparently stable API to introspection.
Introspection currently tells me that pip exports *at least* 32 public
names (and this is without checking for public submodules that aren't
implicitly imported by pip/__init__.py):
import pip; public = set(k for k, v in pip.__dict__.items() if not
k.startswith('_') and (not hasattr(v, __name__) or hasattr(v,
__module__) or v.__name__.startswith(pip.))); print(len(public))
32
If pip really has no stable public API, then it should properly indicate
this under introspection (if it already uses relative imports correctly,
then the easiest ways to achieve that are to just shove everything under a
pip._impl subpackage or shuffle it sideways into a _pip package).
Pip does not use relative imports. Is simply documenting the fact there is
no public API enough? Pushing everything into a _impl or _pip directory
makes me nervous because that's a lot of code churn (and I know there are
people using those APIs, and while they aren't technically stable it feels
like moving things around just for the sake of an _ in the name is
unfriendly to those people.
Either the existing APIs are moved to a different name, or they get
declared stable and pip switches to internally forked APIs any time a
backwards incompatible change is needed for refactoring purposes (see
runpy._run_module_as_main for an example of needing to do this in the
standard library). I've had to directly deal with too many issues arising
from getting this wrong in the past for me to endorse bundling of a module
that doesn't follow this practice with CPython - if introspection indicates
an API is public, then it's public and subject to all standard library
backwards compatibility guarantees, or else we take the pain *once* and
explicitly mark it private by adding a leading underscore rather than
leaving it in limbo (contextlib._GeneratorContextManager is a standard
library example of the latter approach - it used to lack the leading
underscore, suggesting it was a public API when it's really just an
implementation detail of contextlib.contextmanager).
Mere documentation of public vs private generally doesn't cut it, as too
many people use dir(), help() and inspect() rather than the published docs
to explore APIs. The only general exception I'm aware of is test
packages, including the standard library's test package, and for those you
can make the case that having test or tests as a name segment is just
as clear an indicator of something being private as at least one name
segment starting with a leading underscore.
I really this is a fairly big ask for the pip maintainers, but I *don't*
consider Oh, don't use our module API, it isn't stable to be an adequate
answer for something that is bundled with the standard installers. Beyond
that, I don't mind if the answer is to declare the 1.5 API stable or to
sprinkle underscore where appropriate or moving everything to a private
package - the documentation and the naming conventions just need to be
consistent in their private vs public distinctions (although your points do
suggest heavily that the right answer is to accept the burden of backwards
compatibility for all APIs currently marked public, and move towards the
introduction of appropriate private APIs over time through refactoring).
Instead of a milestone I added a PEP439 tag so that we can differentiate
between 1.5 milestone items for PEP439 and not. Ideally we don't need to
drop anything from 1.5 but just in case we do.
I think we should probably target pip 1.5 to release in the beginning of
December? Would need to see what the other team members think, that's a
shorter release cycle then we normally have but I think it'd be good to
have 1.5 out for a month or so to get real world use to make sure it
doesn't need a patch release before inclusion in CPython (assuming the
dates you mentioned are correct).
Just to confuse matters a little bit, Richard has suggested explicitly
creating a bundling PEP as a *competitor* to PEP 439, thus making it easier
to be explicit about our reasons for rejecting bootstrapping in favour of
bundling. I think that's a good way to move this forward, but I won't
actually reject 439 until the competing bundling PEP has been posted
(otherwise people might get the wrong impression that we're moving away
from the idea of making pip install X work out of the box, when we're
really just changing our tactics for achieving that goal).
I also realised what is probably a better idea than python -m getpip for
dealing with the How do I get pip after doing a source build?: add a
get-pip.py utility to Tools/scripts in the
