Re: [Distutils] PEP 439 and pip bootstrap updated
On 12 July 2013 15:11, Nick Coghlan wrote: > In particular, it establishes the infrastructure to have pyvenv > automatically bootstrap the installer into each venv, even when it isn't > installed system wide (which is the key missing feature of pyvenv relative > to virtualenv). > The other thing I will note is that *if* we decide to add an implicit bootstrap later (which I doubt will happen, but you never know), then having "getpip" available as an importable module will also make *that* easier to write (since the heart of the current bootstrap code could be replaced by "from getpip import bootstrap") Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On 12 July 2013 12:12, Richard Jones wrote: > The point of PEP 439 is that the current situation of "but first do > this" for any given 3rd-party package installation was a bad thing and > we desire to move away from it. The PEP therefore proposes to allow > "just do this" to eventually become the narrative. The direction this > conversation is heading is removing that very significant primary > benefit, and I'm not convinced there's any point to the PEP in that > case. > That was never the primary benefit to my mind. The status quo sucks because there is *no* simple answer to "first, do this", not because some kind of bootstrapping is needed. The problem in my view is that the "first, do this" step is currently a mess of various arcane platform dependendent incantations that may or may not work (and may even contradict each other) and can't be readily incorporated into an automatic script because they're not idempotent. Accordingly, I consider simplifying that "first, do this" step to "python -m getpip" to be a major upgrade from the status quo: * unlike curl, wget and "python -c" incantations, it's immediately obvious to a reader what it is supposed to do: "Get pip" * unlike curl, wget and "python -c" incantations, it can be easily made platform independent * unlike curl, wget and "python -c" incantations, it can be easily made idempotent (so it does nothing if it has already been run) * through "getpip.bootstrap" it will provide the infrastructure to easily add automatic bootstrapping to other tools In particular, it establishes the infrastructure to have pyvenv automatically bootstrap the installer into each venv, even when it isn't installed system wide (which is the key missing feature of pyvenv relative to virtualenv). Having the retrieval of pip happen automagically as part of an install command initially sounded nice, but I'm now a firm -1 on that because making it work cleanly in a cross-platform way that doesn't conflict with a manual pip install has proven to require several awkward compromises that make it an ugly solution: * we have to introduce a new install command (pip3 vs pip) to avoid packaging problems on Linux distros * this is then inconsistent with Windows (which doesn't have separate versioning for the Python 3 installation) * we have to introduce new bootstrap arguments to pip * we have to special case installation of pip and its dependencies to avoid odd looking warning messages * the implementation is tricky to explain * it doesn't work nicely with the "py" launcher on Windows (or equivalents that may be added to other platforms) If your reaction is "well, in that case, I don't want to write it anymore", I will be disappointed, but that won't stop me from rejecting this approach and waiting for someone else to volunteer to write the more explicit version based on your existing bootstrap code. I'd prefer not to do that though - I'd prefer it if I can persuade you that "python -m getpip" *is* a major upgrade over the status quo that is worth implementing, and one that adheres to the Zen of Python, in particular: * Explicit is better than implicit * Simple is better than complex * Readability counts * Errors should never pass silently, unless explicitly silenced * In the face of ambiguity, refuse the temptation to guess * If the implementation is hard to explain, it's a bad idea. * If the implementation is easy to explain, it may be a good idea. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Jul 11, 2013, at 10:12 PM, Richard Jones wrote: > The point of PEP 439 is that the current situation of "but first do > this" for any given 3rd-party package installation was a bad thing and > we desire to move away from it. The PEP therefore proposes to allow > "just do this" to eventually become the narrative. The direction this > conversation is heading is removing that very significant primary > benefit, and I'm not convinced there's any point to the PEP in that > case. > > >Richard > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Now that I think about it some more I agree (and this was one of my sticking points with PyEnvs). There's already an API given to people who want to run a command to install pip: ``curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python`` Now that's platform dependent obviously but even then I don't see anyone documenting people should do that before installing things and I do think that blessing a script like that in the stdlib seems kind of pointless. The UX of the PEP as written as whenever you want to install something you run ``pip3 install foo``. The fact that pip _isn't_ bundled with Python and is instead fetched from PyPI is an implementation detail. It provides the major benefit of bundling it with Python without tying packaging to the release cycle of the stdlib (which has proven disastrous with distutils). We should remember that in general people have considered PyEnv that ships with Python 3.3 an inferior alternative to virtualenv largely in part because they have to fetch setuptools and pip prior to using it whereas in virtualenv they do not. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
The point of PEP 439 is that the current situation of "but first do this" for any given 3rd-party package installation was a bad thing and we desire to move away from it. The PEP therefore proposes to allow "just do this" to eventually become the narrative. The direction this conversation is heading is removing that very significant primary benefit, and I'm not convinced there's any point to the PEP in that case. Richard ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
Daniel Holth gmail.com> writes: > I hope we will also arrive at a pip that doesn't need to be individually > installed per venv... You mean, like distil? :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
I hope we will also arrive at a pip that doesn't need to be individually installed per venv... On Jul 11, 2013 6:13 PM, "Paul Moore" wrote: > +1 Explicit is better than implicit. > > Amending venv to automatically install pip (as suggested by Donald) may be > worth doing. I'm +0 on that (with the proviso that there's a --no-pip > option in that case). OTOH, the venv module is very extensible and writing > your own wrapper to import getpip and call bootstrap is pretty much trivial. > > On 11 July 2013 22:48, Nick Coghlan wrote: > >> (Oops, started this yesterday, got distracted and never hit send) >> >> On 11 July 2013 11:09, Richard Jones wrote: >> > >> > On 11 July 2013 06:50, Paul Moore wrote: >> > > I think "python -m pip" should be the canonical form (used in >> documentation, >> > > examples, etc). The unittest module has taken this route, as has >> timeit. >> > > Traditionally, python-dev have been lukewarm about the -m interface, >> but its >> > > key advantage is that it bypasses all the issues around versioned >> > > executables, cross-platform issues, the general dreadfulness of script >> > > wrappers on Windows, etc, in one fell swoop. >> > >> > "python -m pip" does make the bootstrapping a more complex proposition >> > - the stdlib would have to have something called "pip" that could be >> > overridden (while it is actually *running*) by something installed in >> > site-packages. Not easy. >> >> I was thinking about that, and I'm wondering if the most sensible option >> may be to claim the "getpip" name on PyPI for ourselves and then do the >> following: >> >> 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a >> 2.7.x release) >> 2. Install it to site-packages in the "Python launcher for Windows" >> installer for earlier versions >> >> getpip would expose at least one function: >> >> def bootstrap(index_url=None, system_install=False): >> ... >> >> And executing it as a main module would either: >> >> 1. Do nothing, if "import pip" already works >> 2. Call bootstrap with the appropriate arguments >> >> That way, installation instructions can simply say to unconditionally do: >> >> python -m getpip >> >> And that will either: >> >> 1. Report that pip is already installed; >> 2. Bootstrap pip into the user environment; or >> 3. Emit a distro-specific message if the distro packagers want to push >> users to use the system pip instead (since they get to patch the system >> Python and can tweak the system getpip however they want) >> >> The 2.7 change would then be to create a new download that bundles the >> Windows launcher into the Windows installer. >> >> Users aren't stupid - the problem with the status quo is really that the >> bootstrapping instructions are annoyingly complicated and genuinely >> confusing, not that an explicit bootstrapping step is needed in the first >> place. >> >> Cheers, >> Nick. >> >> >> >> >> > >> > Thanks everyone for your brilliant feedback and discussion - I look >> > forward to being able to say something sensible about Windows in the >> > PEP :-) >> >> >> > >> > >> > >> > Richard >> >> >> -- >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> On 11 July 2013 06:50, Paul Moore wrote: >> > I think "python -m pip" should be the canonical form (used in >> documentation, >> > examples, etc). The unittest module has taken this route, as has timeit. >> > Traditionally, python-dev have been lukewarm about the -m interface, >> but its >> > key advantage is that it bypasses all the issues around versioned >> > executables, cross-platform issues, the general dreadfulness of script >> > wrappers on Windows, etc, in one fell swoop. >> >> "python -m pip" does make the bootstrapping a more complex proposition >> - the stdlib would have to have something called "pip" that could be >> overridden (while it is actually *running*) by something installed in >> site-packages. Not easy. >> >> Thanks everyone for your brilliant feedback and discussion - I look >> forward to being able to say something sensible about Windows in the >> PEP :-) >> >> >> Richard >> > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
+1 Explicit is better than implicit. Amending venv to automatically install pip (as suggested by Donald) may be worth doing. I'm +0 on that (with the proviso that there's a --no-pip option in that case). OTOH, the venv module is very extensible and writing your own wrapper to import getpip and call bootstrap is pretty much trivial. On 11 July 2013 22:48, Nick Coghlan wrote: > (Oops, started this yesterday, got distracted and never hit send) > > On 11 July 2013 11:09, Richard Jones wrote: > > > > On 11 July 2013 06:50, Paul Moore wrote: > > > I think "python -m pip" should be the canonical form (used in > documentation, > > > examples, etc). The unittest module has taken this route, as has > timeit. > > > Traditionally, python-dev have been lukewarm about the -m interface, > but its > > > key advantage is that it bypasses all the issues around versioned > > > executables, cross-platform issues, the general dreadfulness of script > > > wrappers on Windows, etc, in one fell swoop. > > > > "python -m pip" does make the bootstrapping a more complex proposition > > - the stdlib would have to have something called "pip" that could be > > overridden (while it is actually *running*) by something installed in > > site-packages. Not easy. > > I was thinking about that, and I'm wondering if the most sensible option > may be to claim the "getpip" name on PyPI for ourselves and then do the > following: > > 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a > 2.7.x release) > 2. Install it to site-packages in the "Python launcher for Windows" > installer for earlier versions > > getpip would expose at least one function: > > def bootstrap(index_url=None, system_install=False): > ... > > And executing it as a main module would either: > > 1. Do nothing, if "import pip" already works > 2. Call bootstrap with the appropriate arguments > > That way, installation instructions can simply say to unconditionally do: > > python -m getpip > > And that will either: > > 1. Report that pip is already installed; > 2. Bootstrap pip into the user environment; or > 3. Emit a distro-specific message if the distro packagers want to push > users to use the system pip instead (since they get to patch the system > Python and can tweak the system getpip however they want) > > The 2.7 change would then be to create a new download that bundles the > Windows launcher into the Windows installer. > > Users aren't stupid - the problem with the status quo is really that the > bootstrapping instructions are annoyingly complicated and genuinely > confusing, not that an explicit bootstrapping step is needed in the first > place. > > Cheers, > Nick. > > > > > > > > Thanks everyone for your brilliant feedback and discussion - I look > > forward to being able to say something sensible about Windows in the > > PEP :-) > > > > > > > > > > Richard > > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > On 11 July 2013 06:50, Paul Moore wrote: > > I think "python -m pip" should be the canonical form (used in > documentation, > > examples, etc). The unittest module has taken this route, as has timeit. > > Traditionally, python-dev have been lukewarm about the -m interface, but > its > > key advantage is that it bypasses all the issues around versioned > > executables, cross-platform issues, the general dreadfulness of script > > wrappers on Windows, etc, in one fell swoop. > > "python -m pip" does make the bootstrapping a more complex proposition > - the stdlib would have to have something called "pip" that could be > overridden (while it is actually *running*) by something installed in > site-packages. Not easy. > > Thanks everyone for your brilliant feedback and discussion - I look > forward to being able to say something sensible about Windows in the > PEP :-) > > > Richard > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Jul 11, 2013, at 6:00 PM, Carl Meyer wrote: > On 07/11/2013 03:48 PM, Nick Coghlan wrote: >> I was thinking about that, and I'm wondering if the most sensible option >> may be to claim the "getpip" name on PyPI for ourselves and then do the >> following: >> >> 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a >> 2.7.x release) >> 2. Install it to site-packages in the "Python launcher for Windows" >> installer for earlier versions >> >> getpip would expose at least one function: >> >>def bootstrap(index_url=None, system_install=False): >>... >> >> And executing it as a main module would either: >> >> 1. Do nothing, if "import pip" already works >> 2. Call bootstrap with the appropriate arguments >> >> That way, installation instructions can simply say to unconditionally do: >> >>python -m getpip >> >> And that will either: >> >> 1. Report that pip is already installed; >> 2. Bootstrap pip into the user environment; or >> 3. Emit a distro-specific message if the distro packagers want to push >> users to use the system pip instead (since they get to patch the system >> Python and can tweak the system getpip however they want) >> >> The 2.7 change would then be to create a new download that bundles the >> Windows launcher into the Windows installer. >> >> Users aren't stupid - the problem with the status quo is really that the >> bootstrapping instructions are annoyingly complicated and genuinely >> confusing, not that an explicit bootstrapping step is needed in the >> first place. > > +1. This sounds far better to me than the implicit bootstrapping. > > Carl > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Generally +1, the one negative point I see is it's kinda a degradation in functionality to need to type ``python -m getpip`` in every PyEnv (coming from virtualenv). Maybe PyEnv can be smart enough to automatically install pip that's installed in the interpreter it's installed from? Maybe that's too much magic and the answer will be that tools like virtualenvwrapper will continue to exist and wrap that for you. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On 07/11/2013 03:48 PM, Nick Coghlan wrote: > I was thinking about that, and I'm wondering if the most sensible option > may be to claim the "getpip" name on PyPI for ourselves and then do the > following: > > 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a > 2.7.x release) > 2. Install it to site-packages in the "Python launcher for Windows" > installer for earlier versions > > getpip would expose at least one function: > > def bootstrap(index_url=None, system_install=False): > ... > > And executing it as a main module would either: > > 1. Do nothing, if "import pip" already works > 2. Call bootstrap with the appropriate arguments > > That way, installation instructions can simply say to unconditionally do: > > python -m getpip > > And that will either: > > 1. Report that pip is already installed; > 2. Bootstrap pip into the user environment; or > 3. Emit a distro-specific message if the distro packagers want to push > users to use the system pip instead (since they get to patch the system > Python and can tweak the system getpip however they want) > > The 2.7 change would then be to create a new download that bundles the > Windows launcher into the Windows installer. > > Users aren't stupid - the problem with the status quo is really that the > bootstrapping instructions are annoyingly complicated and genuinely > confusing, not that an explicit bootstrapping step is needed in the > first place. +1. This sounds far better to me than the implicit bootstrapping. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
+1. No magic side effects will make everyone happier. On Jul 11, 2013 5:48 PM, "Nick Coghlan" wrote: > (Oops, started this yesterday, got distracted and never hit send) > > On 11 July 2013 11:09, Richard Jones wrote: > > > > On 11 July 2013 06:50, Paul Moore wrote: > > > I think "python -m pip" should be the canonical form (used in > documentation, > > > examples, etc). The unittest module has taken this route, as has > timeit. > > > Traditionally, python-dev have been lukewarm about the -m interface, > but its > > > key advantage is that it bypasses all the issues around versioned > > > executables, cross-platform issues, the general dreadfulness of script > > > wrappers on Windows, etc, in one fell swoop. > > > > "python -m pip" does make the bootstrapping a more complex proposition > > - the stdlib would have to have something called "pip" that could be > > overridden (while it is actually *running*) by something installed in > > site-packages. Not easy. > > I was thinking about that, and I'm wondering if the most sensible option > may be to claim the "getpip" name on PyPI for ourselves and then do the > following: > > 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a > 2.7.x release) > 2. Install it to site-packages in the "Python launcher for Windows" > installer for earlier versions > > getpip would expose at least one function: > > def bootstrap(index_url=None, system_install=False): > ... > > And executing it as a main module would either: > > 1. Do nothing, if "import pip" already works > 2. Call bootstrap with the appropriate arguments > > That way, installation instructions can simply say to unconditionally do: > > python -m getpip > > And that will either: > > 1. Report that pip is already installed; > 2. Bootstrap pip into the user environment; or > 3. Emit a distro-specific message if the distro packagers want to push > users to use the system pip instead (since they get to patch the system > Python and can tweak the system getpip however they want) > > The 2.7 change would then be to create a new download that bundles the > Windows launcher into the Windows installer. > > Users aren't stupid - the problem with the status quo is really that the > bootstrapping instructions are annoyingly complicated and genuinely > confusing, not that an explicit bootstrapping step is needed in the first > place. > > Cheers, > Nick. > > > > > > > > Thanks everyone for your brilliant feedback and discussion - I look > > forward to being able to say something sensible about Windows in the > > PEP :-) > > > > > > > > > > Richard > > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > On 11 July 2013 06:50, Paul Moore wrote: > > I think "python -m pip" should be the canonical form (used in > documentation, > > examples, etc). The unittest module has taken this route, as has timeit. > > Traditionally, python-dev have been lukewarm about the -m interface, but > its > > key advantage is that it bypasses all the issues around versioned > > executables, cross-platform issues, the general dreadfulness of script > > wrappers on Windows, etc, in one fell swoop. > > "python -m pip" does make the bootstrapping a more complex proposition > - the stdlib would have to have something called "pip" that could be > overridden (while it is actually *running*) by something installed in > site-packages. Not easy. > > Thanks everyone for your brilliant feedback and discussion - I look > forward to being able to say something sensible about Windows in the > PEP :-) > > > Richard > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
(Oops, started this yesterday, got distracted and never hit send) On 11 July 2013 11:09, Richard Jones wrote: > > On 11 July 2013 06:50, Paul Moore wrote: > > I think "python -m pip" should be the canonical form (used in documentation, > > examples, etc). The unittest module has taken this route, as has timeit. > > Traditionally, python-dev have been lukewarm about the -m interface, but its > > key advantage is that it bypasses all the issues around versioned > > executables, cross-platform issues, the general dreadfulness of script > > wrappers on Windows, etc, in one fell swoop. > > "python -m pip" does make the bootstrapping a more complex proposition > - the stdlib would have to have something called "pip" that could be > overridden (while it is actually *running*) by something installed in > site-packages. Not easy. I was thinking about that, and I'm wondering if the most sensible option may be to claim the "getpip" name on PyPI for ourselves and then do the following: 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a 2.7.x release) 2. Install it to site-packages in the "Python launcher for Windows" installer for earlier versions getpip would expose at least one function: def bootstrap(index_url=None, system_install=False): ... And executing it as a main module would either: 1. Do nothing, if "import pip" already works 2. Call bootstrap with the appropriate arguments That way, installation instructions can simply say to unconditionally do: python -m getpip And that will either: 1. Report that pip is already installed; 2. Bootstrap pip into the user environment; or 3. Emit a distro-specific message if the distro packagers want to push users to use the system pip instead (since they get to patch the system Python and can tweak the system getpip however they want) The 2.7 change would then be to create a new download that bundles the Windows launcher into the Windows installer. Users aren't stupid - the problem with the status quo is really that the bootstrapping instructions are annoyingly complicated and genuinely confusing, not that an explicit bootstrapping step is needed in the first place. Cheers, Nick. > > Thanks everyone for your brilliant feedback and discussion - I look > forward to being able to say something sensible about Windows in the > PEP :-) > > > > Richard -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia On 11 July 2013 06:50, Paul Moore wrote: > I think "python -m pip" should be the canonical form (used in documentation, > examples, etc). The unittest module has taken this route, as has timeit. > Traditionally, python-dev have been lukewarm about the -m interface, but its > key advantage is that it bypasses all the issues around versioned > executables, cross-platform issues, the general dreadfulness of script > wrappers on Windows, etc, in one fell swoop. "python -m pip" does make the bootstrapping a more complex proposition - the stdlib would have to have something called "pip" that could be overridden (while it is actually *running*) by something installed in site-packages. Not easy. Thanks everyone for your brilliant feedback and discussion - I look forward to being able to say something sensible about Windows in the PEP :-) Richard ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On 11 July 2013 18:05, PJ Eby wrote: > On Thu, Jul 11, 2013 at 10:20 AM, Brett Cannon wrote: > > And if people want to promote the -m option then the executable scripts > just > > become a secondary convenience. Plus you can't exactly require > setuptools to > > create those scripts at install-time with Python if that's when they are > > going to be installed. > > You don't need setuptools in order to include .exe wrappers, though: > there's nothing setuptools-specific about the .exe files, they just > run a matching, adjacent 'foo-script.py', which can contain whatever > you want. Just take the appropriate wrapper .exe, and rename it to > whatever 'foo' you want. > > IOW, if you want to ship a pip.exe on windows that just does "from pip > import __main__; __main__()" (or whatever), you can do precisely that, > no setuptools needed. With the launcher, a .py file with the relevant #! line set pretty much covers things. It's not an exe, although there are very few things I know of that need specifically an exe file, and if you want to omit the ".py" suffix when invoking it you need to add .py to PATHEXT. But actual exe launchers are much less critical nowadays, I believe. What *is* important, though, is some level of consistency. Before setuptools promoted the idea of declaraive entry points, distributions shipped with a ridiculous variety of attempts to make cross-platform launchers (many of which didn't work very well). I care a lot more about promoting a consistent cross-platform approach than about arguing for any particular solution... Paul Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Thu, Jul 11, 2013 at 10:20 AM, Brett Cannon wrote: > And if people want to promote the -m option then the executable scripts just > become a secondary convenience. Plus you can't exactly require setuptools to > create those scripts at install-time with Python if that's when they are > going to be installed. You don't need setuptools in order to include .exe wrappers, though: there's nothing setuptools-specific about the .exe files, they just run a matching, adjacent 'foo-script.py', which can contain whatever you want. Just take the appropriate wrapper .exe, and rename it to whatever 'foo' you want. IOW, if you want to ship a pip.exe on windows that just does "from pip import __main__; __main__()" (or whatever), you can do precisely that, no setuptools needed. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Jul 11, 2013, at 11:50 AM, Brett Cannon wrote: > Yes, but you can clear it out of sys.modules before executing runpy to get > the desired effect of falling through to the regular package (runpy wouldn't > import pip.__main__ so you literally just need ``del sys.modules['pip']``). > You could also pull the old pkgutil.extend_path() trick and use the append > method on the _NamespacePath object to directly add the new directory that > pip was installed to and then import pip.runner.main(), but that feels like > more of a hack to me (but then again I'm rather comfortable mucking with the > import system =). Ok, Just making sure :) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Thu, Jul 11, 2013 at 10:52 AM, Donald Stufft wrote: > > On Jul 11, 2013, at 10:47 AM, Brett Cannon wrote: > > I'm just not seeing the downside. We control the stdlib and pip, so we > know the expected interaction and we are purposefully using the override > mechanics so it's not going to get messed up by us if we consciously use it > (and obviously have tests for it). > > > I don't think it's especially a problem for pip. I think Daniel was just > speaking how the behavior you suggested we could exploit to make this > happen has been a major issue for namespace packages in the past using the > other methods. > > However I'm not sure how it's going to work… python -m pip is going to > import the pip namespace package yes? And then when pip is installed it'll > shadow that, but in the original process where we ran python -m pip won't > the namespace package have been cached in sys.modules already? > Yes, but you can clear it out of sys.modules before executing runpy to get the desired effect of falling through to the regular package (runpy wouldn't import pip.__main__ so you literally just need ``del sys.modules['pip']``). You could also pull the old pkgutil.extend_path() trick and use the append method on the _NamespacePath object to directly add the new directory that pip was installed to and then import pip.runner.main(), but that feels like more of a hack to me (but then again I'm rather comfortable mucking with the import system =). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Thu, Jul 11, 2013 at 11:50 AM, Brett Cannon wrote: > > > > On Thu, Jul 11, 2013 at 10:52 AM, Donald Stufft wrote: > >> >> On Jul 11, 2013, at 10:47 AM, Brett Cannon wrote: >> >> I'm just not seeing the downside. We control the stdlib and pip, so we >> know the expected interaction and we are purposefully using the override >> mechanics so it's not going to get messed up by us if we consciously use it >> (and obviously have tests for it). >> >> >> I don't think it's especially a problem for pip. I think Daniel was just >> speaking how the behavior you suggested we could exploit to make this >> happen has been a major issue for namespace packages in the past using the >> other methods. >> >> However I'm not sure how it's going to work… python -m pip is going to >> import the pip namespace package yes? And then when pip is installed it'll >> shadow that, but in the original process where we ran python -m pip won't >> the namespace package have been cached in sys.modules already? >> > > Yes, but you can clear it out of sys.modules before executing runpy to get > the desired effect of falling through to the regular package (runpy > wouldn't import pip.__main__ so you literally just need ``del > sys.modules['pip']``). You could also pull the old pkgutil.extend_path() > trick and use the append method on the _NamespacePath object to directly > add the new directory that pip was installed to and then import > pip.runner.main(), but that feels like more of a hack to me (but then again > I'm rather comfortable mucking with the import system =). > And if you're still worried you can always invalidate the cache of the finder representing the parent directory pip got installed to (or all finder caches if you really want to get jumpy). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Jul 11, 2013, at 10:47 AM, Brett Cannon wrote: > I'm just not seeing the downside. We control the stdlib and pip, so we know > the expected interaction and we are purposefully using the override mechanics > so it's not going to get messed up by us if we consciously use it (and > obviously have tests for it). I don't think it's especially a problem for pip. I think Daniel was just speaking how the behavior you suggested we could exploit to make this happen has been a major issue for namespace packages in the past using the other methods. However I'm not sure how it's going to work… python -m pip is going to import the pip namespace package yes? And then when pip is installed it'll shadow that, but in the original process where we ran python -m pip won't the namespace package have been cached in sys.modules already? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Thu, Jul 11, 2013 at 10:29 AM, Daniel Holth wrote: > On Thu, Jul 11, 2013 at 9:33 AM, Paul Moore wrote: > > On 11 July 2013 13:49, Brett Cannon wrote: > >> > >> The dead-simple, extremely elegant solution (starting in Python 3.4) is > to > >> make pip a namespace package in the stdlib with nothing more than a > >> __main__.py file that installs pip; no checking if it's installed and > then > >> running it, etc, just blindly install pip. Then, if you install pip as a > >> regular package, it takes precedence and what's in the stdlib is > completely > >> ignored (this helps with any possible staleness with the stdlib's > bootstrap > >> script vs. what's in pip, etc.). You don't even need to change the > >> __main__.py in pip as it stands today since namespace packages only > work if > >> no regular package is found. > > > > > > Wow - that is exceptionally cool. I had never realised namespace packages > > would work like this. > > Not exceptionally cool ... and that's why the namespace_package form > is popular, since the first package in a set of namespace packages > that gets it wrong breaks everything. > I'm really not following that sentence. You are saying the idea is bad, but is that in general or for this specific case? And you say it's popular because people get it wrong which breaks everything? And how can namespace packages be popular if they are new to Python 3.3 (the ability to execute them with -m is new in Python 3.4)? Are you talking about pkgutil's extend_path hack because I'm talking about NamespaceLoader in importlib? I'm just not seeing the downside. We control the stdlib and pip, so we know the expected interaction and we are purposefully using the override mechanics so it's not going to get messed up by us if we consciously use it (and obviously have tests for it). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Thu, Jul 11, 2013 at 9:33 AM, Paul Moore wrote: > On 11 July 2013 13:49, Brett Cannon wrote: >> >> The dead-simple, extremely elegant solution (starting in Python 3.4) is to >> make pip a namespace package in the stdlib with nothing more than a >> __main__.py file that installs pip; no checking if it's installed and then >> running it, etc, just blindly install pip. Then, if you install pip as a >> regular package, it takes precedence and what's in the stdlib is completely >> ignored (this helps with any possible staleness with the stdlib's bootstrap >> script vs. what's in pip, etc.). You don't even need to change the >> __main__.py in pip as it stands today since namespace packages only work if >> no regular package is found. > > > Wow - that is exceptionally cool. I had never realised namespace packages > would work like this. Not exceptionally cool ... and that's why the namespace_package form is popular, since the first package in a set of namespace packages that gets it wrong breaks everything. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Thu, Jul 11, 2013 at 9:39 AM, Donald Stufft wrote: > > On Jul 11, 2013, at 9:33 AM, Paul Moore wrote: > > The pip executable script/wrapper currently uses setuptools entry points > and wrapper scripts. I'm not a fan of those, so I'd be happy to see the > change you suggest, but OTOH they have been like that since long before I > was involved with pip, and I have no idea if there are reasons they need to > stay that way. > > > Typically the reasoning is because of the .exe wrapper. > And if people want to promote the -m option then the executable scripts just become a secondary convenience. Plus you can't exactly require setuptools to create those scripts at install-time with Python if that's when they are going to be installed. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Jul 11, 2013, at 9:33 AM, Paul Moore wrote: > The pip executable script/wrapper currently uses setuptools entry points and > wrapper scripts. I'm not a fan of those, so I'd be happy to see the change > you suggest, but OTOH they have been like that since long before I was > involved with pip, and I have no idea if there are reasons they need to stay > that way. Typically the reasoning is because of the .exe wrapper. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On 11 July 2013 13:49, Brett Cannon wrote:
> The dead-simple, extremely elegant solution (starting in Python 3.4) is to
> make pip a namespace package in the stdlib with nothing more than a
> __main__.py file that installs pip; no checking if it's installed and then
> running it, etc, just blindly install pip. Then, if you install pip as a
> regular package, it takes precedence and what's in the stdlib is completely
> ignored (this helps with any possible staleness with the stdlib's bootstrap
> script vs. what's in pip, etc.). You don't even need to change the
> __main__.py in pip as it stands today since namespace packages only work if
> no regular package is found.
>
Wow - that is exceptionally cool. I had never realised namespace packages
would work like this.
> This also means pip3 literally becomes ``import runpy;
> runpy.run_module('pip')``, so that is even easier to maintain (assuming
> pip's bin/ stub isn't already doing that because of backwards-compatibility
> concerns or something with __main__.py or runpy not existing far enough
> back, otherwise it should =).
>
The pip executable script/wrapper currently uses setuptools entry points
and wrapper scripts. I'm not a fan of those, so I'd be happy to see the
change you suggest, but OTOH they have been like that since long before I
was involved with pip, and I have no idea if there are reasons they need to
stay that way.
Paul
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Wed, Jul 10, 2013 at 9:09 PM, Richard Jones wrote:
> On 11 July 2013 06:50, Paul Moore wrote:
> > I think "python -m pip" should be the canonical form (used in
> documentation,
> > examples, etc). The unittest module has taken this route, as has timeit.
> > Traditionally, python-dev have been lukewarm about the -m interface, but
> its
> > key advantage is that it bypasses all the issues around versioned
> > executables, cross-platform issues, the general dreadfulness of script
> > wrappers on Windows, etc, in one fell swoop.
>
> "python -m pip" does make the bootstrapping a more complex proposition
> - the stdlib would have to have something called "pip" that could be
> overridden (while it is actually *running*) by something installed in
> site-packages. Not easy.
>
It's also fraught with historical baggage; remember xmlplus? That was
extremely painful and something I believe everyone was glad to see go away.
Having said that, there are two solutions to this.
The compatible solution with older Python versions is to have the bootstrap
download pip and have it installed as piplib or some other alternative name
that is not masked by a pip stub in the stdlib.
The dead-simple, extremely elegant solution (starting in Python 3.4) is to
make pip a namespace package in the stdlib with nothing more than a
__main__.py file that installs pip; no checking if it's installed and then
running it, etc, just blindly install pip. Then, if you install pip as a
regular package, it takes precedence and what's in the stdlib is completely
ignored (this helps with any possible staleness with the stdlib's bootstrap
script vs. what's in pip, etc.). You don't even need to change the
__main__.py in pip as it stands today since namespace packages only work if
no regular package is found.
In case that didn't make sense, here is the file structure:
python3.4/
pip/
__main__.py # Install pip, nothing more
~/.local/
bin/
pip # Literally a shebang and two lines of Python; see below
lib/python3.4/site-packages
pip/ # As it stands today
__init__.py
__main__.py
...
This also means pip3 literally becomes ``import runpy;
runpy.run_module('pip')``, so that is even easier to maintain (assuming
pip's bin/ stub isn't already doing that because of backwards-compatibility
concerns or something with __main__.py or runpy not existing far enough
back, otherwise it should =).
-Brett
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
