Re: [Distutils] Versioned entry points
On Mar 28, 2014, at 6:51 PM, Nick Coghlan wrote: > On 29 March 2014 07:23, Paul Moore wrote: >> On 28 March 2014 21:06, Nick Coghlan wrote: >>> So consider me in the school that suggests this be a standard installer >>> feature that can be applied to any entry point script, rather than something >>> that varies by project. >> >> Doing that also implies (detecting and) deliberately ignoring any >> versioned entry points created by setup.py. I'd suggest going for a >> rule that if a project defines two entry points with the same >> definitions, where one ends in the pattern -?\d(\.\d)? (in other >> words, -X, -X.Y, X or X.Y) but they are otherwise the same, then >> ignore the versioned one. If we don't do this, we'll likely end up >> with abominations like virtualenv-2.7-3.4.exe. > > We need to be careful with that - scripts may end with a version > number unrelated to the Python version. If we can get agreement on > what we want the *default* idiom to look like (more on that below), > then we can start considering the special cases where that idiom > breaks down (like those where the base script name already ends with a > number, perhaps by inserting a leading "-py" into the suffix in those > cases). > >> I'm not 100% sure whether you're saying that the installer should >> mandate a versioning policy, or if the user can specify. The >> altinstall case suggests we need a user override option. > > I'm biased because there are actually two places where this matters. > It's not just the user scripts directory (everywhere), but also the > system binary directories on Linux distros. > > (Note: I haven't run the ideas below by Slavek or Toshio yet, so take > the finer details with a grain of salt - but the general principles > should be right. I don't believe Slavek is making it to PyCon, but > Toshio is, so we should be able to thrash out some more specifics > there) > > From a Linux distro perspective, these versioned scripts should > arguably have different shebang lines - ignoring the parts that would > remain the same, "pip" would correspond to "python", "pip3" to > "python3" and "pip3.4" to "python3.4". Otherwise you > could change the "python" symlink to point somewhere else, and then > have to figure out why "pip" appeared to succeed, but you couldn't > import things you just installed (because they were installed into the > wrong version). > > For system wide installs, *which* ones to install should also be > possible to largely automate by following a convention based on the > final name element in sys.executable (and that approach would also fix > the problem of potential conflicts in the user scripts directory). > > There's also the fact that for universal (or otherwise cross-version) > wheels, which scripts get installed should be governed by the version > of Python used for the *installation*, rather than the one used to do > the build. > > If we put this under the control of the installer, then there is some > hope of eventually bringing order to the chaos and getting some kind > of consistency in naming schemes and alignment with the shebang lines. > If we leave it up to the individual projects? Not a chance (especially > since we *can't* get it right if the versioned scripts are embedded in > wheel files at build time). > >> There's also the question of if I do "pip install virtualenv >> --use-versioned X,X.Y" and then "pip install -U virtualenv" will the >> installer remember my versioning choices for the upgrade? Doing so >> needs some complex logic. Not doing so means that "pip install -U pip" >> could change the executables ensurepip had chosen to create. > > Yeah, I think driving it based on sys.executable is a better idea, > with a fallback of "-" as the suffix if the final > component of sys.executable doesn't follow one of the recognised > patterns (python, pythonX, pythonX.Y). The reason for using > sys.executable is because that is the thing that can tell us the > interpreter's role *with respect to the larger system*. > > Under that model, on a current Fedora system, we would see the > following behaviour when run under the following interpreters: > > python -> pip, pip2, pip2.7 > python2 -> pip2, pip2.7 > python2.7 -> pip2.7 > python3 -> pip3, pip3.3 > python3.3 -> pip3.3 > pypy -> pip-pypy > > That avoids collisions not only in /usr/bin (or /usr/local/bin if we > change the install directory for scripts on POSIX), but also in > ~/.local/bin (and the per-user directory on Windows). > > On Windows system wide installs, since the Scripts directory path > already includes the Python version, I'd just install all 3 in all > cases. Ditto for virtual environments (even when using an alternate > interpreter implementation). > > That's essentially using ensurepip's behaviour as precedent for how we > *want* Python scripts to behave. > > You may ask "why would you ever do that?", and from my perspective, > the answer is because Fedora is migrating over to using "pip" in th
Re: [Distutils] Versioned entry points
On 29 March 2014 07:23, Paul Moore wrote: > On 28 March 2014 21:06, Nick Coghlan wrote: >> So consider me in the school that suggests this be a standard installer >> feature that can be applied to any entry point script, rather than something >> that varies by project. > > Doing that also implies (detecting and) deliberately ignoring any > versioned entry points created by setup.py. I'd suggest going for a > rule that if a project defines two entry points with the same > definitions, where one ends in the pattern -?\d(\.\d)? (in other > words, -X, -X.Y, X or X.Y) but they are otherwise the same, then > ignore the versioned one. If we don't do this, we'll likely end up > with abominations like virtualenv-2.7-3.4.exe. We need to be careful with that - scripts may end with a version number unrelated to the Python version. If we can get agreement on what we want the *default* idiom to look like (more on that below), then we can start considering the special cases where that idiom breaks down (like those where the base script name already ends with a number, perhaps by inserting a leading "-py" into the suffix in those cases). > I'm not 100% sure whether you're saying that the installer should > mandate a versioning policy, or if the user can specify. The > altinstall case suggests we need a user override option. I'm biased because there are actually two places where this matters. It's not just the user scripts directory (everywhere), but also the system binary directories on Linux distros. (Note: I haven't run the ideas below by Slavek or Toshio yet, so take the finer details with a grain of salt - but the general principles should be right. I don't believe Slavek is making it to PyCon, but Toshio is, so we should be able to thrash out some more specifics there) >From a Linux distro perspective, these versioned scripts should arguably have different shebang lines - ignoring the parts that would remain the same, "pip" would correspond to "python", "pip3" to "python3" and "pip3.4" to "python3.4". Otherwise you could change the "python" symlink to point somewhere else, and then have to figure out why "pip" appeared to succeed, but you couldn't import things you just installed (because they were installed into the wrong version). For system wide installs, *which* ones to install should also be possible to largely automate by following a convention based on the final name element in sys.executable (and that approach would also fix the problem of potential conflicts in the user scripts directory). There's also the fact that for universal (or otherwise cross-version) wheels, which scripts get installed should be governed by the version of Python used for the *installation*, rather than the one used to do the build. If we put this under the control of the installer, then there is some hope of eventually bringing order to the chaos and getting some kind of consistency in naming schemes and alignment with the shebang lines. If we leave it up to the individual projects? Not a chance (especially since we *can't* get it right if the versioned scripts are embedded in wheel files at build time). > There's also the question of if I do "pip install virtualenv > --use-versioned X,X.Y" and then "pip install -U virtualenv" will the > installer remember my versioning choices for the upgrade? Doing so > needs some complex logic. Not doing so means that "pip install -U pip" > could change the executables ensurepip had chosen to create. Yeah, I think driving it based on sys.executable is a better idea, with a fallback of "-" as the suffix if the final component of sys.executable doesn't follow one of the recognised patterns (python, pythonX, pythonX.Y). The reason for using sys.executable is because that is the thing that can tell us the interpreter's role *with respect to the larger system*. Under that model, on a current Fedora system, we would see the following behaviour when run under the following interpreters: python -> pip, pip2, pip2.7 python2 -> pip2, pip2.7 python2.7 -> pip2.7 python3 -> pip3, pip3.3 python3.3 -> pip3.3 pypy -> pip-pypy That avoids collisions not only in /usr/bin (or /usr/local/bin if we change the install directory for scripts on POSIX), but also in ~/.local/bin (and the per-user directory on Windows). On Windows system wide installs, since the Scripts directory path already includes the Python version, I'd just install all 3 in all cases. Ditto for virtual environments (even when using an alternate interpreter implementation). That's essentially using ensurepip's behaviour as precedent for how we *want* Python scripts to behave. You may ask "why would you ever do that?", and from my perspective, the answer is because Fedora is migrating over to using "pip" in the RPM build process for Python packages, rather than calling setup.py directly. This means we can more easily adapt to changes in upstream conventions and automatically take advantage of the upcoming metadata 2.0 enhancements and
Re: [Distutils] Versioned entry points
On 28 March 2014 21:06, Nick Coghlan wrote: > So consider me in the school that suggests this be a standard installer > feature that can be applied to any entry point script, rather than something > that varies by project. Doing that also implies (detecting and) deliberately ignoring any versioned entry points created by setup.py. I'd suggest going for a rule that if a project defines two entry points with the same definitions, where one ends in the pattern -?\d(\.\d)? (in other words, -X, -X.Y, X or X.Y) but they are otherwise the same, then ignore the versioned one. If we don't do this, we'll likely end up with abominations like virtualenv-2.7-3.4.exe. I'm not 100% sure whether you're saying that the installer should mandate a versioning policy, or if the user can specify. The altinstall case suggests we need a user override option. There's also the question of if I do "pip install virtualenv --use-versioned X,X.Y" and then "pip install -U virtualenv" will the installer remember my versioning choices for the upgrade? Doing so needs some complex logic. Not doing so means that "pip install -U pip" could change the executables ensurepip had chosen to create. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
On 29 Mar 2014 06:20, "Vinay Sajip" wrote: > > > Does anyone have any thoughts? > > Note that there is one situation where scripts for multiple Python versions reside in the same directory: per-user site-packages (PEP 370). If you install a package which has scripts and you don't write versioned scripts, a second installation of that package (with a different Python version) will cause the first script to be overwritten. Much of the time this won't matter, but there might be scenarios where it does. > > It may also give a problem when uninstalling, or when verifying the installation (as the hash of an overwritten script may not match what's expected, as per the original installation). > > To avoid overwriting, one would need to write versioned scripts *only* :-) Speaking of which, it's worth describing exactly what we had to do for ensurepip. That has 3 configurations: POSIX system default: pip3, pip3.4 Windows & venv default: pip, pip3, pip3.4 Alt install: pip3.4 There's currently an issue where "make altinstall && make install" is slightly broken, as it won't add the missing "pip3" command. So consider me in the school that suggests this be a standard installer feature that can be applied to any entry point script, rather than something that varies by project. Cheers, Nick. > > Regards, > > Vinay Sajip > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/28/2014 01:18 PM, Vinay Sajip wrote: >> Does anyone have any thoughts? > > Note that there is one situation where scripts for multiple Python > versions reside in the same directory: per-user site-packages (PEP > 370). If you install a package which has scripts and you don't write > versioned scripts, a second installation of that package (with a > different Python version) will cause the first script to be > overwritten. Much of the time this won't matter, but there might be > scenarios where it does. > > It may also give a problem when uninstalling, or when verifying the > installation (as the hash of an overwritten script may not match > what's expected, as per the original installation). > > To avoid overwriting, one would need to write versioned scripts > *only* :-) Heh, sounds like a call for Underdog^W: $ python setup.py altinstall Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1vkAACgkQ+gerLs4ltQ5CLACgodRwkXiS9irK/nP3dKPnlaXO 5GYAoKbJ4NiDFJdHNIWPozIpMzN6+z62 =q7Bh -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
On 28 March 2014 14:56, Paul Moore wrote: > [2] What's the process for agreeing on extension schemas? Do they need > their own PEPs, or what? I was skimming the discussion at the point > when these things got pushed out to extensions, so I missed the > detail. I just found PEP 459. So I guess the question is should the "commands" extension include some sort of "versioned" flag in the wrap_console/wrap_gui data? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
I would like to see just a very simple $ interpolation language. On Mar 28, 2014 11:03 AM, "Paul Moore" wrote: > On 28 March 2014 14:56, Paul Moore wrote: > > [2] What's the process for agreeing on extension schemas? Do they need > > their own PEPs, or what? I was skimming the discussion at the point > > when these things got pushed out to extensions, so I missed the > > detail. > > I just found PEP 459. So I guess the question is should the "commands" > extension include some sort of "versioned" flag in the > wrap_console/wrap_gui data? > > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
> Does anyone have any thoughts? Note that there is one situation where scripts for multiple Python versions reside in the same directory: per-user site-packages (PEP 370). If you install a package which has scripts and you don't write versioned scripts, a second installation of that package (with a different Python version) will cause the first script to be overwritten. Much of the time this won't matter, but there might be scenarios where it does. It may also give a problem when uninstalling, or when verifying the installation (as the hash of an overwritten script may not match what's expected, as per the original installation). To avoid overwriting, one would need to write versioned scripts *only* :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
Definitely the project. On Mar 28, 2014 11:51 AM, "Paul Moore" wrote: > On 28 March 2014 15:32, Daniel Holth wrote: > > I would like to see just a very simple $ interpolation language. > > So you think the project should decide, not the user? Or should the > user be allowed to override? > Paul > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
On Mar 28, 2014, at 11:51 AM, Paul Moore wrote: > On 28 March 2014 15:32, Daniel Holth wrote: >> I would like to see just a very simple $ interpolation language. > > So you think the project should decide, not the user? Or should the > user be allowed to override? > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig For what it’s worth, if the person invoking the pip command picks that would remove a need for a hack we have in place for ensurepip. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
On 28 March 2014 15:32, Daniel Holth wrote: > I would like to see just a very simple $ interpolation language. So you think the project should decide, not the user? Or should the user be allowed to override? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Versioned entry points
One open question with pip's handling of script generation for wheels is what to do with versioned entry points (virtualenv-2.7.exe etc). At the moment, pip explicitly disables distlib's facility to generate "versioned" script wrappers for entry points, and generates exactly what the project metadata specifies[1]. The problem is that some projects use sys.version in setup.py to explicitly specify versioned commands. This does not work for wheels, where the build and install environments are different. There's an open issue for virtualenv, which has hit this problem, and I suspect there are a number of other projects which will hit it if & when they start publishing wheels (I think py.test and nose have the same problem, for a start). The first question is, who should be responsible for deciding whether to have versioned commands anyway? Is that a project decision, or should it be left to the user doing the install? Should the user be allowed to override a project default? If it's a user decision, pip probably needs an additional command line argument to specify whether the user wants versioned commands. We may also want pip to ignore any existing entry point specifications that look like they are versioned. (And of course someone needs to tell the projects that currently specify versioned entry points that they should stop :-)) If it's a project decision, we need a means for the project to record that data in metadata that pip can read. For Metadata 2.0, that means something in the console script metadata extension [2]. For the short term, we may need some sort of convention or addition for entry_points.txt. But I'm not sure who would be affected by something like that (beyond the obvious answer of setuptools and pip...) Does anyone have any thoughts? Paul [1] There's a hack in there to generate versioned entry points for pip and easy_install. [2] What's the process for agreeing on extension schemas? Do they need their own PEPs, or what? I was skimming the discussion at the point when these things got pushed out to extensions, so I missed the detail. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
