subject:"\[Distutils\] What does it mean for Python to bundle pip\?"

On Thu, Aug 22, 2013 at 6:52 AM, Oscar Benjamin
oscar.j.benja...@gmail.com wrote:

 I'm pretty sure the current Windows installer just doesn't bother with
 BLAS/LAPACK libraries. Maybe it will become possible to expose them
 via a separate wheel-distributed PyPI name one day.

Well, the rule of thumb with Windows binaries is that you bundle in
(usually via static linking) all the libs you need -- numpy could have
a semi-optimized LAPACK or not, and the user shouldn't know the
difference at install time. But the trick in this case is that numpy
is used by itself, but also widely used with external C and Fortran
that might want LAPACK. (including scipy, in fact...)

But maybe this is all too much to bite off for pip and wheels. If we
could get to a state where pip install numpy and pip install scipy
would do something reasonable, if not optimized, I think that would be
great! And it's really not a big deal to say:

If you want an optimized LAPACK, etc.  for your system, you need to do
something special/ by hand/ etc...

something special may be as simple as download
numpy_optimized_just_for_this_machine.whl and install it with pip.

All that being said -- OS-X has a moderately complex binary set, what
with universal binaries, so maybe we can have a bit more meta-data
about the architecture supported.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-22 Thread Oscar Benjamin

On 22 August 2013 16:33, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 On Thu, Aug 22, 2013 at 6:52 AM, Oscar Benjamin
 oscar.j.benja...@gmail.com wrote:

 I'm pretty sure the current Windows installer just doesn't bother with
 BLAS/LAPACK libraries. Maybe it will become possible to expose them
 via a separate wheel-distributed PyPI name one day.

 Well, the rule of thumb with Windows binaries is that you bundle in
 (usually via static linking) all the libs you need -- numpy could have
 a semi-optimized LAPACK or not, and the user shouldn't know the
 difference at install time. But the trick in this case is that numpy
 is used by itself, but also widely used with external C and Fortran
 that might want LAPACK. (including scipy, in fact...)

 But maybe this is all too much to bite off for pip and wheels. If we
 could get to a state where pip install numpy and pip install scipy
 would do something reasonable, if not optimized, I think that would be
 great!

Agreed.

And actually 'pip wheel numpy' works fine for me on Windows with MinGW
installed. (I don't even need to patch distutils because
numpy.distutils fixed the MinGW bug three years ago!). There's no
BLAS/LAPACK support but I assume it has the appropriate SSE2 build
which is basically what the win32 superpack gives you.

 And it's really not a big deal to say:

 If you want an optimized LAPACK, etc.  for your system, you need to do
 something special/ by hand/ etc...

 something special may be as simple as download
 numpy_optimized_just_for_this_machine.whl and install it with pip.

Exactly. Speaking personally, I do all my real computation on Linux
clusters managed by scientific software professionals who have
hand-tuned and pre-built a bewildering array of alternative
BLAS/LAPACK setups and numpy etc. to go with. For me having numpy on
Windows is just for developing, debugging etc. so hard-core
optimisation isn't important.


Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

On Thu, Aug 22, 2013 at 9:37 AM, Paul Moore p.f.mo...@gmail.com wrote:

 That is essentially possible now.

 1. Go to Christoph Gohlke's website and download his bdist_wininst
 installers for numpy and scipy.



Exactly. And when all this settles down, hopefully Christoph, and
others, will put up binary wheels and we'll have one stop installing
that supports virtualenv, and pypi discover of good enough binary
wheels.

My point is that is may not be wise to try to support the more complex
builds -- they ARE complex, and trying to suppor it with an auto tool
is a bit much.

Oscar wrote:
 And actually 'pip wheel numpy' works fine for me on Windows with MinGW
installed.

Good start, but the bigger issue is that 'pip install' finds that wheel...

I'm still confused as to the state of all this -- are the tools ready
for project to start posting wheels so that pip can find them?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-22 Thread Paul Moore

On 22 August 2013 23:08, Chris Barker - NOAA Federal
chris.bar...@noaa.govwrote:

 I want to give it a shot for OS-X -- no one seems to want to maintian
 bdist_mpkg, and it's time to move forward...


My impression is that the architecture and fat binary stuff on OSX is the
bit that may bite you. I know little or nothing about OSX, but I'm sure if
you try and report on how you get on, the people on the list will be able
to help you get things sorted and we will be able to get any dark corners
ironed out.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

On Thu, Aug 22, 2013 at 3:52 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 22 August 2013 23:08, Chris Barker - NOAA Federal chris.bar...@noaa.gov

 My impression is that the architecture and fat binary stuff on OSX is the
 bit that may bite you.

exactly.

 I know little or nothing about OSX, but I'm sure if
 you try and report on how you get on, the people on the list will be able to
 help you get things sorted and we will be able to get any dark corners
 ironed out.

yup -- we'll never figure it out if no one uses it...

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-22 Thread Nick Coghlan

On 23 August 2013 08:03, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 On Thu, Aug 22, 2013 at 9:37 AM, Paul Moore p.f.mo...@gmail.com wrote:

 That is essentially possible now.

 1. Go to Christoph Gohlke's website and download his bdist_wininst
 installers for numpy and scipy.

 

 Exactly. And when all this settles down, hopefully Christoph, and
 others, will put up binary wheels and we'll have one stop installing
 that supports virtualenv, and pypi discover of good enough binary
 wheels.

Right - this is exactly my ambition. Have pip+wheel+virtualenv provide
a good enough out-of-the-box experience, but not necessarily support
a fully optimised experience for any given platform. There are too
many different possible full stack integration technologies for it to
make sense for us to try to supersede them all - instead, I'd like to
provide an 80% cross-platform solution that plays well enough with
others to cover the remaining 20%.

The others then includes things like zc.buildout, conda, Linux
distro package managers, Microsoft System Centre, automated
configuration management tools, PaaS providers like OpenShift and
Heroku, container technologies like Docker and the underpinnings of
OpenShift. In this space, the goal of the pip+wheel ecosystem will be
to make it possible to either use a command like pip install
zc.buildout or pip install conda to bootstrap a cross-platform
toolchain, or else to use a platform specific toolchain (like pyp2rpm)
to *consume* the upstream packages and produce nice, policy compliant,
packages automatically.

Managing arbitrary binary dependencies adds a lot of complexity for a
capability that I believe the majority of Python projects don't need.
Even the ones that can use it (like scientific tools) can often
provide a good enough fallback option that will fit within the
constraints of the draft metadata 2.0 standards.

 My point is that is may not be wise to try to support the more complex
 builds -- they ARE complex, and trying to suppor it with an auto tool
 is a bit much.

This is where I think pip install conda shows a lot of promise as a
good, cross-platform solution, at least in the scientific space.
However, the trade-off it makes is that the hash-based packaging
system means you *always* pin your dependencies when building a
package, with all the downsides that entails (mainly in increasing the
complexity of security updates). It's just that in the scientific
space, easily reproducing a particular software stack will often trump
the desire to make security updates easy to deploy with minimal impact
on other components.

That said, I'm considering the idea of adding a variant field to the
compatibility tags for wheel 1.1, along the lines of what Oscar
Benjamin suggested earlier. By default, installers would only find
wheels without a variant defined, but users could opt in to looking
for particular variants. The meaning of the variants field would be
entirely distribution specific. Numpy, for example, could publish:

numpy-1.7.1-cp27-cp22m-win32.whl
numpy-1.7.1-cp27-cp22m-win32-sse.whl
numpy-1.7.1-cp27-cp22m-win32-sse2.whl
numpy-1.7.1-cp27-cp22m-win32-sse3.whl

The only restrictions on the variant tags would be:
1. must be ASCII
2. must not contain '.' or '-' characters

You could even go to the extent of using hashes as variant tags.

 Oscar wrote:
 And actually 'pip wheel numpy' works fine for me on Windows with MinGW
 installed.

 Good start, but the bigger issue is that 'pip install' finds that wheel...

 I'm still confused as to the state of all this -- are the tools ready
 for project to start posting wheels so that pip can find them?

As others have noted:

- definite yes for building your own wheels for internal use
(including simple build caching to speed up virtualenv creation)
- qualified yes for publication on PyPI (i.e. there may still be
rough edges, so don't be too surprised if this still has flaws at this
point, especially on OS X and Linux)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-22 Thread Donald Stufft


On Aug 22, 2013, at 11:17 PM, Nick Coghlan ncogh...@gmail.com wrote:

 - qualified yes for publication on PyPI (i.e. there may still be
 rough edges, so don't be too surprised if this still has flaws at this
 point, especially on OS X and Linux)

PyPI won't even accept binary Wheels for Linux or OSX at the moment.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

On Thu, Aug 22, 2013 at 8:17 PM, Nick Coghlan ncogh...@gmail.com wrote:

 numpy-1.7.1-cp27-cp22m-win32.whl
 numpy-1.7.1-cp27-cp22m-win32-sse.whl
 numpy-1.7.1-cp27-cp22m-win32-sse2.whl
 numpy-1.7.1-cp27-cp22m-win32-sse3.whl

I'm still confused -- how would pip install numpy know which of
these to install?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-22 Thread Donald Stufft


On Aug 23, 2013, at 12:47 AM, Chris Barker - NOAA Federal 
chris.bar...@noaa.gov wrote:

 On Thu, Aug 22, 2013 at 8:17 PM, Nick Coghlan ncogh...@gmail.com wrote:
 
 numpy-1.7.1-cp27-cp22m-win32.whl
 numpy-1.7.1-cp27-cp22m-win32-sse.whl
 numpy-1.7.1-cp27-cp22m-win32-sse2.whl
 numpy-1.7.1-cp27-cp22m-win32-sse3.whl
 
 I'm still confused -- how would pip install numpy know which of
 these to install?

Most likely pip would just ignore anything that has a variant it doesn't 
understand. So it would only see the first one as an available download. 
However an alternative installer that understands the variants could use the 
additional information to select one of the SSE optimized downloads.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-21 Thread Oscar Benjamin

On 21 August 2013 08:04, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
Oscar Benjamin oscar.j.benjamin at gmail.com writes:

I think that they are responsible for installing the f2py script in
each of my Scripts directories. I never use this script and I don't
know what numpy wants with it (my understanding is that the Fortran
parts of numpy were all shifted over to scipy).

IIUC, if a third-party extension wants to use Fortran, the build process
converts it using f2py in to a Python-importable extension. It may be a
feature for distributions that use numpy, even if numpy doesn't use Fortran
itself.

Okay, that makes sense. I'm sure that's not a big problem. It won't
work very well on Windows (the case where wheels are really needed)
anyway since it doesn't have a wrapper script and won't get picked up
by make etc.

2. Tags (not in general, but AIUI numpy distribute a fancy installer that
decides what compiled code to use depending on whether you have certain CPU
features - they may want to retain that, and to do so may prefer to have
more fine-grained tags, which in turn may or may not be possible to
support). I don't think that's a critical issue though.

I guess this is what you mean:
https://github.com/numpy/numpy/blob/master/tools/win32build/cpuid/test.c

Is there no way for them to run a post-install script when pip
installing wheels from PyPI?

I'm not sure that would be enough. The numpy installation checks for various
features available at build time, and then writes numpy source code which is
then installed. When building and installing on the same machine, perhaps no
problem - but there could be problems when installation happens on a
different machine, since the sources written to the wheel at build time
would encode information about the build environment which may not be valid
in the installation environment.

ISTM for numpy to work with wheels, all of this logic would need to move
from build time to run time, but I don't know how pervasive the
source-writing approach is and how much work would be entailed in switching
over to run-time adaptation to the environment.

I may have misunderstood it but looking at this
https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147
I think that the installer ships variants for each architecture and
decides at install time which to place on the target system. If that's
the case then would it be possible for a wheel to ship all variants so
that a post-install script could sort it out (rename/delete) after the
wheel is installed?

Oscar
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-21 Thread Paul Moore

On 21 August 2013 11:29, Oscar Benjamin oscar.j.benja...@gmail.com wrote:

 I may have misunderstood it but looking at this

 https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147
 I think that the installer ships variants for each architecture and
 decides at install time which to place on the target system. If that's
 the case then would it be possible for a wheel to ship all variants so
 that a post-install script could sort it out (rename/delete) after the
 wheel is installed?


Wheel 1.0 does not have the ability to bundle multiple versions (and I
don't think tags are fine-grained enough to cover the differences numpy
need, which are at the do you have the SSE instruction set? level AIUI).
Multi-version wheels are a possible future extension, but I don't know if
anyone has thought about fine-grained tags.

This is precisely the sort of input that the numpy people could provide to
make sure that the wheel design covers their needs.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-21 Thread Oscar Benjamin

On 21 August 2013 11:39, Paul Moore p.f.mo...@gmail.com wrote:
 On 21 August 2013 11:29, Oscar Benjamin oscar.j.benja...@gmail.com wrote:

 I may have misunderstood it but looking at this

 https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147
 I think that the installer ships variants for each architecture and
 decides at install time which to place on the target system. If that's
 the case then would it be possible for a wheel to ship all variants so
 that a post-install script could sort it out (rename/delete) after the
 wheel is installed?

 Wheel 1.0 does not have the ability to bundle multiple versions (and I don't
 think tags are fine-grained enough to cover the differences numpy need,
 which are at the do you have the SSE instruction set? level AIUI).
 Multi-version wheels are a possible future extension, but I don't know if
 anyone has thought about fine-grained tags.

No, but the wheel could do like the current numpy installer and ship
_numpy.pyd.nosse
_numpy.pyd.sse1
_numpy.pyd.sse2
_numpy.pyd.sse3
as platlib files and then a post-install script can check for SSE
support, rename the appropriate file to _numpy.pyd and delete the
other _numpy.pyd.* files.

 This is precisely the sort of input that the numpy people could provide to
 make sure that the wheel design covers their needs.

I'm I right in guessing (since the question keeps being evaded :) )
that a post-install script is not possible with pip+wheel+PyPI?.


Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-21 Thread Donald Stufft


On Aug 21, 2013, at 6:46 AM, Oscar Benjamin oscar.j.benja...@gmail.com wrote:

 On 21 August 2013 11:39, Paul Moore p.f.mo...@gmail.com wrote:
 On 21 August 2013 11:29, Oscar Benjamin oscar.j.benja...@gmail.com wrote:
 
 I may have misunderstood it but looking at this
 
 https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147
 I think that the installer ships variants for each architecture and
 decides at install time which to place on the target system. If that's
 the case then would it be possible for a wheel to ship all variants so
 that a post-install script could sort it out (rename/delete) after the
 wheel is installed?
 
 Wheel 1.0 does not have the ability to bundle multiple versions (and I don't
 think tags are fine-grained enough to cover the differences numpy need,
 which are at the do you have the SSE instruction set? level AIUI).
 Multi-version wheels are a possible future extension, but I don't know if
 anyone has thought about fine-grained tags.
 
 No, but the wheel could do like the current numpy installer and ship
_numpy.pyd.nosse
_numpy.pyd.sse1
_numpy.pyd.sse2
_numpy.pyd.sse3
 as platlib files and then a post-install script can check for SSE
 support, rename the appropriate file to _numpy.pyd and delete the
 other _numpy.pyd.* files.
 
 This is precisely the sort of input that the numpy people could provide to
 make sure that the wheel design covers their needs.
 
 I'm I right in guessing (since the question keeps being evaded :) )
 that a post-install script is not possible with pip+wheel+PyPI?.
 
 
 Oscar
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig

Metadata 2.0 includes the ability to have a post install script, but Wheel is 
not yet using Metadata 2.0 (and it's not yet finalized).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-21 Thread Paul Moore

On 21 August 2013 11:47, Donald Stufft don...@stufft.io wrote:

 Metadata 2.0 includes the ability to have a post install script, but Wheel
 is not yet using Metadata 2.0 (and it's not yet finalized).


But when Metadata 2.0 support is available, what you (Oscar) suggest does
sound like a reasonable approach.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-21 Thread Nick Coghlan

On 21 Aug 2013 20:40, Paul Moore p.f.mo...@gmail.com wrote:

 On 21 August 2013 11:29, Oscar Benjamin oscar.j.benja...@gmail.com
wrote:

 I may have misunderstood it but looking at this

https://github.com/numpy/numpy/blob/master/tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in#L147
 I think that the installer ships variants for each architecture and
 decides at install time which to place on the target system. If that's
 the case then would it be possible for a wheel to ship all variants so
 that a post-install script could sort it out (rename/delete) after the
 wheel is installed?


 Wheel 1.0 does not have the ability to bundle multiple versions (and I
don't think tags are fine-grained enough to cover the differences numpy
need, which are at the do you have the SSE instruction set? level AIUI).
Multi-version wheels are a possible future extension, but I don't know if
anyone has thought about fine-grained tags.

 This is precisely the sort of input that the numpy people could provide
to make sure that the wheel design covers their needs.

I'm reasonably confident the wheel format *doesn't* meet the scientific
community's needs in the general case, and can't be made to do so without a
lot of additional complexity. That's why I explicitly support the
hashdist/conda approach which abandons some of the goals of pip and wheel
(notably, easier handling of security updates and easier conversion to
Linux distro packages) in order to better handle complex binary
dependencies.

From the pip/wheel point of view, the conda ecosystem then becomes just
another downstream system package manager to interoperate with.

Cheers,
Nick.

 Paul

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-21 Thread Paul Moore

On 21 August 2013 21:35, Nick Coghlan ncogh...@gmail.com wrote:

 I'm reasonably confident the wheel format *doesn't* meet the scientific
 community's needs in the general case, and can't be made to do so without a
 lot of additional complexity. That's why I explicitly support the
 hashdist/conda approach which abandons some of the goals of pip and wheel
 (notably, easier handling of security updates and easier conversion to
 Linux distro packages) in order to better handle complex binary
 dependencies.


While the general case may include some specialised situations, in my
view, if the wheel format isn't a suitable replacement for bdist_wininst
(and by implication, cannot be used by the numpy, scipy and similar
projects to deliver Windows binary distributions) - maybe not for
specialised use, but certainly for casual users like myself - then it will
be essentially a failure.

The only reason I am interested in wheels *at all* is as a format that
allows me to pip install all those projects that currently provide
bdist_wininst installers. In the first instance, via wheel convert, but
ultimately by the projects themselves switching from wininst format to
wheel (or via some form of build farm mechanism, it doesn't matter to me,
as long as the wheels are on PyPI).

Note that wheel convert is proof that this is the case right now, so this
is not setting the bar unreasonably high. Nor am I saying that there's even
a problem here at the moment. But if distutils-sig is sending a message
that they don't think wheels are a suitable distribution format for (say)
numpy or scipy, then I will be very disappointed.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-21 Thread Nick Coghlan

On 22 Aug 2013 06:57, Paul Moore p.f.mo...@gmail.com wrote:

 On 21 August 2013 21:35, Nick Coghlan ncogh...@gmail.com wrote:

 I'm reasonably confident the wheel format *doesn't* meet the scientific
community's needs in the general case, and can't be made to do so without a
lot of additional complexity. That's why I explicitly support the
hashdist/conda approach which abandons some of the goals of pip and wheel
(notably, easier handling of security updates and easier conversion to
Linux distro packages) in order to better handle complex binary
dependencies.


 While the general case may include some specialised situations, in my
view, if the wheel format isn't a suitable replacement for bdist_wininst
(and by implication, cannot be used by the numpy, scipy and similar
projects to deliver Windows binary distributions) - maybe not for
specialised use, but certainly for casual users like myself - then it will
be essentially a failure.

Sure, it can replace bdist_wininst. What it *can't* do is tag an
arbitrarily complex dependency tree the way hashdist can. So you're either
going to need fat wheels which figure out the right binary extensions to
use in a post-install hook, or else you're going to have to bless a default
configuration that is published as a wheel.

Since wheel 1.0 has no support for post-install hooks, that option is not
yet available, and I don't expect the idea of blessing a default binary
version to be popular either.


 The only reason I am interested in wheels *at all* is as a format that
allows me to pip install all those projects that currently provide
bdist_wininst installers. In the first instance, via wheel convert, but
ultimately by the projects themselves switching from wininst format to
wheel (or via some form of build farm mechanism, it doesn't matter to me,
as long as the wheels are on PyPI).

 Note that wheel convert is proof that this is the case right now, so
this is not setting the bar unreasonably high. Nor am I saying that there's
even a problem here at the moment. But if distutils-sig is sending a
message that they don't think wheels are a suitable distribution format for
(say) numpy or scipy, then I will be very disappointed.

Wheel is a suitable replacement for bdist_wininst (although anything that
needs install hooks will have to wait for wheel 1.1, which will support
metadata 2.0). It's just not a replacement for what hashdist and conda let
you do when you care more about reproducibility than you do about security
updates.

Cheers,
Nick.


 Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

Paul wrote:
 Given that the installer includes the py.exe launcher, if you leave the
defaults, then at a command prompt python doesn't work. But that's fine,
because py does. And if you have multiple versions of Python installed,
you don't *want* python on PATH, because then you have to manage your PATH.
Why bother when py -2.7 or py -3.3 does what you want with no path
management? Once you want any *other* executables, though, you have to deal
with PATH (especially in the multiple Pythons case). That is a new issue,
and one that hasn't been thought through yet, and we don't have a good
solution.

From a user perspective I think that 'py -3.4 -m pip ...' is an improvement
as it means I can easily install or upgrade for a particular python
installation (I tend to have a few). There's no need to put Scripts on PATH
just to run pip. I think this should be the recommended invocation for
Windows users.

Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Paul Moore

On 20 August 2013 09:13, Oscar Benjamin oscar.j.benja...@gmail.com wrote:

 Paul wrote:
  Given that the installer includes the py.exe launcher, if you leave the
 defaults, then at a command prompt python doesn't work. But that's fine,
 because py does. And if you have multiple versions of Python installed,
 you don't *want* python on PATH, because then you have to manage your PATH.
 Why bother when py -2.7 or py -3.3 does what you want with no path
 management? Once you want any *other* executables, though, you have to deal
 with PATH (especially in the multiple Pythons case). That is a new issue,
 and one that hasn't been thought through yet, and we don't have a good
 solution.

 From a user perspective I think that 'py -3.4 -m pip ...' is an
 improvement as it means I can easily install or upgrade for a particular
 python installation (I tend to have a few). There's no need to put Scripts
 on PATH just to run pip. I think this should be the recommended invocation
 for Windows users.


Thanks - I agree with you (IMO it would be nice to have pip.exe in PATH,
but it's not important enough to change the install process).

Some other points on how the various bundling approaches fit with the
Windows python installer:

1. Will the bundled pip go into the system site-packages or the user
site-packages? Does this depend on whether the user selects install for
all users or install for just me?
2. If pip goes into system site-packages, what happens with the
uninstaller? It doesn't know about pip, so it won't uninstall Python
cleanly. (Not a major point, you can delete the directory manually after
uninstalling, but it's untidy). Maybe the uninstaller should just
unconditionally delete all of site-packages as well as whatever files it
knows were installed. This is a normal issue when installing into the
system Python, but for people who avoid that and use virtualenvs (e.g. me
:-)) it's new (and annoying, as we'll never use the system pip in any
case...)

This raises another point - to an extent, I don't care about any of this,
as I routinely use virtualenvs. But if using pip to manage the system
python is becoming the recommended approach, I'd like to understand what
precisely the recommendation is so that I can see if it's better than what
I currently do - for instance, I've never used --user so I don't know if it
will be of benefit to me. I assume that this will go in the packaging user
guide in due course, but I don't know who will write it (does anyone have
the relevant experience? most people I know recommend virtualenv...)

Maybe the whole automatically bootstrapping on install should be optional
(albeit likely on by default) for people who don't want to install stuff
into their system python anyway?

Paul.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

On 20 August 2013 09:51, Paul Moore p.f.mo...@gmail.com wrote:
 1. Will the bundled pip go into the system site-packages or the user
 site-packages? Does this depend on whether the user selects install for all
 users or install for just me?

If you have get-pip then why not choose at that point whether you want
to install for the system or for all users e.g.: 'py -3.4 -m get-pip
--user' (or perhaps reverse the default)?

 2. If pip goes into system site-packages, what happens with the uninstaller?
 It doesn't know about pip, so it won't uninstall Python cleanly. (Not a
 major point, you can delete the directory manually after uninstalling, but
 it's untidy). Maybe the uninstaller should just unconditionally delete all
 of site-packages as well as whatever files it knows were installed. This is
 a normal issue when installing into the system Python, but for people who
 avoid that and use virtualenvs (e.g. me :-)) it's new (and annoying, as
 we'll never use the system pip in any case...)

Can you not just teach the Python installer to check for pip and
remove it if found?

 This raises another point - to an extent, I don't care about any of this, as
 I routinely use virtualenvs. But if using pip to manage the system python is
 becoming the recommended approach, I'd like to understand what precisely the
 recommendation is so that I can see if it's better than what I currently do
 - for instance, I've never used --user so I don't know if it will be of
 benefit to me. I assume that this will go in the packaging user guide in due
 course, but I don't know who will write it (does anyone have the relevant
 experience? most people I know recommend virtualenv...)

If I could install everything I wanted with pip then virtualenvs would
be more practical. Maybe when wheel distribution becomes commonplace
I'll start doing that. I basically always want to install a large
number of third party packages before I do anything though.

So for me the procedure on ubuntu is something like:
1) install ubuntu
2) sudo apt-get install python-numpy python-scipy python-matplotlib
ipython python-sympy python-dev cython python-pygraph python-tables
python-wxgtk2.8 python-pywt python-sphinx ...

On Windows the procedure is:
1) Install Python
2) Get MSIs for numpy, scipy, wxPython, matplotlib, PyQt, numexpr, ...
3) Setup PATH or create a shell/batch script called 'python' that does
the right thing.
4) Run ez_setup.py and Install pip
5) Patch distutils (http://bugs.python.org/issue12641)
6) Use pip for cython, sympy, ipython, pyreadline, spyder, sphinx,
docutils, line_profiler, coverage, ...
7) Build and install my own commonly used private packages.
8) Get more prebuilt binaries for other awkward packages when
necessary: pytables, numexpr, mayavi, ...

(You can see why some people just install Python(x, y) or EPD right?)

It takes quite a while to do all this and then I have basically all
the packages I want minus a few pippable ones. At this point I don't
really see the point in creating a virtualenv except to test something
that I'm personally developing. Or am I missing something?


Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Paul Moore

On 20 August 2013 11:25, Oscar Benjamin oscar.j.benja...@gmail.com wrote:

 On 20 August 2013 09:51, Paul Moore p.f.mo...@gmail.com wrote:
  1. Will the bundled pip go into the system site-packages or the user
  site-packages? Does this depend on whether the user selects install for
 all
  users or install for just me?

 If you have get-pip then why not choose at that point whether you want
 to install for the system or for all users e.g.: 'py -3.4 -m get-pip
 --user' (or perhaps reverse the default)?


That's not how get-pip is being proposed. It will run automatically on
installation of Python. If it were manually run, *and* if a --user flag was
included, then you would be correct.

 2. If pip goes into system site-packages, what happens with the
 uninstaller?
  It doesn't know about pip, so it won't uninstall Python cleanly. (Not a
  major point, you can delete the directory manually after uninstalling,
 but
  it's untidy). Maybe the uninstaller should just unconditionally delete
 all
  of site-packages as well as whatever files it knows were installed. This
 is
  a normal issue when installing into the system Python, but for people
 who
  avoid that and use virtualenvs (e.g. me :-)) it's new (and annoying, as
  we'll never use the system pip in any case...)

 Can you not just teach the Python installer to check for pip and
 remove it if found?


I'm not sure. That's my point, essentially. Who knows enough about the
Windows installer to do this correctly? If the answer is nobody, then is
a best-efforts approach with some issues that we don't have anyone who
knows how to solve, an acceptable approach? Maybe it is, I'm not claiming
that this is a major issue, but we should note it in the PEP as a
limitation.

 This raises another point - to an extent, I don't care about any of this,
 as
  I routinely use virtualenvs. But if using pip to manage the system
 python is
  becoming the recommended approach, I'd like to understand what precisely
 the
  recommendation is so that I can see if it's better than what I currently
 do
  - for instance, I've never used --user so I don't know if it will be of
  benefit to me. I assume that this will go in the packaging user guide in
 due
  course, but I don't know who will write it (does anyone have the relevant
  experience? most people I know recommend virtualenv...)

 If I could install everything I wanted with pip then virtualenvs would
 be more practical. Maybe when wheel distribution becomes commonplace
 I'll start doing that. I basically always want to install a large
 number of third party packages before I do anything though.

 So for me the procedure on ubuntu is something like:
 1) install ubuntu
 2) sudo apt-get install python-numpy python-scipy python-matplotlib
 ipython python-sympy python-dev cython python-pygraph python-tables
 python-wxgtk2.8 python-pywt python-sphinx ...

 On Windows the procedure is:
 1) Install Python
 2) Get MSIs for numpy, scipy, wxPython, matplotlib, PyQt, numexpr, ...
 3) Setup PATH or create a shell/batch script called 'python' that does
 the right thing.
 4) Run ez_setup.py and Install pip
 5) Patch distutils (http://bugs.python.org/issue12641)
 6) Use pip for cython, sympy, ipython, pyreadline, spyder, sphinx,
 docutils, line_profiler, coverage, ...
 7) Build and install my own commonly used private packages.
 8) Get more prebuilt binaries for other awkward packages when
 necessary: pytables, numexpr, mayavi, ...

 (You can see why some people just install Python(x, y) or EPD right?)

 It takes quite a while to do all this and then I have basically all
 the packages I want minus a few pippable ones. At this point I don't
 really see the point in creating a virtualenv except to test something
 that I'm personally developing. Or am I missing something?


:-)

Yes, it's a pain. My experience is better because I don't use that many
packages that need binaries. For those that I do, I maintain a local cache
of wheels that I convert from bdist_wininst installers using wheel
convert and then pip install works for everything. This is a really slick
experience, but it relies on me maintaining my local repo, and having an
appropriate -i flag added to pip (or I put it in pip.ini). As I work on
multiple PCs, it's still fiddly (I put the cache on my skydrive for ease).

But yes, if I made extensive use of binary extensions, I'd hate this
approach. That's why I keep saying that the biggest win for wheels will be
when they become the common means of distributing Windows binaries on PyPI,
in place of wininst/msi.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Nick Coghlan

On 20 Aug 2013 05:51, Paul Moore p.f.mo...@gmail.com wrote:

 On 20 August 2013 11:25, Oscar Benjamin oscar.j.benja...@gmail.com
wrote:

 On 20 August 2013 09:51, Paul Moore p.f.mo...@gmail.com wrote:
  1. Will the bundled pip go into the system site-packages or the user
  site-packages? Does this depend on whether the user selects install
for all
  users or install for just me?

 If you have get-pip then why not choose at that point whether you want
 to install for the system or for all users e.g.: 'py -3.4 -m get-pip
 --user' (or perhaps reverse the default)?


 That's not how get-pip is being proposed. It will run automatically on
installation of Python. If it were manually run, *and* if a --user flag was
included, then you would be correct.

  2. If pip goes into system site-packages, what happens with the
uninstaller?
  It doesn't know about pip, so it won't uninstall Python cleanly. (Not a
  major point, you can delete the directory manually after uninstalling,
but
  it's untidy). Maybe the uninstaller should just unconditionally delete
all
  of site-packages as well as whatever files it knows were installed.
This is
  a normal issue when installing into the system Python, but for
people who
  avoid that and use virtualenvs (e.g. me :-)) it's new (and annoying, as
  we'll never use the system pip in any case...)

 Can you not just teach the Python installer to check for pip and
 remove it if found?


 I'm not sure. That's my point, essentially. Who knows enough about the
Windows installer to do this correctly? If the answer is nobody, then is
a best-efforts approach with some issues that we don't have anyone who
knows how to solve, an acceptable approach? Maybe it is, I'm not claiming
that this is a major issue, but we should note it in the PEP as a
limitation.

  This raises another point - to an extent, I don't care about any of
this, as
  I routinely use virtualenvs. But if using pip to manage the system
python is
  becoming the recommended approach, I'd like to understand what
precisely the
  recommendation is so that I can see if it's better than what I
currently do
  - for instance, I've never used --user so I don't know if it will be of
  benefit to me. I assume that this will go in the packaging user guide
in due
  course, but I don't know who will write it (does anyone have the
relevant
  experience? most people I know recommend virtualenv...)

 If I could install everything I wanted with pip then virtualenvs would
 be more practical. Maybe when wheel distribution becomes commonplace
 I'll start doing that. I basically always want to install a large
 number of third party packages before I do anything though.

 So for me the procedure on ubuntu is something like:
 1) install ubuntu
 2) sudo apt-get install python-numpy python-scipy python-matplotlib
 ipython python-sympy python-dev cython python-pygraph python-tables
 python-wxgtk2.8 python-pywt python-sphinx ...

 On Windows the procedure is:
 1) Install Python
 2) Get MSIs for numpy, scipy, wxPython, matplotlib, PyQt, numexpr, ...
 3) Setup PATH or create a shell/batch script called 'python' that does
 the right thing.
 4) Run ez_setup.py and Install pip
 5) Patch distutils (http://bugs.python.org/issue12641)
 6) Use pip for cython, sympy, ipython, pyreadline, spyder, sphinx,
 docutils, line_profiler, coverage, ...
 7) Build and install my own commonly used private packages.
 8) Get more prebuilt binaries for other awkward packages when
 necessary: pytables, numexpr, mayavi, ...

 (You can see why some people just install Python(x, y) or EPD right?)

 It takes quite a while to do all this and then I have basically all
 the packages I want minus a few pippable ones. At this point I don't
 really see the point in creating a virtualenv except to test something
 that I'm personally developing. Or am I missing something?


 :-)

 Yes, it's a pain. My experience is better because I don't use that many
packages that need binaries. For those that I do, I maintain a local cache
of wheels that I convert from bdist_wininst installers using wheel
convert and then pip install works for everything. This is a really slick
experience, but it relies on me maintaining my local repo, and having an
appropriate -i flag added to pip (or I put it in pip.ini). As I work on
multiple PCs, it's still fiddly (I put the cache on my skydrive for ease).

 But yes, if I made extensive use of binary extensions, I'd hate this
approach. That's why I keep saying that the biggest win for wheels will be
when they become the common means of distributing Windows binaries on PyPI,
in place of wininst/msi.

Scientific users will always be better off with something like
hashdist/conda, since that ignores platform interoperability and easy
security updates in favour of hash based reproducibility. Continuum
Analytics also already take care of providing the prebuilt binary versions.

The pip ecosystem is more appropriate for pure Python code and relatively
simple C extensions

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Antoine Pitrou

Paul Moore p.f.moore at gmail.com writes:
 
 That's not how get-pip is being proposed. It will run automatically on 
 installation of Python. If it were manually run, *and* if a --user flag was
 included, then you would be correct.

+1 for providing a --user flag to python -m getpip. It is important to be
able to install things without root access, and without having to create a venv.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

On 20 August 2013 14:49, Nick Coghlan ncogh...@gmail.com wrote:

 On 20 Aug 2013 05:51, Paul Moore p.f.mo...@gmail.com wrote:

 But yes, if I made extensive use of binary extensions, I'd hate this
 approach. That's why I keep saying that the biggest win for wheels will be
 when they become the common means of distributing Windows binaries on PyPI,
 in place of wininst/msi.

 Scientific users will always be better off with something like
 hashdist/conda, since that ignores platform interoperability and easy
 security updates in favour of hash based reproducibility. Continuum
 Analytics also already take care of providing the prebuilt binary versions.

Hashdist looks useful but it's for people who will build everything
from source (as is basically required in the HPC environments for
which it is designed). This is still problematic on Windows (which is
never used for HPC).

Conda looks interesting though, I'll give that a try soon.

 The pip ecosystem is more appropriate for pure Python code and relatively
 simple C extensions (including cffi bindings).

The core extensions that I would want to put into each and every
virtualenv are things like numpy and matplotlib. These projects have
been reliably providing binary installers for Windows for years and
I'm sure that they will soon be distributing wheels. The current PyPI
binaries for numpy are here:
https://pypi.python.org/pypi/numpy
Is it not a fairly simple change to make it so that they're also
uploading wheels?

BTW is there any reason for numpy et al not to start distributing
wheels now? Is any part of the wheel
specification/tooling/infrastructure not complete yet?


Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Steve Dower

Oscar Benjamin wrote:
 Paul wrote:
 Given that the installer includes the py.exe launcher, if you leave the
 defaults, then at a command prompt python doesn't work. But that's fine,
 because py does. And if you have multiple versions of Python installed, you
 don't *want* python on PATH, because then you have to manage your PATH. Why
 bother when py -2.7 or py -3.3 does what you want with no path 
 management?
 Once you want any *other* executables, though, you have to deal with PATH
 (especially in the multiple Pythons case). That is a new issue, and one that
 hasn't been thought through yet, and we don't have a good solution.

 From a user perspective I think that 'py -3.4 -m pip ...' is an improvement as
 it means I can easily install or upgrade for a particular python installation 
 (I
 tend to have a few). There's no need to put Scripts on PATH just to run pip. I
 think this should be the recommended invocation for Windows users.

Crazy idea:

py install other args
(or 'py --install ...', but I think 'py install' is currently invalid and could 
be used?)

which the launcher executes identically to:

py -m pip install other args

(Implicitly extended to other relevant commands... I'm not proposing a complete 
list.)

Pros:
* allows implicit bootstrapping on first use (from a bundled pip, IMO, in case 
no network is available)
* multiple Python versions are handled nicely and consistently ('py -3.3 
install ...')
* can minimize officially supported API surface (as Paul described at the start 
of this thread)
* pip becomes an internal implementation detail that can be entirely replaced
* one less character of typing (slightly tongue-in-cheek, but some people count 
:) )

Cons:
* doesn't apply on *nix (or does/could it?)
* requires the most new code of any of the options
* more difficult to update the launcher than a user-installed package
* others that I can't think of because I'm suffering from confirmation bias?

Thoughts?

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Paul Moore

On 20 August 2013 16:09, Oscar Benjamin oscar.j.benja...@gmail.com wrote:

 BTW is there any reason for numpy et al not to start distributing
 wheels now? Is any part of the wheel
 specification/tooling/infrastructure not complete yet?


Not really. It's up to them to do so, though. Maybe their toolset makes
that more difficult, I don't believe they use setuptools, for example -
that's their problem, but it may not be one they are interested in solving
:-(

The biggest issues outstanding are:

1. Script handling, which is a bit flaky still (but I don't think that
affects numpy)
2. Tags (not in general, but AIUI numpy distribute a fancy installer that
decides what compiled code to use depending on whether you have certain CPU
features - they may want to retain that, and to do so may prefer to have
more fine-grained tags, which in turn may or may not be possible to
support). I don't think that's a critical issue though.

Getting numpy et al on board would be a huge win - if wheels can satisfy
their needs, we could be pretty sure we haven't missed anything. And it
gets a big group of users involved, giving us a lot more real world use
cases. Feel like sounding the numpy community out on the idea?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Brett Cannon

On Tue, Aug 20, 2013 at 11:10 AM, Steve Dower steve.do...@microsoft.comwrote:

 Oscar Benjamin wrote:
  Paul wrote:
  Given that the installer includes the py.exe launcher, if you leave the
  defaults, then at a command prompt python doesn't work. But that's
 fine,
  because py does. And if you have multiple versions of Python
 installed, you
  don't *want* python on PATH, because then you have to manage your PATH.
 Why
  bother when py -2.7 or py -3.3 does what you want with no path
 management?
  Once you want any *other* executables, though, you have to deal with
 PATH
  (especially in the multiple Pythons case). That is a new issue, and one
 that
  hasn't been thought through yet, and we don't have a good solution.
 
  From a user perspective I think that 'py -3.4 -m pip ...' is an
 improvement as
  it means I can easily install or upgrade for a particular python
 installation (I
  tend to have a few). There's no need to put Scripts on PATH just to run
 pip. I
  think this should be the recommended invocation for Windows users.

 Crazy idea:

 py install other args
 (or 'py --install ...', but I think 'py install' is currently invalid and
 could be used?)

 which the launcher executes identically to:

 py -m pip install other args

 (Implicitly extended to other relevant commands... I'm not proposing a
 complete list.)

 Pros:
 * allows implicit bootstrapping on first use (from a bundled pip, IMO, in
 case no network is available)


Nick already killed this idea. Richard's original PEP proposed this and the
idea eventually was shot down.

-Brett


 * multiple Python versions are handled nicely and consistently ('py -3.3
 install ...')
 * can minimize officially supported API surface (as Paul described at the
 start of this thread)
 * pip becomes an internal implementation detail that can be entirely
 replaced
 * one less character of typing (slightly tongue-in-cheek, but some people
 count :) )

 Cons:
 * doesn't apply on *nix (or does/could it?)
 * requires the most new code of any of the options
 * more difficult to update the launcher than a user-installed package
 * others that I can't think of because I'm suffering from confirmation
 bias?

 Thoughts?

 Cheers,
 Steve
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

On 20 August 2013 16:21, Paul Moore p.f.mo...@gmail.com wrote:
 On 20 August 2013 16:09, Oscar Benjamin oscar.j.benja...@gmail.com wrote:

 BTW is there any reason for numpy et al not to start distributing
 wheels now? Is any part of the wheel
 specification/tooling/infrastructure not complete yet?

 Not really. It's up to them to do so, though. Maybe their toolset makes that
 more difficult, I don't believe they use setuptools, for example - that's
 their problem, but it may not be one they are interested in solving :-(

They seem to be using setuptools commands here:
https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py#L48
https://github.com/numpy/numpy/blob/master/setupegg.py

 The biggest issues outstanding are:

 1. Script handling, which is a bit flaky still (but I don't think that
 affects numpy)

I think that they are responsible for installing the f2py script in
each of my Scripts directories. I never use this script and I don't
know what numpy wants with it (my understanding is that the Fortran
parts of numpy were all shifted over to scipy).

 2. Tags (not in general, but AIUI numpy distribute a fancy installer that
 decides what compiled code to use depending on whether you have certain CPU
 features - they may want to retain that, and to do so may prefer to have
 more fine-grained tags, which in turn may or may not be possible to
 support). I don't think that's a critical issue though.

I guess this is what you mean:
https://github.com/numpy/numpy/blob/master/tools/win32build/cpuid/test.c

Is there no way for them to run a post-install script when pip
installing wheels from PyPI?

 Getting numpy et al on board would be a huge win - if wheels can satisfy
 their needs, we could be pretty sure we haven't missed anything. And it gets
 a big group of users involved, giving us a lot more real world use cases.
 Feel like sounding the numpy community out on the idea?

Maybe. I'm not usually on their mailing lists but I'd be willing to
find out what they think. First I need to be clear that I know what
I'm talking about though!


Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Steve Dower

Brett Cannon wrote:
 Steve Dower wrote:
 Oscar Benjamin wrote:
 Paul wrote:
 Given that the installer includes the py.exe launcher, if you leave the
 defaults, then at a command prompt python doesn't work. But that's fine,
 because py does. And if you have multiple versions of Python installed, 
 you
 don't *want* python on PATH, because then you have to manage your PATH. Why
 bother when py -2.7 or py -3.3 does what you want with no path 
 management?
 Once you want any *other* executables, though, you have to deal with PATH
 (especially in the multiple Pythons case). That is a new issue, and one 
 that
 hasn't been thought through yet, and we don't have a good solution.

 From a user perspective I think that 'py -3.4 -m pip ...' is an improvement 
 as
 it means I can easily install or upgrade for a particular python 
 installation (I
 tend to have a few). There's no need to put Scripts on PATH just to run 
 pip. I
 think this should be the recommended invocation for Windows users.
 Crazy idea:
 
 py install other args
 (or 'py --install ...', but I think 'py install' is currently invalid and 
 could be used?)
 
 which the launcher executes identically to:
 
 py -m pip install other args
 
 (Implicitly extended to other relevant commands... I'm not proposing a 
 complete list.)
 
 Pros:
 * allows implicit bootstrapping on first use (from a bundled pip, IMO, in 
 case no network is available)
 
 Nick already killed this idea. Richard's original PEP proposed this and the 
 idea eventually was shot down.

Are you referring to the whole idea or just the implicit bootstrap? I followed 
the discussions about the latter, and it seemed that the problem was trying to 
bootstrap pip using pip.

This is different (bootstrap pip from py) and I believe it does not have the 
same issues.

 -Brett

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-20 Thread Nick Coghlan

On 20 Aug 2013 11:02, Steve Dower steve.do...@microsoft.com wrote:

 Brett Cannon wrote:
  Steve Dower wrote:
  Oscar Benjamin wrote:
  Paul wrote:
  Given that the installer includes the py.exe launcher, if you leave
the
  defaults, then at a command prompt python doesn't work. But that's
fine,
  because py does. And if you have multiple versions of Python
installed, you
  don't *want* python on PATH, because then you have to manage your
PATH. Why
  bother when py -2.7 or py -3.3 does what you want with no path
management?
  Once you want any *other* executables, though, you have to deal with
PATH
  (especially in the multiple Pythons case). That is a new issue, and
one that
  hasn't been thought through yet, and we don't have a good solution.
 
  From a user perspective I think that 'py -3.4 -m pip ...' is an
improvement as
  it means I can easily install or upgrade for a particular python
installation (I
  tend to have a few). There's no need to put Scripts on PATH just to
run pip. I
  think this should be the recommended invocation for Windows users.
  Crazy idea:
 
  py install other args
  (or 'py --install ...', but I think 'py install' is currently invalid
and could be used?)
 
  which the launcher executes identically to:
 
  py -m pip install other args
 
  (Implicitly extended to other relevant commands... I'm not proposing a
complete list.)
 
  Pros:
  * allows implicit bootstrapping on first use (from a bundled pip, IMO,
in case no network is available)
 
  Nick already killed this idea. Richard's original PEP proposed this and
the idea eventually was shot down.

 Are you referring to the whole idea or just the implicit bootstrap? I
followed the discussions about the latter, and it seemed that the problem
was trying to bootstrap pip using pip.

That and potentially relying on network access at runtime, as well as
potentially running into permissions issues depending on where Python was
installed. Implicit bootstrap just seems like a recipe for hard to debug
failure modes.

 This is different (bootstrap pip from py) and I believe it does not have
the same issues.

It certainly avoids some of them, but not all.

I think Paul Moore has the right idea: treat scripts on Windows as a
distinct problem deserving of a separate PEP. That general solution will
then apply to pip as well. In the meantime py -m pip feels like a
tolerable Windows specific workaround.

Cheers,
Nick.


  -Brett

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] What does it mean for Python to bundle pip?

While debate on PEP 439 has died down, I've been thinking about what it
actually means to end users for Python to bundle pip. If we have an
explicit bootstrapping command like get_pip, how much practical benefit is
there for that to be bundled vs having a well-known location for such a
script (which is more or less what we have now)?

Also, there is the whole question of how much we want to lock down what
pip currently provides, as official functionality. (Particularly given the
renewed energy around projects like distlib/distil, etc).

I'd like to propose that as a first step, we agree and document a *user
interface* that will be officially supported going forward. It would be
based on (a subset of) the pip interface, so that a bundle pip strategy
is still the immediate way forward, but by standardising the interface
rather than the implementation, we retain the option to change things
behind the scenes if, for example, distil suddenly takes over the world.

Users get the benefit of a properly sanctioned one true way to manage
packages. And we reserve enough flexibility that we don't accidentally
accept constraints that are stronger than we'd like and stifle innovation.

To be specific, what I propose is that we agree on the following:

1. Python installations provide a bootstrap command (tentatively python -m
get_pip, I'm not sure this needs to be directly executable, the -m
interface is probably fine). This command may be a no-op if for example we
opt for full bundling (and it will always be a no-op after the first run).
2. Once bootstrapped, Python will provide a pip command (or pip3?)
accessible whenever the python command is available (worded this way
because the Windows installer doesn't put python on PATH by default).
3. A specific subset of the current pip interface is provided (to be
documented). I'd suggest we need a minimum of pip install, pip list, pip
uninstall, pip install -U (for upgrades). I don't know what proportion of
the rest of the pip API (various options, requirement files, environment
variables and ini files) we want to lock down - I suggest we start minimal
and add items as needed.
4. We may want to add a separate note that python -m pip will do the same
as the pip command, and may be needed to self-upgrade the pip command
(python -m pip -U pip).

We can note for 3.4 that the pip command will be implemented using the
current pip project, and so all of the other features of pip will be
available, but with the proviso that any interfaces not explicitly
mentioned in the standard documentation may be changed or removed as pip
changes, and are not protected by Python's compatibility rules. That gives
users a reasonable level of stability, while still allowing us the room to
innovate.

Does this sound reasonable? It may be obvious to everyone but me that this
is what we're expecting - or I may even be proposing *more* than people
expect. But I think something along the lines of an interface over
implementation definition like this would be reasonable.

If this is a useful proposal and no-one else is already working on
something similar, I'm willing to write this up as a PEP.

Paul.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-19 Thread Donald Stufft

I'm only briefly checking my email before going back to sleep but just thought 
I'd mention I have a successor to PEP439 mostly written and I was planning on 
finishing it up based on some feedback i'd gotten and publishing it this week.

More specific response when I'm not half asleep.

On Aug 19, 2013, at 7:31 AM, Paul Moore p.f.mo...@gmail.com wrote:

 While debate on PEP 439 has died down, I've been thinking about what it 
 actually means to end users for Python to bundle pip. If we have an explicit 
 bootstrapping command like get_pip, how much practical benefit is there for 
 that to be bundled vs having a well-known location for such a script (which 
 is more or less what we have now)?
 
 Also, there is the whole question of how much we want to lock down what pip 
 currently provides, as official functionality. (Particularly given the 
 renewed energy around projects like distlib/distil, etc).
 
 I'd like to propose that as a first step, we agree and document a *user 
 interface* that will be officially supported going forward. It would be based 
 on (a subset of) the pip interface, so that a bundle pip strategy is still 
 the immediate way forward, but by standardising the interface rather than the 
 implementation, we retain the option to change things behind the scenes if, 
 for example, distil suddenly takes over the world.
 
 Users get the benefit of a properly sanctioned one true way to manage 
 packages. And we reserve enough flexibility that we don't accidentally accept 
 constraints that are stronger than we'd like and stifle innovation.
 
 To be specific, what I propose is that we agree on the following:
 
 1. Python installations provide a bootstrap command (tentatively python -m 
 get_pip, I'm not sure this needs to be directly executable, the -m interface 
 is probably fine). This command may be a no-op if for example we opt for full 
 bundling (and it will always be a no-op after the first run).
 2. Once bootstrapped, Python will provide a pip command (or pip3?) 
 accessible whenever the python command is available (worded this way 
 because the Windows installer doesn't put python on PATH by default).
 3. A specific subset of the current pip interface is provided (to be 
 documented). I'd suggest we need a minimum of pip install, pip list, pip 
 uninstall, pip install -U (for upgrades). I don't know what proportion of the 
 rest of the pip API (various options, requirement files, environment 
 variables and ini files) we want to lock down - I suggest we start minimal 
 and add items as needed.
 4. We may want to add a separate note that python -m pip will do the same 
 as the pip command, and may be needed to self-upgrade the pip command 
 (python -m pip -U pip).
 
 We can note for 3.4 that the pip command will be implemented using the 
 current pip project, and so all of the other features of pip will be 
 available, but with the proviso that any interfaces not explicitly mentioned 
 in the standard documentation may be changed or removed as pip changes, and 
 are not protected by Python's compatibility rules. That gives users a 
 reasonable level of stability, while still allowing us the room to innovate.
 
 Does this sound reasonable? It may be obvious to everyone but me that this is 
 what we're expecting - or I may even be proposing *more* than people expect. 
 But I think something along the lines of an interface over implementation 
 definition like this would be reasonable.
 
 If this is a useful proposal and no-one else is already working on something 
 similar, I'm willing to write this up as a PEP.
 
 Paul.
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-19 Thread Nick Coghlan

On 19 Aug 2013 07:54, Donald Stufft don...@stufft.io wrote:

 I'm only briefly checking my email before going back to sleep but just
thought I'd mention I have a successor to PEP439 mostly written and I was
planning on finishing it up based on some feedback i'd gotten and
publishing it this week.

 More specific response when I'm not half asleep.

Donald sent me the preliminary draft last week, so I can share the short
version of the proposal:

* Bootstrap included in the standard library, executable as python -m
getpip. This ensures a clear distinction between files managed by the
system installer (i.e. getpip) and those managed by pip (including pip
itself)
* Installs the bundled copy of pip by default, but has an option to do a
network install of the latest from PyPI
* Updates of the installed version use pip install --upgrade pip as normal
* Bundled version inside getpip will be updated when pip is updated (so
CPython maintenance releases may have newer versions of pip)
* If you build from source, you need to run the bootstrap explicitly to get
pip
* Windows and Mac OS X installers will bootstrap the bundled pip by default
(opt-out) with a network update option (opt-in)

The following two points aren't in Donald's PEP yet, and are the main
reason the initial draft wasn't published more widely:

* We'll ask the Linux distros to define a circular runtime dependency
between python and python-pip (I spoke to Toshio about that at Flock last
weekend and Fedora is amenable, but we haven't asked any Debian or Ubuntu
folks yet)
* pyvenv will install pip by default (with an option to opt-out)

The main open question is how we uninstall Python without leaving pip
managed files around. If an appropriate command doesn't already exist, pip
may need to grow a python -m pip uninstall --all --force command that the
Python uninstallers can invoke.

The bootstrap will also need to be able to control whether or not script
wrappers are installed and the name they use and/or key them off
sys.implementation somehow.

Cheers,
Nick.

 On Aug 19, 2013, at 7:31 AM, Paul Moore p.f.mo...@gmail.com wrote:

  While debate on PEP 439 has died down, I've been thinking about what it
actually means to end users for Python to bundle pip. If we have an
explicit bootstrapping command like get_pip, how much practical benefit is
there for that to be bundled vs having a well-known location for such a
script (which is more or less what we have now)?
 
  Also, there is the whole question of how much we want to lock down
what pip currently provides, as official functionality. (Particularly given
the renewed energy around projects like distlib/distil, etc).
 
  I'd like to propose that as a first step, we agree and document a *user
interface* that will be officially supported going forward. It would be
based on (a subset of) the pip interface, so that a bundle pip strategy
is still the immediate way forward, but by standardising the interface
rather than the implementation, we retain the option to change things
behind the scenes if, for example, distil suddenly takes over the world.
 
  Users get the benefit of a properly sanctioned one true way to manage
packages. And we reserve enough flexibility that we don't accidentally
accept constraints that are stronger than we'd like and stifle innovation.
 
  To be specific, what I propose is that we agree on the following:
 
  1. Python installations provide a bootstrap command (tentatively
python -m get_pip, I'm not sure this needs to be directly executable, the
-m interface is probably fine). This command may be a no-op if for example
we opt for full bundling (and it will always be a no-op after the first
run).
  2. Once bootstrapped, Python will provide a pip command (or pip3?)
accessible whenever the python command is available (worded this way
because the Windows installer doesn't put python on PATH by default).
  3. A specific subset of the current pip interface is provided (to be
documented). I'd suggest we need a minimum of pip install, pip list, pip
uninstall, pip install -U (for upgrades). I don't know what proportion of
the rest of the pip API (various options, requirement files, environment
variables and ini files) we want to lock down - I suggest we start minimal
and add items as needed.
  4. We may want to add a separate note that python -m pip will do the
same as the pip command, and may be needed to self-upgrade the pip
command (python -m pip -U pip).
 
  We can note for 3.4 that the pip command will be implemented using the
current pip project, and so all of the other features of pip will be
available, but with the proviso that any interfaces not explicitly
mentioned in the standard documentation may be changed or removed as pip
changes, and are not protected by Python's compatibility rules. That gives
users a reasonable level of stability, while still allowing us the room to
innovate.
 
  Does this sound reasonable? It may be obvious to everyone but me that
this is what we're

Re: [Distutils] What does it mean for Python to bundle pip?

On 19 August 2013 15:28, Nick Coghlan ncogh...@gmail.com wrote:

  More specific response when I'm not half asleep.

 Donald sent me the preliminary draft last week, so I can share the short
 version of the proposal:

 * Bootstrap included in the standard library, executable as python -m
 getpip. This ensures a clear distinction between files managed by the
 system installer (i.e. getpip) and those managed by pip (including pip
 itself)
 * Installs the bundled copy of pip by default, but has an option to do a
 network install of the latest from PyPI
 * Updates of the installed version use pip install --upgrade pip as
 normal
 * Bundled version inside getpip will be updated when pip is updated (so
 CPython maintenance releases may have newer versions of pip)
 * If you build from source, you need to run the bootstrap explicitly to
 get pip
 * Windows and Mac OS X installers will bootstrap the bundled pip by
 default (opt-out) with a network update option (opt-in)

 The following two points aren't in Donald's PEP yet, and are the main
 reason the initial draft wasn't published more widely:


Thanks for the update, Nick. I'll wait for Donald's comments and the
published version of the PEP before commenting in detail, but the following
points come to mind for me at least:

1. I would still like to see *some* clear statement of what the guaranteed
interface users can rely on is. Maybe I'm suggesting more than anyone else
is willing to commit to, but I *do* think that if we're suggesting that
there is now a standard Python installer, we should give users some
indication of what it actually is. For example, I think it's reasonable as
a user to assume that pip install foo will work for the foreseeable
future - but not that (for example) pip bundle will.

2. We need to be careful to define exactly when and how the pip command
is present. Don't forget that on Windows, the python command is not on
PATH by default (and the existence of the launcher means that it really
doesn't need to be). I would suggest that we say something like The pip
command will be installed alongside the python command, and will be
available when python is[1]. We should also probably note that versioned
variants of pip will be provided matching the versioned copies of the
python command that are available (e.g., pip3/pip3.4 on Unix, maybe none at
all on Windows...). Unless of course we want to use a different scheme for
pip, in which case we need to agree on what that will be.

3. This also begs the question of whether pip.exe gets installed in the
Scripts subdirectory on Windows, as at present - if it does, we'll have
to be very careful indeed over how we word the instructions, as it's *easy*
for users to have python.exe on PATH but not have Scripts\pip.exe on there
:-(

Paul.

[1] I don't like that wording, but I can't find a better way of putting it
that covers all cases but isn't too wordy :-(
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-19 Thread Nick Coghlan

On 19 August 2013 10:08, Paul Moore p.f.mo...@gmail.com wrote:
 2. We need to be careful to define exactly when and how the pip command is
 present. Don't forget that on Windows, the python command is not on PATH
 by default (and the existence of the launcher means that it really doesn't
 need to be). I would suggest that we say something like The pip command
 will be installed alongside the python command, and will be available when
 python is[1]. We should also probably note that versioned variants of pip
 will be provided matching the versioned copies of the python command that
 are available (e.g., pip3/pip3.4 on Unix, maybe none at all on Windows...).
 Unless of course we want to use a different scheme for pip, in which case we
 need to agree on what that will be.

In 3.3+, I believe the Windows installer does PATH modification by
default. In 3.4+ it will likely do PATHEXT modification, too.

 3. This also begs the question of whether pip.exe gets installed in the
 Scripts subdirectory on Windows, as at present - if it does, we'll have to
 be very careful indeed over how we word the instructions, as it's *easy* for
 users to have python.exe on PATH but not have Scripts\pip.exe on there :-(

We should just add Scripts to the PATH in the installer as well.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

On 19 August 2013 22:21, Donald Stufft don...@stufft.io wrote:

 My current draft explicitly ensures that the end result is the same as if
 someone installed pip manually. There will be no importable pip from the
 standard library.


OK, in which case I think you need to explain in the PEP why this is a
benefit compared to users just manually installing pip by hand. In
particular why doesn't pip just supply getpip.py for download?

You also need to address the fact that installing pip (whether by hand or
via the bootstrap) on Windows, does not normally make the pip command
available (because Python34\Scripts is not on %PATH%). Benefits: install
process matches the current one, disadvantages: all user documentation
needs to add some level of verbiage about how to add pip to %PATH%.

It also specifically excludes pip from the backwards compat and other
 governance related issues of CPython.


I agree with this, but I think we should offer some core stability
guarantees - even if it's only that python -m pip install X will install
X... At the moment, the PEP doesn't really offer any tangible benefits for
the end user.

I'm not sure if it's the place of this PEP, but *something* needs to
document the reasons why we are even proposing that pip gets bundled with
Python. From an end user perspective, the immediate impression is that
everything's going to be hugely better, because we'll have a package
manager that can install, upgrade, uninstall, etc, built into Python. From
there on, however, the picture is one of a series of little disappointments
- I need to explicitly bootstrap. Oh, OK., And now I have to set PATH.
Yech, but fair enough. And wait, the interface isn't stable? What?[1]

The various technical issues have started to obscure the original
user-facing reasons why we want to do this at all. If we don't document
those reasons, we risk ending up with a technical solution that doesn't
actually address the original issue. Which would be a shame.

I can understand if Donald feels that the original rationale isn't
something he wants to cover (I don't know if he was involved in the
discussions - IIRC, they were at PyCon, and I know for example that *I*
don't have a clear picture of all the details, as I wasn't there...) Maybe
Nick would be better placed to ad a Why are we doing this? section.

This is an outdated draft, it needs updated which once I have time in the
 next few days i'll update it and post it but this is my draft
 https://github.com/dstufft/peps/blob/master/pip-bundling.rst


I hope the above is useful, and not out of date with where you plan on
going with your updates. Apologies if I'm jumping the gun here.

Paul.

[1] Yes, I know it is stable - but the docs will say not covered by Python
stability guarantees which gives the *impression* not stable :-(
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

On 19 August 2013 22:37, Nick Coghlan ncogh...@gmail.com wrote:

 On 19 August 2013 10:08, Paul Moore p.f.mo...@gmail.com wrote:
  2. We need to be careful to define exactly when and how the pip
 command is
  present. Don't forget that on Windows, the python command is not on
 PATH
  by default (and the existence of the launcher means that it really
 doesn't
  need to be). I would suggest that we say something like The pip command
  will be installed alongside the python command, and will be available
 when
  python is[1]. We should also probably note that versioned variants of
 pip
  will be provided matching the versioned copies of the python command that
  are available (e.g., pip3/pip3.4 on Unix, maybe none at all on
 Windows...).
  Unless of course we want to use a different scheme for pip, in which
 case we
  need to agree on what that will be.

 In 3.3+, I believe the Windows installer does PATH modification by
 default. In 3.4+ it will likely do PATHEXT modification, too.


Add python to PATH is off by default. And I think it only adds
C:\Python33 to PATH, not C:\Python33\Scripts (which isn't even created by
the installer). I haven't tested this, though.

PATHEXT modification is handled by the code for Register Python
extensions which is on by default. But that's separate from add python to
PATH.

Given that the installer includes the py.exe launcher, if you leave the
defaults, then at a command prompt python doesn't work. But that's fine,
because py does. And if you have multiple versions of Python installed,
you don't *want* python on PATH, because then you have to manage your PATH.
Why bother when py -2.7 or py -3.3 does what you want with no path
management? Once you want any *other* executables, though, you have to deal
with PATH (especially in the multiple Pythons case). That is a new issue,
and one that hasn't been thought through yet, and we don't have a good
solution.

There's a reason that python.exe was not on PATH by default for all these
years. The launcher let us avoid most of the issues, and having it optional
and off by default solved most of the remaining issues. But all of the
problems still apply equally to pip.exe - and we don't have
workarounds/solutions yet for that.

At a minimum, if you want to set PATH to include
C:\Python33;C:\Python33\Scripts by default, this should be proposed on
python-dev. And be prepared for questions like what if you install
multiple Pythons?, how will uninstall work?, etc. They have been raised
before and never answered satisfactorily. TBH, I doubt it'll be resolved in
time for beta 1.

 3. This also begs the question of whether pip.exe gets installed in the
  Scripts subdirectory on Windows, as at present - if it does, we'll
 have to
  be very careful indeed over how we word the instructions, as it's *easy*
 for
  users to have python.exe on PATH but not have Scripts\pip.exe on there
 :-(

 We should just add Scripts to the PATH in the installer as well.


See above.

I'm -1 on changing any of the current installer PATH handling for Windows
as a side-effect of this proposal. I think we need to devise a solution
that works with the current behaviour (which might mean installing a
pip.exe alongside python.exe, and documenting the command as pip, if you
have python on PATH, otherwise py -m pip).

OTOH, if you want to raise the issue of fixing the current Python install
layout and process for Windows, then I'd be fine with that. The current
layout is messy, and annoyingly different than Unix. But (a) it's an issue
for python-dev, and (b) I don't think you should expect that debate to be
resolved in time to hold this proposal up for it.

Paul.

PS I'm carefully *not* mentioning the differing layout used by
virtualenv/venv in the above, because it will muddy the water. But it is
probably relevant. And if the system Python is on PATH and you activate a
virtualenv, there could be some difficult interactions there - that's why I
explicitly don't add my system Python to PATH, or install packages into it.
I think we need some end user best practice advice, if we're going to start
advocating using a pip installed in the system Python (maybe always use
--user is enough). Hmm, I'm starting to find more questions than answers
here. Time to stop for a while...
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

2013-08-19 Thread Donald Stufft


On Aug 19, 2013, at 5:43 PM, Paul Moore p.f.mo...@gmail.com wrote:

 On 19 August 2013 22:21, Donald Stufft don...@stufft.io wrote:
 My current draft explicitly ensures that the end result is the same as if 
 someone installed pip manually. There will be no importable pip from the 
 standard library.
 
 OK, in which case I think you need to explain in the PEP why this is a 
 benefit compared to users just manually installing pip by hand. In particular 
 why doesn't pip just supply getpip.py for download?
 
 You also need to address the fact that installing pip (whether by hand or via 
 the bootstrap) on Windows, does not normally make the pip command available 
 (because Python34\Scripts is not on %PATH%). Benefits: install process 
 matches the current one, disadvantages: all user documentation needs to add 
 some level of verbiage about how to add pip to %PATH%.
 
 It also specifically excludes pip from the backwards compat and other 
 governance related issues of CPython.
 
 I agree with this, but I think we should offer some core stability guarantees 
 - even if it's only that python -m pip install X will install X... At the 
 moment, the PEP doesn't really offer any tangible benefits for the end user.

The benefit is that the installers will run the bootstrap during the install 
process, so ``pip`` works out of the box. The typical end user will not be 
running ``python -m getpip``.

 
 I'm not sure if it's the place of this PEP, but *something* needs to document 
 the reasons why we are even proposing that pip gets bundled with Python. From 
 an end user perspective, the immediate impression is that everything's going 
 to be hugely better, because we'll have a package manager that can install, 
 upgrade, uninstall, etc, built into Python. From there on, however, the 
 picture is one of a series of little disappointments - I need to explicitly 
 bootstrap. Oh, OK., And now I have to set PATH. Yech, but fair enough. 
 And wait, the interface isn't stable? What?[1]


As said above they typically won't have to manually bootstrap it. I 
specifically excludes the PATH stuff from the PEP because in my mind that's a 
separate issue. I haven't used windows in a long enough time I don't feel 
qualified to speak on it (And it's not an issue under OSX which is the other 
installer distributed by Python). However that problem affects all installed 
distributions with command line scripts on Windows and probably should have a 
more generic solution.

 
 The various technical issues have started to obscure the original user-facing 
 reasons why we want to do this at all. If we don't document those reasons, we 
 risk ending up with a technical solution that doesn't actually address the 
 original issue. Which would be a shame.
 
 I can understand if Donald feels that the original rationale isn't something 
 he wants to cover (I don't know if he was involved in the discussions - IIRC, 
 they were at PyCon, and I know for example that *I* don't have a clear 
 picture of all the details, as I wasn't there...) Maybe Nick would be better 
 placed to ad a Why are we doing this? section.

I have 7 paragraphs of rationale. Im not sure what additional information 
should be added to them in order to cover it? Part of what I hope to do before 
official release is reword things to be a bit more clear.

 
 This is an outdated draft, it needs updated which once I have time in the 
 next few days i'll update it and post it but this is my draft 
 https://github.com/dstufft/peps/blob/master/pip-bundling.rst
 
 I hope the above is useful, and not out of date with where you plan on going 
 with your updates. Apologies if I'm jumping the gun here.
 
 Paul. 
 
 [1] Yes, I know it is stable - but the docs will say not covered by Python 
 stability guarantees which gives the *impression* not stable :-(

Perhaps the wording should explicitly mention that it is not covered by 
Python's stability guarantees but is instead covered by pip's backwards compat 
policy (and then we need to actually close the loop on that :V).


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?