[Distutils] Serverside Dependency Resolution and Virtualenv Build Server

[Thomas Güttler]

I wrote down a tought about Serverside Dependency Resolution and Virtualenv 
Build Server


What do you think?

Latest version: https://github.com/guettli/virtualenv-build-server

virtualenv-build-server
###

Rough roadmap how a server to build virtualenvs for the python programming 
language could be implemented.

Highlevel goal
--

Make creating new virtual envionments for the Python programming language easy 
and fast. 

Input: fuzzy requirements like this: django>=1.8, requests=>2.7

Output: virtualenv with packages installed.

Two APIs


 #. Resolve fuzzy requirements to a fixed set of packages with exactly pinned 
versions.
 #. Read fixed set of packages. Build virtualenv according to given platform.


Steps
-

Steps:

#. Client sends list of fuzzy requirements to server: 

   * I need: django>=1.8, requests=>2.7, ...

 
#. Server solves the fuzzy requirements to a fixed set of requirememnts: 
django==1.8.2, requests==2.8.1, ...

#. Client reads the fixed set of requirements.

#. Optional: Client sends fixed set of requirements to the server. Telling him 
the plattform

   * My platform: sys.version==2.7.6 and sys.platform=linux2

#. Server builds a virtualenv according to the fixed set of requirements.

#. Server sends the environment to the client

#. Client unpacks the data and has a usable virtualenv

Benefits


Speed: 

* There is only one round-trip from client to server. If the dependencies get 
resolved on the client the client would need to download the available version 
information.
* Caching: If the server gets input parameters (fuzzy requirements and platform 
information) which he has seen before, he can return the cached result from the 
previous request.

Possible Implementations


APIs

Both APIs could be implementated by a webservice/Rest interface passing json or 
yaml.

Serverside
==


Implementation Strategie "PostgreSQL"
.

Since the API is de-coupled from the internals the implementation could be 
exchanged without the need for changes at the client side.

I suggest using the PostgreSQL und resolving the dependcy graph using SQL (WITH 
RECURSIVE).

The package and version data gets stored in PostgreSQL via ORM (Django or 
SQLAlchemy).

The version numbers need to be normalized to ascii to allow fast comparision.

Related: https://www.python.org/dev/peps/pep-0440/

Implementation Strategie "Node.js"
..

I like python, but I am not married with it. Why not use a different tools that 
is already working? Maybe the node package manager: https://www.npmjs.com/

Questions
-
Are virtualenv relocatable? AFAIK they are not.

General Thoughts


 * Ignore Updates. Focus on creating new virtualenvs. The server can do caching 
and that's why I prefer creating virtualenvs which never get updated. They get 
created and removed (immutable).
 

I won't implement it


This idea is in the public domain. If you are young and brave or old and wise: 
Go ahead, try to implement it. Please communicate early and often. Ask on 
mailing-lists or me for feedback. Good luck :-)

I love feedback
---

Please tell me want you like or dislike:

 * typos and spelling stuff (I am not a native speaker)
 * alternative implementation strategies.
 * existing software which does this (even if implemented in a different 
programming language).
 * ...
 
-- 
http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Donald Stufft]

On November 7, 2015 at 8:56:02 AM, Ralf Gommers (ralf.gomm...@gmail.com) wrote:
> On Sat, Nov 7, 2015 at 2:02 PM, Paul Moore wrote:
>  
> > On 7 November 2015 at 01:26, Chris Barker - NOAA Federal
> > wrote:
> > > So what IS supposed to be used in the development workflow? The new
> > > mythical build system?
> >
>  
> I'd like to point out again that this is not just about development
> workflow. This is just as much about simply *installing* from a local git
> repo, or downloaded sources/sdist.
>  
> The "pip install . should reinstall" discussion in
> https://github.com/pypa/pip/issues/536 is also pretty much the same
> argument.

I think that everyone on that ticket has agreed that ``pip install .`` (where . 
is any local path) should reinstall. I think the thing that is being asked for 
here though is for pip to use that directory as the build directory, rather 
than copying everything to a temporary directory and using that. I’m hesitant 
to do that because it’s going to add another slightly different way that things 
could be installed and I’m trying to reduce those (and instead have two “paths” 
for installation, the normal one and the development one).

IOW, I think in development ``-e`` is the right answer if you want to build and 
use the local directory. Otherwise you shouldn’t expect it to modify your 
current directory or the tarball at all. I do think we can make sure that 
specifying a build directory and instructing us not to clean it will function 
to have incremental builds though.

>  
> Fair question. Unfortunately, the answer is honestly that there's no
> > simple answer - pip is not a bad option, but it's not its core use
> > case so there are some rough edges.
>  
>  
> My impression is that right now pip's core use-case is not "installing",
> but "installing from PyPi (and similar repos". There are a lot of rough
> edges around installing from anything on your own hard drive.

This is probably true just in the fact that the bulk of the time when people 
use it, they are using it to install from a remote repository. There are rough 
edges for stuff on your own hard drive, but I think we can clean them up 
though, we just need to figure out what the answer is for each of those rough 
cases.

>  
>  
> > I'd argue that the best way to use
> > pip is with pip install -e, but others in this thread have said that
> > doesn't suit their workflow, which is fine. I don't know of any other
> > really good options, though.
> >
> > I think it would be good to see if we can ensure pip is useful for
> > this use case as well, all I was pointing out was that people
> > shouldn't assume that it "should" work right now, and that changing it
> > to work might involve some trade-offs that we don't want to make, if
> > it compromises the core functionality of installing packages.
> >
>  
> It might be helpful to describe the actual trade-offs then, because as far
> as I can tell no one has actually described how this would either hurt
> another use-case or make pip internals much more complicated.
>  
> Ralf
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>  

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Paul Moore]

On 7 November 2015 at 01:26, Chris Barker - NOAA Federal
 wrote:
> So what IS supposed to be used in the development workflow? The new
> mythical build system?

Fair question. Unfortunately, the answer is honestly that there's no
simple answer - pip is not a bad option, but it's not its core use
case so there are some rough edges. I'd argue that the best way to use
pip is with pip install -e, but others in this thread have said that
doesn't suit their workflow, which is fine. I don't know of any other
really good options, though.

I think it would be good to see if we can ensure pip is useful for
this use case as well, all I was pointing out was that people
shouldn't assume that it "should" work right now, and that changing it
to work might involve some trade-offs that we don't want to make, if
it compromises the core functionality of installing packages.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
> Maybe it would be clearer to drop the comment and newline handling
> stuff from the core requirement specifier syntax (declaring that
> newlines are simply a syntax error), and assume that there's some
> higher-level framing protocol taking care of that stuff?


that sounds right to me.



>   https://gist.github.com/njsmith/ed74851c0311e858f0f7
>
> (Nice to see Metadata-Version: 2.0 getting some real-world use! I
> guess???


the use of 2.0 already is odd
see here for old discussion in the wheel tracker:
https://bitbucket.org/pypa/wheel/issues/96/metadata-version-20
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Ralf Gommers]

On Sat, Nov 7, 2015 at 2:02 PM, Paul Moore  wrote:

> On 7 November 2015 at 01:26, Chris Barker - NOAA Federal
>  wrote:
> > So what IS supposed to be used in the development workflow? The new
> > mythical build system?
>

I'd like to point out again that this is not just about development
workflow. This is just as much about simply *installing* from a local git
repo, or downloaded sources/sdist.

The "pip install . should reinstall" discussion in
https://github.com/pypa/pip/issues/536 is also pretty much the same
argument.

Fair question. Unfortunately, the answer is honestly that there's no
> simple answer - pip is not a bad option, but it's not its core use
> case so there are some rough edges.


My impression is that right now pip's core use-case is not "installing",
but "installing from PyPi (and similar repos". There are a lot of rough
edges around installing from anything on your own hard drive.


> I'd argue that the best way to use
> pip is with pip install -e, but others in this thread have said that
> doesn't suit their workflow, which is fine. I don't know of any other
> really good options, though.
>
> I think it would be good to see if we can ensure pip is useful for
> this use case as well, all I was pointing out was that people
> shouldn't assume that it "should" work right now, and that changing it
> to work might involve some trade-offs that we don't want to make, if
> it compromises the core functionality of installing packages.
>

It might be helpful to describe the actual trade-offs then, because as far
as I can tell no one has actually described how this would either hurt
another use-case or make pip internals much more complicated.

Ralf
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Paul Moore]

On 7 November 2015 at 13:55, Ralf Gommers  wrote:
> On Sat, Nov 7, 2015 at 2:02 PM, Paul Moore  wrote:
>>
>> On 7 November 2015 at 01:26, Chris Barker - NOAA Federal
>>  wrote:
>> > So what IS supposed to be used in the development workflow? The new
>> > mythical build system?
>
> I'd like to point out again that this is not just about development
> workflow. This is just as much about simply *installing* from a local git
> repo, or downloaded sources/sdist.

Possibly I'm misunderstanding here.

> The "pip install . should reinstall" discussion in
> https://github.com/pypa/pip/issues/536 is also pretty much the same
> argument.

Well, that one is about pip reinstalling if you install from a local
directory, and not skipping the install if the local directory version
is the same as the installed version. As I noted there, I'm OK with
this, it seems reasonable to me to say that if someone has a directory
of files, they may have updated something but not (yet) bumped the
version.

The debate over there has gone on to whether we force reinstall for a
local *file* (wheel or sdist) which I'm less comfortable with. But
that's is being covered over there.

The discussion *here* is, I thought, about skipping build steps when
possible because you can reuse build artifacts. That's not "should pip
do the install?", but rather "*how* should pip do the install?"
Specifically, to reuse build artifacts it's necessaryto *not* do what
pip currently does for all (non-editable) installs, which is to
isolate the build in a temporary directory and do a clean build.
That's a sensible debate to have, but it's very different from the
issue you referenced.

IMO, the discussions currently are complex enough that isolating
independent concerns is crucial if anyone is to keep track. (It
certainly is for me!)

>> Fair question. Unfortunately, the answer is honestly that there's no
>> simple answer - pip is not a bad option, but it's not its core use
>> case so there are some rough edges.
>
> My impression is that right now pip's core use-case is not "installing", but
> "installing from PyPi (and similar repos". There are a lot of rough edges
> around installing from anything on your own hard drive.

Not true. The rough edges are around installing things where (a) you
don't want to rely in the invariant that name and version uniquely
identify an installation (that's issue 536) and (b) where you don't
want to do a clean build, because building is complex, slow, or
otherwise something you want to optimise (that's this discussion).

I routinely download wheels and use them to install. I also sometimes
download sdists and install from them, although 99.99% of the time, I
download them, build them into wheels and install them from wheels. It
*always* works exactly as I'd expect. But if I'm doing development, I
use -e. That seems to be the problem here, there are rough edges if
you want a development workflow that doesn't rely on editable
installs. I think that's what I already said :-)

>> I'd argue that the best way to use
>> pip is with pip install -e, but others in this thread have said that
>> doesn't suit their workflow, which is fine. I don't know of any other
>> really good options, though.
>>
>> I think it would be good to see if we can ensure pip is useful for
>> this use case as well, all I was pointing out was that people
>> shouldn't assume that it "should" work right now, and that changing it
>> to work might involve some trade-offs that we don't want to make, if
>> it compromises the core functionality of installing packages.
>
> It might be helpful to describe the actual trade-offs then, because as far
> as I can tell no one has actually described how this would either hurt
> another use-case or make pip internals much more complicated.

1. (For issue 536, not this thread) Pip and users can't rely on the
invariant that name and version uniquely identify a release. You could
have version 1.2dev4 installed, and it may have come from your local
working directory (with changes you made) or from a wheel that's on
your local hard drive that you built last week, or from the release on
PyPI you made last month. All 3 may behave differently. Also wheel
caching is based on name/version - it would need to be switched off in
cases where name/version doesn't guarantee repeatable code.
2. (For here) Builds are not isolated from what's in the development
directory. So if you have your sdist definition wrong, what you build
locally may work, but when you release it it may fail. Obviously that
can be fixed by proper development and testing practices, but pip is
designed currently to isolate builds to protect against mistakes like
this, we'd need to remove that protection for cases where we wanted to
do in-place builds.
3. The logic inside pip for doing builds is already pretty tricky.
Adding code to sometimes build in place and sometimes in a temporary
directory is going to 

Re: [Distutils] New PEP : dependency specification

[Robert Collins]

On unknown names, current pkg_resources raises SyntaxError. So I think
we need to update the spec from that perspective to be clear. All PEP
426 defined variables are handled without error by current
pkg_resources/markerlib implementation. Perhaps the new variables
should raise rather than evaluating to '' / 0 ? Some discussion /
thought would be good. Certainly when used and evaluated by an
existing pkg_resources they will error - so perhaps we should not add
any new variables at this point?

So, if we don't unify this with the wheel encoding of extras, it will
require multiple parsers indefinitely. I need to think more about
whether it makes sense or not.

Wheel certainly needs a way to say 'this distribution exports extras
X, Y, Z (and their respective dependencies)'. flit and other tools
producing wheels need the same facility.

https://www.python.org/dev/peps/pep-0427 doesn't define this use of
markers; but pip and wheel have collaborated on it. PEP-345 doesn't
describe Provides-Extra, which pkg_resources uses when parsing
.dist-info directories as well (it determines which extra variables
get substituted into the set of requires to determine the values of
the extras...). So there's basically still a bunch of underspecified
behaviours out there in the wild, and since my strategy is to minimal
variation vs whats there, we need to figure out the right places to
split things to layer this well.

Specifying a new variable of 'extra' is fairly easy: we need to decide
on the values it will take, and thats well defined but layer crossing:
when processing a dependency with one or more extras, you need to loop
over all the dependency specifications once with each extra defined
(including I suspect for completeness '' for the non-extras) and then
union together the results.

So at this layer I think we could say that:
 - extra is only valid if the context that is interpreting the
specification defines it
 - when invalid it will raise SyntaxError

This allows a single implementation to handle .dist-info directories
as it does today, while specifying it more fully. It leaves it open in
future for us to continue modelling exported extras as
marker-filtered-specifications + a Provides-Extra, or to move to
exported extras as something in a hash in a richer serialisation
format, or some third thing. This is good I think.

I do like the idea of comments and line continuations being removed.
We can then explicitly say that this DSL is going to be embedded in a
larger context such as requirements.txt files, requires headers etc,
and that those contexts may provide multi line handling as desired.

I'll apply Nathaniel's excellent review details + this on Monday and
issue an update.

-Rob

On 8 November 2015 at 05:08, Marcus Smith  wrote:
>>
>> I'm not sure this is the syntax that I would have come up with, but I
>> guess it's too late to put the genie back in the bottle, so this PEP
>> should have some way to cope with these things?
>
>
> why would this PEP deal with this?
> the higher level PEP that builds on top of this would bump the wheel
> metadata version (probably to 3.0, due to the jump to 2.0 already)
> *then* tools will cope with this based on the metadata version
>



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
>  PEP-345 doesn't
> describe Provides-Extra, which pkg_resources uses when parsing
> .dist-info directories as well


fwiw, this provides a bit of history on the "Provides-Extra":
https://github.com/pypa/interoperability-peps/issues/44
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
> I'm not sure this is the syntax that I would have come up with, but I
> guess it's too late to put the genie back in the bottle, so this PEP
> should have some way to cope with these things?


why would this PEP deal with this?
the higher level PEP that builds on top of this would bump the wheel
metadata version (probably to 3.0, due to the jump to 2.0 already)
*then* tools will cope with this based on the metadata version
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Ralf Gommers]

On Sat, Nov 7, 2015 at 3:57 PM, Paul Moore  wrote:

> On 7 November 2015 at 13:55, Ralf Gommers  wrote:
> > On Sat, Nov 7, 2015 at 2:02 PM, Paul Moore  wrote:
> >>
> >> On 7 November 2015 at 01:26, Chris Barker - NOAA Federal
> >>  wrote:
> >> > So what IS supposed to be used in the development workflow? The new
> >> > mythical build system?
> >
> > I'd like to point out again that this is not just about development
> > workflow. This is just as much about simply *installing* from a local git
> > repo, or downloaded sources/sdist.
>
> Possibly I'm misunderstanding here.
>

I had an example above of installing into different venvs. Full rebuilds
for that each time are very expensive. And this whole thread is basically
about `pip install .`, not about inplace builds for development.

As another example of why even for a single build/install it's helpful to
just let the build system do what it wants to do instead of first copying
stuff over, here are some timing results. This is for PyWavelets, which
isn't that complicated a build (mostly pure Python, 1 Cython extension):

1. python setup.py install:   40 s
2. pip install . --upgrade --no-deps:   58 s
# OK, (2) is slow due to using shutil, to be fixed to work like (3):
3. python setup.py sdist:  8 s
pip install dist/PyWavelets0.4.0.dev0+da1c6b4.tar.gz:  41 s
# so total time for (3) will be 41 + 8 = 49 s
# and a better alternative to (1)
4. python setup.py bdist_wheel:  34 s
pip install dist/PyWavelets-xxx.whl:   6 s
# so total time for (3) will be 34 + 6 = 40 s

Not super-scientific, but the conclusion is clear: what pip does is a lot
slower than what for me is the expected behavior. And note that without the
Cython compile, the difference in timing will get even larger.

That expected behavior is:
  a) Just ask the build system to spit out a wheel (without any magic)
  b) Install that wheel (always)



> > The "pip install . should reinstall" discussion in
> > https://github.com/pypa/pip/issues/536 is also pretty much the same
> > argument.
>
> Well, that one is about pip reinstalling if you install from a local
> directory, and not skipping the install if the local directory version
> is the same as the installed version. As I noted there, I'm OK with
> this, it seems reasonable to me to say that if someone has a directory
> of files, they may have updated something but not (yet) bumped the
> version.
>
> The debate over there has gone on to whether we force reinstall for a
> local *file* (wheel or sdist) which I'm less comfortable with. But
> that's is being covered over there.
>
> The discussion *here* is, I thought, about skipping build steps when
> possible because you can reuse build artifacts. That's not "should pip
> do the install?", but rather "*how* should pip do the install?"
> Specifically, to reuse build artifacts it's necessaryto *not* do what
> pip currently does for all (non-editable) installs, which is to
> isolate the build in a temporary directory and do a clean build.
> That's a sensible debate to have, but it's very different from the
> issue you referenced.
>
> IMO, the discussions currently are complex enough that isolating
> independent concerns is crucial if anyone is to keep track. (It
> certainly is for me!)
>

Agreed that the discussions are complex now. But imho they're mostly
complex because the basic principles of what pip should be doing are not
completely clear, at least to me. If it's "build a wheel, install the
wheel" then a lot of things become simpler.


>> Fair question. Unfortunately, the answer is honestly that there's no
> >> simple answer - pip is not a bad option, but it's not its core use
> >> case so there are some rough edges.
> >
> > My impression is that right now pip's core use-case is not "installing",
> but
> > "installing from PyPi (and similar repos". There are a lot of rough edges
> > around installing from anything on your own hard drive.
>
> Not true. The rough edges are around installing things where (a) you
> don't want to rely in the invariant that name and version uniquely
> identify an installation (that's issue 536) and (b) where you don't
> want to do a clean build, because building is complex, slow, or
> otherwise something you want to optimise (that's this discussion).
>
> I routinely download wheels and use them to install. I also sometimes
> download sdists and install from them, although 99.99% of the time, I
> download them, build them into wheels and install them from wheels. It
> *always* works exactly as I'd expect. But if I'm doing development, I
> use -e. That seems to be the problem here, there are rough edges if
> you want a development workflow that doesn't rely on editable
> installs. I think that's what I already said :-)
>

It always works as you expect because you're very familiar with how things
work I suspect. I honestly started working on docs/code to make people use

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

The actual question is: which problem are you trying to solve *that
current users are actually experiencing*?

I'm -1 on removing the "pip" command.  "python -m pip" is frankly not a
reasonable subtitution if we want to *promote* pip.

> * The above gets *really* confusing when ``pipX`` or ``pip`` do not agree with
>   what ``pythonX`` and ``python`` point to.

That's a problem for foreign package managers and distributions. Let
them deal with it.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Donald Stufft]

On November 7, 2015 at 5:45:24 PM, Paul Moore (p.f.mo...@gmail.com) wrote:
> On 7 November 2015 at 16:33, Ralf Gommers wrote:
> > I had an example above of installing into different venvs. Full rebuilds for
> > that each time are very expensive.
>  
> Why doesn't wheel caching solve this problem? That's what it's *for*, surely?
>  

I’m pretty sure we don’t cache wheels for local file paths.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Donald Stufft]

On November 7, 2015 at 6:07:47 PM, Paul Moore (p.f.mo...@gmail.com) wrote:
> On 7 November 2015 at 22:47, Donald Stufft wrote:
> > Only if we think we can trust the version numbers to be unique from random 
> > paths on the  
> file system.
>  
> Precisely. And that's the sort of trade-off that Ralf was asking to be
> clarified. Here, the trade off is that if we *are* allowed to rely on
> the fact that name/version uniquely identifies the build, then we can
> optimise build times via wheel cacheing. If we can't make that
> assumption, we can't do the optimisation.
>  
> The request here seems to be that we provide the best of both worlds -
> provide optimal builds *without* making the assumptions we use for the
> "install a released version" case.
> Paul
> 

Well, you can get the optimized builds by not copying the path into a temporary 
location when you do ``pip install .`` and just letting the build system handle 
whether or not it caches the build output between multiple runs.

I don’t want to start doing this, because I want to make a different change 
that will make it harder (impossible?) to do that. I want to reduce the “paths” 
that an installation can go down. Right now we have:

1. I have a wheel and pip installs it.
2. I have an sdist and pip turns it into a wheel and then pip installs it.
3. I have an sdist and pip installs it.
4. I have a directory and pip installs it.
5. I have a directory and pip installs it in editable mode.

The outcome of all of these types of installs are subtly different and we’ve 
had a number of users regularly get confused when they act differently over the 
years. I do not think it’s possible to make (5) act like anything else because 
it is inherently different, however I think we can get to the point that 1-4 
all act the exact same way. and I think the way to do it is to change these so 
instead it is like:

1. I have a wheel and pip installs it.
2. I have an sdist and pip turns it into a wheel and then pip installs it.
3. I have a directory and pip turns it into a sdist and then pip turns that 
sdist into a wheel and then pip installs it.
4. I have a directory and pip installs it in editable mode.

Essentially, this is removing two “different” types of installations, one where 
we install directly from a sdist (without ever going through a wheel) and one 
where we install directly from a path (without ever going through a sdist or a 
wheel). Omitting the whole editable mode from the consideration, we get to a 
point where installs ONLY ever happen to go from a “Arbitrary Directory” to an 
Sdist to a Wheel to installation and the only real differences are at what 
point in that process the item we’re trying to install is already at.

Of course development/editable installs are always going to be weird because 
they are in-place.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Paul Moore]

On 7 November 2015 at 22:21, Antoine Pitrou  wrote:
> The actual question is: which problem are you trying to solve *that
> current users are actually experiencing*?

Typically, people using "pip" to install stuff, and finding it gets
installed into the "wrong" Python installation (i.e., not the one they
expected). I'm not clear myself on how this happens, but it seems to
be common on some Linux distros (and I think on OSX as well) where
system and user-installed Pythons get confused.

Whether removing the pip command in favour of explicitly using the
name of the python you want to install into is a reasonable solution,
or an over-reaction, is what we're trying to establish. But it is a
very real problem and we see a fair number of bug reports based on it.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

On Sat, 7 Nov 2015 19:16:55 -0500
Donald Stufft  wrote:
> 
> The largest problem comes when ``python`` and ``pip`` disagree about which 
> Python is being invoked.

As a said, this is a problem for package managers and distributions.
"pip" isn't the only affected command, e.g. "pydoc" is as well.

> What should the command be to install into PyPy 2.4.0?

If you are using a virtualenv (or a conda environment, assuming you
did a conda package for pypy), just "pip".

> What if someone has /usr/bin/python2.7
> and /usr/bin/pip2.7 and they then install another Python 2.7
> into /usr/local/bin/python2.7 but they don’t have pip installed there?

Why wouldn't they? I thought the plan is to have "pip" bundled with
every recent Python version? AFAIR someone even said it was a bug if pip
wasn't installed together with Python...

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[David Mertz]

I found I really did typically have that problem that Paul describes pretty
often until I switched to using predominantly conda.  I would always make
symlinks for pip2 and pip3 (and maybe for pip3.3 vs. pip3.4) to make sure
things went the right places.

I suppose this problem was largely because I didn't really use virtualenv
much, and now that I'm a conda person the environments come "for free" with
the installation.

On Sat, Nov 7, 2015 at 3:41 PM, Paul Moore  wrote:

> On 7 November 2015 at 22:21, Antoine Pitrou  wrote:
> > The actual question is: which problem are you trying to solve *that
> > current users are actually experiencing*?
>
> Typically, people using "pip" to install stuff, and finding it gets
> installed into the "wrong" Python installation (i.e., not the one they
> expected). I'm not clear myself on how this happens, but it seems to
> be common on some Linux distros (and I think on OSX as well) where
> system and user-installed Pythons get confused.
>
> Whether removing the pip command in favour of explicitly using the
> name of the python you want to install into is a reasonable solution,
> or an over-reaction, is what we're trying to establish. But it is a
> very real problem and we see a fair number of bug reports based on it.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
*David Mertz, Ph.D.*
*Senior Software Engineer and Senior Trainer*
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Paul Moore]

On 7 November 2015 at 22:47, Donald Stufft  wrote:
> Only if we think we can trust the version numbers to be unique from random 
> paths on the file system.

Precisely. And that's the sort of trade-off that Ralf was asking to be
clarified. Here, the trade off is that if we *are* allowed to rely on
the fact that name/version uniquely identifies the build, then we can
optimise build times via wheel cacheing. If we can't make that
assumption, we can't do the optimisation.

The request here seems to be that we provide the best of both worlds -
provide optimal builds *without* making the assumptions we use for the
"install a released version" case.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Donald Stufft]

On November 7, 2015 at 7:12:59 PM, Antoine Pitrou (solip...@pitrou.net) wrote:
> On Sat, 7 Nov 2015 23:41:25 +
> Paul Moore wrote:
> > On 7 November 2015 at 22:21, Antoine Pitrou wrote:
> > > The actual question is: which problem are you trying to solve *that
> > > current users are actually experiencing*?
> >
> > Typically, people using "pip" to install stuff, and finding it gets
> > installed into the "wrong" Python installation (i.e., not the one they
> > expected). I'm not clear myself on how this happens, but it seems to
> > be common on some Linux distros (and I think on OSX as well) where
> > system and user-installed Pythons get confused.
>  
> Well, the problem is that "python -m pip" isn't any better. If you
> don't know what the current "pip" is, then chances are you don't know
> what the current "python" is, either.
>  
> (I'm not trying to deny the issue, I sometimes wonder what "pip" will
> install into exactly, but removing the command in favour of a "-m"
> switch wouldn't do any any good IMO, and it would make Python package
> management "even more baroque" than it currently is)
>  

The largest problem comes when ``python`` and ``pip`` disagree about which 
Python is being invoked. A further problem is that we also need a way beyond 
just X.Y to differentiate what version of Python something is being installed 
into. What should the command be to install into PyPy 2.4.0? What about PyPy3 
2.4.0? What if someone has /usr/bin/python2.7 and /usr/bin/pip2.7 and they then 
install another Python 2.7 into /usr/local/bin/python2.7 but they don’t have 
pip installed there?

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Paul Moore]

On 7 November 2015 at 16:33, Ralf Gommers  wrote:
> I had an example above of installing into different venvs. Full rebuilds for
> that each time are very expensive.

Why doesn't wheel caching solve this problem? That's what it's *for*, surely?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Donald Stufft]

On November 7, 2015 at 5:46:53 PM, Paul Moore (p.f.mo...@gmail.com) wrote:
> On 7 November 2015 at 22:46, Donald Stufft wrote:
> > On November 7, 2015 at 5:45:24 PM, Paul Moore (p.f.mo...@gmail.com) wrote:
> >> On 7 November 2015 at 16:33, Ralf Gommers wrote:
> >> > I had an example above of installing into different venvs. Full rebuilds 
> >> > for
> >> > that each time are very expensive.
> >>
> >> Why doesn't wheel caching solve this problem? That's what it's *for*, 
> >> surely?
> >>
> >
> > I’m pretty sure we don’t cache wheels for local file paths.
>  
> So is this an argument that we should?
> Paul
> 

Only if we think we can trust the version numbers to be unique from random 
paths on the file system.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Paul Moore]

On 7 November 2015 at 22:46, Donald Stufft  wrote:
> On November 7, 2015 at 5:45:24 PM, Paul Moore (p.f.mo...@gmail.com) wrote:
>> On 7 November 2015 at 16:33, Ralf Gommers wrote:
>> > I had an example above of installing into different venvs. Full rebuilds 
>> > for
>> > that each time are very expensive.
>>
>> Why doesn't wheel caching solve this problem? That's what it's *for*, surely?
>>
>
> I’m pretty sure we don’t cache wheels for local file paths.

So is this an argument that we should?
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Nathaniel Smith]

On Sat, Nov 7, 2015 at 2:44 PM, Paul Moore  wrote:
> On 7 November 2015 at 16:33, Ralf Gommers  wrote:
>> I had an example above of installing into different venvs. Full rebuilds for
>> that each time are very expensive.
>
> Why doesn't wheel caching solve this problem? That's what it's *for*, surely?

The wheel cache maps (name, version) -> wheel. If I hand you a source
directory, you may not even be able to determine the (name, version)
except via building a wheel (depending on the resolution to that other
thread about egg_info/dist-info commands). And it's certainly not true
in general that you can trust the (name, version) from a working
directory to indicating anything meaningful -- e.g. every commit to
pip mainline right now creates a new different "pip 8.0.0.dev0". So
what would you even use for your cache key? I don't see how wheel
caching can really help here.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Nathaniel Smith]

On Sat, Nov 7, 2015 at 6:57 AM, Paul Moore  wrote:
> 2. (For here) Builds are not isolated from what's in the development
> directory. So if you have your sdist definition wrong, what you build
> locally may work, but when you release it it may fail. Obviously that
> can be fixed by proper development and testing practices, but pip is
> designed currently to isolate builds to protect against mistakes like
> this, we'd need to remove that protection for cases where we wanted to
> do in-place builds.

I agree that it would be nice to make sdist generation more reliable
and tested by default, but I don't think this quite works as a
solution.

1) There's no guarantee that building an sdist from some dirty working
tree will produce anything like what you'd have for a release sdist,
or even a clean isolated build. (E.g. a very common mistake is adding
a new file to the working directory for forgetting to run 'git/hg
add'. To protect against this, you have to either have to have a build
system that's smart enough to talk to the VCS when figuring out what
files to include, or better yet you have to work from a clean
checkout.) And as currently specified these "isolated" build trees
might even end up including partial build detritus from previous
in-place builds, copied from the source directory into the temporary
directory.

2) Sometimes people will want to download an sdist, unpack it, and
then run 'pip install .' from it. In your proposal this would require
first building a new sdist from the unpacked working tree. But there's
no guarantee that you can generate an sdist from an sdist. None of the
proposals for a new build system interface have contemplated adding an
"sdist" command, and even if they did, then a clever sdist command
might well fail, e.g. because it is only designed to build sdists from
a checkout with full VCS metadata that it can use to figure out what
files to include :-).

3) And anyway, it's pretty weird logically to include a mandatory
sdist command inside an interface that 99% of the time will be working
*from* an sdist :-). The rule of thumb I've used for the build
interface stuff so far is that it should be the minimal stuff that is
needed to provide a convenient interface for people who just want to
install packages, because the actual devs on a particular project can
use whatever project/build-system-specific interfaces make sense for
their workflow. And end-users don't build sdists. But for the
operations that pip does provide, like 'pip wheel' and 'pip install',
they should be usable by devs, because devs will use them.

> 3. The logic inside pip for doing builds is already pretty tricky.
> Adding code to sometimes build in place and sometimes in a temporary
> directory is going to make it even more complex. That might not be a
> concern for end users, but it makes maintaining pip harder, and risks
> there being subtle bugs in the logic that could bite end users. If you
> want specifics, I can't give them at the moment, because I don't know
> what the code to do the proposed in-place building would look like.

Yeah, this is always a concern for any change. The tradeoff is that
you get to delete the code for "downloading" unpacked directories into
a temporary directory (which currently doesn't even use sdist -- it
just blindly copies everything, including e.g. the full git history).
And you get to skip specifying a standard build-an-sdist interface
that pip and every build system backend would all have to support and
interoperate on.

Basically AFAICT the logic should be:

1) Arrange for the existence of a build directory:
  If building from a directory:
great, we have one, use that
  else if building from a file/url:
download it and unpack it, then use that
2) do the build using the build directory
3) if it's a temporary directory and the build succeeded, clean up

(Possibly with some complications like providing options for people to
specify a non-temporary directory to use for unpacking downloaded
sdists.)

It might need a bit of refactoring so that the "arrange for the
existence of a build directory" step returns the chosen build
directory instead of taking it as a parameter like I assume it does
now, but it doesn't seem like the intrinsic complexity is very high.

> I hope that helps. It's probably not as specific or explicit as you'd
> like, but to be fair, nor is the proposal.
>
> What we currently have on the table is "If 'pip (install/wheel) .' is
> supposed to become the standard way to build things, then it should
> probably build in-place by default." For my personal use cases, I
> don't actually agree with any of that, but my use cases are not even
> remotely like those of numpy developers, so I don't want to dismiss
> the requirement. But if it's to go anywhere, it needs to be better
> explained.
>
> Just to be clear, *my* position (for projects simpler than numpy and
> friends) is:
>
> 1. The standard way to install should be "pip install ".

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Donald Stufft]

On November 7, 2015 at 6:16:34 PM, Donald Stufft (don...@stufft.io) wrote:
> > I want to reduce the “paths” that an installation can go down. 

I decided I’d make a little visual aid to help explain what I mean here 
(omitting development/editable installs because they are weird and will always 
be weird)!

Here’s essentially the way that installs can happen right now 
https://caremad.io/s/Ol1TuV6R9K/. Each of these types of installations act 
subtly different in ways that are not very obvious to most people.

Here’s what I want it to be: https://caremad.io/s/uJYeVzBlQG/. In this way no 
matter what a user is installing from (Wheel, Source Dist, Directory) the 
outcome will be the same and there won’t be subtly different behaviors based on 
what is being provided.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Nathaniel Smith]

On Sat, Nov 7, 2015 at 3:16 PM, Donald Stufft  wrote:
[...]
> The outcome of all of these types of installs are subtly different and we’ve 
> had a number of users regularly get confused when they act differently over 
> the years. I do not think it’s possible to make (5) act like anything else 
> because it is inherently different, however I think we can get to the point 
> that 1-4 all act the exact same way. and I think the way to do it is to 
> change these so instead it is like:
>
> 1. I have a wheel and pip installs it.
> 2. I have an sdist and pip turns it into a wheel and then pip installs it.
> 3. I have a directory and pip turns it into a sdist and then pip turns that 
> sdist into a wheel and then pip installs it.
> 4. I have a directory and pip installs it in editable mode.

I wrote some more detailed comments on this idea in the reply I just
posted to Paul's message, but briefly, the alternative way to approach
this would be:

1. I have a wheel and pip installs it
2. I have an sdist and pip unpacks it into a directory and builds a
wheel from that directory and then pip installs it.
3. I have a directory and pip builds a wheel from that directory and
then pip installs it.
4. I have a directory and pip installs it in editable mode.

This is actually simpler, because we've eliminated the "create an
sdist" operation and replaced it with the far-more-trivial "unpack an
sdist". And it isn't even a replacement, because your 2 and my 2 are
actually identical when you look at what it means to turn an sdist
into a wheel :-).

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Donald Stufft]

On November 7, 2015 at 6:43:50 PM, Nathaniel Smith (n...@pobox.com) wrote:
> On Sat, Nov 7, 2015 at 3:16 PM, Donald Stufft wrote:
> [...]
> > The outcome of all of these types of installs are subtly different and 
> > we’ve had a number  
> of users regularly get confused when they act differently over the years. I 
> do not think  
> it’s possible to make (5) act like anything else because it is inherently 
> different,  
> however I think we can get to the point that 1-4 all act the exact same way. 
> and I think the  
> way to do it is to change these so instead it is like:
> >
> > 1. I have a wheel and pip installs it.
> > 2. I have an sdist and pip turns it into a wheel and then pip installs it.
> > 3. I have a directory and pip turns it into a sdist and then pip turns that 
> > sdist into a wheel  
> and then pip installs it.
> > 4. I have a directory and pip installs it in editable mode.
>  
> I wrote some more detailed comments on this idea in the reply I just
> posted to Paul's message, but briefly, the alternative way to approach
> this would be:
>  
> 1. I have a wheel and pip installs it
> 2. I have an sdist and pip unpacks it into a directory and builds a
> wheel from that directory and then pip installs it.
> 3. I have a directory and pip builds a wheel from that directory and
> then pip installs it.
> 4. I have a directory and pip installs it in editable mode.
>  
> This is actually simpler, because we've eliminated the "create an
> sdist" operation and replaced it with the far-more-trivial "unpack an
> sdist". And it isn't even a replacement, because your 2 and my 2 are
> actually identical when you look at what it means to turn an sdist
> into a wheel :-).
>  

The problem is that an sdist and a directory are not the same things even 
though they may trivially appear to be. A very common problem people run into 
right now is that they don’t adjust their MANIFEST.in so that some new file 
they’ve added gets included in the sdist. In the current system and your 
proposed system if someone types ``pip install .`` that just silently works. 
Then they go “Ok great, my package works” and they create a sdist and send that 
off… except the sdist is broken because it’s missing that file they needed.

Since we’ve disabled the ability to delete + reupload files to PyPI I get 
probably once or twice a week someone contacting me asking if I can let them 
re-upload a file because they created an sdist that was missing a file. A 
decent number of those told me that they had “tested” it by running ``pip 
install .`` or ``setup.py install`` into a fresh virtual environment and that 
it had worked.

It’s true that the MANIFEST.in system exacerbates this problem by being a 
pretty crummy and error prone system to begin with, however the same thing is 
going to exist for any system where you have a path that builds a sdist (and 
may or may not include a file in that sdist) and a path that goes direct to 
wheel.

It might not only be files that didn’t get to be included because of a mistake 
either. Some files might not get generated until sdist build time, something 
like LXML generates .c files from Cython sources at sdist creation time and 
then they build a Wheel from those .c files. They do this to prevent people 
from needing to have Cython available on a machine other than a development 
machine. In your proposed work flow their “build wheel” command needs to be 
able to deal with the fact that the .c files may or may not be available (and 
will need to figure out a way to indicate that Cython is a build dependency if 
they are not). In my proposed workflow their wheel build command gets to be 
simpler, it only needs to deal with .c files and their sdist command gets used 
to create the .c files while the sdist is being generated.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

On Sat, 7 Nov 2015 23:41:25 +
Paul Moore  wrote:
> On 7 November 2015 at 22:21, Antoine Pitrou  wrote:
> > The actual question is: which problem are you trying to solve *that
> > current users are actually experiencing*?
> 
> Typically, people using "pip" to install stuff, and finding it gets
> installed into the "wrong" Python installation (i.e., not the one they
> expected). I'm not clear myself on how this happens, but it seems to
> be common on some Linux distros (and I think on OSX as well) where
> system and user-installed Pythons get confused.

Well, the problem is that "python -m pip" isn't any better. If you
don't know what the current "pip" is, then chances are you don't know
what the current "python" is, either.

(I'm not trying to deny the issue, I sometimes wonder what "pip" will
install into exactly, but removing the command in favour of a "-m"
switch wouldn't do any any good IMO, and it would make Python package
management "even more baroque" than it currently is)

Regarrds

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Donald Stufft]

On November 7, 2015 at 7:22:23 PM, Antoine Pitrou (solip...@pitrou.net) wrote:
> On Sat, 7 Nov 2015 19:16:55 -0500
> Donald Stufft wrote:
> >
> > The largest problem comes when ``python`` and ``pip`` disagree about which 
> > Python  
> is being invoked.
>  
> As a said, this is a problem for package managers and distributions.
> "pip" isn't the only affected command, e.g. "pydoc" is as well.

Package managers and distributions aren’t the only place that Python comes 
from. People might have installed their own Python (such as via pyenv or even 
manually) or they may be using two different package managers one that comes 
with pip by default and one that doesn’t.

This probably does also affect pydoc but I would suggest that more people are 
invoking pip in various “weird” situation than are invoking pydoc. I know 
personally I’ve *never* invoked pydoc. In fact, the pyvenv script has been 
deprecated and is going to be removed in Python 3.8 in favor of `python -m 
venv` for similar reasons that I've described here.

>  
> > What should the command be to install into PyPy 2.4.0?
>  
> If you are using a virtualenv (or a conda environment, assuming you
> did a conda package for pypy), just "pip”.

And if you’re not using a virtual environment or a conda environment? If you 
have /usr/bin/python and /usr/bin/pypy how should I install something into 
PyPy? We can’t just pretend that the only time someone wants to install into 
PyPy is inside of a virtual environment.

>  
> > What if someone has /usr/bin/python2.7
> > and /usr/bin/pip2.7 and they then install another Python 2.7
> > into /usr/local/bin/python2.7 but they don’t have pip installed there?
>  
> Why wouldn't they? I thought the plan is to have "pip" bundled with
> every recent Python version? AFAIR someone even said it was a bug if pip
> wasn't installed together with Python…

Python 2.7.9+ and Python 3.4+ come with ensurepip which can be used to install 
pip into an environment. In Python 3.4+ it will be installed by default by the 
Makefile and by the OSX/Windows installers. In Python 2.7.9+ it’s only 
installed by default in the OSX/Windows installers but it is not in the 
Makefile. So how might they not get pip?

* They installed from a Makefile (or their distribution did) and they either 
accepted the 2.7 default or they disabled it when installing on 3.4.
* They are using a version of Python that didn’t come with pip.
* They uninstalled pip (because pip isn’t part of the standard library, it’s 
just another Python package that can be uninstalled or upgraded).
* They installed into a virtual environment that was created without pip being 
installed.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Vincent Povirk]

> Typically, people using "pip" to install stuff, and finding it gets
> installed into the "wrong" Python installation (i.e., not the one they
> expected). I'm not clear myself on how this happens, but it seems to
> be common on some Linux distros (and I think on OSX as well) where
> system and user-installed Pythons get confused.

FWIW, the approach I'm taking for the oneget provider is that, if
there's more than one version of python into which you "can" install a
package (the definition of "can install" is a bit hairy, but for right
now it means the versions with wheels if there are any wheels), it'll
print a message listing the installs you can use. You can then specify
an install using -PythonVersion 3.4 or -PythonLocation
c:\python34\python.exe.

The -PythonVersion switch implicitly uses a wildcard compare (3.4.*)
even though the wildcard is not specified.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Nathaniel Smith]

On Thu, Nov 5, 2015 at 1:08 PM, Donald Stufft  wrote:
> Another possible option is to modify pip so that instead of installing into
> site-packages we instead create an "executable zip file" which is a simple zip
> file that has all of pip inside of it and a top level __main__.py. Python can
> directly execute this as if it were installed. We would no longer have any
> command except for ``pip`` (which is really this executable zip file). This
> script would default to installing into ``python``, and you can direct it to
> install into a different python using a (new) flag of ``-p`` or ``--python``
> where you can specify either a full path or a command that we'll search $PATH
> for.

I'm not sure if I like this idea or not, but I think it's an
interesting one. I'd frame it differently, though. It seems to me that
the core idea is:

A basic problem that pip has to solve, before it can do anything else,
is that it has to identify the python environment that it's supposed
to be working on. Right now, the standard way it does this is by
looking at sys.executable -- so pip is always doing an odd dance where
it's rearranging the universe around itself, and conversely, every
environment needs to have its own copy of pip inside itself. An
alternative approach would be to totally decouple the host python used
to execute pip from the target python that pip acts upon, on the
grounds that these are logically distinct things. (As a thought
experiment you can even imagine a package manager called, say, 'cpip',
which is a standalone program written in C or some other
non-Python-language, but that happens to know how to manipulate Python
environments. I'm not saying porting pip to another language makes is
in any way a good idea, just that imagining such a thing is a useful
exercise to clarify the difference between the host environment and
the target environment.) In this approach, you'd have a program called
'pip', and when run it uses some set of rules to figure out which
environment it should work on, consulting command line arguments,
environment variables, the current PATH, whatever.

(Windows has its own complicated system that I don't understand, but
for Unix-likes, the default rule would probably be: search the path
for "python" + sys.argv[0][3:], and use that. So you could install a
main pip executable + symlinks named pip3, pip3.4, etc., and invoking
each of them would automatically target the same environments you get
when you run python, python3, python3.4, etc.)

Having decoupled things like this, then it doesn't really matter how
pip is distributed, so long as there is a pip program and it works.
It'd be in a similar position to, say, how mercurial is shipped. You
could use pyinstaller to make a fully standalone package for Windows
that didn't even depend on the system python install, and Debian could
ship a version that's hosted by the system default python, and if you
want to have multiple pip's installed then you could do that just like
you do now by installing them into their own venvs.

Other interesting consequences:

- pip could get more aggressive about dropping *host* support for
older versions of python -- e.g. eventually pip itself could be
written in pure python 3 while still preserving support for installing
stuff into a python 2 target environment. Or less aggressively, you
could imagine dropping host support for 2.6 ahead of dropping target
support for 2.6.

- If pip is a thing that lives "outside" environments instead of
"inside" them, then it might be natural for it to grow some interface
tools for working with environments directly, becoming kinda the
one-stop-friendly-UI for Python developers. E.g.

   pip new-env my-env/ -p pypy3 -r requirements.txt

to create a new virtual environment and install stuff into it in one
go. (Probably pip is going to have to gain some related functionality
anyway to install setup-requires into isolated build environments...)
Obviously there's a desire not to shove everything in the world into
pip, but having a single friendly frontend to
installation/virtualenv/venv is *really* nice for newbies. And
obviously this would a be "someday maybe" thing at best, just an
interesting possibility down the road.

- The eventual interface seems nice enough...

path/to/python -m pip
-> pip -p path/to/python

pip3, pip3.5
-> still would work in my suggested interface

path/to/venv/bin/pip
-> pip -p path/to/venv/bin/python
or pip -E path/to/venv

though the transition would be tricky/painful.

I don't have any conclusion, I just think it's just an interesting
idea to think about.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Nathaniel Smith]

On Sat, Nov 7, 2015 at 4:02 PM, Donald Stufft  wrote:
> On November 7, 2015 at 6:43:50 PM, Nathaniel Smith (n...@pobox.com) wrote:
>> On Sat, Nov 7, 2015 at 3:16 PM, Donald Stufft wrote:
>> [...]
>> > The outcome of all of these types of installs are subtly different and 
>> > we’ve had a number
>> of users regularly get confused when they act differently over the years. I 
>> do not think
>> it’s possible to make (5) act like anything else because it is inherently 
>> different,
>> however I think we can get to the point that 1-4 all act the exact same way. 
>> and I think the
>> way to do it is to change these so instead it is like:
>> >
>> > 1. I have a wheel and pip installs it.
>> > 2. I have an sdist and pip turns it into a wheel and then pip installs it.
>> > 3. I have a directory and pip turns it into a sdist and then pip turns 
>> > that sdist into a wheel
>> and then pip installs it.
>> > 4. I have a directory and pip installs it in editable mode.
>>
>> I wrote some more detailed comments on this idea in the reply I just
>> posted to Paul's message, but briefly, the alternative way to approach
>> this would be:
>>
>> 1. I have a wheel and pip installs it
>> 2. I have an sdist and pip unpacks it into a directory and builds a
>> wheel from that directory and then pip installs it.
>> 3. I have a directory and pip builds a wheel from that directory and
>> then pip installs it.
>> 4. I have a directory and pip installs it in editable mode.
>>
>> This is actually simpler, because we've eliminated the "create an
>> sdist" operation and replaced it with the far-more-trivial "unpack an
>> sdist". And it isn't even a replacement, because your 2 and my 2 are
>> actually identical when you look at what it means to turn an sdist
>> into a wheel :-).
>>
>
> The problem is that an sdist and a directory are not the same things even 
> though they may trivially appear to be. A very common problem people run into 
> right now is that they don’t adjust their MANIFEST.in so that some new file 
> they’ve added gets included in the sdist. In the current system and your 
> proposed system if someone types ``pip install .`` that just silently works. 
> Then they go “Ok great, my package works” and they create a sdist and send 
> that off… except the sdist is broken because it’s missing that file they 
> needed.
>
> Since we’ve disabled the ability to delete + reupload files to PyPI I get 
> probably once or twice a week someone contacting me asking if I can let them 
> re-upload a file because they created an sdist that was missing a file. A 
> decent number of those told me that they had “tested” it by running ``pip 
> install .`` or ``setup.py install`` into a fresh virtual environment and that 
> it had worked.
>
> It’s true that the MANIFEST.in system exacerbates this problem by being a 
> pretty crummy and error prone system to begin with, however the same thing is 
> going to exist for any system where you have a path that builds a sdist (and 
> may or may not include a file in that sdist) and a path that goes direct to 
> wheel.
>
> It might not only be files that didn’t get to be included because of a 
> mistake either. Some files might not get generated until sdist build time, 
> something like LXML generates .c files from Cython sources at sdist creation 
> time and then they build a Wheel from those .c files. They do this to prevent 
> people from needing to have Cython available on a machine other than a 
> development machine. In your proposed work flow their “build wheel” command 
> needs to be able to deal with the fact that the .c files may or may not be 
> available (and will need to figure out a way to indicate that Cython is a 
> build dependency if they are not). In my proposed workflow their wheel build 
> command gets to be simpler, it only needs to deal with .c files and their 
> sdist command gets used to create the .c files while the sdist is being 
> generated.

I'm not sure how to respond, because I sympathize and agree with all
of these points, but I just think that the trade-offs are such that
pip is the wrong place to try and fix this.

Even if pip always copies the source tree to a temp dir, or even
builds an sdist and unpacks it to a temp dir, then this doesn't
actually guarantee that the final distribution will work, because of
the reasons I mentioned in my other email -- you can still forget to
check things in, have random detritus in your working directory
(orphaned .pyc files create all kinds of fun, since python will
happily import them even if the corresponding .py file has been
deleted), etc.

Which isn't to say that it's hopeless to try and improve matters, but
I don't think we should do so at the expense of adding otherwise
unneeded complexity to the pip <-> project-build-system interface.
("Otherwise unneeded" because nothing else in pip cares about
generating sdists.)

And just in general, your plan to improve matters has an ocean-boiling
feel to it to me, 

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Nathaniel Smith]

On Sat, Nov 7, 2015 at 4:03 PM, Paul Moore  wrote:
> I see building a wheel as a release activity. As such, it should
> produce a reproducible result, and so should not be affected by
> arbitrary state in the development directory. I don't know whether you
> consider "ensuring the wheels aren't wrong" as aimed at developers or
> at end users, it seems to me that both parties benefit.
>
> Personally, I'm deeply uncomfortable about *ever* encountering, or
> producing (as a developer) sdists or wheels with the same version
> number but functional differences. I am OK with installing a
> development version (i.e., direct from a development directory into a
> site-packages, either as -e or as a normal install) where the version
> number doesn't change even though the code does, but for me the act of
> producing release artifacts (wheels and sdists) should freeze the
> version number.

The problem with this is that we want to get rid of "direct installs"
entirely, and move to doing wheel-based installs always -- direct
installs require that every build system has to know about every
possible install configuration, and it's just not viable.

I think the way to approach this is to assume that 'pip install
' will always 100% of the time involve a wheel; the
distinction is that sometimes that wheel is treated as a reliable
artifact that can be cached etc., and that sometimes it's treated as a
temporary intermediate format that's immediately discarded.

(As a separate point I do think it would be good to encourage people
to use + versions like 1.2+dev for VCS trees, or better yet
1.2+dev., to emphasize that these are not real reliable version
numbers. (Recall that PEP 440 defines + as defining a "local version"
that's explicitly somewhat unreliable, not allowed on index servers,
etc.) But even pip itself doesn't follow this rule right now so for
the foreseeable future we'll have to assume that source directories
have unreliable version numbers.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Nathaniel Smith]

On Sat, Nov 7, 2015 at 12:32 PM, Robert Collins
 wrote:
> On unknown names, current pkg_resources raises SyntaxError. So I think
> we need to update the spec from that perspective to be clear. All PEP
> 426 defined variables are handled without error by current
> pkg_resources/markerlib implementation. Perhaps the new variables
> should raise rather than evaluating to '' / 0 ? Some discussion /
> thought would be good. Certainly when used and evaluated by an
> existing pkg_resources they will error - so perhaps we should not add
> any new variables at this point?

I just thought of one possible strategy for allowing future extensions
without opening the door wide to every typo: add a variable which
contains the version number of the environment marker evaluation
system, define the and/or operators as being short-circuiting, and
keep the rule that trying to access an unknown variable name raises an
error. Then you could use write things like:

# 'foo is only really needed if some_new_attr > "2", but doesn't hurt otherwise.
# So when using an old installation tool that doesn't know about some_new_attr,
# install it unconditionally.
Requires-Dist: foo; marker_handling >= "2" and some_new_attr > 2
Requires-Dist: foo; marker_handling < "2"

> So, if we don't unify this with the wheel encoding of extras, it will
> require multiple parsers indefinitely. I need to think more about
> whether it makes sense or not.
>
> Wheel certainly needs a way to say 'this distribution exports extras
> X, Y, Z (and their respective dependencies)'. flit and other tools
> producing wheels need the same facility.
>
> https://www.python.org/dev/peps/pep-0427 doesn't define this use of
> markers; but pip and wheel have collaborated on it. PEP-345 doesn't
> describe Provides-Extra, which pkg_resources uses when parsing
> .dist-info directories as well (it determines which extra variables
> get substituted into the set of requires to determine the values of
> the extras...). So there's basically still a bunch of underspecified
> behaviours out there in the wild, and since my strategy is to minimal
> variation vs whats there, we need to figure out the right places to
> split things to layer this well.
>
> Specifying a new variable of 'extra' is fairly easy: we need to decide
> on the values it will take, and thats well defined but layer crossing:
> when processing a dependency with one or more extras, you need to loop
> over all the dependency specifications once with each extra defined
> (including I suspect for completeness '' for the non-extras) and then
> union together the results.
>
> So at this layer I think we could say that:
>  - extra is only valid if the context that is interpreting the
> specification defines it
>  - when invalid it will raise SyntaxError
>
> This allows a single implementation to handle .dist-info directories
> as it does today, while specifying it more fully. It leaves it open in
> future for us to continue modelling exported extras as
> marker-filtered-specifications + a Provides-Extra, or to move to
> exported extras as something in a hash in a richer serialisation
> format, or some third thing. This is good I think.

This seems like a good strategy to me.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[David Mertz]

On Nov 7, 2015 7:30 PM, "Nathaniel Smith"  wrote:
> alternative approach would be to totally decouple the host python used
> to execute pip from the target python that pip acts upon, on the
> grounds that these are logically distinct things. (As a thought
> experiment you can even imagine a package manager called, say, 'cpip',
> which is a standalone program written in C or some other
> non-Python-language, but that happens to know how to manipulate Python
> environments.

This is a great idea! It exists, and it is spelled "conda". Well, it's
written in some particular version of Python, but the package and
environment management is completely decoupled. You can in install R
packages or Julia packages or different Python versions.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig