Re: [Distutils] Serverside Dependency Resolution and Virtualenv Build Server

[Wes Turner]

* It is [currently [#PEP426JSONLD)] necessary to run setup.py with each
given destination platform, because parameters are expanded within the
scope of setup.py.
  * Because of this, client side dependency resolution (with a given
platform) is currently the only viable option for something like this

...

* Build: Docker, Tox (Dox) to build package(s)
  * Each assembly of packages is / could be a package with a setup.py
(and/or a requirements.txt)
 * And tests:
   * http://conda.pydata.org/docs/building/meta-yaml.html#test-section
* Release: DevPi
   * http://doc.devpi.net/latest/
* conda env environment.yml YAML:
http://conda.pydata.org/docs/using/envs.html
  * [x] conda packages
  * [x] pip packages
  * [ ] system packages (configuration management)

And then, really, Is there a stored version of this instance of a named
Docker image?
#reproducibility #linkedreproducibility

On Sat, Nov 7, 2015 at 8:37 AM, Thomas Güttler  wrote:

> I wrote down a tought about Serverside Dependency Resolution and
> Virtualenv Build Server
>
>
> What do you think?
>
> Latest version: https://github.com/guettli/virtualenv-build-server
>
> virtualenv-build-server
> ###
>
> Rough roadmap how a server to build virtualenvs for the python programming
> language could be implemented.
>
> Highlevel goal
> --
>
> Make creating new virtual envionments for the Python programming language
> easy and fast.
>
> Input: fuzzy requirements like this: django>=1.8, requests=>2.7
>
> Output: virtualenv with packages installed.
>
> Two APIs
> 
>
>  #. Resolve fuzzy requirements to a fixed set of packages with exactly
> pinned versions.
>  #. Read fixed set of packages. Build virtualenv according to given
> platform.
>
>
> Steps
> -
>
> Steps:
>
> #. Client sends list of fuzzy requirements to server:
>
>* I need: django>=1.8, requests=>2.7, ...
>
>
> #. Server solves the fuzzy requirements to a fixed set of requirememnts:
> django==1.8.2, requests==2.8.1, ...
>
> #. Client reads the fixed set of requirements.
>
> #. Optional: Client sends fixed set of requirements to the server. Telling
> him the plattform
>
>* My platform: sys.version==2.7.6 and sys.platform=linux2
>
> #. Server builds a virtualenv according to the fixed set of requirements.
>
> #. Server sends the environment to the client
>
> #. Client unpacks the data and has a usable virtualenv
>
> Benefits
> 
>
> Speed:
>
> * There is only one round-trip from client to server. If the dependencies
> get resolved on the client the client would need to download the available
> version information.
> * Caching: If the server gets input parameters (fuzzy requirements and
> platform information) which he has seen before, he can return the cached
> result from the previous request.
>
> Possible Implementations
> 
>
> APIs
> 
> Both APIs could be implementated by a webservice/Rest interface passing
> json or yaml.
>
> Serverside
> ==
>
>
> Implementation Strategie "PostgreSQL"
> .
>
> Since the API is de-coupled from the internals the implementation could be
> exchanged without the need for changes at the client side.
>
> I suggest using the PostgreSQL und resolving the dependcy graph using SQL
> (WITH RECURSIVE).
>
> The package and version data gets stored in PostgreSQL via ORM (Django or
> SQLAlchemy).
>
> The version numbers need to be normalized to ascii to allow fast
> comparision.
>
> Related: https://www.python.org/dev/peps/pep-0440/
>
> Implementation Strategie "Node.js"
> ..
>
> I like python, but I am not married with it. Why not use a different tools
> that is already working? Maybe the node package manager:
> https://www.npmjs.com/
>
> Questions
> -
> Are virtualenv relocatable? AFAIK they are not.
>
> General Thoughts
> 
>
>  * Ignore Updates. Focus on creating new virtualenvs. The server can do
> caching and that's why I prefer creating virtualenvs which never get
> updated. They get created and removed (immutable).
>
>
> I won't implement it
> 
>
> This idea is in the public domain. If you are young and brave or old and
> wise: Go ahead, try to implement it. Please communicate early and often.
> Ask on mailing-lists or me for feedback. Good luck :-)
>
> I love feedback
> ---
>
> Please tell me want you like or dislike:
>
>  * typos and spelling stuff (I am not a native speaker)
>  * alternative implementation strategies.
>  * existing software which does this (even if implemented in a different
> programming language).
>  * ...
>
> --
> http://www.thomas-guettler.de/
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  

Re: [Distutils] The future of invoking pip

[James Bennett]

On Mon, Nov 9, 2015 at 3:41 PM, Chris Barker  wrote:

> pip is a special case -- for MOST python command line tools, the user does
> not care which python it is running with -- if it works, it works.
>
> the failure case we are trying to address here is when "pip install" works
> sjtu fine -- it finds and installs the package into the python pip is
> associated with -- it just doesn't do what the user wants and expects!
>

I still feel like it's just kicking the problem down the line. Switching
from 'pip install' to 'python -m pip install' doesn't actually solve the
issue of how easy we've made it for people to create non-functional Python
installations (non-functional in the sense that they will mysteriously
"work but not really work"). All this switch does is hand the problem over
to the *next* tool the user happens to invoke. So "pip is special" doesn't
really work as a rebuttal, at least to me.

The real problem to solve is management of multi-Python installations; I
don't really know right now *how* to solve it, but slapping one band-aid
onto the problem by changing pip won't accomplish that and so will break a
lot of existing conventions and documentation for what seems like at best
very minor gains.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Chris Barker - NOAA Federal]

> In this situation, if pip's default for finding the python environment
> were to search the path for 'python' instead of using sys.executable,

One trick here -- PATH may not be the same everywhere. For instance,
on OS-X, the environment GUI programs get is entirely independent of
the shell. So, as a rule, PATH will not be "right". I don't know if
anyone tries to run pip from an IDE, but it might be worth keeping in
mind-- this issue comes up all the time with setting the Python used
by an IDE.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Nathaniel Smith]

On Mon, Nov 9, 2015 at 6:11 PM, Robert Collins
 wrote:
> On 10 November 2015 at 15:03, Nathaniel Smith  wrote:
>> On Sun, Nov 8, 2015 at 5:28 PM, Robert Collins
>>  wrote:
>>> +The use of a command line API rather than a Python API is a little
>>> +contentious. Fundamentally anything can be made to work, and Robert wants 
>>> to
>>> +pick something thats sufficiently lowest common denominator that
>>> +implementation is straight forward on all sides. Picking a CLI for that 
>>> makes
>>> +sense because all build systems will need a CLI for end users to use 
>>> anyway.
>>
>> I agree that this is not terribly important, and anything can be made
>> to work. Having pondered it all for a few more weeks though I think
>> that the "entrypoints-style" interface actually is unambiguously
>> better, so let me see about making that case.
>>
>> What's at stake?
>> --
>>
>> Option 1, as in Robert's PEP:
>>
>> The build configuration file contains a string like "flit
>> --dump-build-description" (or whatever), which names a command to run,
>> and then a protocol for running this command to get information on the
>> actual build system interface. Build operations are performed by
>> executing these commands as subprocesses.
>>
>> Option 2, my preference:
>>
>> The build configuration file contains a string like
>> "flit:build_system_api" (or whatever) which names a Python object
>> accessed like
>>
>>   import flit
>>   flit.build_system_api
>>
>> (This is the same syntax used for naming entry points.) Which would
>> then have attributes and methods describing the actual build system
>> interface. Build operations are performed by calling these methods.
>
> Option 3 expressed by Donald on IRC

Where is this IRC channel, btw? :-)

> (and implied by his 'smaller step'
> email - hard code the CLI).
>
> A compromise position from 'setup.py  the 'setup.py' step in pypa.json, but define the rest as a fixed
> contract, e.g. with subcommands like wheel, metadata etc. This drops
> the self describing tool blob and the caching machinery.

So this would give up on having schema versioning for the API, I guess?

> I plan on using that approach in my next draft.
>
> Your point about bugs etc is interesting, but the use of stdin etc in
> a dedicated Python API also needs to be specified.

Yes, but this specification is trivial:

"Stdin is unspecified, and stdout/stderr can be used for printing
status messages, errors, etc. just like you're used to from every
other build system in the world."

Similarly, we still have to specify how what the different operations
are, what arguments they take, how they signal errors, etc. The point
though is this specification will be shorter and simpler if we're
specifying Python APIs than if we're specifying IPC APIs, because with
a Python API we get to assume the existence of things like data
structures and kwargs and exceptions and return values instead of
having to build them from scratch.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Nathaniel Smith]

On Tue, Nov 10, 2015 at 3:29 PM, James Bennett  wrote:
> On Mon, Nov 9, 2015 at 3:41 PM, Chris Barker  wrote:
>>
>> pip is a special case -- for MOST python command line tools, the user does
>> not care which python it is running with -- if it works, it works.
>>
>> the failure case we are trying to address here is when "pip install" works
>> sjtu fine -- it finds and installs the package into the python pip is
>> associated with -- it just doesn't do what the user wants and expects!
>
>
> I still feel like it's just kicking the problem down the line. Switching
> from 'pip install' to 'python -m pip install' doesn't actually solve the
> issue of how easy we've made it for people to create non-functional Python
> installations (non-functional in the sense that they will mysteriously "work
> but not really work"). All this switch does is hand the problem over to the
> *next* tool the user happens to invoke. So "pip is special" doesn't really
> work as a rebuttal, at least to me.

There are lots of ways that Python installations can be broken. As
another example, I helped someone today whose bug report turned out to
boil down to: they used 'pip install' to upgrade a package, and got
upgraded .py files, but somehow their old .pyc files were still around
and in use, so they were still seeing bugs from the old version. The
solution was to manually delete all the .pyc files [1]. No idea how
they managed that, it has nothing to do with this thread, it's just an
example of how infinitely weird installation/configuration problems
get out at the long tail. Python's installed base is large enough that
one-in-a-million cases happen every day...

What's special about pip is that it totally violates DRY: for a
functional python installation, each way of spawning a python
interpreter needs to have a corresponding stub script to spawn pip,
and the shebang line of that stub script has to point to the
corresponding python interpreter. If any part of this complex
assemblage gets out of sync or missing, then your installation is
broken.

(Very few tools have this kind of consistency requirement, because
very few tools are as tightly tied to a single python environment as
pip is -- who cares which virtualenv hg runs out of, it does the same
thing either way. Also, if you do discover that some virtualenv is
missing a script that it should have, then the way to fix that is...
run pip. Kinda a problem if the missing script *is* pip.)

I totally get why people dislike the ergonomics of 'python -m pip',
but we can also acknowledge that it does solve a real technical
problem: it strictly reduces the number of things that can go wrong,
in a tool that's down at the base of the stack.

-n

[1] https://groups.google.com/d/msg/pystatsmodels/KcSzNqDxv-Q/CCim-Tz_BwAJ

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Paul Moore]

On 10 November 2015 at 04:03, Marcus Smith  wrote:
> although I wasn't arguing for it in that context, but rather just using it
> to be clear that a python api approach could still be used with build
> environment isolation

Which is a good point - it's easy enough to write adapters from one
convention to another (I'm inclined to think it's easier to adapt a
Python API to a CLI interface than the other way around, but I may be
wrong about that).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] BDFL Delegates for distutils-sig PEPs

[Donald Stufft]

On November 10, 2015 at 1:58:38 AM, Nick Coghlan (ncogh...@gmail.com) wrote:
> On 10 November 2015 at 16:14, Nick Coghlan wrote:
> > On 30 October 2015 at 16:27, Marcus Smith wrote:
> >>>
> >>> =
> >>> Whenever a new PEP is put forward on distutils-sig, any PyPA core
> >>> reviewer that believes they are suitably experienced to make the final
> >>> decision on that PEP may offer to serve as the BDFL's delegate (or
> >>> "PEP czar") for that PEP. If their self-nomination is accepted by the
> >>> other PyPA core reviewer, the lead PyPI maintainer and the lead
> >>> CPython representative on distutils-sig, then they will have the
> >>> authority to approve (or reject) that PEP.
> >>> =
> >>
> >>
> >> Nick:
> >> just be clear, if nobody nominates themselves, then you still remain (by
> >> default) the active delegate who's responsible for ruling on a Pypa-related
> >> PEP?
> >
> > For anything PyPI related, it defaults to being Donald's call as lead
> > maintainer, for other interoperability specs, it defaults to me.
>  
> Although I'll also note that whenever Donald wants to handle an
> interoperability PEP himself (as with the dependency specifier one),
> I'm highly unlikely to object :)
>  

And of course, it’s likely that a lot of PyPI PEPs will end up having someone 
other than myself as the BDFL-Delegate since it’s likely a fair number of them 
will be written by me heh.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Robert Collins]

On 11 November 2015 at 19:49, Nick Coghlan  wrote:
> On 11 November 2015 at 16:19, Robert Collins  
> wrote:
...>> pip is going to be invoking a CLI *no matter what*. Thats a hard
>> requirement unless Python's very fundamental import behaviour changes.
>> Slapping a Python API on things is lipstick on a pig here IMO: we're
>> going to have to downgrade any richer interface; and by specifying the
>> actual LCD as the interface it is then amenable to direct exploration
>> by users without them having to reverse engineer an undocumented thunk
>> within pip.
>
> I'm not opposed to documenting how pip talks to its worker CLI - I
> just share Nathan's concerns about locking that down in a PEP vs
> keeping *that* CLI within pip's boundary of responsibilities, and
> having a documented Python interface used for invoking build systems.

I'm also very wary of something that would be an attractive nuisance.
I've seen nothing suggesting that a Python API would be anything but:
 - it won't be usable [it requires the glue to set up an isolated
context, which is buried in pip] in the general case
 - no matter what we do, pip can't benefit from it beyond the
subprocess interface pip needs, because pip *cannot* import and use
the build interface

tl;dr - I think making the case that the layer we define should be a
Python protocol rather than a subprocess protocol requires some really
strong evidence. We're *not* dealing with the same moving parts that
typical Python stuff requires.

> However, I've now realised that we're not constrained even if we start
> with the CLI interface, as there's still a migration path to a Python
> API based model:
>
> Now: documented CLI for invoking build systems
> Future: documented Python API for invoking build systems, default
> fallback invokes the documented CLI

Or we just issue an updated bootstrap schema, and there's no fallback
or anything needed.

> So the CLI documented in the PEP isn't *necessarily* going to be the
> one used by pip to communicate into the build environment - it may be
> invoked locally within the build environment.

No, it totally will be. Exactly as setup.py is today. Thats
deliberate: The *new* thing we're setting out to enable is abstract
build systems, not reengineering pip.

The future - sure, someone can write a new thing, and the necessary
capability we're building in to allow future changes will allow a new
PEP to slot in easily and take on that [non trivial and substantial
chunk of work]. (For instance, how do you do compiler and build system
specific options when you have a CLI to talk to pip with)?

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Robert Collins]

I've pushed up a major edit resulting from various reviews + IRC discussion.

tl;dr:
 - replaced the ABNF grammar with a parsley grammar, which is
functionally the same but we can directly execute. sample code doing
that is now also included in the PEP.
 - added some basic tests
 - dropped the new variables that depended on now deprecated platform.dist
 - made the fallback from PEP-440 to string comparisons better worded
 - reinstated the ~= and === operators with appropriate warnings about
adoption within markers. They shouldn't have been removed from version
comparisons in the first place, so that was a bugfix.
 - fixed a bug in the marker language that happened somewhere in
PEP-426. In PEP 345 markers were strictly 'LHS OP RHS' followed by AND
or OR and then another marker expression. The grammar in PEP-426
however allowed things like "(python_version) and
(python_version=='2.7')" which I believe wasn't actually the intent -
truthy values are not defined there. So the new grammar does not allow
("fred" and "bar") or other such things - and and or are exclusively
between well defined expressions now.

-Rob

commit 874538b3e6a5dd6421517e519e63162c3fc30194
Author: Robert Collins 
Date:   Fri Nov 6 16:25:57 2015 +1300

Define dependency specifications formally.

diff --git a/dependency-specification.rst b/dependency-specification.rst
new file mode 100644
index 000..8b38550
--- /dev/null
+++ b/dependency-specification.rst
@@ -0,0 +1,522 @@
+:PEP: XX
+:Title: Dependency specification for Python Software Packages
+:Version: $Revision$
+:Last-Modified: $Date$
+:Author: Robert Collins 
+:BDFL-Delegate: Donald Stufft 
+:Discussions-To: distutils-sig 
+:Status: Draft
+:Type: Standards Track
+:Content-Type: text/x-rst
+:Created: 11-Nov-2015
+:Post-History: XX
+
+
+Abstract
+
+
+This PEP specifies the language used to describe dependencies for packages.
+It draws a border at the edge of describing a single dependency - the
+different sorts of dependencies and when they should be installed is a higher
+level problem. The intent is provide a building block for higher layer
+specifications.
+
+The job of a dependency is to enable tools like pip [#pip]_ to find the right
+package to install. Sometimes this is very loose - just specifying a name, and
+sometimes very specific - referring to a specific file to install. Sometimes
+dependencies are only relevant in one platform, or only some versions are
+acceptable, so the language permits describing all these cases.
+
+The language defined is a compact line based format which is already in
+widespread use in pip requirements files, though we do not specify the command
+line option handling that those files permit. There is one caveat - the
+URL reference form, specified in PEP-440 [#pep440]_ is not actually
+implemented in pip, but since PEP-440 is accepted, we use that format rather
+than pip's current native format.
+
+Motivation
+==
+
+Any specification in the Python packaging ecosystem that needs to consume
+lists of dependencies needs to build on an approved PEP for such, but
+PEP-426 [#pep426]_ is mostly aspirational - and there are already existing
+implementations of the dependency specification which we can instead adopt.
+The existing implementations are battle proven and user friendly, so adopting
+them is arguably much better than approving an aspirational,
unconsumed, format.
+
+Specification
+=
+
+Examples
+
+
+All features of the language shown with a name based lookup::
+
+requests [security,tests] >= 2.8.1, == 2.8.* ; python_version < "2.7.10"
+
+A minimal URL based lookup::
+
+pip @ 
https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
+
+Concepts
+
+
+A dependency specification always specifies a distribution name. It may
+include extras, which expand the dependencies of the named distribution to
+enable optional features. The version installed can be controlled using
+version limits, or giving the URL to a specific artifact to install. Finally
+the dependency can be made conditional using environment markers.
+
+Grammar
+---
+
+We first cover the grammar briefly and then drill into the semantics of each
+section later.
+
+A distribution specification is written in ASCII text. We use a parsley
+[#parsley]_ grammar to provide a precise grammar. It is expected that the
+specification will be embedded into a larger system which offers framing such
+as comments, multiple line support via continuations, or other such features.
+
+The full grammar including annotations to build a useful parse tree is
+included at the end of the PEP.
+
+Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note:
+URI is defined in std-66 [#std66]_::
+
+version_cmp   = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='
+version   = wsp* ( letterOrDigit | '-' | 

Re: [Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Robert Collins]

On 11 November 2015 at 18:53, Nick Coghlan  wrote:
> On 11 November 2015 at 08:44, Nathaniel Smith  wrote:
>> On Mon, Nov 9, 2015 at 6:11 PM, Robert Collins
>>  wrote:
>>> On 10 November 2015 at 15:03, Nathaniel Smith  wrote:
>> Similarly, we still have to specify how what the different operations
>> are, what arguments they take, how they signal errors, etc. The point
>> though is this specification will be shorter and simpler if we're
>> specifying Python APIs than if we're specifying IPC APIs, because with
>> a Python API we get to assume the existence of things like data
>> structures and kwargs and exceptions and return values instead of
>> having to build them from scratch.
>
> I think the potentially improved quality of error handling arising
> from a Python API based approach is well worth taking into account.
> When the backend interface is CLI based, you're limited to:
>
> 1. The return code
> 2. Typically unstructured stderr output
>
> This isn't like HTTP+JSON, where there's an existing rich suite of
> well-defined error codes to use, and an ability to readily include
> error details in the reply payload.
>
> The other thing is that if the core interface is Python API based,
> then if no hook is specified, there can be a default provider in pip
> that knows how to invoke the setup.py CLI (or perhaps even implements
> looking up the CLI to invoke from the source tree metadata).

Its richer, which is both a positive and a negative. I appreciate the
arguments, but I'm not convinced at this point.

pip is going to be invoking a CLI *no matter what*. Thats a hard
requirement unless Python's very fundamental import behaviour changes.
Slapping a Python API on things is lipstick on a pig here IMO: we're
going to have to downgrade any richer interface; and by specifying the
actual LCD as the interface it is then amenable to direct exploration
by users without them having to reverse engineer an undocumented thunk
within pip.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Serverside Dependency Resolution and Virtualenv Build Server

[Thomas Güttler]

Am 10.11.2015 um 21:54 schrieb Wes Turner:
> * It is [currently [#PEP426JSONLD)] necessary to run setup.py with each given 
> destination platform, because parameters are expanded within the scope of 
> setup.py.

OK


>* Because of this, client side dependency resolution (with a given 
> platform) is currently the only viable option for something like this

Are you sure that this conclusion is the only solution?

A server could create a new container/VM to run setup.py.

Then the install_requires can be cached (for this  plattform).

Maybe I am missing something, but still think server side dependency resolution 
is possible.

Please tell me what's wrong with my conclusion.



> 
> ...
> 
> * Build: Docker, Tox (Dox) to build package(s)
>* Each assembly of packages is / could be a package with a setup.py 
> (and/or a requirements.txt)
>   * And tests:
> * http://conda.pydata.org/docs/building/meta-yaml.html#test-section
> * Release: DevPi
> * http://doc.devpi.net/latest/
> * conda env environment.yml YAML: http://conda.pydata.org/docs/using/envs.html
>* [x] conda packages
>* [x] pip packages
>* [ ] system packages (configuration management)
> 
> And then, really, Is there a stored version of this instance of a named 
> Docker image?
> #reproducibility #linkedreproducibility

I don't fully understand the above.

I guess you had the container/VM solution in mind, too.

There is a new topic in your mail which I will reply to in a new thread.

Regards,
  Thomas Güttler


-- 
http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Nick Coghlan]

On 10 November 2015 at 09:24, Chris Barker  wrote:
> As it happens, I am in the middle of a intro class that's using python3.4 or
> 3.5 right now -- and I am telling everyone to do:
>
> python3 -m pip install
>
> Yes, plain old "pip install" is nicer, but only a little bit, and the
> biggest reason we really, really want that to still work is that there are a
> LOT of instructions all over the web telling people to do that -- so really
> too bad if it doesn't work! But the reality is that it often DOESN'T work
> now! and when it doesn't newbies really have no idea why in the heck not!
>
> personally, I think the best approach is to deprecate plain old "pip
> install" -- if it's not there as an option, I expect no one will find it odd
> that to install something for python, you might use python to do that!

Long thread, so I'm picking a semi-random spot to chime in, rather
than replying to every part individually. I think there are a few
different aspects worth considering here:

1. The ergonomics of "pip install X" are really nice, it's by far the
most common instruction given in project documentation, and we're
still in the process of updating documentation to recommend it over
"easy_install X" (or sometimes even over running "./setup.py install"
directly). We *want* it to be the right answer for installing Python
packages, especially in single-installation scenarios.


2. There's one particular reason I *didn't* specify it as the default
recommendation in https://docs.python.org/3/installing/: until Python
3.5, the "pip" executable wasn't placed on the PATH on Windows by
default, even if you'd enabled PATH modification when installing
Python (the problem is that the installers for Python 3.3 and 3.4 only
add the directory where CPython itself resides to PATH, but not the
Scripts directory where "pip.exe" ends up, while even earlier versions
didn't offer the option to modify PATH automatically at all)

So that meant "python -m pip" gave me the broadest coverage - it
worked for everything except system level Python 3 installation on
*nix systems.

With the path issue being fixed in Python 3.5, we're now in the
situation where "pip install X" based instructions will work in all of
the following "single installation" scenarios:

- any activated virtual environment (including conda ones)
- Windows Python 3.5+ installations (with PATH modification selected
at install time)
- *nix Python 2 installations (including Mac OS X and Linux)

"pip install X" still doesn't work for system Python 3 installations
on *nix systems - you need "pip3 install X" or "python3 -m pip install
X" there

Windows Python 2 installations require manual PATH modifications
regardless, but it's more common for people to know how to make
"python -m pip install X" work, than it is for them to remember to
also add the "Scripts" directory needed to make "pip install X" work.


3. I've started thinking of pip as a "plugin manager" for Python. This
is mostly a consequence of work, since many of the challenges we face
in dealing with language specific package management systems are akin
to the ones we encounter with the plugin management systems in web
browsers and IDEs like Eclipse. (It also helps to more clearly
differentiate it from conda, which has the much broader role of being
a data analysis environment manager)

The problem with pip in its current form for that role is that it's
installed as a standalone tool, but isn't natively multi-version
aware. Accordingly, the ideas I like the most are the ones that
suggest taking it down the path of the "py" launcher - make it
natively multi-version aware, and have it choose a well-defined
default target version if multiple versions are available.

Longer term, it may even make sense to take the "python" command on
*nix systems in that direction, or, at the very least, make "py" a
cross-platform invocation technique:
https://mail.python.org/pipermail/linux-sig/2015-October/00.html

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Nick Coghlan]

On 11 November 2015 at 16:11, Robert Collins  wrote:
>  - fixed a bug in the marker language that happened somewhere in
> PEP-426. In PEP 345 markers were strictly 'LHS OP RHS' followed by AND
> or OR and then another marker expression. The grammar in PEP-426
> however allowed things like "(python_version) and
> (python_version=='2.7')" which I believe wasn't actually the intent -
> truthy values are not defined there. So the new grammar does not allow
> ("fred" and "bar") or other such things - and and or are exclusively
> between well defined expressions now.

Right, Vinay pointed out that the use of parentheses for grouping
wasn't well defined in 345, but instead grew implicitly out of their
evaluation as a Python subset. I attempted to fix that in 426, but
didn't intend to allow non-comparisons as operands for AND and OR.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Nick Coghlan]

On 11 November 2015 at 16:19, Robert Collins  wrote:
> On 11 November 2015 at 18:53, Nick Coghlan  wrote:
>> On 11 November 2015 at 08:44, Nathaniel Smith  wrote:
>>> On Mon, Nov 9, 2015 at 6:11 PM, Robert Collins
>>>  wrote:
 On 10 November 2015 at 15:03, Nathaniel Smith  wrote:
>>> Similarly, we still have to specify how what the different operations
>>> are, what arguments they take, how they signal errors, etc. The point
>>> though is this specification will be shorter and simpler if we're
>>> specifying Python APIs than if we're specifying IPC APIs, because with
>>> a Python API we get to assume the existence of things like data
>>> structures and kwargs and exceptions and return values instead of
>>> having to build them from scratch.
>>
>> I think the potentially improved quality of error handling arising
>> from a Python API based approach is well worth taking into account.
>> When the backend interface is CLI based, you're limited to:
>>
>> 1. The return code
>> 2. Typically unstructured stderr output
>>
>> This isn't like HTTP+JSON, where there's an existing rich suite of
>> well-defined error codes to use, and an ability to readily include
>> error details in the reply payload.
>>
>> The other thing is that if the core interface is Python API based,
>> then if no hook is specified, there can be a default provider in pip
>> that knows how to invoke the setup.py CLI (or perhaps even implements
>> looking up the CLI to invoke from the source tree metadata).
>
> Its richer, which is both a positive and a negative. I appreciate the
> arguments, but I'm not convinced at this point.
>
> pip is going to be invoking a CLI *no matter what*. Thats a hard
> requirement unless Python's very fundamental import behaviour changes.
> Slapping a Python API on things is lipstick on a pig here IMO: we're
> going to have to downgrade any richer interface; and by specifying the
> actual LCD as the interface it is then amenable to direct exploration
> by users without them having to reverse engineer an undocumented thunk
> within pip.

I'm not opposed to documenting how pip talks to its worker CLI - I
just share Nathan's concerns about locking that down in a PEP vs
keeping *that* CLI within pip's boundary of responsibilities, and
having a documented Python interface used for invoking build systems.

However, I've now realised that we're not constrained even if we start
with the CLI interface, as there's still a migration path to a Python
API based model:

Now: documented CLI for invoking build systems
Future: documented Python API for invoking build systems, default
fallback invokes the documented CLI

So the CLI documented in the PEP isn't *necessarily* going to be the
one used by pip to communicate into the build environment - it may be
invoked locally within the build environment.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Wayne Werner]

With all of the weirdness involved, it makes me wonder - could there be a
better way? If we waved our hands and were able to magically make Python
package management perfect, what would that look like?

Would that kind of discussion even be valuable?

On Tue, Nov 10, 2015, 6:22 PM Nathaniel Smith  wrote:


I totally get why people dislike the ergonomics of 'python -m pip',
but we can also acknowledge that it does solve a real technical
problem: it strictly reduces the number of things that can go wrong,
in a tool that's down at the base of the stack.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Nick Coghlan]

On 11 November 2015 at 08:44, Nathaniel Smith  wrote:
> On Mon, Nov 9, 2015 at 6:11 PM, Robert Collins
>  wrote:
>> On 10 November 2015 at 15:03, Nathaniel Smith  wrote:
> Similarly, we still have to specify how what the different operations
> are, what arguments they take, how they signal errors, etc. The point
> though is this specification will be shorter and simpler if we're
> specifying Python APIs than if we're specifying IPC APIs, because with
> a Python API we get to assume the existence of things like data
> structures and kwargs and exceptions and return values instead of
> having to build them from scratch.

I think the potentially improved quality of error handling arising
from a Python API based approach is well worth taking into account.
When the backend interface is CLI based, you're limited to:

1. The return code
2. Typically unstructured stderr output

This isn't like HTTP+JSON, where there's an existing rich suite of
well-defined error codes to use, and an ability to readily include
error details in the reply payload.

The other thing is that if the core interface is Python API based,
then if no hook is specified, there can be a default provider in pip
that knows how to invoke the setup.py CLI (or perhaps even implements
looking up the CLI to invoke from the source tree metadata).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig