This is the update I promised...full prose, since enough has changed
that a diff would be noise.
:PEP: XX
:Title: Build system abstraction for pip/conda etc
:Version: $Revision$
:Last-Modified: $Date$
:Author: Robert Collins <rbtcoll...@hp.com>,
Nathaniel Smith <n...@pobox.com>
:BDFL-Delegate: Donald Stufft <don...@stufft.io>
:Discussions-To: distutils-sig <distutils-sig@python.org>
:Status: Draft
:Type: Standards Track
:Content-Type: text/x-rst
:Created: 26-Oct-2015
:Post-History: XX
:Requires: The new dependency-specification PEP.
Abstract
========
This PEP specifies a programmatic interface for pip [#pip]_ and other
distribution or installation tools to use when working with Python
source trees (both the developer tree - e.g. the git tree - and source
distributions).
The programmatic interface allows decoupling of pip from its current
hard dependency on setuptools [#setuptools]_ able for two
key reasons:
1. It enables new build systems that may be much easier to use without
requiring them to even appear to be setuptools.
2. It facilitates setuptools itself changing its user interface without
breaking pip, giving looser coupling.
The interface needed to permit pip to install build systems also enables pip to
install build time requirements for packages which is an important step in
getting pip to full feature parity with the installation components of
easy-install.
As PEP-426 [#pep426]_ is draft, we cannot utilise the metadata format it
defined. However PEP-427 wheels are in wide use and fairly well specified, so
we have adopted the METADATA format from that for specifying distribution
dependencies. However something was needed for communicating bootstrap
requirements and build requirements - but a thin JSON schema is sufficient
when overlaid over the new dependency specification PEP.
Motivation
==========
There is significant pent-up frustration in the Python packaging ecosystem
around the current lock-in between build system and pip. Breaking that lock-in
is better for pip, for setuptools, and for other build systems like flit
[#flit]_.
Specification
=============
Overview
--------
Build tools will be located by reading a file ``pypa.json`` from the root
directory of the source tree. That file describes how to get the build tool
and the name of the command to run to invoke the tool.
All tools will be expected to conform to a single command line interface
modelled on pip's existing use of the setuptools setup.py interface.
pypa.json
---------
The file ``pypa.json`` acts as neutron configuration file for pip and other
tools that want to build source trees to consult for configuration. The
absence of a ``pypa.json`` file in a Python source tree implies a setuptools
or setuptools compatible build system.
The JSON has the following schema. Extra keys are ignored.
schema
The version of the schema. This PEP defines version "1". Defaults to "1"
when absent. All tools reading the file must error on an unrecognised
schema version.
bootstrap_requires
Optional list of dependency specifications [#dependencyspec] that must be
installed before running the build tool. For instance, if using flit, then
the requirements might be::
bootstrap_requires: ["flit"]
build_command
A mandatory Python format string [#strformat]_ describing the command to
run. For instance, if using flit then the build command might be::
build_command: ["flit"]
If using a command which is a runnable module fred::
{PYTHON} -m fred
Process interface
-----------------
The command to run is defined by a simple Python format string [#strformat]_.
This permits build systems with dedicated scripts and those that are invoked
using "python -m somemodule".
Processes will be run with the current working directory set to the root of
the source tree.
When run, processes should not read from stdin - while pip currently runs
build systems with stdin connected to it's own stdin, stdout and stderr are
redirected and no communication with the user is possible.
As usual with processes, a non-zero exit status indicates an error.
Available variables
-------------------
PYTHON
The Python interpreter in use. This is important to enable calling things
which are just Python entry points.
${PYTHON} -m foo
Subcommands
-----------
There are a number of separate subcommands that build systems must support.
The examples below use a build_command of ``flit`` for illustrative purposes.
build_requires
Query build requirements. Build requirements are returned as a JSON
document with one key ``build_requires`` consisting of a list of
dependency specifications [#dependencyspec]_. Additional keys must be
ignored. The build_requires command is the only command run without
setting up a build environment.
Example command::
flit build_requires
metadata
Query project metadata. The metadata and only the metadata should
be output on stdout. pip would run metadata just once to determine what
other packages need to be downloaded and installed. The metadata is output
as a wheel METADATA file per PEP-427 [#pep427]_.
Note that the metadata generated by the metadata command, and the metadata
present in a generated wheel must be identical.
Example command::
flit metadata
wheel -d OUTPUT_DIR
Command to run to build a wheel of the project. OUTPUT_DIR will point to
an existing directory where the wheel should be output. Stdout and stderr
have no semantic meaning. Only one file should be output - if more are
output then pip would pick an arbitrary one to consume.
Example command::
flit wheel -d /tmp/pip-build_1234
develop
Command to do an in-place 'development' installation of the project.
Stdout and stderr have no semantic meaning.
Not all build systems will be able to perform develop installs. If a build
system cannot do develop installs, then it should error when run. Note
that doing so will cause use operations like ``pip install -e foo`` to
fail.
The build environment
---------------------
Except for the build_requires command, all commands are run within a build
environment. No specific implementation is required, but a build environment
must achieve the following requirements.
1. All dependencies specified by the project's build_requires must be
available for import from within ``$PYTHON``.
1. All command-line scripts provided by the build-required packages must be
present in ``$PATH``.
A corollary of this is that build systems cannot assume access to any Python
package that is not declared as a build_requires or in the Python standard
library.
Hermetic builds
---------------
This specification does not prescribe whether builds should be hermetic or not.
Existing build tools like setuptools will use installed versions of build time
requirements (e.g. setuptools_scm) and only install other versions on version
conflicts or missing dependencies. However its likely that better consistency
can be created by always isolation builds and using only the specified
dependencies.
However there are nuanced problems there - such as how can users force the
avoidance of a bad version of a build requirement which meets some packages
dependencies. Future PEPs may tackle this problem, but it is not currently in
scope - it does not affect the metadata required to coordinate between build
systems and things that need to do builds, and thus is not PEP material.
Upgrades
--------
'pypa.json' is versioned to permit future changes without requiring
compatibility.
The sequence for upgrading either of schemas in a new PEP will be:
1. Issue new PEP defining an updated schema. If the schema is not entirely
backward compatible then a new version number must be defined.
2. Consumers (e.g. pip) implement support for the new schema version.
3. Package authors opt into the new schema when they are happy to introduce a
dependency on the version of 'pip' (and potentially other consumers) that
introduced support for the new schema version.
The *same* process will take place for the initial deployment of this PEP:-
the propogation of the capability to use this PEP without a `setuptools shim`_
will be largely gated by the adoption rate of the first version of pip that
supports it.
Static metadata in sdists
-------------------------
This PEP does not tackle the current inability to trust static metadata in
sdists. That is a separate problem to identifying and consuming the build
system that is in use in a source tree, whether it came from an sdist or not.
Handling of compiler options
----------------------------
Handling of different compiler options is out of scope for this specification.
pip currently handles compiler options by appending user supplied strings to
the command line it runs when running setuptools. This approach is sufficient
to work with the build system interface defined in this PEP, with the
exception that globally specified options will stop working globally as
different build systems evolve. That problem can be solved in pip (or conda or
other installers) without affecting interoperability.
In the long term, wheels should be able to express the difference between
wheels built with one compiler or options vs another, and that is PEP
material.
Examples
========
An example 'pypa.json' for using flit::
{"bootstrap_requires": ["flit"],
"build_command": ["flit"]}
When 'pip' reads this it would prepare an environment with flit in it before
trying to use flit.
Because flit doesn't have setup-requires support today,
`flit build_requires` would just output a constant string::
{"build_requires": []}
`flit metadata` would interrogate `flit.ini` and marshal the metadata into
a wheel METADATA file and output that on stdout.
`flit wheel` would need to accept a `-d` parameter that tells it where
to output the
wheel (pip needs this).
Backwards Compatibility
=======================
Older pips will remain unable to handle alternative build systems.
This is no worse than the status quo - and individual build system
projects can decide whether to include a shim ``setup.py`` or not.
All existing build systems that can product wheels and do develop installs
should be able to run under this abstraction and will only need a specific
adapter for them constructed and published on PyPI.
In the absence of a ``pypa.json`` file, tools like pip should assume a
setuptools build system and use setuptools commands directly.
Network effects
---------------
Projects that adopt build systems that are not setuptools compatible - that
is that they have no setup.py, or the setup.py doesn't accept commands that
existing tools try to use - will not be installable by those existing tools.
Where those projects are used by other projects, this effect will cascade.
In particular, because pip does not handle setup-requires today, any project
(A) that adopts a setuptools-incompatible build system and is consumed as a
setup-requirement by a second project (B) which has not itself transitioned to
having a pypa.json will make B uninstallable by any version of pip. This is
because setup.py in B will trigger easy-install when 'setup.py egg_info' is
run by pip, and that will try and fail to install A.
As such we recommend that tools which are currently used as setup-requires
either ensure that they keep a `setuptools shim`_ or find their consumers and
get them all to upgrade to the use of a `pypa.json` in advance of moving
themselves. Pragmatically that is impossible, so the advice is to keep a
setuptools shim indefinitely - both for projects like pbr, setuptools_scm and
also projects like numpy.
setuptools shim
---------------
It would be possible to write a generic setuptools shim that looks like
``setup.py`` and under the hood uses ``pypa.json`` to drive the builds. This
is not needed for pip to use the system, but would allow package authors to
use the new features while still retaining compatibility with older pip
versions.
Rationale
=========
This PEP started with a long mailing list thread on distutils-sig [#thread]_.
Subsequent to that a online meeting was held to debug all the positions folk
had. Minutes from that were posted to the list [#minutes]_.
This specification is a translation of the consensus reached there into PEP
form, along with some arbitrary choices on the minor remaining questions.
The basic heuristic for the design has to been to focus on introducing an
abstraction without requiring development not strictly tied to the
abstraction. Where the gap is small to improvements, or the cost of using the
existing interface is very high, then we've taken on having the improvement as
a dependency, but otherwise defered such to future iterations.
We chose wheel METADATA files rather than defining a new specification,
because pip can already handle wheel .dist-info directories which encode all
the necessary data in a METADATA file. PEP-426 can't be used as it's still
draft, and defining a new metadata format, while we should do that, is a
separate problem. Using a directory on disk would not add any value to the
interface (pip has to do that today due to limitations in the setuptools
CLI).
The use of 'develop' as a command is because there is no PEP specifying the
interoperability of things that do what 'setuptools develop' does - so we'll
need to define that before pip can take on the responsibility for doing the
'develop' step. Once thats done we can issue a successor PEP to this one.
The use of a command line API rather than a Python API is a little
contentious. Fundamentally anything can be made to work, and the pip
maintainers have spoken strongly in favour of retaining a process based
interface - something that is mature and robust in pip today.
The choice of JSON as a file format is a compromise between several
constraints. Firstly there is no stdlib YAML interpreter, nor one for any of
the other low-friction structured file formats. Secondly, INIParser is a poor
format for a number of reasons, primarily that it has very minimal structure -
but pip's maintainers are not fond of it. JSON is in the stdlib, has
sufficient structure to permit embedding anything we want in future without
requiring embedded DSL's.
Donald suggested using ``setup.cfg`` and the existing setuptools command line
rather than inventing something new. While that would permit interoperability
with less visible changes, it requires nearly as much engineering on the pip
side - looking for the new key in setup.cfg, implementing the non-installed
environments to run the build in. And the desire from other build system
authors not to confuse their users by delivering something that looks like but
behaves quite differently to setuptools seems like a bigger issue than pip
learning how to invoke a custom build tool.
The metadata and wheel commands are required to have consistent metadata to
avoid a race condition that could otherwise happen where pip reads the
metadata, acts on it, and then the resulting wheel has incompatible
requirements. That race is exploited today by packages using PEP-426
environment markers, to work with older pip versions that do not support
environment markers. That exploit is not needed with this PEP, because either
the setuptools shim is in use (with older pip versions), or an environment
marker ready pip is in use. The setuptools shim can take care of exploiting
the difference older pip versions require.
References
==========
.. [#pip] pip, the recommended installer for Python packages
(http://pip.readthedocs.org/en/stable/)
.. [#setuptools] setuptools, the defacto Python package build system
(https://pythonhosted.org/setuptools/)
.. [#flit] flit, a simple way to put packages in PyPI
(http://flit.readthedocs.org/en/latest/)
.. [#pypi] PyPI, the Python Package Index
(https://pypi.python.org/)
.. [#shellvars] Shellvars, an implementation of shell variable rules for Python.
(https://github.com/testing-cabal/shellvars)
.. [#pep426] PEP-426, Python distribution metadata.
(https://www.python.org/dev/peps/pep-0426/)
.. [#pep427] PEP-427, Python distribution metadata.
(https://www.python.org/dev/peps/pep-0427/)
.. [#thread] The kick-off thread.
(https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html)
.. [#minutes] The minutes.
(https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html)
.. [#strformat] The Python string formatting syntax.
(https://docs.python.org/3.1/library/string.html#format-string-syntax)
.. [#dependencyspec] Dependency specification language PEP.
(https://github.com/pypa/interoperability-peps/pull/56)
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
--
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
