Re: [Distutils] setup_requires: the obvious option(?)

[Nick Coghlan]

On 2 September 2016 at 13:30, Antony Lee  wrote:
>> Similarly, it wouldn't astonish me if we eventually see an emergent
>> practice of people writing pyproject.toml.in files for complex
>> projects, in order to move some particular forms of complexity away
>> from build time and towards development time - this would be a similar
>> practice to folks using autoconf to generate a project's C Makefile.
>
> This actually formulates much better than I could have done the reasons why
> I dislike PEP518: it's only going to lead to reinventing the wheel (AKA
> autoconf, which is a pretty big wheel to reinvent).

Unlike autoconf, we don't need to support building arbitrary C/C++
projects - rather, we just want people to have a backend independent
way to tell Python-centric toolchains how to invoke their *existing*
build system (whether that's autoconf/make, CMake, Scons, waf, Meson,
yotta, etc).

It's the current *lack* of that ability to readily integrate with
existing build tools (whether written in Python or not) that prompts
people to reinvent the world.

Defining and support PEP 518 means that a possible future workflow for
autoconf using projects would be something like:

$ ./configure && make pyproject.toml && pip install -e .

In that kind of scenario, the "tell Python tools how to build our
Python bindings" file becomes just another output of a project's
existing build system, which is entirely feasible with a well
documented static format, but impractical with the current
underspecified and underdocumented setup.py based approach.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Antony Lee]

>
> Similarly, it wouldn't astonish me if we eventually see an emergent
> practice of people writing pyproject.toml.in files for complex
> projects, in order to move some particular forms of complexity away
> from build time and towards development time - this would be a similar
> practice to folks using autoconf to generate a project's C Makefile.


This actually formulates much better than I could have done the reasons why
I dislike PEP518: it's only going to lead to reinventing the wheel (AKA
autoconf, which is a pretty big wheel to reinvent).
Antony
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Nick Coghlan]

On 31 August 2016 at 07:04, Daniel Holth  wrote:
> In other systems I've worked on I sometimes have make-like rules that
> automatically rebuild static metadata depending on other files, like copying
> a version number between a .json and an .xml file - reprogramming the system
> that uses the .xml file is not an option. For example a rule could watch
> certain files in the .git directory to regenerate a version number
> automatically as part of the build if .git changed, and do nothing if the
> .git directory was absent per a tarball dist. This works pretty well. Once
> you have a system that's easy to customize with make-like rules there are
> all sorts of trivial build or housekeeping tasks you might decide to do
> which would never be considered in a more difficult to customize system.

CPython's own build process takes this to extremes by sometimes
bootstrapping a version of Python without an import system in order to
refreeze importlib before continuing with building the normal version.
Argument Clinic (which generates C function argument handling
preambles from specially formatted comments) similarly needs a
pre-existing Python build in order to run.

However, we successfully hide that complexity from folks that just
want to build their own Python from source by checking in the
generated files.

Similarly, it wouldn't astonish me if we eventually see an emergent
practice of people writing pyproject.toml.in files for complex
projects, in order to move some particular forms of complexity away
from build time and towards development time - this would be a similar
practice to folks using autoconf to generate a project's C Makefile.

Party of the beauty of the pyproject.toml format is that as
publication and installation system maintainers, we don't need to care
about this problem beyond ensuring that "maintain it by hand" is a
viable and fully supported option for the majority of projects - as
long as the resulting file gets checked in, folks are free to
autogenerate from another data source if they choose to do so.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Daniel Holth]

On Tue, Aug 30, 2016 at 4:06 PM Donald Stufft  wrote:

>
> On Aug 30, 2016, at 2:32 PM, Daniel Holth  wrote:
>
> name, version, packages, install_requires, extras_require, description,
> license, classifiers, keywords, author, url, entry_points.
>
>
> Out of these, a number of them are regularly dynamic for people’s setup.py
> as is. The version number is often dynamically computed based on things
> like git tags, packages can be computed based on Python version,
> install_requires and extras_requires regularly get computed based on things
> like Python version, OS, etc (although environment markers eases some of
> this) but also things like “We support Numpy >= for building from source,
> but once you’ve built from source you only support Numpy >=
> VERSION_YOU_BUILT_AGAINST”.
>
> Outside of “name”, it’s not entirely unreasonable to find reasons why a
> lot of things need to be dynamic. Although there’s a difference in what
> needs to be dynamic when pulling from a VCS and when pulling from a sdist,
> and currently there’s not really a whole lot of difference in terms of
> capability or how they are handled.
>

Of course pip continues to call egg_info before trusting the metadata from
any sdist and 90k pypi projects say this isn't changing. Once you need
dynamic static metadata, madness.

In other systems I've worked on I sometimes have make-like rules that
automatically rebuild static metadata depending on other files, like
copying a version number between a .json and an .xml file - reprogramming
the system that uses the .xml file is not an option. For example a rule
could watch certain files in the .git directory to regenerate a version
number automatically as part of the build if .git changed, and do nothing
if the .git directory was absent per a tarball dist. This works pretty
well. Once you have a system that's easy to customize with make-like rules
there are all sorts of trivial build or housekeeping tasks you might decide
to do which would never be considered in a more difficult to customize
system.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Donald Stufft]

> On Aug 30, 2016, at 2:32 PM, Daniel Holth  wrote:
> 
> name, version, packages, install_requires, extras_require, description, 
> license, classifiers, keywords, author, url, entry_points. 

Out of these, a number of them are regularly dynamic for people’s setup.py as 
is. The version number is often dynamically computed based on things like git 
tags, packages can be computed based on Python version, install_requires and 
extras_requires regularly get computed based on things like Python version, OS, 
etc (although environment markers eases some of this) but also things like “We 
support Numpy >= for building from source, but once you’ve built from source 
you only support Numpy >= VERSION_YOU_BUILT_AGAINST”.

Outside of “name”, it’s not entirely unreasonable to find reasons why a lot of 
things need to be dynamic. Although there’s a difference in what needs to be 
dynamic when pulling from a VCS and when pulling from a sdist, and currently 
there’s not really a whole lot of difference in terms of capability or how they 
are handled.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Daniel Holth]

On Tue, Aug 30, 2016 at 1:13 PM Thomas Kluyver  wrote:

> On Tue, Aug 30, 2016, at 05:51 PM, Antony Lee wrote:
>
> I am not really a fan of PEP518 in general.  Basically, the idea of
> setup.py is that declarative languages are not sufficient to express a
> build system (and AFAICT this is always going to be the case for
> expressing, say, compiler flags for extensions), so I'd rather just accept
> that and stick everything in setup.py instead of adding more parameter
> files.  What if someone wants dynamic build dependencies?
>
>
> Dynamic build deps aren't precluded - the idea is that the build system
> can discover additional dependencies when it runs, while the static
> build-system field specifies just what's required to run the build system
> itself.  However, the build system interface was split out into separate
> PEPs (517 & 516 are alternatives) to allow 518 to go forwards.
>
> I take totally the opposite view: we should make as much metadata as
> possible declarative, even though we know we can't define a totally general
> build system with purely declarative information.
>

This comes up over and over again because we've been living with this
system for long enough that the build script and metadata are together both
in setup.py and in people's brains. But there is a whole lot of stuff that
makes perfect sense in a static file. For example: name, version, packages,
install_requires, extras_require, description, license, classifiers,
keywords, author, url, entry_points. The only thing that would cause
trouble is if the system had no available build script.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Thomas Kluyver]

On Tue, Aug 30, 2016, at 05:51 PM, Antony Lee wrote:
> I am not really a fan of PEP518 in general.  Basically, the idea of
> setup.py is that declarative languages are not sufficient to express a
> build system (and AFAICT this is always going to be the case for
> expressing, say, compiler flags for extensions), so I'd rather just
> accept that and stick everything in setup.py instead of adding more
> parameter files.  What if someone wants dynamic build dependencies?

Dynamic build deps aren't precluded - the idea is that the build system
can discover additional dependencies when it runs, while the static build-
system field specifies just what's required to run the build system
itself.  However, the build system interface was split out into separate
PEPs (517 & 516 are alternatives) to allow 518 to go forwards.

I take totally the opposite view: we should make as much metadata as
possible declarative, even though we know we can't define a totally
general build system with purely declarative information.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Antony Lee]

2016-08-30 5:08 GMT-07:00 Erik Bray :

> I mean this sort of already exists but it's spelled:
>
> from setuptools import Distribution
> Distribution({'setup_requires': ['numpy'])
>
> Granted it's non-obvious and doesn't have the needed_for flag, which I
> like.  It's not entirely clear how needed_for would work though.  For
> example, what if the package you're requiring provides the command
> that you need that package to run?
>

needed_for just textually checks sys.argv[1], and does so before the call
to setup() itself happens, so that's not a problem.

The same can be done by subclassing commands, and there can be some
> corner cases where that gets extra tricky (Cython comes to mind).


I personally don't like subclassing commands at all (it's not very
composable -- what happens if two projects both attempt to subclass
build_ext?).  But even then, how is Cython an issue?

I'm glad you mentioned Daniel Holth's setup-requires hack.  Although I
> haven't used it myself directly I generally like the concept.
>

I am not really a fan of PEP518 in general.  Basically, the idea of
setup.py is that declarative languages are not sufficient to express a
build system (and AFAICT this is always going to be the case for
expressing, say, compiler flags for extensions), so I'd rather just accept
that and stick everything in setup.py instead of adding more parameter
files.  What if someone wants dynamic build dependencies?


> Yeah, setup_requires is a mess, but I'd be skeptical of solving the
> problem by depending on any new features in setuptools :/
>

This could also go into pip (perhaps a better solution now that pip is
de-facto stdlib (via ensurepip))... we'll need something new somewhere
anyways.


> Best,
> Erik
>

Best,

Antony
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Daniel Holth]

On Tue, Aug 30, 2016 at 8:09 AM Erik Bray  wrote:

> On Mon, Aug 29, 2016 at 7:29 PM, Antony Lee  wrote:
> > Hi all,
> >
> > The `setup_requires` option to `setup()` is well-known to suffer from
> > multiple issues.  Most importantly, as it is a keyword argument to
> > `setup()`, it appears too late for modules that may need to be imported
> for
> > the build to occur (e.g., Cython, for which support must explicitly
> provided
> > by setuptools itself rather than by letting Cython hook into it);
> > additionally, there are various contorsions that people go to to avoid
> some
> > `setup_requires` when not building the package (such as checking the
> value
> > of `sys.argv`).  `setup_requires` also uses `easy_install` rather than
> > `pip`, but I do not see why this could not be fixed; let's focus on the
> > first issue instead.
> >
> > If `setup_requires` appears too late to be useful, the obvious(?) option
> is
> > to move it earlier: provide a function, say,
> `setuptools.setup_requires()`,
> > that should be called *before* `setup()` itself, e.g.:
> >
> > from setuptools import setup, setup_requires
> > setup_requires("numpy", needed_for=["build_ext"])
> > try:
> > import numpy as np
> > except ImportError:
> > np = None
> > setup(..., include_dirs=[np.get_include()] if np else [])
>
> I mean this sort of already exists but it's spelled:
>
> from setuptools import Distribution
> Distribution({'setup_requires': ['numpy'])
>
> Granted it's non-obvious and doesn't have the needed_for flag, which I
> like.  It's not entirely clear how needed_for would work though.  For
> example, what if the package you're requiring provides the command
> that you need that package to run?
>
> The same can be done by subclassing commands, and there can be some
> corner cases where that gets extra tricky (Cython comes to mind).
>
> > When `setup.py` is invoked, either directly or by pip, upon the call to
> > `setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at
> > least one requirement is missing, `setup_requires()` calls asks pip to
> > install the required packages (similarly to
> > `https://bitbucket.org/dholth/setup-requires`
> ) in a temporary directory,
> and
> > the whole Python process replaces itself (in the `os.execv()` sense) by a
> > new call to `python setup.py` with this temporary directory prepended to
> the
> > PYTHONPATH.  In this new process, the arguments to `setup_requires()` are
> > now available and we can proceed to the rest of `setup.py`.
> >
> > I feel like this idea is natural enough that someone must already have
> come
> > up with it... but I may be missing something :-)
>
> I'm glad you mentioned Daniel Holth's setup-requires hack.  Although I
> haven't used it myself directly I generally like the concept.
>
> Yeah, setup_requires is a mess, but I'd be skeptical of solving the
> problem by depending on any new features in setuptools :/


In June I updated my setup_requires hack to be a PEP 518 implementation. It
reads build-system.requires out of pyproject.toml and installs them into an
isolated directory if necessary. The basic feature is 14 lines of code not
counting blanks, and you use it by prepending the code to your setup.py.
Once pip implements PEP 518 that code becomes a no-op.

Antony is right that the main problem with setup_requires is that it
happens too late, or that you have to write a setuptools extension to use
it. Most people do not know that it is possible to write a setuptools
extension, let alone want to write one.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Erik Bray]

On Mon, Aug 29, 2016 at 7:29 PM, Antony Lee  wrote:
> Hi all,
>
> The `setup_requires` option to `setup()` is well-known to suffer from
> multiple issues.  Most importantly, as it is a keyword argument to
> `setup()`, it appears too late for modules that may need to be imported for
> the build to occur (e.g., Cython, for which support must explicitly provided
> by setuptools itself rather than by letting Cython hook into it);
> additionally, there are various contorsions that people go to to avoid some
> `setup_requires` when not building the package (such as checking the value
> of `sys.argv`).  `setup_requires` also uses `easy_install` rather than
> `pip`, but I do not see why this could not be fixed; let's focus on the
> first issue instead.
>
> If `setup_requires` appears too late to be useful, the obvious(?) option is
> to move it earlier: provide a function, say, `setuptools.setup_requires()`,
> that should be called *before* `setup()` itself, e.g.:
>
> from setuptools import setup, setup_requires
> setup_requires("numpy", needed_for=["build_ext"])
> try:
> import numpy as np
> except ImportError:
> np = None
> setup(..., include_dirs=[np.get_include()] if np else [])

I mean this sort of already exists but it's spelled:

from setuptools import Distribution
Distribution({'setup_requires': ['numpy'])

Granted it's non-obvious and doesn't have the needed_for flag, which I
like.  It's not entirely clear how needed_for would work though.  For
example, what if the package you're requiring provides the command
that you need that package to run?

The same can be done by subclassing commands, and there can be some
corner cases where that gets extra tricky (Cython comes to mind).

> When `setup.py` is invoked, either directly or by pip, upon the call to
> `setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at
> least one requirement is missing, `setup_requires()` calls asks pip to
> install the required packages (similarly to
> `https://bitbucket.org/dholth/setup-requires`) in a temporary directory, and
> the whole Python process replaces itself (in the `os.execv()` sense) by a
> new call to `python setup.py` with this temporary directory prepended to the
> PYTHONPATH.  In this new process, the arguments to `setup_requires()` are
> now available and we can proceed to the rest of `setup.py`.
>
> I feel like this idea is natural enough that someone must already have come
> up with it... but I may be missing something :-)

I'm glad you mentioned Daniel Holth's setup-requires hack.  Although I
haven't used it myself directly I generally like the concept.

Yeah, setup_requires is a mess, but I'd be skeptical of solving the
problem by depending on any new features in setuptools :/

Best,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] setup_requires: the obvious option(?)

[Antony Lee]

Hi all,

The `setup_requires` option to `setup()` is well-known to suffer from
multiple issues.  Most importantly, as it is a keyword argument to
`setup()`, it appears too late for modules that may need to be imported for
the build to occur (e.g., Cython, for which support must explicitly
provided by setuptools itself rather than by letting Cython hook into it);
additionally, there are various contorsions that people go to to avoid some
`setup_requires` when not building the package (such as checking the value
of `sys.argv`).  `setup_requires` also uses `easy_install` rather than
`pip`, but I do not see why this could not be fixed; let's focus on the
first issue instead.

If `setup_requires` appears too late to be useful, the obvious(?) option is
to move it earlier: provide a function, say, `setuptools.setup_requires()`,
that should be called *before* `setup()` itself, e.g.:

from setuptools import setup, setup_requires
setup_requires("numpy", needed_for=["build_ext"])
try:
import numpy as np
except ImportError:
np = None
setup(..., include_dirs=[np.get_include()] if np else [])

When `setup.py` is invoked, either directly or by pip, upon the call to
`setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at
least one requirement is missing, `setup_requires()` calls asks pip to
install the required packages (similarly to `
https://bitbucket.org/dholth/setup-requires`) in a temporary directory, and
the whole Python process replaces itself (in the `os.execv()` sense) by a
new call to `python setup.py` with this temporary directory prepended to
the PYTHONPATH.  In this new process, the arguments to `setup_requires()`
are now available and we can proceed to the rest of `setup.py`.

I feel like this idea is natural enough that someone must already have come
up with it... but I may be missing something :-)

Best,
Antony
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig