Re: [Distutils] Sources of truth

[Robert Collins]

On 12 October 2015 at 18:36, Nathaniel Smith  wrote:
> Hi all,
>
> Again trying to split out some more focused discussion from the big
> thread about sdists...
>
> One big theme there has been the problem of "sources of truth": e.g.
> in current sdists, there is a PKG-INFO file that has lots of static
> metadata in it, but because the "real" version of that metadata is in
> setup.py, everyone ignores PKG-INFO.
>
> A clear desideratum for a new sdist format is that we avoid this
> problem, by having static metadata that is actually trustworthy. I see
> two fundamentally different strategies that we might use to accomplish
> this. In time honored mailing list tradition, these are of course the
> one that I hear other people advocating and the one that I like ;-).



> The first strategy is: sdists and the wheels they generate logically
> share the same metadata; so, we need some mechanism to enforce that

This is false: they don't share the same metadata. Some portions are
the same, but deps, supported platforms, those will differ (and
perhaps more than that).

In particular, an sdist doesn't have a dependency on an ABI, and a
wheel doesn't have a dependency on an API. Some APIs are ABIs
(approximately true for all pure Python packages, for instance), but
some are not (numpy).

> The second strategy is: put static metadata in both sdists and wheels,
> but treat them as logically distinct things: the static metadata in
> sdists is the source of truth for information *about that sdist*
> (sdist name, sdist version, sdist description, sdist authors, etc.),
> and the static metadata in wheels is the source of truth for
> information about that wheel, but we think of these as distinct things
> and don't pretend that we can statically guarantee that they will
> match. I mean, in practice, they basically always will match.

The analgous current data won't match for pbr using packages when we
fix https://bugs.launchpad.net/pbr/+bug/1502692 (older pip's don't
support PEP-426 environment markers, but don't error when they are
used either, leading to silent failure to install dependencies).

Now, you might say 'hey, but the new shiny will support markers from
day one'. Well the problem is backwards compat: we're going to have
future things that change, and the more we split things out the more
the changes are likely to need skewed results like this approach to
deal with it.

...
> the sdist name instead of the wheel name, it can actually do it

but the sdist and the wheel have to have the same name- or do you mean
the filename on disk, vs the distribution name?

> reliably in a totally static way, without having to run arbitrary code
> to validate this. OTOH pip will always have to be prepared to handle
> the possibility of mismatch between what it was expecting based on the
> sdist metadata and what it actually got after building it, so we might
> as well acknowledge that in our mental model.
>
> One potential advantage of this approach is that we might be able to
> talk ourselves into trusting the existing PKG-INFO as providing static
> metadata about the sdist, and thus PyPI at least could start trusting
> it for things like the "description" field, and if we define a new

The challenge is the 40K broken packages up there on PyPI. Basically
pip has a bugfix for any of:
sdists built using distutils
sdists built using random build systems that don't understand what an
sdist is (e.g. automake)
sdists built using versions of setuptools that had a bug in this area

There is no corrective mechanism for broken packages other than
route-around-it-while-you-ask-the-author-to-upload-a-fix.

So I think to tackle the 'please trust the metadata in the sdist'
problem, one needs to have a graceful ramp-up of that trust with
robust backoff mechanisms that don't involve 50% of PyPI users hating
on that one old project in the corner everyone has a dep on but that
is actually moribund and not doing uploads. I can imagine several such
routes, including a crowdsourced blacklist - but its going to be (like
we're dealing with with the automatic wheel cache already) years of
bug reports until things age out.

> sdist format then it would be possible to generate its static metadata
> from current setup.py files (e.g. by modifying setuptools's sdist
> command). Contrast this with the other approach, where getting any
> kind of static source-of-truth would require rewriting almost all
> existing setup.py files.

We already generate static metadata from current setup.py files:
setup.py egg_info does precisely that. There, bug fixed ;).

> The challenge, of course, is that there are a few places where pip
> actually does need to know something about wheels based on examining
> an sdist -- in particular name and version and (controversially)
> dependencies. But this can/should be addressed explicitly, e.g. by
> writing down a special rule about the name and version fields.

I'm sorry, I don't follow.

-Rob


-- 
Robert Collins 

Re: [Distutils] What metadata does pip actually need about sdists?

[Nathaniel Smith]

On Sun, Oct 11, 2015 at 10:55 PM, Marcus Smith  wrote:
>>
>> So instead, the current plan is that we're going
>> to drop the libraries inside a wheel and upload it to PyPI:
>
> aha... ok, now it's clearer where you're coming from.
> but using what platform in the wheel tag?
> linux wheels are blocked currently

Windows is the first goal, likely OS X after that (OS X provides a bit
more in the base system so distributing these external dependencies
isn't as urgent, but it would be good eventually), and yeah, doing
this for Linux would obviously require unblocking Linux wheels.

(We do have a plan for unblocking them, but the critical person is a
bit busy so I'm not sure how long it will be before it can be unveiled
:-). It's not really a secret, the plan is just: take advantage of
Continuum and Enthought's hard-won knowledge from distributing
hundreds of thousands of pre-built python package binaries on Linux to
define a named "common Linux platform" abi tag that is known to be
abi-compatible with >99.99% of Linux installs out there. This doesn't
prevent the use of other abi tags -- e.g. distro-specific ones -- or
the development of more detailed metadata about "distro dependencies",
and wouldn't require that every single Linux box be willing to install
wheels with this abi tag, but it would at least bring Linux support up
to parity with the other platforms.)

> regardless of the tagging though, I'm not so sure about using PyPI like this
> to hold all these "python" wrapper packages for binary extensions

If someone else invents a better solution before this is implemented
then of course we'd switch to that instead...

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What metadata does pip actually need about sdists?

[Paul Moore]

On 12 October 2015 at 07:14, Nathaniel Smith  wrote:
> On Sun, Oct 11, 2015 at 10:55 PM, Marcus Smith  wrote:
>>>
>>> So instead, the current plan is that we're going
>>> to drop the libraries inside a wheel and upload it to PyPI:
>>
>> aha... ok, now it's clearer where you're coming from.
>> but using what platform in the wheel tag?
>> linux wheels are blocked currently
>
> Windows is the first goal, likely OS X after that (OS X provides a bit
> more in the base system so distributing these external dependencies
> isn't as urgent, but it would be good eventually), and yeah, doing
> this for Linux would obviously require unblocking Linux wheels.

I'm running out of time to keep up with this mega-thread, so I'm
starting to only skim at this point. Apologies if I've missed
something that has already been said as a result...

My view is that if you distribute a library (e.g., openblas) in a
wheel, then it's essentially a "Python dependency" at that point,
insofar as it can participate in the standard package dependency
process in exactly the same way as (say) numpy itself does. So if you
use that approach, I no longer see *any* reason why you can't work
within the existing dependency process. Is that right?

> We do have a plan for unblocking them, but the critical person is a
> bit busy so I'm not sure how long it will be before it can be unveiled
> :-).

Obviously, as Marcus said, you need Linux wheels unblocked, and that
means being able to specify compatibility cleanly, but as you say
here, you're looking at that.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Robert Collins]

On 12 October 2015 at 21:15, Thomas Kluyver  wrote:
> On 12 October 2015 at 05:37, Robert Collins 
> wrote:
>>
>> And a bugfix :) - I didn't link to the docs for the build system
>> interface we have today -
>>
>> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
>
>
> I'm happy to see that the command line interface pip requires from setup.py
> is now documented. But the first thing it describes is the egg_info command,
> and the description of what that involves is basically 'whatever setuptools
> does'. Egg info is not any kind of standard, AFAIK - unlike dist info. One

https://www.python.org/dev/peps/pep-0314/ describes PKG_INFO, which
can in principle include dependency data. I don't know why they're not
being fully populated - I suspect its another case of
forward-looking-spec-not-quite-right-and-thus-never-used. Someone with
more history on Python packaging will have to fill that in. One option
would be to push on that.

> of my main goals in writing flit is to build wheels without involving
> setuptools at all, so I'm probably never going to implement that.

Well thats fair enough. but: pip uses pkg_resources to find out whats
installed and what it depends on, and that pulls out of egg-info on
the file system for non-wheel-installs (dist-info for wheel installs).
So, if that isn't well enough documented, it would be entirely
reasonable to make a PEP to capture it so that you can emit it with
confidence. There's already buy-in from everyone in the ecosystem to
use PEPs to document interoperability impacting changes across these
tools, so I can't imagine it being a contentious proposal.

> I also don't want to have a setup.py in the VCS at all, because it's an
> invitation for people to run setup.py whatever, and then file bug reports
> that it doesn't do exactly what setuptools/distutils would do.

Ok, so I understand that frustration. You're going to get those
through pip too though - right now there's a giant escape clause in
pip where arbitrary options are passed through to setup.py, because we
don't have semantic modelling of all the things that people want to
do. And until thats achieved - and the options aged out - folk are
going to use those options and then be surprised when flit using
packages don't behave the same way.

> This is what I like about Nathaniel's proposal. By looking for a new file,
> it clearly indicates that this is a new style Python source tree, without
> potential confusion over what setup.py does, and it lets us specify a simple
> build system interface based on standardised pieces like wheels rather than
> messing around with the details of setuptools.

The problem with 'simple' is that we've got a rich (not just old:
fully featured) interface. Building something that isn't complected
out from behind that is both valuable and hard. An 85% solution or 95%
solution will retain the trap-doors that let folk tickle the system
from the outside, and thats going to lead back to the same bug reports
you don't want IMO :/.

I'm not against a spec that avoids the need for each tree to have a
setup.py - I just don't think its going to be the end of the story in
any deep fashion :).

Here's a draft one I'd be happy with (with no boilerplate or
anything). For commands we can define a small number environment
variables like e.g. OUTPUT_PATH and PYTHON have those supplied when
invoking the command.

Python packaging config in $root/pypa.yaml

Defined keys:

version:
  # not needed yet, since its a new file, defaults to 1.
setup-requires:
  - requirement
  - requirement[extra]
  - requirement:markers
  - requirement[extra]:markers
build-tool:
  # basic command that will spit a hunk of json back to the caller defining the
  # commands to use with the build tool.


Build tool output json (in yaml format for ease of reading):
version:
  # not needed yet, defaults to 1
egg-info:
   # comand to run to generate an egg-info directory
   # only used if dist-info is not defined
dist-info:
   # command to run to generate a dist-info directory
wheel:
   # command to build a wheel
develop:
   # command to do an in-place installation
install:
  # command to do a direct installation
  # only used if wheeling failed
provided-by:
  # the requirement that provides the build system.
  # this is used to facilitate caching the build tool output
  # so that we don't need to probe everytime. (e.g. if the
  # same dependency version is present in two cases
  # we can presume the output - I threw this in as an
  # obvious tweak for large dependency chains but it is
  # probably not worth it for typical 10-20 package things

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Sources of truth

[Paul Moore]

On 12 October 2015 at 08:23, Nathaniel Smith  wrote:
> I mean the distribution name - there's no way to guarantee that
> building foo-1.0.zip won't spit out bar-7.4.whl, where by "no way" I
> mean "it's literally undecideable". I mean, if someone actually did
> this it would be super weird and we would all shun them, but our code
> and specs still need to be prepared for the possibility.

But the whole point of a spec is to *make* that guarantee. Pip doesn't
expect to ever work with random code that can do literally anything,
that's silly. Pip works with code that conforms to a spec.

At the moment, the spec is "it does what setuptools/distutils does",
which is a lousy spec because (a) nobody else can implement the spec
without reading the huge mess that is distutils/setuptools, (b) it's
too broad (we don't *really* need every corner case that distutils
covers) and (c) distutils/setuptools don't satisfy the needs of a
number of communities, notably the scientific community.

So we want to define a more usable spec for the future. Pip will
always have to deal with backward compatibility, and all of the hacks
that implies, but we can at least say "if a project declares that it
follows the new spec, we can rely on certain things". One of those
things we can rely on is that building foo-1.0.zip won't spit out
bar-7.4.whl - that would be part of the contract of that spec.

The debate is over on the one hand, we want to be able to rely on when
writing packaging tools, vs what flexibility we want to have when
writing build tools. But it's not over ignoring the other use cases,
it's about agreeing a *contract* that satisfies the needs of *both*
packaging tools and build tools.

At the moment, the dial is 100% over to the packaging side - builders
have zero flexibility, the rule is "use distutils and setuptools".
That doesn't feel like zero flexibility, because distutils/setuptools
let you do a lot of things. But when you do finally hit the limits
you're stopped cold. It also is a bad choice for packaging tools,
because distutils and setuptools are a dreadful API for automation. So
nobody wins.

Two suggestions that have been around for a long while that give more
flexibility to build tools while at the same time giving packaging
tools a better interface to work with are:

1. Declarative setup - you have to put your metadata in file X in
format Y, and it's static. But when you do, your build tool can do
whatever it wants as long as it spits out binaries that confirm to
your declared metadata. The downside to this is that it's totally
static, and developers don't like that ("what if I want to generate my
version number from my VCS tags?")
2. A fixed API - we document a set of commands that packaging tools
can use and build tools have to provide. This is where we talk about
documenting the distutils commands that pip relies on ("setup.py
egg-info", "setup.py install", "setup.py develop"...) This one usually
falls down because nobody likes the idea of writing a "dummy" setup.py
that translates the interface, and because no-one has done anything
more with this idea than suggest documenting the setuptools commands
that pip *currently* uses, even though these are probably a lousy API
design. It also does very little for packaging tools *other* than pip
(for example, it does nothing for PyPI, which cannot accept an API
that requires running user-supplied code).

The key drivers here have been about defining something that packaging
tools can use effectively, and build tools can cope with being
required to provide. While not constraining *either* type of tool
beyond the minimum needed for the contract.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What metadata does pip actually need about sdists?

[Nathaniel Smith]

On Sun, Oct 11, 2015 at 10:49 PM, Donald Stufft  wrote:
> FTR, I plan on making some sort of auto builder for PyPI so it’s possible
> that we can get pip to a point where almost all things it downloads are
> binary wheels and we don’t need to worry too much about needing to optimize
> the sdist case.

That would be 1000% awesomesauce.

> I also think that part of the problem with egg-info and
> setup.y’s and pip’s attempted use of them is just down to the setup.py
> interface being pretty horrible and setup.py’s serving too many masters
> where it needs to be both VCS entry point, sdist metadata entry point,
> installation entry point, and wheel building entry point.

Yeah, I'm glad that Robert linked
  https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
because I hadn't seen it, and it's great that it's documented, but
what reading that documentation mostly convinced me is that I
definitely could not write a new implementation of that interface and
get it right without significant pain and trial-and-error :-). I
mean... you have to support both --single-version-externally-managed
enabled and disabled? develop has to know the right arcane magic to
get the directory added to sys.path? (how do you know where in
sys.path to add the magic?) what is egg_info even supposed to do --
the linked to docs don't actually define what egg metadata looks like,
and IIRC it's not standardized, is it? (as opposed to the dist-info
used in wheels)

Like, obviously all that should be needed to implement a new build
system is the ability to generate a wheel, right? Wheels are
well-documented and one honking great idea.

My current best guess at where we want to end up with sdists is:

- there's a static file like the _pypackage/_pypackage.cfg in my
original proposal (ignore the name, we can change it, I don't care)
- it's written in some simple well-specified language like json or
toml (if you look at the spec, toml is basically json encoded in an
INI-style syntax -- it's definitely nowhere near as complex as yaml)
- it includes static metadata for name, version (required for pypi
uploads, optional otherwise to accomodate VCS checkouts), description,
long_description, authors, URLs, trove, etc. etc., but not
install_depends
- it also includes the setup-dependencies / setup-dependencies-dynamic
/ build-wheel / build-in-place entries/hooks from my original proposal
(or the moral equivalent)
- it lives inside the single top-level directory next to the source
code (instead of having a meta/ + source/ split, because that way it
works for VCS checkouts too and AFAICT there aren't any other
advantages to this split. We can afford to steal a single name --
everyone use used to having a setup.py or Makefile or Gemfile or
whatever cluttering up the top-level of their source, and this is just
a replacement for that.)

That gives you static metadata for PyPI and solves the setup_depends
problem while simultaneously cleaning up the "fugly" setup.py
interface so that it becomes easy to write your own build system. (I
think instead of just arguing about tests, we should have big debates
here about which of the 3-4 highly-regarded competing build systems we
should be recommending to newbies. But this will only happen if we
make designing a new build system something that dozens of people do
and a few take seriously, which requires making it accessible.)

I *think* the above interface also makes it possible to write a shim
in either direction (a setup.py that speaks the current "fugly"
interface to old versions of pip while using the new files to actually
do its work, or a new-style sdist that delegates all the actual work
to setup.py), and even for setuptools to start generating sdists that
are simultaneously both old-style and new-style for compatibility.

So I think we can at least solve the metadata-for-PyPI problem, the
setup_requires problem, and the let-a-thousand-build-systems-bloom
problems now, without solving the static-install_requires-in-sdists
problem (but also without creating any particular obstacles to solving
it in the future, if it's even possible). And in the mean time we can
keep working on autobuilders and linux wheels that will remove much of
the demand for automagic sdist building -- it's difficult to make
predictions, especially about the future, so who knows, maybe in the
end we'll decide we just don't care that much about
static-install_requires-in-sdists.

> I also think that it would be a terrible idea to have the science stack
> leave the “standard” Python packaging ecosystem and go their own way and I
> think it’d make the science packages essentially useless to the bulk of the
> non science users. I think a lot of the chasing rainbows like stuff comes
> mostly from: We have some desires from our experiences but we haven’t yet
> taken the time to fully flesh out what the impact of those desires are, nor
> are any of us science stack users (or contributors) to my 

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Robert Collins]

On 12 October 2015 at 19:23, Ralf Gommers  wrote:
>
>
> On Mon, Oct 12, 2015 at 6:37 AM, Robert Collins 
> wrote:
>>
>> On 12 October 2015 at 17:06, Robert Collins 
>> wrote:
>> > EWOW, huge thread.
>> >
>> > I've read nearly all of it but in order not to make it massively
>> > worse, I'm going to reply to all the points I think need raising in
>> > one mail :).
>>
>> And a bugfix :) - I didn't link to the docs for the build system
>> interface we have today -
>>
>> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
>
>
> From that link:
> """
> In order for pip to install a package from source, setup.py must implement
> the following commands:
> ...
> The install command should implement the complete process of installing the
> package to the target directory XXX.
>  """
> That just sounds so wrong. You want the build system to build, not install.

Right so - with the automatic wheel cache we added, that is only used
if building a wheel failed. So you can consider it to be legacy cruft
- the preferred interface is build-a-wheel then install-that-wheel.

> And if "install" actually means "build to a tempdir so pip can copy it over
> it to its final location", then how does that address something like
> installing docs to a different dir than the package itself?

Well, how do we install docs to a different dir with wheels? If we've
got an answer for that, I think we're in a good position to figure
something out for install (even if that is 'fix your package so it can
build wheels).

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Sources of truth

[Nathaniel Smith]

On Sun, Oct 11, 2015 at 11:00 PM, Robert Collins
 wrote:
> On 12 October 2015 at 18:36, Nathaniel Smith  wrote:
[...]
>> the sdist name instead of the wheel name, it can actually do it
>
> but the sdist and the wheel have to have the same name- or do you mean
> the filename on disk, vs the distribution name?

I mean the distribution name - there's no way to guarantee that
building foo-1.0.zip won't spit out bar-7.4.whl, where by "no way" I
mean "it's literally undecideable". I mean, if someone actually did
this it would be super weird and we would all shun them, but our code
and specs still need to be prepared for the possibility. IIUC this is
why PyPI can't trust PKG-INFO: 99.9% of the time the metadata in
PKG-INFO matches what you will get when you run setup.py, but right
now PyPI wants to know what setup.py will do, and there's no way to
know if it will be the same as what PKG-INFO says, so it just doesn't
trust PKG-INFO.

OTOH if we redefine PyPI's goal as being, figure out what's in
PKG-INFO (or whatever replaces it), and declare that it's okay (for
PyPI's purposes) if that doesn't match what the build system will
eventually do, then that's a viable way forward.

>> reliably in a totally static way, without having to run arbitrary code
>> to validate this. OTOH pip will always have to be prepared to handle
>> the possibility of mismatch between what it was expecting based on the
>> sdist metadata and what it actually got after building it, so we might
>> as well acknowledge that in our mental model.
>>
>> One potential advantage of this approach is that we might be able to
>> talk ourselves into trusting the existing PKG-INFO as providing static
>> metadata about the sdist, and thus PyPI at least could start trusting
>> it for things like the "description" field, and if we define a new
>
> The challenge is the 40K broken packages up there on PyPI. Basically
> pip has a bugfix for any of:
> sdists built using distutils
> sdists built using random build systems that don't understand what an
> sdist is (e.g. automake)
> sdists built using versions of setuptools that had a bug in this area
>
> There is no corrective mechanism for broken packages other than
> route-around-it-while-you-ask-the-author-to-upload-a-fix.

IIUC what PyPI wants to do with PKG-INFO is read out stuff like the
description and trove classifiers fields. Are there really 40K sdists
on PyPI that have PKG-INFO files and where those files contain
incorrect descriptions and so forth? I mean, obviously someone would
have to check :-) But it seems unlikely, since almost everyone uploads
by running 'sdist upload' or twine or something similarly automated.

> So I think to tackle the 'please trust the metadata in the sdist'
> problem, one needs to have a graceful ramp-up of that trust with
> robust backoff mechanisms that don't involve 50% of PyPI users hating
> on that one old project in the corner everyone has a dep on but that
> is actually moribund and not doing uploads. I can imagine several such
> routes, including a crowdsourced blacklist - but its going to be (like
> we're dealing with with the automatic wheel cache already) years of
> bug reports until things age out.
>
>> sdist format then it would be possible to generate its static metadata
>> from current setup.py files (e.g. by modifying setuptools's sdist
>> command). Contrast this with the other approach, where getting any
>> kind of static source-of-truth would require rewriting almost all
>> existing setup.py files.
>
> We already generate static metadata from current setup.py files:
> setup.py egg_info does precisely that. There, bug fixed ;).

I'm pretty sure that merely making it so 'setup.py sdist' created a
file that contained the output from egg_info would not solve the
current problem. That's pretty much exactly what the existing PKG-INFO
*is*, isn't it? Yet apparently no-one trusts it.

>> The challenge, of course, is that there are a few places where pip
>> actually does need to know something about wheels based on examining
>> an sdist -- in particular name and version and (controversially)
>> dependencies. But this can/should be addressed explicitly, e.g. by
>> writing down a special rule about the name and version fields.
>
> I'm sorry, I don't follow.

E.g., we can document that if you have a sdist foo-1.0, then pip and
similar tools will expect this to generate a foo-1.0 wheel (but be
prepared to do something sensible if this doesn't happen, like give an
error message or whatever). That's really all pip needs, right?

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Daniel Holth]

I could be wrong but if egg-info actually generated a dist-info directory
it would probably still work.

On Mon, Oct 12, 2015 at 8:50 AM Thomas Kluyver  wrote:

> On 12 October 2015 at 05:37, Robert Collins 
> wrote:
>
>> And a bugfix :) - I didn't link to the docs for the build system
>> interface we have today -
>>
>> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
>>
>
> I'm happy to see that the command line interface pip requires from
> setup.py is now documented. But the first thing it describes is the
> egg_info command, and the description of what that involves is basically
> 'whatever setuptools does'. Egg info is not any kind of standard, AFAIK -
> unlike dist info. One of my main goals in writing flit is to build wheels
> without involving setuptools at all, so I'm probably never going to
> implement that.
>
> I also don't want to have a setup.py in the VCS at all, because it's an
> invitation for people to run setup.py whatever, and then file bug reports
> that it doesn't do exactly what setuptools/distutils would do.
>
> This is what I like about Nathaniel's proposal. By looking for a new file,
> it clearly indicates that this is a new style Python source tree, without
> potential confusion over what setup.py does, and it lets us specify a
> simple build system interface based on standardised pieces like wheels
> rather than messing around with the details of setuptools.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Thomas Kluyver]

On 12 October 2015 at 05:37, Robert Collins 
wrote:

> And a bugfix :) - I didn't link to the docs for the build system
> interface we have today -
> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
>

I'm happy to see that the command line interface pip requires from setup.py
is now documented. But the first thing it describes is the egg_info
command, and the description of what that involves is basically 'whatever
setuptools does'. Egg info is not any kind of standard, AFAIK - unlike dist
info. One of my main goals in writing flit is to build wheels without
involving setuptools at all, so I'm probably never going to implement that.

I also don't want to have a setup.py in the VCS at all, because it's an
invitation for people to run setup.py whatever, and then file bug reports
that it doesn't do exactly what setuptools/distutils would do.

This is what I like about Nathaniel's proposal. By looking for a new file,
it clearly indicates that this is a new style Python source tree, without
potential confusion over what setup.py does, and it lets us specify a
simple build system interface based on standardised pieces like wheels
rather than messing around with the details of setuptools.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Ralf Gommers]

On Mon, Oct 12, 2015 at 6:37 AM, Robert Collins 
wrote:

> On 12 October 2015 at 17:06, Robert Collins 
> wrote:
> > EWOW, huge thread.
> >
> > I've read nearly all of it but in order not to make it massively
> > worse, I'm going to reply to all the points I think need raising in
> > one mail :).
>
> And a bugfix :) - I didn't link to the docs for the build system
> interface we have today -
> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
>

>From that link:
"""
In order for pip to install a package from source, setup.py must implement
the following commands:
...
The install command should implement the complete process of installing the
package to the target directory XXX.
 """
That just sounds so wrong. You want the build system to build, not install.
And if "install" actually means "build to a tempdir so pip can copy it over
it to its final location", then how does that address something like
installing docs to a different dir than the package itself?

+1 for your main point of focusing more on enabling other build systems
though.

Ralf



> -Rob
>
>
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Wes Turner]

On Oct 11, 2015 11:07 PM, "Robert Collins" 
wrote:
>
> EWOW, huge thread.
>
> I've read nearly all of it but in order not to make it massively
> worse, I'm going to reply to all the points I think need raising in
> one mail :).
>
> Top level thoughts here, more point fashion with only rough editing
> below the fold.
>
> I realise many things - like the issue between different wheels of the
> same package consuming different numpy abis - have been touched on,
> but AFAICT they are entirely orthogonal to the proposal, which was to
> solve 'be able to use arbitrary build systems and still install with
> pip'.
>
> Of the actual problems with using arbitrary build systems, 99% of them
> seem to boil down to 'setup-requires isn't introspectable by pip
> (https://github.com/pypa/pip/issues/1820 ). - If it was, then
> alternative build systems could be depended on reasonably; and the
> mooted thunk from setuptools CLI to arbitrary build system would be
> viable.
>
> It is, in principle a matter of one patch to teach pip *a* way to do
> this (and then any and all build systems that want to can utilise it).
> https://github.com/rbtcollins/pip/tree/declarative is a POC I did - my
> next steps on that were to discuss the right ecosystem stuff for it -
> e.g. should pip consume it via setuptools, or should pip support it as
> *the way* and other systems including setuptools can choose to use it?

as a standard RDF graph representation,
JSON-LD would be uniquely portable here.

"PEP 426: Define a JSON-LD context as part of the proposal"
https://github.com/pypa/interoperability-peps/issues/31

>
> A related but separate thing is being able to *exclusively* install
> things without setuptools present - I've filed
> https://github.com/pypa/pip/issues/3175 about that, but I think its
> -much- lower priority than reliably enabling third party build tools.

peep may not need setuptools?
* SHA256
* --no-deps
* https://pypi.python.org/pypi/peep
* wheels

>
> -Rob
>
> 
>
>
> "
> solved many of the hard problems here -- e.g. it's no longer necessary
> that a build system also know about every possible installation
> configuration -- so pretty much all we really need from a build system
> is that it have some way to spit out standard-compliant wheels.
> "
>
> Actually pip still punts a *lot* here - we have bypasses to let things
> like C compiler flags be set during wheel build, and when thats done
> we don't cache the wheels (or even try to build wheels).
>
> "
> While ``distutils`` / ``setuptools`` have taken us a long way, they
> suffer from three serious problems: ...
> (c) you are forced to use them anyway, because they provide the
> standard interface for installing python packages expected by both
> users and installation tools like ``pip``."
>
> I don't understand the claim of (c) here - its entirely possible to
> write a package that doesn't use setuptools and have it do the right
> thing - pip uses a subprocess to drive package installation, and the
> interface is documented. The interface might be fugly as, but it
> exists and works. It is missing setup-requires handling, but so is
> setup.py itself. The only thing we'd really need to do AFAICT is make
> our setuptools monkeypatching thunk handle setuptools not being
> installed (which would be a sensible thing to Just Do anyhow).
>
> "
> - query for build dependencies
> - run a build, producing wheels as output
> - set up the current source tree so that it can be placed on
>   ``sys.path`` in "develop mode"
> "
>
> So we have that already. setup.py egg-info, setup.py bdist_wheel,
> setup.py develop.
>
> "A version 1-or-greater format source tree can be identified by the
> presence of a file ``_pypackage/_pypackage.cfg``.
> "
>
> I really don't like this. Its going to be with us forever, and its
> intrusive (its visible), and so far isn't shown to be fixing anything.
>
>
> "to scatter files around willy-nilly never works, so we adopt the
> convention that names starting with an underscore are reserved for
> official use, and non-underscored names are available for
> idiosyncratic use by individual projects."
>
> I can see the motivation here, but is it really solving a problem we have?
>
>
> On the specifics of the format: I don't want to kibbitz over strawman
> aspects at this point.
>
> Having the extension mechanism be both pip specific and in Python
> means that we're going to face significant adoption issues: the former
> because pip is not by any means the only thing around - and some
> distros have until very recently been actively hostile to pip (which
> in turn means we need to wait a decade or two for them to age-out and
> stop being used). The latter because we'll face all the headaches of
> running arbitrary untrusted code and dealing with two deps with
> different versions of the same hook and so on: I think its an
> intrinsically unsafe design.
>
> @dstufft "problem with numpy.distutils, as I know you’re aware!). We
> could 

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Thomas Kluyver]

On 12 October 2015 at 11:01, Robert Collins 
wrote:

> Python packaging config in $root/pypa.yaml
>
> Defined keys:
> 
> ...
> build-tool:
>   # basic command that will spit a hunk of json back to the caller
> defining the
>   # commands to use with the build tool.
> 
>
> Build tool output json (in yaml format for ease of reading):
>

I would be quite happy with something along the lines of this proposal,
though I'd bikeshed about some of the details. I like the idea of the
source tree having a single reference to the build tool, and the build tool
describing itself to pip. I'd probably use references to Python
functions/modules rather than specifying shell-style commands, though; it
seems like there's less to go wrong that way.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Robert Collins]

On 13 October 2015 at 06:23, Nathaniel Smith  wrote:
> On Oct 12, 2015 10:16 AM, "Robert Collins" 
> wrote:
...
>> So until we solve the problems related to unloading something loaded
>> into Python and loading a different version in and all the related
>> pain that can occur - I think using Python function calls is a
>> non-starter.
>
> I don't see the contradiction here. If you look at the original draft PEP
> then it exactly specifies that builds get isolated environments, and build
> tools are supposed to spawn a child and then have that child do a function
> call using whatever mechanism they prefer.

Ok, so here's a worked example to let us debug the disconnect.

Given:

A@1.0: setup-requires: S~=1.0
 install-requires: B
B@1.0: setup-requires: S~=2.0
S@1.0: no dependencies at all.
S@2.0: no dependencies at all.

and no binaries of A or B... then:

  pip install A

will do the following (key bits related to this proposal only, happy path only):

- download the A@1.0 sdist
- read pypa.yaml and read the setup-requires + build-tool keys
- download S@1.0 and prepare a built version of it
- place S@1.0 into PYTHONPATH and the built versions bin into PATH
   - run the build-tool command to determine how to use it
   - run the resulting wheel-build command to build a wheel for A@1.0
- read the wheel metadata in to determine A's install-requires
- download the B@1.0 sdist
- read pypa.yaml and read the setup-requires + build-tool keys
- download S@2.0 and prepare a built version of it
- place S@2.0 into PYTHONPATH and the built versions bin into PATH
   - run the build-tool command to determine how to use it
   - run the resulting wheel-build command to build a wheel for B@1.0
- read the wheel metadata in to determine B's install-requires
- install the B wheel into the target environment
- install the A wheel into the target environment

Note the two places where PYTHONPATH and PATH need to be something
other than the environment of pip itself. pip may not be installed in
the target environment (or may be a different version than the pip
installed there). I don't understand how you propose to have S@1.0 and
S@2.0 co-exist in pip's Python process.

-Rob
-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Robert Collins]

On 13 October 2015 at 01:52, Daniel Holth  wrote:
> I could be wrong but if egg-info actually generated a dist-info directory it
> would probably still work.

I'd worry about fallout - since pip doesn't need to change (much)
either way, I'd be inclined to be specific and allow setuptools to
merely define the bootstrap code and opt in to later changes without
pressure.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Meeting info re: sdists

[Nathaniel Smith]

Hangouts URL:
  https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a

Shared document for notes:
  
https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Robert Collins]

On 13 October 2015 at 02:33, Thomas Kluyver  wrote:
> On 12 October 2015 at 11:01, Robert Collins 
> wrote:
>>
>> Python packaging config in $root/pypa.yaml
>>
>> Defined keys:
>> 
>> ...
>> build-tool:
>>   # basic command that will spit a hunk of json back to the caller
>> defining the
>>   # commands to use with the build tool.
>> 
>>
>> Build tool output json (in yaml format for ease of reading):
>
>
> I would be quite happy with something along the lines of this proposal,
> though I'd bikeshed about some of the details. I like the idea of the source
> tree having a single reference to the build tool, and the build tool
> describing itself to pip. I'd probably use references to Python
> functions/modules rather than specifying shell-style commands, though; it
> seems like there's less to go wrong that way.

One of the fundamental things that emerged during the review of the
design of my static setup-requires implementation in pip was that
setuptools behaviour of not installing setup requirements into the
target environment was deliberate design: it permits the use of
different, mutually incompatible, versions of a given setup
requirement by packages in the same dependency tree. E.g. imagine A
and B both use setuptools-vcs, and setuptools-vcs does an incompatible
2.0 release. When A upgrades to that and B hasn't, if B
install-requires A, pip installing B needs to install both those
setuptools-vcs versions transiently, not permanently. (Even if one
version is already installed, the build-time actions around the other
of A|B need to have the other version installed). [My branch of pip
doesn't do this - its one of the differences between proof of concept
and production ready]

So until we solve the problems related to unloading something loaded
into Python and loading a different version in and all the related
pain that can occur - I think using Python function calls is a
non-starter.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Nathaniel Smith]

On Oct 12, 2015 10:16 AM, "Robert Collins" 
wrote:
>
> On 13 October 2015 at 02:33, Thomas Kluyver  wrote:
> > On 12 October 2015 at 11:01, Robert Collins 
> > wrote:
> >>
> >> Python packaging config in $root/pypa.yaml
> >>
> >> Defined keys:
> >> 
> >> ...
> >> build-tool:
> >>   # basic command that will spit a hunk of json back to the caller
> >> defining the
> >>   # commands to use with the build tool.
> >> 
> >>
> >> Build tool output json (in yaml format for ease of reading):
> >
> >
> > I would be quite happy with something along the lines of this proposal,
> > though I'd bikeshed about some of the details. I like the idea of the
source
> > tree having a single reference to the build tool, and the build tool
> > describing itself to pip. I'd probably use references to Python
> > functions/modules rather than specifying shell-style commands, though;
it
> > seems like there's less to go wrong that way.
>
> One of the fundamental things that emerged during the review of the
> design of my static setup-requires implementation in pip was that
> setuptools behaviour of not installing setup requirements into the
> target environment was deliberate design: it permits the use of
> different, mutually incompatible, versions of a given setup
> requirement by packages in the same dependency tree. E.g. imagine A
> and B both use setuptools-vcs, and setuptools-vcs does an incompatible
> 2.0 release. When A upgrades to that and B hasn't, if B
> install-requires A, pip installing B needs to install both those
> setuptools-vcs versions transiently, not permanently. (Even if one
> version is already installed, the build-time actions around the other
> of A|B need to have the other version installed). [My branch of pip
> doesn't do this - its one of the differences between proof of concept
> and production ready]
>
> So until we solve the problems related to unloading something loaded
> into Python and loading a different version in and all the related
> pain that can occur - I think using Python function calls is a
> non-starter.

I don't see the contradiction here. If you look at the original draft PEP
then it exactly specifies that builds get isolated environments, and build
tools are supposed to spawn a child and then have that child do a function
call using whatever mechanism they prefer.

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig