Re: [Distutils] PEP 517 again

On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore wrote: > Me too. At the moment, I only expect two backends of any substance - > your flit backend and xoviat's setuptools one. But I only know of one > frontend, namely pip - and talk of projects like tox or twine acting > as frontends never seems to g

Re: [Distutils] PEP 517 again

On 29 August 2017 at 01:13, Chris Barker wrote: > On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore wrote: >> >> Me too. At the moment, I only expect two backends of any substance - >> your flit backend and xoviat's setuptools one. But I only know of one >> frontend, namely pip - and talk of projects

Re: [Distutils] PEP 517 again

On Mon, 28 Aug 2017 at 16:29 Nathaniel Smith wrote: > On Mon, Aug 28, 2017 at 1:27 PM, Thomas Kluyver > wrote: > > On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote: > > > > > Then end the debate by letting the PEP authors decide the return type, > and > > > write a paragraph explaining why

Re: [Distutils] PEP 517 again

On 2017-08-29 01:20, Paul Moore wrote: On 29 August 2017 at 01:13, Chris Barker wrote: On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore wrote: Me too. At the moment, I only expect two backends of any substance - your flit backend and xoviat's setuptools one. But I only know of one frontend, name

Re: [Distutils] PEP 517 again

On 29 August 2017 at 17:22, Phil Austin wrote: > On 2017-08-29 01:20, Paul Moore wrote: >> >> On 29 August 2017 at 01:13, Chris Barker wrote: >>> If the build API is clean and supported enough, I can see conda using it. >>> >>> just sayin' >> >> >> Thanks - that's really good to know. So *is* the

Re: [Distutils] PEP 517 again

> conda-build doesn't produce or consume wheels. It works by creating a > clean > > conda environment and running a shell script to install the python > package > > into that environment To be really clear -- conda build doesn't directly use ANY build/install system. All files produced by the i

Re: [Distutils] PEP 517 again

On 29 August 2017 at 19:45, Chris Barker wrote: > But it would be nice if packages didn't NEED to be pip installed -- but I > think that's an issue with the wheel spec (that may well be solved). Wheels don't need pip to install them - the spec has always been clear on how to "manually" install a

Re: [Distutils] PEP 517 again

> For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution). I hate to reiterate too much, but I think that you are correct here: NotImplemented is not Pythonic. It's a complete hack that I

Re: [Distutils] PEP 517 again

Hi, On Tue, Aug 29, 2017 at 8:30 PM, xoviat wrote: >> For me, using NotImplemented is a misuse of the singleton since I know >> what it's meant to be used for (and so I cringe every time I hear it brought >> up as a solution). > > I hate to reiterate too much, but I think that you are correct her

Re: [Distutils] PEP 517 again

So while I was working on pip, I noticed a parameter called 'autobuilding' in wheel.py. What this does (I think) is that if pip decides that it needs to be enabled (which is an implementation detail not relevant here), it adds the following to the "setup.py bdist_wheel" call: ["--python-tag", pyth

Re: [Distutils] PEP 517 again

This appears to be related to https://github.com/pypa/pip/pull/3225, where pip would generate overbroad wheel names in the cache. So to proceed on this issue, setuptools will assume that generated wheels are valid for the specific python implementation only (eg. wheels generated for CPython will no

Re: [Distutils] PEP 517 again

On 31 August 2017 at 14:22, xoviat wrote: > This appears to be related to https://github.com/pypa/pip/pull/3225, where > pip would generate overbroad wheel names in the cache. So to proceed on this > issue, setuptools will assume that generated wheels are valid for the > specific python implementa

Re: [Distutils] PEP 517 again

On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan wrote: > On 31 August 2017 at 14:22, xoviat wrote: >> Again, let me repeat that: wheels generated using setuptools are valid for >> CPython only if build on CPython. This is not the current setuptools >> behavior but will be for all setuptools build b

Re: [Distutils] PEP 517 again

Will do. Better to hash this out sooner than later. On Aug 31, 2017 12:36 AM, "Nathaniel Smith" wrote: > On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan wrote: > > On 31 August 2017 at 14:22, xoviat wrote: > >> Again, let me repeat that: wheels generated using setuptools are valid > for > >> CPy

Re: [Distutils] PEP 517 again

On 31 August 2017 at 06:36, Nathaniel Smith wrote: > On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan wrote: >> On 31 August 2017 at 14:22, xoviat wrote: >>> Again, let me repeat that: wheels generated using setuptools are valid for >>> CPython only if build on CPython. This is not the current setu

Re: [Distutils] PEP 517 again

> that neither pip nor the setuptools backend should not change the tags > it applies to wheels by default). I'm a bit confused -- are we talking about the backwards compatible path to the future -- or the end-game? In short -- I'm sure we'll have to do some hacky stuff to keep backwards compatib

Re: [Distutils] PEP 517 again

On 31 August 2017 at 16:41, Chris Barker - NOAA Federal wrote: > As to using pip to build wheels -- there is good reason to do that > now, but in s post PEP 517 world, one would call the build system > directly to build a wheel-- after all, all pip should be doing is > calling the build system an

Re: [Distutils] PEP 517 again

> On Aug 31, 2017, at 11:41 AM, Chris Barker - NOAA Federal > wrote: > >> that neither pip nor the setuptools backend should not change the tags >> it applies to wheels by default). > > I'm a bit confused -- are we talking about the backwards compatible > path to the future -- or the end-game?

Re: [Distutils] PEP 517 again

I'm not against renaming a wheel to adjust the tags, but it is a strange caching strategy. wheel is due for a 'retag' feature that does a more rigorous job of changing tags. The build system should name its wheels correctly. Normally the pyNN tag would be used for a pure distribution and the cpNN

Re: [Distutils] PEP 517 again

On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore wrote: > > As to using pip to build wheels -- there is good reason to do that > > now, but in s post PEP 517 world, one would call the build system > > directly to build a wheel-- after all, all pip should be doing is > > calling the build system anywa

Re: [Distutils] PEP 517 again

On Thu, Aug 31, 2017 at 9:04 AM, Donald Stufft wrote: > I'm a bit confused -- are we talking about the backwards compatible > path to the future -- or the end-game? > > > Pip purposely overrides what setuptools tags the wheel with in order to > make it extremely specific to the Python it is curre

Re: [Distutils] PEP 517 again

> On Aug 31, 2017, at 12:58 PM, Chris Barker wrote: > > In fact, I see PEP 517 as a step toward more wheels, and fewer sdists for > distribution. Everything should ideally still have a sdist. — Donald Stufft ___ Distutils-SIG maillist - Distut

Re: [Distutils] PEP 517 again

On 31 August 2017 at 17:50, Chris Barker wrote: >> So I still expect pip wheel to be useful in a post-PEP 517 world. > > Maybe so -- but all pip should be doing is passing off the job to the > back-end. It also handles locating the appropriate version on PyPI, downloading (including caching the d

Re: [Distutils] PEP 517 again

On Thu, Aug 31, 2017 at 12:58 PM Chris Barker wrote: > On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore wrote: > >> > As to using pip to build wheels -- there is good reason to do that >> > now, but in s post PEP 517 world, one would call the build system >> > directly to build a wheel-- after all, a

Re: [Distutils] PEP 517 again

On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal wrote: > The package manager should manage the package, not built it, or change it. > > Surely the build system should know how to correctly name the wheel it builds. It's probably worth mentioning the specific problem that motivated pi

Re: [Distutils] PEP 517 again

Don't be afraid of extras. They have not been PEP'd but are required for a useful Python packaging system. install_requires is just a shorthand for extras with the "" empty string as their name. Too bad we couldn't agree on ; vs : as the delimiter though. On Thu, Aug 31, 2017 at 2:04 PM Nathaniel

Re: [Distutils] PEP 517 again

There are two points that I would note here: 1. setuptools handling of environment markers in wheels was actually broken until recently. 2. Even now, people still prefer runtime detection of environment markers to work around pip's upgrade behavior and old setuptools versions. There needs to be a p

Re: [Distutils] PEP 517 again

On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith wrote: > > Surely the build system should know how to correctly name the wheel it > builds. > > It's probably worth mentioning the specific problem that motivated pip > to start doing this. > > It used to be standard, and is still quite common, fo

Re: [Distutils] PEP 517 again

I don't buy the argument that pip needs to keep separate caches for different implementations. According to the wheel specification, a wheel is valid for the Python implementations that it specifies. This is therefore a blatant violation by setuptools of an existing specification that, in theory, s

Re: [Distutils] PEP 517 again

One thing to keep in mind is that there are quite a few projects on pypa with pure python source distributions uploaded that will not be updated and people may still desire to use. We want pip to be able to still build and install them. That is the challenge! But the motivating use case here was

Re: [Distutils] PEP 517 again

On 1 September 2017 at 07:29, Chris Barker wrote: > But I do think we should be clear about what is a hack for backward > compatibility, and what is part of the designed functionality. Right, and I think it's pretty clear that the problem xoviat raised is with the way pip's implicit local artifac

Re: [Distutils] PEP 517 again

> On Sep 1, 2017, at 12:23 AM, Nick Coghlan wrote: > > 1. Wheel download cache (if this gets a hit, don't even check the package > repo) > 2. Check the package repo for a wheel file (and if you find one, add > it to the download cache) > 3. Local build cache (make this interpreter implementatio

Re: [Distutils] PEP 517 again

On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan wrote: since it > doesn't reliably distinguish between "this cached wheel was downloaded > from a repository" and "this wheel was generated locally with a > particular version of Python". It shouldn't have to. sigh. > PEP 517 deliberately doesn't l

Re: [Distutils] PEP 517 again

I don't think this is a big deal. Compatibility is a guess, wheel tags do not and are not intended to guarantee compatibility especially when the wheel was built locally for caching purposes and not intentionally published. Anything can happen during a build. PEP 517 won't change this. So right no

Re: [Distutils] PEP 517 again

On Fri, Sep 1, 2017 at 11:30 AM, Chris Barker wrote: > I'm still confused -- if setuptools ( invoked by pip) is producing > incorrectly named wheels -- surely that's a bug-fix/workaround that should > go into setuptools? > > If the build is being run by pip, then doesn't setuptools have all the i

Re: [Distutils] PEP 517 again

On 1 September 2017 at 19:30, Chris Barker wrote: > I'm still confused -- if setuptools ( invoked by pip) is producing > incorrectly named wheels -- surely that's a bug-fix/workaround that should > go into setuptools? > > If the build is being run by pip, then doesn't setuptools have all the info

Re: [Distutils] PEP 517 again

> On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: > > On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan > wrote: > > since it > doesn't reliably distinguish between "this cached wheel was downloaded > from a repository" and "this wheel was generated locally with a > parti

Re: [Distutils] PEP 517 again

One more issue that has come up is that "--no-user-cfg" seems to be passed to the egg_info invocation if the "isolated" parameter is enabled. I don't understand what this does, but it is again not defined in the PEP 517 interface. Should we always pass this parameter or should we never pass it? 20

Re: [Distutils] PEP 517 again

Donald, This was your work in https://github.com/pypa/pip/pull/2169. Unfortunately the comments were quite sparse. 2017-09-02 18:25 GMT-05:00 xoviat : > One more issue that has come up is that "--no-user-cfg" seems to be passed > to the egg_info invocation if the "isolated" parameter is enabled

Re: [Distutils] PEP 517 again

Whatever it was, removing it seems to have had no effect on the tests. I will remove it unless someone has an objection. 2017-09-02 18:26 GMT-05:00 xoviat : > Donald, > > > This was your work in https://github.com/pypa/pip/pull/2169. > Unfortunately the comments were quite sparse. > > 2017-09-02

Re: [Distutils] PEP 517 again

On Sat, Sep 2, 2017 at 5:17 PM xoviat wrote: > Whatever it was, removing it seems to have had no effect on the tests. I > will remove it unless someone has an objection. > Just FYI, I wouldn't take the tests still passing as a major signal. I've noticed there are even common code paths / functio

Re: [Distutils] PEP 517 again

Just an update for everyone here: 1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much mo

Re: [Distutils] PEP 517 again

Great discussions on this PEP over the proposal, here and in the archives. I've just now gotten plugged in to the distutils distro, at a seemingly perfect time... somewhat new to Python packaging and still learning. Many of the issues observed by users in the mail chain are reasons why we had m

Re: [Distutils] PEP 517 again

On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: > Just an update for everyone here: > > 1. We're currently waiting on the implementation of the 'dist_info" command > in the wheel project. > 2. Once that is done we can switch pip over to reading dist-info rather than > egg_info. > 3. Then we can mov

Re: [Distutils] PEP 517 again

On 3 September 2017 at 05:42, Donald Stufft wrote: > On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: >> Do the Linux distros use pip to build their packages? > > Not that I am aware of. Fedora's build macros for Python projects currently rely on running setup.py directly, but we've been consider

Re: [Distutils] PEP 517 again

Wheel has always implemented dist info by converting from egg info. It is necessary to invoke all of the egg info writers provided by setuptools or tons of packages will break. But so far dist info has only been generated as part of building a wheel. On Sun, Sep 3, 2017, 22:01 Nathaniel Smith wro

Re: [Distutils] PEP 517 again

Nathaniel: Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that. 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :

Re: [Distutils] PEP 517 again

Also if someone with pip write access could please discuss and hopefully merge my initial PR on pip, I would very much appreciate it. Paul seems to be short on time. 2017-09-04 19:09 GMT-05:00 xoviat : > Nathaniel: > > Pip requires egg_info to discover dependencies of source distributions so > th

Re: [Distutils] PEP 517 again

On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: > Nathaniel: > > Pip requires egg_info to discover dependencies of source distributions so > that it can build wheels all at once after downloading the requirements. I > need to move pip off of egg_info as soon as possible and dist_info is > required t

Re: [Distutils] PEP 517 again

> The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. Yes. That is absolutely correct. > But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. The simplest

Re: [Distutils] PEP 517 again

In any case, we're going to need this for prepare_metadata, so the question you should ask is: what are the reasons for *not* merging this? I haven't heard any so far but that doesn't mean that they don't exist. If there are none, then I don't see why we cannot merge my wheel PR and do a release.

Re: [Distutils] PEP 517 again

On Mon, Sep 4, 2017 at 6:08 PM, xoviat wrote: > In any case, we're going to need this for prepare_metadata, so the question > you should ask is: what are the reasons for *not* merging this? I haven't > heard any so far but that doesn't mean that they don't exist. If there are > none, then I don't

Re: [Distutils] PEP 517 again

On Mon, Sep 4, 2017 at 5:51 PM, xoviat wrote: >> The only reason I can think of that setuptools would need a dist_info > command would be to implement the PEP 517 prepare_wheel_metadata hook. > > Yes. That is absolutely correct. > >> But this hook is optional and in fact provides no value right no

Re: [Distutils] PEP 517 again

The PR that I am taking about is not for pip but for the wheel project. On Sep 4, 2017 8:19 PM, "Chris Jerdonek" wrote: > On Mon, Sep 4, 2017 at 6:08 PM, xoviat wrote: > > In any case, we're going to need this for prepare_metadata, so the > question > > you should ask is: what are the reasons f

Re: [Distutils] PEP 517 again

On Mon, Sep 4, 2017 at 6:41 PM, xoviat wrote: > The PR that I am taking about is not for pip but for the wheel project. Okay, well you started the thread asking something similar for your pip PR (see below). I'm sure similar considerations hold for the wheel project. On Mon, Sep 4, 2017 at 5:11

Re: [Distutils] PEP 517 again

Supposedly there is some meeting tomorrow concerning the wheel project that will determine the fate of dist_info. So that is why I bought it up. On Sep 4, 2017 9:00 PM, "Chris Jerdonek" wrote: > On Mon, Sep 4, 2017 at 6:41 PM, xoviat wrote: > > The PR that I am taking about is not for pip but f

Re: [Distutils] PEP 517 again

On Mon, Sep 4, 2017 at 6:00 AM, Nick Coghlan wrote: > >> Do the Linux distros use pip to build their packages? > > > > Not that I am aware of. > > Fedora's build macros for Python projects currently rely on running > setup.py directly, but we've been considering switching to pip instead > since 2

<    1   2