Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

Nick:

That's generally a good idea, but one significant problem that can occur is
that the Python import system will cache certain libraries, people will run
"pip install," and then they will expect such libraries to be available. I
don't even know exactly how the caching for the import system works, so I
don't want to go and make claims about it that may be incorrect (maybe you
do?). However, it is important to keep that in mind when considering an API.

2017-10-21 6:15 GMT-05:00 Nick Coghlan :

> On 21 October 2017 at 20:03, Paul Moore  wrote:
>
>> Likely the biggest problems will be for people who call into the pip
>> resolver and build APIs, as I don't know of any alternatives out
>> there. But they were *definitely* breaking anyway, as we've made major
>> changes to that code (and will be making more).
>>
>> Also, I should note that we didn't take this decision lightly. We
>> don't have any particular objection in principle to having a supported
>> stable pip API, it's just that we don't have anything even close to
>> the resources needed to define a supported API, maintain it with
>> acceptable backward compatibility guarantees, and support users who
>> will inevitably be using it in unexpected and creative ways (we don't
>> even have the resources to support the limited use of pip's internals
>> that we see today). Also, pip was never designed originally as a
>> library, so we *would* have to design that API from scratch. As I
>> alluded to above, the existing internals code makes some strong
>> assumptions about how it's called - assumptions that would be
>> unacceptable in library code, but are fine in an application's
>> internal code.
>>
>
> (Note: this is entirely speculative, and I have no idea how hard it would
> be, so please read it as the question it's intended to be)
>
> Do you know if there any key APIs (like installation) that could be turned
> into wrappers around pip CLI calls in order to mitigate some of the impact?
>
> The reason I ask is because it's unlikely anyone else is going to
> understand how to emulate the previous functionality better than the pip
> devs would, and if there's an API for those particular invocations, than
> they can be covered directly by pip's test suite.
>
> Plus if there are previous API capabilities that *can't* currently be
> emulated via the CLI, then the pip devs are the folks in the best position
> to add the necessary CLI enhancements (such as the ones Noah asked about
> for doing a more selective `pip list`).
>
> If that's an approach you might be amenable to, then a 10.0 pre-release
> could be a good time to solicit PRs from folks that were using particular
> APIs and would be prepared to invest time in defining comparable CLI
> wrappers for them.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

The short answer is that someone complained about the pip API not doing
what they wanted it to do, so we broke the API to make it clear to everyone
that the API is not something that should be depended on.

2017-10-20 17:53 GMT-05:00 Richard Jones :

> Hiya Paul,
>
> There's a bunch of tooling out there using pip's internals to extending
> pip's functionality. Could you please provide a some reasoning as to why
> they're all going to be broken at pip 10, and possibly some guidance on how
> to get that functionality back?
>
>
> Cheers,
>
>  Richard
>
>
> On 21 October 2017 at 00:22, Paul Moore  wrote:
>
>> We're in the process of starting to plan for a release of pip (the
>> long-awaited pip 10). We're likely still a month or two away from a
>> release, but now is the time for people to start ensuring that
>> everything works for them. One key change in the new version will be
>> that all of the internal APIs of pip will no longer be available, so
>> any code that currently calls functions in the "pip" namespace will
>> break. Calling pip's internal APIs has never been supported, and
>> always carried a risk of such breakage, so projects doing so should,
>> in theory, be prepared for such things. However, reality is not always
>> that simple, and we are aware that people will need time to deal with
>> the implications.
>>
>> Just in case it's not clear, simply finding where the internal APIs
>> have moved to and calling them under the new names is *not* what
>> people should do. We can't stop people calling the internal APIs,
>> obviously, but the idea of this change is to give people the incentive
>> to find a supported approach, not just to annoy people who are doing
>> things we don't want them to ;-)
>>
>> So please - if you're calling pip's internals in your code, take the
>> opportunity *now* to check out the in-development version of pip, and
>> ensure your project will still work when pip 10 is released.
>>
>> And many thanks to anyone else who helps by testing out the new
>> version, as well :-)
>>
>> Thanks,
>> Paul
>>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

The "installing collected packages" output is not always correct. The
"successfully installed [package list]" is always correct
(see pypa/pip/issues/4724 for an example).

I understand your requirements, but is the practical implication that pip
should offer a command that outputs packages that are potentially
incorrect? What happens when someone relies on that and then files a bug
report?

2017-10-20 13:48 GMT-05:00 Noah Kantrowitz <n...@coderanger.net>:

> While I understand that pip itself has to be very careful about edge cases
> and all the pathological things you can do in setup.py, as a higher-level
> tooling author my priorities are on the happy path UX and speed is a big
> factor there. So yes, using PackageFinder is potentially inaccurate, but
> it's also _usually_ accurate :) Anyways, if there is true concern that
> finder-based approaches are too risky, probably don't offer it in the pip
> list output.
>
> --Noah
>
> > On Oct 20, 2017, at 11:43 AM, xoviat <xov...@gmail.com> wrote:
> >
> > A correct dry-run implementation will do about the same amount of work
> as installing to a temporary directory right now. In the future, that could
> be optimized, but any patch to the finder doesn't actually detect the
> requirements correctly (as they're not necessarily known until after the
> wheels are built).
> >
> > 2017-10-20 13:41 GMT-05:00 Noah Kantrowitz <n...@coderanger.net>:
> > Installing to a temp dir is really not an option for automated tooling
> (if nothing else, it takes way too long). `pip list --outdated` does
> already get fairly close to this (and doesn't install anything I suspect
> you can actually get a lot closer than you think) but it calculates for all
> packages (read: is slow) and doesn't give a good way to restrict things
> (hence that hack-y script which is a modified version of the pip list
> code). This is 100% a hard requirement for config management systems and if
> not fixed in pip, will require continued use of internal APIs. I would
> recommend just making pip list take a set of install-compatible
> names/version patterns and apply that as a filter in a similar way to what
> I've done there.
> >
> > --Noah
> >
> > > On Oct 20, 2017, at 11:35 AM, xoviat <xov...@gmail.com> wrote:
> > >
> > > There's no dry-run functionality that I know of so far. However, you
> could use the following:
> > >
> > > pip install --prefix=tmpdir
> > >
> > > This command is actually about the same speed as a proper
> implementation, because we can't actually know what we're installing until
> we build the requirements.
> > >
> > > 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz <n...@coderanger.net>:
> > > So as someone on the tooling side, is there any kind of install
> dry-run yet? I've got https://github.com/poise/
> poise-python/blob/master/lib/poise_python/resources/python_
> package.rb#L34-L78 which touches a tn of internals. Basically I need
> a way to know exactly what versions `pip install` would have used in a
> given situation without actually changing the system. Happy for a better
> solution!
> > >
> > > --Noah
> > >
> > > > On Oct 20, 2017, at 6:22 AM, Paul Moore <p.f.mo...@gmail.com> wrote:
> > > >
> > > > We're in the process of starting to plan for a release of pip (the
> > > > long-awaited pip 10). We're likely still a month or two away from a
> > > > release, but now is the time for people to start ensuring that
> > > > everything works for them. One key change in the new version will be
> > > > that all of the internal APIs of pip will no longer be available, so
> > > > any code that currently calls functions in the "pip" namespace will
> > > > break. Calling pip's internal APIs has never been supported, and
> > > > always carried a risk of such breakage, so projects doing so should,
> > > > in theory, be prepared for such things. However, reality is not
> always
> > > > that simple, and we are aware that people will need time to deal with
> > > > the implications.
> > > >
> > > > Just in case it's not clear, simply finding where the internal APIs
> > > > have moved to and calling them under the new names is *not* what
> > > > people should do. We can't stop people calling the internal APIs,
> > > > obviously, but the idea of this change is to give people the
> incentive
> > > > to find a supported approach, not just to annoy people who are doing
> > > > things 

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

A correct dry-run implementation will do about the same amount of work as
installing to a temporary directory right now. In the future, that could be
optimized, but any patch to the finder doesn't actually detect the
requirements correctly (as they're not necessarily known until after the
wheels are built).

2017-10-20 13:41 GMT-05:00 Noah Kantrowitz <n...@coderanger.net>:

> Installing to a temp dir is really not an option for automated tooling (if
> nothing else, it takes way too long). `pip list --outdated` does already
> get fairly close to this (and doesn't install anything I suspect you can
> actually get a lot closer than you think) but it calculates for all
> packages (read: is slow) and doesn't give a good way to restrict things
> (hence that hack-y script which is a modified version of the pip list
> code). This is 100% a hard requirement for config management systems and if
> not fixed in pip, will require continued use of internal APIs. I would
> recommend just making pip list take a set of install-compatible
> names/version patterns and apply that as a filter in a similar way to what
> I've done there.
>
> --Noah
>
> > On Oct 20, 2017, at 11:35 AM, xoviat <xov...@gmail.com> wrote:
> >
> > There's no dry-run functionality that I know of so far. However, you
> could use the following:
> >
> > pip install --prefix=tmpdir
> >
> > This command is actually about the same speed as a proper
> implementation, because we can't actually know what we're installing until
> we build the requirements.
> >
> > 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz <n...@coderanger.net>:
> > So as someone on the tooling side, is there any kind of install dry-run
> yet? I've got https://github.com/poise/poise-python/blob/master/lib/
> poise_python/resources/python_package.rb#L34-L78 which touches a tn
> of internals. Basically I need a way to know exactly what versions `pip
> install` would have used in a given situation without actually changing the
> system. Happy for a better solution!
> >
> > --Noah
> >
> > > On Oct 20, 2017, at 6:22 AM, Paul Moore <p.f.mo...@gmail.com> wrote:
> > >
> > > We're in the process of starting to plan for a release of pip (the
> > > long-awaited pip 10). We're likely still a month or two away from a
> > > release, but now is the time for people to start ensuring that
> > > everything works for them. One key change in the new version will be
> > > that all of the internal APIs of pip will no longer be available, so
> > > any code that currently calls functions in the "pip" namespace will
> > > break. Calling pip's internal APIs has never been supported, and
> > > always carried a risk of such breakage, so projects doing so should,
> > > in theory, be prepared for such things. However, reality is not always
> > > that simple, and we are aware that people will need time to deal with
> > > the implications.
> > >
> > > Just in case it's not clear, simply finding where the internal APIs
> > > have moved to and calling them under the new names is *not* what
> > > people should do. We can't stop people calling the internal APIs,
> > > obviously, but the idea of this change is to give people the incentive
> > > to find a supported approach, not just to annoy people who are doing
> > > things we don't want them to ;-)
> > >
> > > So please - if you're calling pip's internals in your code, take the
> > > opportunity *now* to check out the in-development version of pip, and
> > > ensure your project will still work when pip 10 is released.
> > >
> > > And many thanks to anyone else who helps by testing out the new
> > > version, as well :-)
> > >
> > > Thanks,
> > > Paul
> > > ___
> > > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

There's no dry-run functionality that I know of so far. However, you could
use the following:

pip install --prefix=tmpdir

This command is actually about the same speed as a proper implementation,
because we can't actually know what we're installing until we build the
requirements.

2017-10-20 12:42 GMT-05:00 Noah Kantrowitz :

> So as someone on the tooling side, is there any kind of install dry-run
> yet? I've got https://github.com/poise/poise-python/blob/master/lib/
> poise_python/resources/python_package.rb#L34-L78 which touches a tn
> of internals. Basically I need a way to know exactly what versions `pip
> install` would have used in a given situation without actually changing the
> system. Happy for a better solution!
>
> --Noah
>
> > On Oct 20, 2017, at 6:22 AM, Paul Moore  wrote:
> >
> > We're in the process of starting to plan for a release of pip (the
> > long-awaited pip 10). We're likely still a month or two away from a
> > release, but now is the time for people to start ensuring that
> > everything works for them. One key change in the new version will be
> > that all of the internal APIs of pip will no longer be available, so
> > any code that currently calls functions in the "pip" namespace will
> > break. Calling pip's internal APIs has never been supported, and
> > always carried a risk of such breakage, so projects doing so should,
> > in theory, be prepared for such things. However, reality is not always
> > that simple, and we are aware that people will need time to deal with
> > the implications.
> >
> > Just in case it's not clear, simply finding where the internal APIs
> > have moved to and calling them under the new names is *not* what
> > people should do. We can't stop people calling the internal APIs,
> > obviously, but the idea of this change is to give people the incentive
> > to find a supported approach, not just to annoy people who are doing
> > things we don't want them to ;-)
> >
> > So please - if you're calling pip's internals in your code, take the
> > opportunity *now* to check out the in-development version of pip, and
> > ensure your project will still work when pip 10 is released.
> >
> > And many thanks to anyone else who helps by testing out the new
> > version, as well :-)
> >
> > Thanks,
> > Paul
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

+1 on pre-release wheels. I've seen the process in action
with SciPy, and it helped to catch at least one significant bug.

2017-10-20 8:30 GMT-05:00 Paul Moore :

> On 20 October 2017 at 14:26, Matthew Brett 
> wrote:
> > Thanks for the heads-up.
> >
> > Will y'all be doing a PyPI pre-release so we can test with `pip
> > install --pre -U pip`?
>
> We've not yet decided on that. Traditionally I don't think we have
> done so, but I'm inclined to think it's a good idea. It might not be
> until noticeably closer to the release, though...
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Claim/remove pypi package named sc0ns

[xoviat]

If I understand correctly, the guidelines say that its possible to
remove/take over a package if the owner doesn't dispute it. If the owner
disputes it, then I don't think it's possible. Please let me know whether
this assumption is correct.

2017-10-10 10:35 GMT-05:00 Bill Deegan :

> Greetings,
>
> There's a package named sc0ns which is an ancient packaging of a SCons
> release:
>
> https://pypi.python.org/pypi/sc0ns
>
> vs.
>
> https://pypi.python.org/pypi/SCons
>
> Is it possible to have the sc0ns package removed/hidden?
> And/or is there a way to contact that package owner to request same?
>
> Thanks,
> Bill Deegan
> SCons Project Co-Manager
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Claiming Ownership of Existing Package

[xoviat]

All:

I would like to claim ownership of the "flint" package on pypi so that I
can upload a new package with the same name. The existing project page
states that it is abandoned.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

After thinking again about that possibilities that we've discussed here, I
realized that a previously proposed alternative would eliminate external
build-time dependencies and allow us to merge setuptools with distutils: an
"ensuresetuptools" module. This was proposed by @zooba, but basically the
idea would be to bundle a wheel of setuptools (setuptools is
py2.py3.none-any) that Python could install without requiring network
access or other modules. If distutils is required during the build process,
then this idea should conform to all of the requirements proposed here and
take distutils off of the CPython release schedule.

Although it may seem more complicated at first, once implemented,
maintenance from the CPython side would be minimal (a bot could update the
wheel, although I'm not sure whether this is done with ensurepip), there is
precedent for this (with ensurepip), end-users could continue to use
distutils without any modifications to their scripts (ensuresetuptools
could be run during the installation process, and even if it isn't, then a
single command line could install distutils/setuptools), and it would allow
simplified setuptools maintenance (no monkeypatching).

2017-09-30 20:14 GMT-05:00 Donald Stufft <don...@stufft.io>:

> I think that the CPython builds a python executable, then uses that built
> executable to finish the installation.
>
>
> On Sep 30, 2017, at 9:11 PM, xoviat <xov...@gmail.com> wrote:
>
> It would be nice to know whether this information is correct, or whether I
> hold an invalid belief.
>
> 2017-09-30 20:09 GMT-05:00 xoviat <xov...@gmail.com>:
>
>> I have personally not built Python myself (though I've built many an
>> extension), but what I can say is that I got the idea from Larry Hastings.
>> According to him (this if for the Gilectomy fork):
>>
>> "Second, as you hack on the Gilectomy you may break your "python"
>> executable rather badly. This is of course expected. However, the python
>> Makefile itself depends on having a working local python interpreter, so
>> when you break that you often break your build too."
>>
>> 2017-09-30 19:59 GMT-05:00 Donald Stufft <don...@stufft.io>:
>>
>>>
>>>
>>> On Sep 30, 2017, at 3:52 PM, xoviat <xov...@gmail.com> wrote:
>>>
>>> I don't think CPython needs to bundle all of its build-time
>>> dependencies. That principle doesn't really apply to other Python programs
>>> nor most other programs in general. AFAIK, CPython already has a build-time
>>> dependency on another, external, Python, so it wouldn't be too much to
>>> require the external Python to have setuptools installed with something
>>> like pyproject.toml (other programming languages usually bootstrap
>>> themselves with previous versions of the language along with some
>>> associated build tools).
>>>
>>>
>>> As far as I can tell, CPython does *not* have a build time dependency on
>>> having Python available. I just spun up a bare alpine linux container and
>>> compiled CPython `master` branch on it. As far as I can tell the only
>>> Python that exists in this container is the one I just compiled.
>>>
>>> That means that in order for CPython to depend on distutils to build as
>>> you indicate, it would also need to start depending on an existing version
>>> of Python being available. I don’t think that’s a great idea. I think
>>> Python should not depend on Python to build.
>>>
>>
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

It would be nice to know whether this information is correct, or whether I
hold an invalid belief.

2017-09-30 20:09 GMT-05:00 xoviat <xov...@gmail.com>:

> I have personally not built Python myself (though I've built many an
> extension), but what I can say is that I got the idea from Larry Hastings.
> According to him (this if for the Gilectomy fork):
>
> "Second, as you hack on the Gilectomy you may break your "python"
> executable rather badly. This is of course expected. However, the python
> Makefile itself depends on having a working local python interpreter, so
> when you break that you often break your build too."
>
> 2017-09-30 19:59 GMT-05:00 Donald Stufft <don...@stufft.io>:
>
>>
>>
>> On Sep 30, 2017, at 3:52 PM, xoviat <xov...@gmail.com> wrote:
>>
>> I don't think CPython needs to bundle all of its build-time dependencies.
>> That principle doesn't really apply to other Python programs nor most other
>> programs in general. AFAIK, CPython already has a build-time dependency on
>> another, external, Python, so it wouldn't be too much to require the
>> external Python to have setuptools installed with something like
>> pyproject.toml (other programming languages usually bootstrap themselves
>> with previous versions of the language along with some associated build
>> tools).
>>
>>
>> As far as I can tell, CPython does *not* have a build time dependency on
>> having Python available. I just spun up a bare alpine linux container and
>> compiled CPython `master` branch on it. As far as I can tell the only
>> Python that exists in this container is the one I just compiled.
>>
>> That means that in order for CPython to depend on distutils to build as
>> you indicate, it would also need to start depending on an existing version
>> of Python being available. I don’t think that’s a great idea. I think
>> Python should not depend on Python to build.
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

I have personally not built Python myself (though I've built many an
extension), but what I can say is that I got the idea from Larry Hastings.
According to him (this if for the Gilectomy fork):

"Second, as you hack on the Gilectomy you may break your "python"
executable rather badly. This is of course expected. However, the python
Makefile itself depends on having a working local python interpreter, so
when you break that you often break your build too."

2017-09-30 19:59 GMT-05:00 Donald Stufft <don...@stufft.io>:

>
>
> On Sep 30, 2017, at 3:52 PM, xoviat <xov...@gmail.com> wrote:
>
> I don't think CPython needs to bundle all of its build-time dependencies.
> That principle doesn't really apply to other Python programs nor most other
> programs in general. AFAIK, CPython already has a build-time dependency on
> another, external, Python, so it wouldn't be too much to require the
> external Python to have setuptools installed with something like
> pyproject.toml (other programming languages usually bootstrap themselves
> with previous versions of the language along with some associated build
> tools).
>
>
> As far as I can tell, CPython does *not* have a build time dependency on
> having Python available. I just spun up a bare alpine linux container and
> compiled CPython `master` branch on it. As far as I can tell the only
> Python that exists in this container is the one I just compiled.
>
> That means that in order for CPython to depend on distutils to build as
> you indicate, it would also need to start depending on an existing version
> of Python being available. I don’t think that’s a great idea. I think
> Python should not depend on Python to build.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

I don't know whether this will convince anyone otherwise, but at least from
my perspective, a build-time dependency is different than a run-time
dependency. Assuming that other parts of CPython don't have a run-time
dependency on distutils, I don't think CPython needs to bundle all of its
build-time dependencies. That principle doesn't really apply to other
Python programs nor most other programs in general. AFAIK, CPython already
has a build-time dependency on another, external, Python, so it wouldn't be
too much to require the external Python to have setuptools installed with
something like pyproject.toml (other programming languages usually
bootstrap themselves with previous versions of the language along with some
associated build tools).

2017-09-29 22:43 GMT-05:00 Donald Stufft <don...@stufft.io>:

>
>
> > On Sep 29, 2017, at 11:21 PM, xoviat <xov...@gmail.com> wrote:
> >
> > I don't think writing shims for distutils is as easy as you make it
> sound. In fact, I'd venture to say that it's an intractable problem because
> of the difficulty of knowing the number of distutils hacks that are in the
> wild.
> >
>
>
> For the compiler code? It’s probably pretty tractable, which is I think
> the main thing that needs maintenance.
>
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

I don't think writing shims for distutils is as easy as you make it sound.
In fact, I'd venture to say that it's an intractable problem because of the
difficulty of knowing the number of distutils hacks that are in the wild.

2017-09-29 22:15 GMT-05:00 Donald Stufft :

>
>
> On Sep 29, 2017, at 5:16 PM, Steve Dower  wrote:
>
> I'm happy enough with this approach, my only problem with it is that I
> don't want to be maintaining two versions (the new one that we want people
> to change to, and the old one so that people keep working with new versions
> of Python without having to change to the new one).
>
> If we don't make a very clear statement that "distutils is no longer being
> maintained, it probably won't build valid extensions on 3.[7/8/whatever],
> and we mean it this time" then it doesn't count. I insist on at least
> removing the distutils tests from the test suite, since as long as those
> need to pass I can never stop maintaining the stdlib version. And that puts
> this in the realm of python-dev, which means a PEP to formally deprecate
> distutils (that I'm happy to co-author) and recommend an alternative, as
> well as the alternative path needed to build the core extension modules on
> non-Windows platforms.
>
>
> As long as CPython itself depends on distutils I don’t think we can break
> or remove distutils, but we can move make it a shim over something that is
> better architecturally and easier to maintain. The tests should still pass
> in this case because they’d essentially be testing that our shim is
> functioning correctly.
>
> I also don’t think we can spin distutils out of the stdlib as long as
> CPython depends on it.
>
> I see two paths forward:
>
> 1) We come up with something better *in the standard library*, implement
> it, move CPython to using that in the build process, and then
> deprecate/remove distutils from the stdlib and have it maintained outside.
>
> 2) We come up with something better outside of the standard library,
> implement it, vendor it into distutils/_whatevercoollib, and turn the
> relevant distutils APIs into shims over the new API.
>
>
> Of the two, I think that (2) is going to be the least painful for users.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

Distutils has many problems as you said, but by direct implication, if
setuptools is "intimately tied" to distutils, then setuptools has these
problems as well (and I think that it does).

> I don’t really have a good answer for that. Probably it’s best to keep
setuptools moving along as it’s doing now without making major
architectural changes to it.

The answer (I think) is clear. Setuptools should take over the distutils
namespace, even on older Python versions, and even when there is already a
distutils in the standard library. If setuptools is imported first, then it
can rewrite the import paths such that the setuptools version of distutils
is used.

This would extract and consolidate what has become a de-facto single
project (distutils +  setuptools) into one repository, reducing duplication
of effort.

2017-09-29 15:06 GMT-05:00 Donald Stufft :

>
>
> On Sep 29, 2017, at 2:31 AM, Nick Coghlan  wrote:
>
> This is one of those changes where the longer term process improvement
> benefits are already reasonably clear to the folks involved in
> maintaining the software (i.e. we think it will provide a better end
> user experience overall if distutils switches to being updated and
> versioned based on the setuptools release cycle rather than the
> reference interpreter & standard library one), so the main thing the
> PEP process will need to ensure is that we're providing a sufficiently
> non-disruptive transition plan for getting from the current state to
> the more desirable state.
>
>
> I’m not sure that it’s going to be helpful to take distutils out of the
> standard library. It has many problems, and being in the standard library
> is one of them sure, but the more insidious problems are inherent in it’s
> entire design which we can’t really fix without breaking all of the things.
> The main blocker to improvement is primarily just that after two decades of
> existence tons of random setup.py scripts out there have reached in and
> messed with sometimes even super internal parts of distutils.
>
> There are bits of distutils that are super useful (Steve indicated the
> compiler support as one of them), and I think a better path forward is to
> do like we’ve done with the packaging library. Make a compiler package that
> can handle those bits and recommend that projects that want to handle
> compilation just reuse them (of course not mandate it!). Then we can focus
> first on creating a good API in that package without inheriting the
> problems of distutils, and then we can maybe work on bundling that inside
> of distutils and turn the distutils API into a shim over that (or just
> leaving it alone).
>
> This leaves the question of what happens to setuptools since it is
> intimately tied to distutils. I don’t really have a good answer for that.
> Probably it’s best to keep setuptools moving along as it’s doing now
> without making major architectural changes to it. It inherited a lot of the
> problems of distutils where people reach in and muck around with it’s
> internals so it’s easy to break things if you make too many changes.
>
> Therefore we might be better to just mostly leave distutils/setuptools
> progressing as they are today and focus on a brighter future in the
> greenfield landscape of PEP 517.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

I understand the motivation behind that PEP, but we're not really
deprecating distutils, just moving it out into setuptools. The goal should
be "not to break anyone's code."

2017-09-29 11:25 GMT-05:00 Brett Cannon <br...@python.org>:

> Actually, you can't remove any module in the stdlib that exists in both
> Python 2.7 and 3.5 until after Python 2.7 is no longer supported:
> https://www.python.org/dev/peps/pep-0004/#for-modules-
> existing-in-both-python-2-7-and-python-3-5 .
>
> On Thu, 28 Sep 2017 at 23:31 Nick Coghlan <ncogh...@gmail.com> wrote:
>
>> On 29 September 2017 at 07:42, Matthias Klose <d...@ubuntu.com> wrote:
>> > On 27.09.2017 11:37, Nick Coghlan wrote:
>> >> On 27 September 2017 at 12:30, xoviat <xov...@gmail.com> wrote:
>> >>>> In short, I think your proposal is a good one, but how can we
>> allocate
>> >>>> manpower?
>> >>>
>> >>> (issue31595 on bugs.python.org)
>> >>>
>> >>> So what do others think of this? My sense of things is that people
>> are open
>> >>> to the idea, but there isn't a plan to make it happen.
>> >>
>> >> As long as Jason is amenable to the idea on the setuptools side, and
>> >> pip's trick of injecting setuptools into a process to change the
>> >> behaviour of distutils imports continues to work, then this seems like
>> >> a reasonable idea to me.
>> >>
>> >> There will still be folks that want to patch distutils itself, but I
>> >> don't see that as being substantially different from folks still
>> >> patching the standard library Tcl/Tk bindings, even though there are a
>> >> number of popular non-stdlib GUI toolkits.
>> >
>> > In this case, we should deprecate distutils in python3.7 and remove it
>> in 3.8.
>>
>> Aye, that seems reasonable to me as well (especially if 3.7+ also
>> offers an "ensuresetuptools" module), and, looking at the recent(-ish)
>> commit history at
>> https://github.com/python/cpython/commits/master/Lib/distutils, I'd
>> suggest we ask Steve Dower if he'd be willing to be BDFL-Delegate for
>> the related PEP.
>>
>> This is one of those changes where the longer term process improvement
>> benefits are already reasonably clear to the folks involved in
>> maintaining the software (i.e. we think it will provide a better end
>> user experience overall if distutils switches to being updated and
>> versioned based on the setuptools release cycle rather than the
>> reference interpreter & standard library one), so the main thing the
>> PEP process will need to ensure is that we're providing a sufficiently
>> non-disruptive transition plan for getting from the current state to
>> the more desirable state.
>>
>> Between Steve ("Don't break Windows"), you ("Don't break Debian (et
>> al)"), me ("Don't break Fedora (et al)"), Donald ("Don't break pip"),
>> and Jason ("Don't break setuptools"/"Don't lumber setuptools with an
>> unreasonable ongoing maintenance burden"), along with everyone else on
>> distutils-sig, python-ideas, and python-dev, I think we'll be able to
>> make a reasonable assessment of what counts as "too disruptive".
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

> So how do we get there? I would think a fork and merge of both
setuptools and distutils would make sense. Preserve the building part
of the API, so folks could simply replace the import line an their
setup.py files to use the new tool.

There's no need to even replace the line in setup.py. Setuptools currently
has the full capability to override the distutils namespace when it's
imported first. I think if we extract distutils into setuptools, it's not
difficult to list setuptools in install_requires or otherwise on Python 3.7
and above. And I think it will eventually be better for setuptools because
the monkeypatch can be avoided. This might even solve the bootstrapping
problem.


2017-09-28 11:08 GMT-05:00 Chris Barker - NOAA Federal <
chris.bar...@noaa.gov>:

>  distutils works fine for its original purpose (building components for
>  the system Python in Linux distros),
>
> What does Linux have to with it? In the eagle days, I found it most
> helpful for Windows, actually. And it's very helpful for OS-X as well.
>
> It was also great for pure python bundling and installing.
>
> But regardless of original intent, it's clear that distutils has not
> turned out to be as cleanly extendible as is needed.
>
> Setuptools has its issues as well -- personally it has always driven
> me crazy with its mingling of run-time vs build-time vs install time
> functionality.
>
> I think it would be helpful to be explicit about the final goals
> before talking about whether and how to fork distutils.
>
> What I'd love to see is a build tool that is compatible with the
> "modern" packaging paradigm-- pip, pypi, wheels. ( and other systems
> like rpm, conda, )
>
> Essentially setuptools without the cruft. ( no easy install,
> pkg_resources, ...)
>
> This could be maintained as a third party package, like the current
> setuptools. If/when it becomes ubiquitous, an "ensure" style module
> could be added to the stdlib.
>
> So how do we get there? I would think a fork and merge of both
> setuptools and distutils would make sense. Preserve the building part
> of the API, so folks could simply replace the import line an their
> setup.py files to use the new tool.
>
> Distutils and setuptools remain in maintainable only mode for probably
> a long time :-(
>
> -CHB
>
>
>
>  so we still need to avoid
>  breaking that. setuptools is only essential if you want full support
>  for modern *Python* level packaging features (PEP 376 install
>  metadata, venv compatibility, wheel files, etc), and a lot of Linux
>  system components simply don't worry about those things, and rely on
>  their system level equivalents instead (e.g. the RPM/deb databases,
>  chroots and containers, RPM/deb files)
> 
>  However, what *could* be interesting is a proposal to move distutils
>  to the "ensurepip" model, where rather than maintaining distutils
>  directly as part of CPython, the CPython build process instead runs
>  setuptools/distutils from a bundled wheel file. Doing that would
>  entail having setuptools actually start installing a copy of distutils
>  into site-packages: older CPython releases would ignore it by default
>  (since the stdlib version would shadow it), while 3.7+ would offer
>  either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools"
>  (bikeshed to be painted via the PEP process) to install the
>  setuptools-provided version.
> 
>  I'm not claiming actually doing that would be particularly easy - I
>  just think it's the most viable path to get us away from the current
>  version coupling between the build infrastructure in distutils and the
>  runtime support infrastructure in the rest of the standard library,
>  and to avoid maintaining two distinct copies of distutils indefinitely
>  (one in the stdlib, one in setuptools).
> 
>  That approach wouldn't even entail any *new* bundling at the CPython
>  level, as while it's currently formally an implementation detail
>  (pending potential removal in a post-PEP-517 world), setuptools is
>  already bundled as part of the support infrastructure for ensurepip:
>  https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled
> 
>  Cheers,
>  Nick.
> 
>  --
>  Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>  ___
>  Distutils-SIG maillist  -  Distutils-SIG@python.org
>  https://mail.python.org/mailman/listinfo/distutils-sig
> >>>
> >>>
> >>> ___
> >>> Distutils-SIG maillist  -  Distutils-SIG@python.org
> >>> https://mail.python.org/mailman/listinfo/distutils-sig
> >>>
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> ___
> 

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

That's actually an interesting idea though: for Python 3.7 distutils ->
_distutils (and then setuptools is required for building). For/against?

2017-09-27 14:46 GMT-05:00 xoviat <xov...@gmail.com>:

> No. Setuptools is what projects without a build_backend in pyproject.toml
> get. Not distutils. We should make it clear now that the distutils
> namespace belongs to setuptools except for when building cpython.
>
> On Sep 27, 2017 2:33 PM, "Ned Deily" <n...@python.org> wrote:
>
>> On Sep 27, 2017, at 13:31, Steve Dower <steve.do...@python.org> wrote:
>> > setuptools is totally welcome in my book to simply copy the compiler
>> infrastructure we already have from core and never look back. It really
>> does need to be maintained separately from CPython, especially on Windows
>> where we continue to get innovation in the targeted tools. I know it's a
>> big ask, and it's one that I can't personally commit real time to (though I
>> obviously will as much as possible), but I do think it is necessary for our
>> ecosystem to not be tied to CPython release cycles.
>>
>> Whatever is done, keep in mind that currently distutils is required to
>> build Python itself, e.g. the standard library.  And that at least one
>> important project, numpy, already subclasses distutils.
>>
>> --
>>   Ned Deily
>>   n...@python.org -- []
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

No. Setuptools is what projects without a build_backend in pyproject.toml
get. Not distutils. We should make it clear now that the distutils
namespace belongs to setuptools except for when building cpython.

On Sep 27, 2017 2:33 PM, "Ned Deily"  wrote:

> On Sep 27, 2017, at 13:31, Steve Dower  wrote:
> > setuptools is totally welcome in my book to simply copy the compiler
> infrastructure we already have from core and never look back. It really
> does need to be maintained separately from CPython, especially on Windows
> where we continue to get innovation in the targeted tools. I know it's a
> big ask, and it's one that I can't personally commit real time to (though I
> obviously will as much as possible), but I do think it is necessary for our
> ecosystem to not be tied to CPython release cycles.
>
> Whatever is done, keep in mind that currently distutils is required to
> build Python itself, e.g. the standard library.  And that at least one
> important project, numpy, already subclasses distutils.
>
> --
>   Ned Deily
>   n...@python.org -- []
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

/cc  @jaraco

2017-09-27 11:34 GMT-05:00 xoviat <xov...@gmail.com>:

> So since PEP 517 is going into setuptools anyway, we can
> sys.path.insert(0, NEW_DISTUTILS_LOCATION) and then import distutils in the
> interface before setup.py is loaded. Seems simple enough.
>
> 2017-09-27 11:27 GMT-05:00 Nick Coghlan <ncogh...@gmail.com>:
>
>> On 27 September 2017 at 23:35, Donald Stufft <don...@stufft.io> wrote:
>> > I suspect this might be difficult, but I haven’t actually tried it. Our
>> > trick (and really setup tools itself relies on this too) is that
>> setuptools
>> > monkey patches distutils at runtime to make distutils into setuptools.
>> It
>> > only has to patch a few bits and pieces right now, but I suspect that if
>> > this happened it’d have to monkey patch the entire namespace. It would
>> > probably also break people who mix and match imports from setuptools and
>> > distutils.
>>
>> Right, my assumption was that pip would need to ensure that setuptools
>> took over the distutils namespace before any of the code in setup.py
>> ran, so that any monkeypatches and state modifications in other code
>> would all take place in the setuptools provided copy, and the standard
>> library's copy would never even get loaded.
>>
>> However, I don't think that's a new requirement: I believe you already
>> need to do that, since the first line might be "from distutils import
>> setup".
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[xoviat]

So since PEP 517 is going into setuptools anyway, we can sys.path.insert(0,
NEW_DISTUTILS_LOCATION) and then import distutils in the interface before
setup.py is loaded. Seems simple enough.

2017-09-27 11:27 GMT-05:00 Nick Coghlan :

> On 27 September 2017 at 23:35, Donald Stufft  wrote:
> > I suspect this might be difficult, but I haven’t actually tried it. Our
> > trick (and really setup tools itself relies on this too) is that
> setuptools
> > monkey patches distutils at runtime to make distutils into setuptools. It
> > only has to patch a few bits and pieces right now, but I suspect that if
> > this happened it’d have to monkey patch the entire namespace. It would
> > probably also break people who mix and match imports from setuptools and
> > distutils.
>
> Right, my assumption was that pip would need to ensure that setuptools
> took over the distutils namespace before any of the code in setup.py
> ran, so that any monkeypatches and state modifications in other code
> would all take place in the setuptools provided copy, and the standard
> library's copy would never even get loaded.
>
> However, I don't think that's a new requirement: I believe you already
> need to do that, since the first line might be "from distutils import
> setup".
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Extracting distutils into setuptools

[xoviat]

This was a comment by @zooba (Steve Dower):

> (FWIW, I think it makes *much* more sense for setuptools to fix this by
simply forking all of distutils and never looking back. But since we don't
live in that world yet, it went into distutils.)

And here is my response:

> Since you mention it, I agree with that proposal. But currently we have
core developers contributing to distutils and @jaraco contributing to
setuptools. @jaraco is quite competent, but I doubt that he would be able
to maintain an independent fork of distutils by himself.

> In short, I think your proposal is a good one, but how can we allocate
manpower?

(issue31595 on bugs.python.org)

So what do others think of this? My sense of things is that people are open
to the idea, but there isn't a plan to make it happen.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] string types for paths in PEP 517

[xoviat]

+1

2017-09-05 3:21 GMT-05:00 Paul Moore :

> On 5 September 2017 at 09:00, Thomas Kluyver  wrote:
> > I considered this. It's *potentially* a problem, but I think we should
> > not try to deal with it for now:
> >
> > - Normally, temp files will go in /tmp - so it should be fine to
> > construct paths of entirely ascii characters.
> > - Frontends that want the wheel to end up elsewhere can ask for it in a
> > tmp directory first and then move it, so there's a workaround if it
> > becomes an issue.
> > - We already have workarounds for the commonest case of UTF-8 paths + C
> > locale: ignore the locale and treat paths as UTF-8.
> > - The 'right' way to deal with it on Unix is to make all paths bytes,
> > which would introduce a similar issue on Windows. If paths have to be
> > bytes in some situations and unicode in others, both frontends and
> > backends need extra complexity to handle that.
> > - If your non-ascii username breaks stuff on Python 2... Python 3 is
> > ready to make your life easier.
>
> +1 on this
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Supposedly there is some meeting tomorrow concerning the wheel project that
will determine the fate of dist_info. So that is why I bought it up.

On Sep 4, 2017 9:00 PM, "Chris Jerdonek" <chris.jerdo...@gmail.com> wrote:

> On Mon, Sep 4, 2017 at 6:41 PM, xoviat <xov...@gmail.com> wrote:
> > The PR that I am taking about is not for pip but for the wheel project.
>
> Okay, well you started the thread asking something similar for your
> pip PR (see below). I'm sure similar considerations hold for the wheel
> project.
>
> On Mon, Sep 4, 2017 at 5:11 PM, xoviat <xov...@gmail.com> wrote:
> > Also if someone with pip write access could please discuss and hopefully
> > merge my initial PR on pip, I would very much appreciate it. Paul seems
> to
> > be short on time.
>
> --Chris
>
>
> >
> > On Sep 4, 2017 8:19 PM, "Chris Jerdonek" <chris.jerdo...@gmail.com>
> wrote:
> >>
> >> On Mon, Sep 4, 2017 at 6:08 PM, xoviat <xov...@gmail.com> wrote:
> >> > In any case, we're going to need this for prepare_metadata, so the
> >> > question
> >> > you should ask is: what are the reasons for *not* merging this? I
> >> > haven't
> >> > heard any so far but that doesn't mean that they don't exist. If there
> >> > are
> >> > none, then I don't see why we cannot merge my wheel PR and do a
> release.
> >>
> >> FYI, you're not the only one waiting for PR's to be merged. There are
> >> 56 other PR's, some of which have already been approved by repo
> >> members with commit access. I would try to be a little more patient.
> >> Otherwise, everyone can be emailing the list saying the same thing and
> >> asking why their PR isn't being merged.
> >>
> >> --Chris
> >>
> >> >
> >> > 2017-09-04 19:51 GMT-05:00 xoviat <xov...@gmail.com>:
> >> >>
> >> >> > The only reason I can think of that setuptools would need a
> dist_info
> >> >> command would be to implement the PEP 517 prepare_wheel_metadata
> hook.
> >> >>
> >> >> Yes. That is absolutely correct.
> >> >>
> >> >> > But this hook is optional and in fact provides no value right now,
> so
> >> >> it can't be a blocker for anything.
> >> >>
> >> >> The simplest way to start on this issue is to replace egg_info in pip
> >> >> with
> >> >> prepare_metadata_for_build_wheel. It is absolutely a historical
> >> >> artifact but
> >> >> I need to work on one issue at a time. The next issue will be
> replacing
> >> >> egg_info in pip with prepare_metadata_for_build_wheel.
> >> >>
> >> >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith <n...@pobox.com>:
> >> >>>
> >> >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xov...@gmail.com> wrote:
> >> >>> > Nathaniel:
> >> >>> >
> >> >>> > Pip requires egg_info to discover dependencies of source
> >> >>> > distributions
> >> >>> > so
> >> >>> > that it can build wheels all at once after downloading the
> >> >>> > requirements. I
> >> >>> > need to move pip off of egg_info as soon as possible and dist_info
> >> >>> > is
> >> >>> > required to do that.
> >> >>>
> >> >>> "Requires" is a strong word -- AFAIK this is just a historical
> >> >>> artifact. I don't really know what you're talking about in the
> second
> >> >>> sentence.
> >> >>>
> >> >>> The only reason I can think of that setuptools would need a
> dist_info
> >> >>> command would be to implement the PEP 517 prepare_wheel_metadata
> hook.
> >> >>> But this hook is optional and in fact provides no value right now,
> so
> >> >>> it can't be a blocker for anything. (In fact I can't see any reason
> >> >>> why pip would ever call it before the resolver lands.) So either (a)
> >> >>> there's some other reason you want a dist_info command, (b) there's
> >> >>> some reason I'm missing why prepare_wheel_metadata matters, or (c)
> one
> >> >>> of us is misunderstanding something :-).
> >> >>>
> >> >>> -n
> >> >>>
> >> >>>

Re: [Distutils] PEP 517 again

[xoviat]

The PR that I am taking about is not for pip but for the wheel project.

On Sep 4, 2017 8:19 PM, "Chris Jerdonek" <chris.jerdo...@gmail.com> wrote:

> On Mon, Sep 4, 2017 at 6:08 PM, xoviat <xov...@gmail.com> wrote:
> > In any case, we're going to need this for prepare_metadata, so the
> question
> > you should ask is: what are the reasons for *not* merging this? I haven't
> > heard any so far but that doesn't mean that they don't exist. If there
> are
> > none, then I don't see why we cannot merge my wheel PR and do a release.
>
> FYI, you're not the only one waiting for PR's to be merged. There are
> 56 other PR's, some of which have already been approved by repo
> members with commit access. I would try to be a little more patient.
> Otherwise, everyone can be emailing the list saying the same thing and
> asking why their PR isn't being merged.
>
> --Chris
>
> >
> > 2017-09-04 19:51 GMT-05:00 xoviat <xov...@gmail.com>:
> >>
> >> > The only reason I can think of that setuptools would need a dist_info
> >> command would be to implement the PEP 517 prepare_wheel_metadata hook.
> >>
> >> Yes. That is absolutely correct.
> >>
> >> > But this hook is optional and in fact provides no value right now, so
> >> it can't be a blocker for anything.
> >>
> >> The simplest way to start on this issue is to replace egg_info in pip
> with
> >> prepare_metadata_for_build_wheel. It is absolutely a historical
> artifact but
> >> I need to work on one issue at a time. The next issue will be replacing
> >> egg_info in pip with prepare_metadata_for_build_wheel.
> >>
> >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith <n...@pobox.com>:
> >>>
> >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xov...@gmail.com> wrote:
> >>> > Nathaniel:
> >>> >
> >>> > Pip requires egg_info to discover dependencies of source
> distributions
> >>> > so
> >>> > that it can build wheels all at once after downloading the
> >>> > requirements. I
> >>> > need to move pip off of egg_info as soon as possible and dist_info is
> >>> > required to do that.
> >>>
> >>> "Requires" is a strong word -- AFAIK this is just a historical
> >>> artifact. I don't really know what you're talking about in the second
> >>> sentence.
> >>>
> >>> The only reason I can think of that setuptools would need a dist_info
> >>> command would be to implement the PEP 517 prepare_wheel_metadata hook.
> >>> But this hook is optional and in fact provides no value right now, so
> >>> it can't be a blocker for anything. (In fact I can't see any reason
> >>> why pip would ever call it before the resolver lands.) So either (a)
> >>> there's some other reason you want a dist_info command, (b) there's
> >>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
> >>> of us is misunderstanding something :-).
> >>>
> >>> -n
> >>>
> >>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith <n...@pobox.com>:
> >>> >>
> >>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xov...@gmail.com> wrote:
> >>> >> > Just an update for everyone here:
> >>> >> >
> >>> >> > 1. We're currently waiting on the implementation of the
> 'dist_info"
> >>> >> > command
> >>> >> > in the wheel project.
> >>> >> > 2. Once that is done we can switch pip over to reading dist-info
> >>> >> > rather
> >>> >> > than
> >>> >> > egg_info.
> >>> >> > 3. Then we can move the backend over to setuptools. Because Jacob
> >>> >> > has a
> >>> >> > much
> >>> >> > more efficient release system than pip, I anticipate having a
> >>> >> > release of
> >>> >> > setuptools first and then we can switch pip over to requiring a
> >>> >> > newer
> >>> >> > setuptools via PEP 518.
> >>> >>
> >>> >> I don't think pip actually has any use for the PEP 517
> >>> >> prepare_wheel_metadata hook right now though? Historically 'setup.py
> >>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
> >>> >> install', but with a PEP 517 backend that's irrelevant because
> >>> >> 'setup.py install' is never used. And in the future when pip has a
> >>> >> real resolver, then prepare_wheel_metadata should allow some
> >>> >> optimizations. But right now, prepare_wheel_metadata is completely
> >>> >> useless AFAIK.
> >>> >>
> >>> >> So why is 'setup.py dist_info' a blocker for things?
> >>> >>
> >>> >> -n
> >>> >>
> >>> >> --
> >>> >> Nathaniel J. Smith -- https://vorpus.org
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Nathaniel J. Smith -- https://vorpus.org
> >>
> >>
> >
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

In any case, we're going to need this for prepare_metadata, so the question
you should ask is: what are the reasons for *not* merging this? I haven't
heard any so far but that doesn't mean that they don't exist. If there are
none, then I don't see why we cannot merge my wheel PR and do a release.

2017-09-04 19:51 GMT-05:00 xoviat <xov...@gmail.com>:

> > The only reason I can think of that setuptools would need a dist_info
> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>
> Yes. That is absolutely correct.
>
> > But this hook is optional and in fact provides no value right now, so
> it can't be a blocker for anything.
>
> The simplest way to start on this issue is to replace egg_info in pip with
> prepare_metadata_for_build_wheel. It is absolutely a historical artifact
> but I need to work on one issue at a time. The next issue will be replacing
> egg_info in pip with prepare_metadata_for_build_wheel.
>
> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith <n...@pobox.com>:
>
>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xov...@gmail.com> wrote:
>> > Nathaniel:
>> >
>> > Pip requires egg_info to discover dependencies of source distributions
>> so
>> > that it can build wheels all at once after downloading the
>> requirements. I
>> > need to move pip off of egg_info as soon as possible and dist_info is
>> > required to do that.
>>
>> "Requires" is a strong word -- AFAIK this is just a historical
>> artifact. I don't really know what you're talking about in the second
>> sentence.
>>
>> The only reason I can think of that setuptools would need a dist_info
>> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>> But this hook is optional and in fact provides no value right now, so
>> it can't be a blocker for anything. (In fact I can't see any reason
>> why pip would ever call it before the resolver lands.) So either (a)
>> there's some other reason you want a dist_info command, (b) there's
>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
>> of us is misunderstanding something :-).
>>
>> -n
>>
>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith <n...@pobox.com>:
>> >>
>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xov...@gmail.com> wrote:
>> >> > Just an update for everyone here:
>> >> >
>> >> > 1. We're currently waiting on the implementation of the 'dist_info"
>> >> > command
>> >> > in the wheel project.
>> >> > 2. Once that is done we can switch pip over to reading dist-info
>> rather
>> >> > than
>> >> > egg_info.
>> >> > 3. Then we can move the backend over to setuptools. Because Jacob
>> has a
>> >> > much
>> >> > more efficient release system than pip, I anticipate having a
>> release of
>> >> > setuptools first and then we can switch pip over to requiring a newer
>> >> > setuptools via PEP 518.
>> >>
>> >> I don't think pip actually has any use for the PEP 517
>> >> prepare_wheel_metadata hook right now though? Historically 'setup.py
>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
>> >> install', but with a PEP 517 backend that's irrelevant because
>> >> 'setup.py install' is never used. And in the future when pip has a
>> >> real resolver, then prepare_wheel_metadata should allow some
>> >> optimizations. But right now, prepare_wheel_metadata is completely
>> >> useless AFAIK.
>> >>
>> >> So why is 'setup.py dist_info' a blocker for things?
>> >>
>> >> -n
>> >>
>> >> --
>> >> Nathaniel J. Smith -- https://vorpus.org
>> >
>> >
>>
>>
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> The only reason I can think of that setuptools would need a dist_info
command would be to implement the PEP 517 prepare_wheel_metadata hook.

Yes. That is absolutely correct.

> But this hook is optional and in fact provides no value right now, so
it can't be a blocker for anything.

The simplest way to start on this issue is to replace egg_info in pip with
prepare_metadata_for_build_wheel. It is absolutely a historical artifact
but I need to work on one issue at a time. The next issue will be replacing
egg_info in pip with prepare_metadata_for_build_wheel.

2017-09-04 19:34 GMT-05:00 Nathaniel Smith <n...@pobox.com>:

> On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xov...@gmail.com> wrote:
> > Nathaniel:
> >
> > Pip requires egg_info to discover dependencies of source distributions so
> > that it can build wheels all at once after downloading the requirements.
> I
> > need to move pip off of egg_info as soon as possible and dist_info is
> > required to do that.
>
> "Requires" is a strong word -- AFAIK this is just a historical
> artifact. I don't really know what you're talking about in the second
> sentence.
>
> The only reason I can think of that setuptools would need a dist_info
> command would be to implement the PEP 517 prepare_wheel_metadata hook.
> But this hook is optional and in fact provides no value right now, so
> it can't be a blocker for anything. (In fact I can't see any reason
> why pip would ever call it before the resolver lands.) So either (a)
> there's some other reason you want a dist_info command, (b) there's
> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
> of us is misunderstanding something :-).
>
> -n
>
> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith <n...@pobox.com>:
> >>
> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xov...@gmail.com> wrote:
> >> > Just an update for everyone here:
> >> >
> >> > 1. We're currently waiting on the implementation of the 'dist_info"
> >> > command
> >> > in the wheel project.
> >> > 2. Once that is done we can switch pip over to reading dist-info
> rather
> >> > than
> >> > egg_info.
> >> > 3. Then we can move the backend over to setuptools. Because Jacob has
> a
> >> > much
> >> > more efficient release system than pip, I anticipate having a release
> of
> >> > setuptools first and then we can switch pip over to requiring a newer
> >> > setuptools via PEP 518.
> >>
> >> I don't think pip actually has any use for the PEP 517
> >> prepare_wheel_metadata hook right now though? Historically 'setup.py
> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
> >> install', but with a PEP 517 backend that's irrelevant because
> >> 'setup.py install' is never used. And in the future when pip has a
> >> real resolver, then prepare_wheel_metadata should allow some
> >> optimizations. But right now, prepare_wheel_metadata is completely
> >> useless AFAIK.
> >>
> >> So why is 'setup.py dist_info' a blocker for things?
> >>
> >> -n
> >>
> >> --
> >> Nathaniel J. Smith -- https://vorpus.org
> >
> >
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Also if someone with pip write access could please discuss and hopefully
merge my initial PR on pip, I would very much appreciate it. Paul seems to
be short on time.

2017-09-04 19:09 GMT-05:00 xoviat <xov...@gmail.com>:

> Nathaniel:
>
> Pip requires egg_info to discover dependencies of source distributions so
> that it can build wheels all at once after downloading the requirements. I
> need to move pip off of egg_info as soon as possible and dist_info is
> required to do that.
>
> 2017-09-03 21:00 GMT-05:00 Nathaniel Smith <n...@pobox.com>:
>
>> On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xov...@gmail.com> wrote:
>> > Just an update for everyone here:
>> >
>> > 1. We're currently waiting on the implementation of the 'dist_info"
>> command
>> > in the wheel project.
>> > 2. Once that is done we can switch pip over to reading dist-info rather
>> than
>> > egg_info.
>> > 3. Then we can move the backend over to setuptools. Because Jacob has a
>> much
>> > more efficient release system than pip, I anticipate having a release of
>> > setuptools first and then we can switch pip over to requiring a newer
>> > setuptools via PEP 518.
>>
>> I don't think pip actually has any use for the PEP 517
>> prepare_wheel_metadata hook right now though? Historically 'setup.py
>> egg-info' was needed to kluge around unwanted behavior in 'setup.py
>> install', but with a PEP 517 backend that's irrelevant because
>> 'setup.py install' is never used. And in the future when pip has a
>> real resolver, then prepare_wheel_metadata should allow some
>> optimizations. But right now, prepare_wheel_metadata is completely
>> useless AFAIK.
>>
>> So why is 'setup.py dist_info' a blocker for things?
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Nathaniel:

Pip requires egg_info to discover dependencies of source distributions so
that it can build wheels all at once after downloading the requirements. I
need to move pip off of egg_info as soon as possible and dist_info is
required to do that.

2017-09-03 21:00 GMT-05:00 Nathaniel Smith <n...@pobox.com>:

> On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xov...@gmail.com> wrote:
> > Just an update for everyone here:
> >
> > 1. We're currently waiting on the implementation of the 'dist_info"
> command
> > in the wheel project.
> > 2. Once that is done we can switch pip over to reading dist-info rather
> than
> > egg_info.
> > 3. Then we can move the backend over to setuptools. Because Jacob has a
> much
> > more efficient release system than pip, I anticipate having a release of
> > setuptools first and then we can switch pip over to requiring a newer
> > setuptools via PEP 518.
>
> I don't think pip actually has any use for the PEP 517
> prepare_wheel_metadata hook right now though? Historically 'setup.py
> egg-info' was needed to kluge around unwanted behavior in 'setup.py
> install', but with a PEP 517 backend that's irrelevant because
> 'setup.py install' is never used. And in the future when pip has a
> real resolver, then prepare_wheel_metadata should allow some
> optimizations. But right now, prepare_wheel_metadata is completely
> useless AFAIK.
>
> So why is 'setup.py dist_info' a blocker for things?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Just an update for everyone here:

1. We're currently waiting on the implementation of the 'dist_info" command
in the wheel project.
2. Once that is done we can switch pip over to reading dist-info rather
than egg_info.
3. Then we can move the backend over to setuptools. Because Jacob has a
much more efficient release system than pip, I anticipate having a release
of setuptools first and then we can switch pip over to requiring a newer
setuptools via PEP 518.

2017-09-02 19:51 GMT-05:00 Chris Jerdonek <chris.jerdo...@gmail.com>:

> On Sat, Sep 2, 2017 at 5:17 PM xoviat <xov...@gmail.com> wrote:
>
>> Whatever it was, removing it seems to have had no effect on the tests. I
>> will remove it unless someone has an objection.
>>
>
> Just FYI, I wouldn't take the tests still passing as a major signal. I've
> noticed there are even common code paths / functional scenarios that aren't
> under test.
>
> --Chris
>
>
>
>
>> 2017-09-02 18:26 GMT-05:00 xoviat <xov...@gmail.com>:
>>
>>> Donald,
>>>
>>>
>>> This was your work in https://github.com/pypa/pip/pull/2169.
>>> Unfortunately the comments were quite sparse.
>>>
>>> 2017-09-02 18:25 GMT-05:00 xoviat <xov...@gmail.com>:
>>>
>>>> One more issue that has come up is that "--no-user-cfg" seems to be
>>>> passed to the egg_info invocation if the "isolated" parameter is enabled. I
>>>> don't understand what this does, but it is again not defined in the PEP 517
>>>> interface. Should we always pass this parameter or should we never pass it?
>>>>
>>>> 2017-09-02 14:42 GMT-05:00 Donald Stufft <don...@stufft.io>:
>>>>
>>>>>
>>>>> On Sep 1, 2017, at 2:30 PM, Chris Barker <chris.bar...@noaa.gov>
>>>>> wrote:
>>>>>
>>>>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncogh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> since it
>>>>>> doesn't reliably distinguish between "this cached wheel was downloaded
>>>>>> from a repository" and "this wheel was generated locally with a
>>>>>> particular version of Python".
>>>>>
>>>>>
>>>>> It shouldn't have to. sigh.
>>>>>
>>>>>>
>>>>>  PEP 517 deliberately doesn't let
>>>>>> frontends do that as part of the initial build process (instead, if
>>>>>> they want to adjust the tags, they need to do it as a post-processing
>>>>>> step).
>>>>>>
>>>>>> Since PEP 517 breaks the current workaround for the caching scheme
>>>>>> being inaccurate, the most suitable response is to instead fix pip's
>>>>>> caching scheme to use a two tier local cache:
>>>>>
>>>>>
>>>>> I'm still confused -- if setuptools ( invoked  by pip) is producing
>>>>> incorrectly named wheels -- surely that's a bug-fix/workaround that should
>>>>> go into setuptools?
>>>>>
>>>>> If the build is being run by pip, then doesn't setuptools have all the
>>>>> info about the system that pip has?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Someone building a wheel for distribution is likely intimately aware
>>>>> of that project, and can take care to ensure that the wheel is built in
>>>>> such a way that it is giving people the most optimal behavior. Pip is auto
>>>>> building wheels without human intervention, and as such there is nobody
>>>>> there to make sure that we’re not accidentally creating a too-broad wheel,
>>>>> so we want to ensure that we have some mechanism in place for not re-using
>>>>> the wheel across boundaries that might cause issues.
>>>>>
>>>>>
>>>>>
>>>>> we also have plenty of PyPI users that
>>>>>> explicitly *opt out* of using publisher-provided pre-built binaries.
>>>>>> While Linux distributions are the most common example (see [1] for
>>>>>> Fedora's policy, for example), we're not the only ones that have that
>>>>>> kind of rule in place.
>>>>>
>>>>>
>>>>> But this is an argument for why pypi should host sdists, and the build
>>>>> tools should build sdists, but not why pip should auto-build them.
>>>>&

Re: [Distutils] PEP 541

[xoviat]

I would actually like to transfer the "pywin32" project to Paul, Glyph,
myself and friends because nothing has been uploaded there, but I thought
it wasn't possible. Maybe it is?

2017-09-02 6:05 GMT-05:00 nassim daoud :

> Hello,
>
> There is a friend of mine wondering, if abandoned PyPI projects can be
> claimed. And if so what the procedure in order to do that.
>
> My best regards,
>
> Daoud Nassim
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Whatever it was, removing it seems to have had no effect on the tests. I
will remove it unless someone has an objection.

2017-09-02 18:26 GMT-05:00 xoviat <xov...@gmail.com>:

> Donald,
>
>
> This was your work in https://github.com/pypa/pip/pull/2169.
> Unfortunately the comments were quite sparse.
>
> 2017-09-02 18:25 GMT-05:00 xoviat <xov...@gmail.com>:
>
>> One more issue that has come up is that "--no-user-cfg" seems to be
>> passed to the egg_info invocation if the "isolated" parameter is enabled. I
>> don't understand what this does, but it is again not defined in the PEP 517
>> interface. Should we always pass this parameter or should we never pass it?
>>
>> 2017-09-02 14:42 GMT-05:00 Donald Stufft <don...@stufft.io>:
>>
>>>
>>> On Sep 1, 2017, at 2:30 PM, Chris Barker <chris.bar...@noaa.gov> wrote:
>>>
>>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncogh...@gmail.com> wrote:
>>>
>>> since it
>>>> doesn't reliably distinguish between "this cached wheel was downloaded
>>>> from a repository" and "this wheel was generated locally with a
>>>> particular version of Python".
>>>
>>>
>>> It shouldn't have to. sigh.
>>>
>>>>
>>>  PEP 517 deliberately doesn't let
>>>> frontends do that as part of the initial build process (instead, if
>>>> they want to adjust the tags, they need to do it as a post-processing
>>>> step).
>>>>
>>>> Since PEP 517 breaks the current workaround for the caching scheme
>>>> being inaccurate, the most suitable response is to instead fix pip's
>>>> caching scheme to use a two tier local cache:
>>>
>>>
>>> I'm still confused -- if setuptools ( invoked  by pip) is producing
>>> incorrectly named wheels -- surely that's a bug-fix/workaround that should
>>> go into setuptools?
>>>
>>> If the build is being run by pip, then doesn't setuptools have all the
>>> info about the system that pip has?
>>>
>>>
>>>
>>>
>>> Someone building a wheel for distribution is likely intimately aware of
>>> that project, and can take care to ensure that the wheel is built in such a
>>> way that it is giving people the most optimal behavior. Pip is auto
>>> building wheels without human intervention, and as such there is nobody
>>> there to make sure that we’re not accidentally creating a too-broad wheel,
>>> so we want to ensure that we have some mechanism in place for not re-using
>>> the wheel across boundaries that might cause issues.
>>>
>>>
>>>
>>> we also have plenty of PyPI users that
>>>> explicitly *opt out* of using publisher-provided pre-built binaries.
>>>> While Linux distributions are the most common example (see [1] for
>>>> Fedora's policy, for example), we're not the only ones that have that
>>>> kind of rule in place.
>>>
>>>
>>> But this is an argument for why pypi should host sdists, and the build
>>> tools should build sdists, but not why pip should auto-build them.
>>>
>>>>
>>> Condo-forge, for example, almost always builds from source -- sometimes
>>> an sdist
>>> from pypi, sometimes a source distribution from github or wherever the
>>> package is hosted. And sometimes from a git tag ( last resort).
>>>
>>>
>>>
>>> Pip supports more systems than Conda does, and we do that by relying on
>>> auto building support. Pip supports systems that don’t have a wheel
>>> compatibility tag defined for them, and for which we’re unlikely to ever
>>> have any wheels published for (much less wide spread). It’s pretty easy to
>>> cover Windows/macOS/Some Linux systems, but when you start talking about
>>> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets
>>> extremely long.
>>>
>>> Pip works in all these situations, and it does so by relying on building
>>> from source.
>>>
>>>
>>>
>>> Do the Linux distros use pip to build their packages?
>>>
>>>
>>>
>>> Not that I am aware of.
>>>
>>>
>>> I tried to do that with conda-packages, and failed due to pip's caching
>>>  behavior-- it probably would have worked fine in production, but when I
>>> was developing the build script, I couldn't reliably get pip to ignore
>>> cached wheels from previous experimental builds.
>>>
>>>
>>>
>>> Adding —no-cache-dir disables all of pip’s caching.
>>>
>>> —
>>> Donald Stufft
>>>
>>>
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Donald,


This was your work in https://github.com/pypa/pip/pull/2169. Unfortunately
the comments were quite sparse.

2017-09-02 18:25 GMT-05:00 xoviat <xov...@gmail.com>:

> One more issue that has come up is that "--no-user-cfg" seems to be passed
> to the egg_info invocation if the "isolated" parameter is enabled. I don't
> understand what this does, but it is again not defined in the PEP 517
> interface. Should we always pass this parameter or should we never pass it?
>
> 2017-09-02 14:42 GMT-05:00 Donald Stufft <don...@stufft.io>:
>
>>
>> On Sep 1, 2017, at 2:30 PM, Chris Barker <chris.bar...@noaa.gov> wrote:
>>
>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncogh...@gmail.com> wrote:
>>
>> since it
>>> doesn't reliably distinguish between "this cached wheel was downloaded
>>> from a repository" and "this wheel was generated locally with a
>>> particular version of Python".
>>
>>
>> It shouldn't have to. sigh.
>>
>>>
>>  PEP 517 deliberately doesn't let
>>> frontends do that as part of the initial build process (instead, if
>>> they want to adjust the tags, they need to do it as a post-processing
>>> step).
>>>
>>> Since PEP 517 breaks the current workaround for the caching scheme
>>> being inaccurate, the most suitable response is to instead fix pip's
>>> caching scheme to use a two tier local cache:
>>
>>
>> I'm still confused -- if setuptools ( invoked  by pip) is producing
>> incorrectly named wheels -- surely that's a bug-fix/workaround that should
>> go into setuptools?
>>
>> If the build is being run by pip, then doesn't setuptools have all the
>> info about the system that pip has?
>>
>>
>>
>>
>> Someone building a wheel for distribution is likely intimately aware of
>> that project, and can take care to ensure that the wheel is built in such a
>> way that it is giving people the most optimal behavior. Pip is auto
>> building wheels without human intervention, and as such there is nobody
>> there to make sure that we’re not accidentally creating a too-broad wheel,
>> so we want to ensure that we have some mechanism in place for not re-using
>> the wheel across boundaries that might cause issues.
>>
>>
>>
>> we also have plenty of PyPI users that
>>> explicitly *opt out* of using publisher-provided pre-built binaries.
>>> While Linux distributions are the most common example (see [1] for
>>> Fedora's policy, for example), we're not the only ones that have that
>>> kind of rule in place.
>>
>>
>> But this is an argument for why pypi should host sdists, and the build
>> tools should build sdists, but not why pip should auto-build them.
>>
>>>
>> Condo-forge, for example, almost always builds from source -- sometimes
>> an sdist
>> from pypi, sometimes a source distribution from github or wherever the
>> package is hosted. And sometimes from a git tag ( last resort).
>>
>>
>>
>> Pip supports more systems than Conda does, and we do that by relying on
>> auto building support. Pip supports systems that don’t have a wheel
>> compatibility tag defined for them, and for which we’re unlikely to ever
>> have any wheels published for (much less wide spread). It’s pretty easy to
>> cover Windows/macOS/Some Linux systems, but when you start talking about
>> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets
>> extremely long.
>>
>> Pip works in all these situations, and it does so by relying on building
>> from source.
>>
>>
>>
>> Do the Linux distros use pip to build their packages?
>>
>>
>>
>> Not that I am aware of.
>>
>>
>> I tried to do that with conda-packages, and failed due to pip's caching
>>  behavior-- it probably would have worked fine in production, but when I
>> was developing the build script, I couldn't reliably get pip to ignore
>> cached wheels from previous experimental builds.
>>
>>
>>
>> Adding —no-cache-dir disables all of pip’s caching.
>>
>> —
>> Donald Stufft
>>
>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

One more issue that has come up is that "--no-user-cfg" seems to be passed
to the egg_info invocation if the "isolated" parameter is enabled. I don't
understand what this does, but it is again not defined in the PEP 517
interface. Should we always pass this parameter or should we never pass it?

2017-09-02 14:42 GMT-05:00 Donald Stufft :

>
> On Sep 1, 2017, at 2:30 PM, Chris Barker  wrote:
>
> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan  wrote:
>
> since it
>> doesn't reliably distinguish between "this cached wheel was downloaded
>> from a repository" and "this wheel was generated locally with a
>> particular version of Python".
>
>
> It shouldn't have to. sigh.
>
>>
>  PEP 517 deliberately doesn't let
>> frontends do that as part of the initial build process (instead, if
>> they want to adjust the tags, they need to do it as a post-processing
>> step).
>>
>> Since PEP 517 breaks the current workaround for the caching scheme
>> being inaccurate, the most suitable response is to instead fix pip's
>> caching scheme to use a two tier local cache:
>
>
> I'm still confused -- if setuptools ( invoked  by pip) is producing
> incorrectly named wheels -- surely that's a bug-fix/workaround that should
> go into setuptools?
>
> If the build is being run by pip, then doesn't setuptools have all the
> info about the system that pip has?
>
>
>
>
> Someone building a wheel for distribution is likely intimately aware of
> that project, and can take care to ensure that the wheel is built in such a
> way that it is giving people the most optimal behavior. Pip is auto
> building wheels without human intervention, and as such there is nobody
> there to make sure that we’re not accidentally creating a too-broad wheel,
> so we want to ensure that we have some mechanism in place for not re-using
> the wheel across boundaries that might cause issues.
>
>
>
> we also have plenty of PyPI users that
>> explicitly *opt out* of using publisher-provided pre-built binaries.
>> While Linux distributions are the most common example (see [1] for
>> Fedora's policy, for example), we're not the only ones that have that
>> kind of rule in place.
>
>
> But this is an argument for why pypi should host sdists, and the build
> tools should build sdists, but not why pip should auto-build them.
>
>>
> Condo-forge, for example, almost always builds from source -- sometimes an
> sdist
> from pypi, sometimes a source distribution from github or wherever the
> package is hosted. And sometimes from a git tag ( last resort).
>
>
>
> Pip supports more systems than Conda does, and we do that by relying on
> auto building support. Pip supports systems that don’t have a wheel
> compatibility tag defined for them, and for which we’re unlikely to ever
> have any wheels published for (much less wide spread). It’s pretty easy to
> cover Windows/macOS/Some Linux systems, but when you start talking about
> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets
> extremely long.
>
> Pip works in all these situations, and it does so by relying on building
> from source.
>
>
>
> Do the Linux distros use pip to build their packages?
>
>
>
> Not that I am aware of.
>
>
> I tried to do that with conda-packages, and failed due to pip's caching
>  behavior-- it probably would have worked fine in production, but when I
> was developing the build script, I couldn't reliably get pip to ignore
> cached wheels from previous experimental builds.
>
>
>
> Adding —no-cache-dir disables all of pip’s caching.
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

I don't buy the argument that pip needs to keep separate caches for
different implementations. According to the wheel specification, a wheel is
valid for the Python implementations that it specifies. This is therefore a
blatant violation by setuptools of an existing specification that, in
theory, should be ignored by pip.

That said, setuptools is currently the de facto build system, meaning that
we cannot simply break the status quo. So here's my proposal:

1. For now, wheels are renamed if and only if the backend is setuptools.
2. In the future, we introduce a new setup() function that is imported from
a different module path that requires compliance with the specification.
3. Once the vast majority of projects are using the new setup function, we
can adjust the pep 517 setuptools build backend to use the most restrictive
tag for the old function
4. We can finally remove autobuilding from pip.

One thing to keep in mind is that there are quite a few projects on pypa
with pure python source distributions uploaded that will not be updated and
people may still desire to use. We want pip to be able to still build and
install them.

2017-08-31 16:29 GMT-05:00 Chris Barker :

> On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith  wrote:
>
>> > Surely the build system should know how to correctly name the wheel it
>> builds.
>>
>> It's probably worth mentioning the specific problem that motivated pip
>> to start doing this.
>>
>> It used to be standard, and is still quite common, for setup.py
>> scripts to contain stuff like:
>>
>> install_requires = [...]
>> if sys.version_info < (3, 4):
>> install_requires += [...]
>> if platform.python_implementation() == "PyPy":
>> install_requires += [...]
>>
>> setup(..., install_requires=install_requires)
>>
>> This kind of logic in setup.py worked fine in the old days when all
>> you did was 'setup.py install', but then wheels came along
>
>
> And indeed, setuptools originally used easy_install, which was part of
> setuptools...
>
> and
>> retroactively turned lots of setup.py scripts from working into
>> broken. The problem is that with this kind of setup.py,
>>
>
>
>> But it will take a while for existing setup.py files transition to
>> using those, and in the mean time pip can't assume that a random wheel
>> generated by 'setup.py bdist_wheel' has accurate Python tags.
>>
>
> This was my original point -- I understand that we want "pip install" to
> continue to work for, hopefully, everything it works for now.
>
> But I do think we should be clear about what is a hack for backward
> compatibility, and what is part of the designed functionality.
>
> Sorry to be poking at all this from the fringes (Not having been all that
> involved in a very long discussion), it's just that the whole
>
> distutils--setuptools--pip--distribute--setuptools--pip
>
> stack has a LOT of legacy cruft, and I'm  concerned that the efforts for
> backward compatibility may end up leading us to another poorly de-coupled
> design.
>
>
>> Hopefully new legacy-free backends will get this right from the start.
>>
>
> exactly -- let's keep the "backward compatibility hack" labels clear!
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

There are two points that I would note here:
1. setuptools handling of environment markers in wheels was actually broken
until recently.
2. Even now, people still prefer runtime detection of environment markers
to work around pip's upgrade behavior and old setuptools versions. There
needs to be a public effort to change this behavior.

For #2, it's really not a clean solution, but it would be more effective to
scan the bytecode of setup.py and then warn users that this behavior is
deprecated. I expect that you would get a lot of complaints from package
authors but that is the point.

2017-08-31 13:03 GMT-05:00 Nathaniel Smith :

> On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal
>  wrote:
> > The package manager should manage the package, not built it, or change
> it.
> >
> > Surely the build system should know how to correctly name the wheel it
> builds.
>
> It's probably worth mentioning the specific problem that motivated pip
> to start doing this.
>
> It used to be standard, and is still quite common, for setup.py
> scripts to contain stuff like:
>
> install_requires = [...]
> if sys.version_info < (3, 4):
> install_requires += [...]
> if platform.python_implementation() == "PyPy":
> install_requires += [...]
>
> setup(..., install_requires=install_requires)
>
> This kind of logic in setup.py worked fine in the old days when all
> you did was 'setup.py install', but then wheels came along and
> retroactively turned lots of setup.py scripts from working into
> broken. The problem is that with this kind of setup.py, setuptools has
> *no idea* that the install_requires you gave it would have been
> different if you had run setup.py with a different version of Python,
> so when it has to assign Python tags to a built wheel it guesses wrong
> and uses ones that are too general.
>
> The right way to do this is to use PEP 508 environment markers:
> https://www.python.org/dev/peps/pep-0508/#environment-markers
> or the non-standard extras hack:
> https://wheel.readthedocs.io/en/latest/#defining-
> conditional-dependencies
> Both of these let you export the whole requirements-choosing logic
> into the wheel metadata, so that it can be evaluated at install time
> instead of build time.
>
> But it will take a while for existing setup.py files transition to
> using those, and in the mean time pip can't assume that a random wheel
> generated by 'setup.py bdist_wheel' has accurate Python tags.
>
> Hopefully new legacy-free backends will get this right from the start.
> For example flit makes it impossible to get this wrong.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Will do. Better to hash this out sooner than later.

On Aug 31, 2017 12:36 AM, "Nathaniel Smith" <n...@pobox.com> wrote:

> On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> > On 31 August 2017 at 14:22, xoviat <xov...@gmail.com> wrote:
> >> Again, let me repeat that: wheels generated using setuptools are valid
> for
> >> CPython only if build on CPython. This is not the current setuptools
> >> behavior but will be for all setuptools build backend calls (I assume
> legacy
> >> will remain the same).
> >
> > While I do think your proposal would work (on the assumption that
> > folks aren't use "pip wheel" to generate their wheel files for
> > upload),
>
> I use 'pip wheel' to generate wheel files for upload... (I like to
> generate an sdist and then build a wheel from that, and 'pip wheel
> sdist.tar.gz' is more convenient than manually unpacking and running
> bdist_wheel. )
>
> > an alternative approach with a lower risk of unintended side
> > effects would be for *pip* to either rename the autobuilt file before
> > adding it to the cache, or else to adjust its caching strategy a bit
> > to internally separate a shared wheel download cache from
> > per-interpreter-compatibility-tag caches for locally built wheel
> > files.
>
> +1
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

This appears to be related to https://github.com/pypa/pip/pull/3225, where
pip would generate overbroad wheel names in the cache. So to proceed on
this issue, setuptools will assume that generated wheels are valid for the
specific python implementation only (eg. wheels generated for CPython will
not be valid for pypy).

Again, let me repeat that: wheels generated using setuptools are valid for
CPython only if build on CPython. This is not the current setuptools
behavior but will be for all setuptools build backend calls (I assume
legacy will remain the same).

Jacob, Paul, Donald, is everyone okay with that? And does everyone
understand the reason for this?

2017-08-30 18:29 GMT-05:00 xoviat <xov...@gmail.com>:

> So while I was working on pip, I noticed a parameter called 'autobuilding'
> in wheel.py. What this does (I think) is that if pip decides that it needs
> to be enabled (which is an implementation detail not relevant here), it
> adds the following to the "setup.py bdist_wheel" call:
>
> ["--python-tag", python_tag]
>
> This switches the output of the bdist_wheel filename to include something
> like "cp36" rather than "py36," which indicates the Python implementation.
> The question is this: this behavior isn't captured (I think) in the
> build_wheel function, possibly (probably) for good reason. So that means
> once everything goes through the PEP 517 protocol, pip will not be able to
> let setuptools know about this information. Which means that we need to
> decide:
>
> Should the PEP 517 backend for setuptools assume "cp36" in python_tag
> OR
> Should the PEP 517 backend for setuptools assume "py36" in python_tag
>
>
> 2017-08-29 18:51 GMT-05:00 Matthew Brett <matthew.br...@gmail.com>:
>
>> Hi,
>>
>> On Tue, Aug 29, 2017 at 8:30 PM, xoviat <xov...@gmail.com> wrote:
>> >> For me, using NotImplemented is a misuse of the singleton since I know
>> >> what it's meant to be used for (and so I cringe every time I hear it
>> brought
>> >> up as a solution).
>> >
>> > I hate to reiterate too much, but I think that you are correct here:
>> > NotImplemented is not Pythonic. It's a complete hack that I would think
>> > people would like to quarantine to the limited use case of binary
>> > operations. Even with the poorly contextualized email that Guido
>> received, I
>> > think it was clear that he thought it wasn't appropriate outside of
>> that use
>> > case either (I don't think better context would have changed that).
>> >
>> > Personally, I was willing to overlook that for the sake of a speedy
>> > resolution. But reading the email chain, it seems that only Thomas and
>> > Daniel are currently in favor of using NotImplemented, which is a shift
>> from
>> > the earlier situation. So now it seems that NotImplemented is not
>> likely to
>> > be used.
>>
>> Just for the record, I thought Nathaniel's NotImplemented suggestion
>> was a good one, and his explanation of why, was clear and convincing,
>> but he's already accepted the alternative, of a custom error, so I
>> don't think there's much point on going back over that.
>>
>> Also just for the record, I'd like to warmly thank y'all for your
>> great patience in this long discussion.
>>
>> Cheers,
>>
>> Matthew
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

So while I was working on pip, I noticed a parameter called 'autobuilding'
in wheel.py. What this does (I think) is that if pip decides that it needs
to be enabled (which is an implementation detail not relevant here), it
adds the following to the "setup.py bdist_wheel" call:

["--python-tag", python_tag]

This switches the output of the bdist_wheel filename to include something
like "cp36" rather than "py36," which indicates the Python implementation.
The question is this: this behavior isn't captured (I think) in the
build_wheel function, possibly (probably) for good reason. So that means
once everything goes through the PEP 517 protocol, pip will not be able to
let setuptools know about this information. Which means that we need to
decide:

Should the PEP 517 backend for setuptools assume "cp36" in python_tag
OR
Should the PEP 517 backend for setuptools assume "py36" in python_tag


2017-08-29 18:51 GMT-05:00 Matthew Brett <matthew.br...@gmail.com>:

> Hi,
>
> On Tue, Aug 29, 2017 at 8:30 PM, xoviat <xov...@gmail.com> wrote:
> >> For me, using NotImplemented is a misuse of the singleton since I know
> >> what it's meant to be used for (and so I cringe every time I hear it
> brought
> >> up as a solution).
> >
> > I hate to reiterate too much, but I think that you are correct here:
> > NotImplemented is not Pythonic. It's a complete hack that I would think
> > people would like to quarantine to the limited use case of binary
> > operations. Even with the poorly contextualized email that Guido
> received, I
> > think it was clear that he thought it wasn't appropriate outside of that
> use
> > case either (I don't think better context would have changed that).
> >
> > Personally, I was willing to overlook that for the sake of a speedy
> > resolution. But reading the email chain, it seems that only Thomas and
> > Daniel are currently in favor of using NotImplemented, which is a shift
> from
> > the earlier situation. So now it seems that NotImplemented is not likely
> to
> > be used.
>
> Just for the record, I thought Nathaniel's NotImplemented suggestion
> was a good one, and his explanation of why, was clear and convincing,
> but he's already accepted the alternative, of a custom error, so I
> don't think there's much point on going back over that.
>
> Also just for the record, I'd like to warmly thank y'all for your
> great patience in this long discussion.
>
> Cheers,
>
> Matthew
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> For me, using NotImplemented is a misuse of the singleton since I know
what it's meant to be used for (and so I cringe every time I hear it
brought up as a solution).

I hate to reiterate too much, but I think that you are correct here:
NotImplemented is not Pythonic. It's a complete hack that I would think
people would like to quarantine to the limited use case of binary
operations. Even with the poorly contextualized email that Guido received,
I think it was clear that he thought it wasn't appropriate outside of that
use case either (I don't think better context would have changed that).

Personally, I was willing to overlook that for the sake of a speedy
resolution. But reading the email chain, it seems that only Thomas and
Daniel are currently in favor of using NotImplemented, which is a shift
from the earlier situation. So now it seems that NotImplemented is not
likely to be used.

2017-08-29 14:07 GMT-05:00 Paul Moore :

> On 29 August 2017 at 19:45, Chris Barker  wrote:
> > But it would be nice if packages didn't NEED to be pip installed -- but I
> > think that's an issue with the wheel spec (that may well be solved).
>
> Wheels don't need pip to install them - the spec has always been clear
> on how to "manually" install a wheel (unzip it and move a few things
> around), and the wheel package itself has a simple "wheel install"
> command.
>
> If you want to create wrapper scripts for entry points, you need to do
> a bit more work, but distlib offers that functionality (that's what
> pip uses).
>
> One point about PEP 517 is that it provides a standard way to ask
> *any* build system to make a wheel. So it's in theory possible to
> install any package without pip (of course, if you try to handle all
> the various corner cases, extra quirks, etc that come up in the real
> world, you end up reinventing pip ;-)).
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> If so, let the trumpets sound, and the heralds declare that "return
NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's
preference right ;-)

For better or for worse, the trumpets seem to be sounding against this idea
(Nathaniel seemed okay with whatever Donald and Nick thought was
appropriate).

2017-08-28 15:27 GMT-05:00 Thomas Kluyver :

> On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote:
>
> Then end the debate by letting the PEP authors decide the return type, and
> write a paragraph explaining why the other options were rejected. It is not
> going to make a big difference.
>
>
> Will that work now? Are we all so tired of this endless war that people
> will sign a peace treaty written by the people whose names are on the PEP
> (Nathaniel & me)?
>
> If so, let the trumpets sound, and the heralds declare that "return
> NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's
> preference right ;-)
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> But I'm suspicious of the rationale that *there will be fewer frontends so
they should have more responsibility*.

To be fair, pip is currently struggling to keep up with project
requirements as it is, which has caused some frustration in the community
(not that the frustration isn't wrong, but I don't think it's not going to
resolve any problems). I don't see how having even more frontends will
resolve these manpower issues.

2017-08-28 14:32 GMT-05:00 Thomas Kluyver :

> On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
> > Maybe we go fully to Nick's proposal that we don't mandate any sort of
> > consistency constraints in the PEP. That would mean pip *has* to go
> > sdist->wheel (because pip does need consistent behaviour), and
> > xoviat's setuptools backend can skip building a sdist on the way to
> > building a wheel. It also means that the build_wheel hook is
> > essentially unreliable in isolation, and that all frontends will
> > likely have to do the build_sdist->build_wheel with fallback to
> > inplace build_wheel dance that pip does. But we expect many less
> > frontends than backends, so maybe that's the right trade-off?
>
> I've mentioned this before, but I have little faith in our ability to
> predict that one side of an interface will be far more numerous than the
> other. This is partly from my experience with Jupyter, where we got that
> prediction completely wrong. But I also think it's quite plausible that
> many frontend tools will want to use this interface (to argue against
> myself, most will be Python tools, so they could theoretically share a
> common wrapper module - but I have some doubts about whether they will).
>
> I'm not saying we shouldn't do this - as Donald is firmly in favour and
> you're wavering, it seems the easier option to wrap the discussion up.
> But I'm suspicious of the rationale that *there will be fewer frontends
> so they should have more responsibility*.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Personally, my plan for the setuptools backend will be to build a source
distribution (essentially using the command-line interface), extract into a
tmpdir, and then build a wheel (essentially using the command line
interface). So if pip calls build_sdist and then build_wheel, there will be
two source distributions built (one by pip and one by setuptools) before
building a wheel. There is not another way to do this that will comply with
the specification because setuptools cannot currently be trusted to build a
wheel directly.

2017-08-28 12:15 GMT-05:00 Donald Stufft :

>
> On Aug 28, 2017, at 12:29 PM, Chris Barker  wrote:
>
> On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft  wrote:
>
>> Donald, what do you think? IIRC, you were most keen on going
>> sdist->wheel where possible, and I don't think you've commented on
>> Paul's suggestion yet (apologies if I've overlooked a response).
>>
>>
>> I still think it should, and prefer pip to attempt to build a sdist prior
>> to building a wheel when we’re coming from a VCS directory. I think that is
>> going to be the most robust mechanism with the least amount of surprising
>> behavior for end users.
>>
>
> why? even as it stands, doesn't setuptools dump everything into the build
> dir anyway? The "Creating of an sdist" really seems like a build-system
> problem, not a package manager problem to me.
>
> > "when we’re coming from a VCS directory."
>
> whether this is a VCS directory or some other source seems to me like
> something pip should not need to know...
>
>
>
> Differences between what files are in a sdist and what files are in a VCS
> directory will lead to different behaviors on install which makes ``pip
> install .`` and ``build-a-sdist && pip install the-sdist.tar.gz`` behave
> differently. Attempting to funnel everything through the same VCS -> sdist
> -> wheel path makes it less likely for these kinds of issues to occur.
>
> This is not a fully resolvable problem, and it is going to happen
> basically anytime you have two independent lists of what files get put into
> a sdist and what files get installed. It is not unique to setuptools nor is
> it a result of the way distutils/setuptools works— they only expose it more
> obviously because of their relevant APIs. Infact, both enscons and flit
> have this same problem (although flit has gone to some length to minimize
> the issue, so it’s somewhat hard, but not impossible, to actually trigger
> it).
>
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

The difference is that an empty list for requires would indicate that
building is possible but none would not. Those two values are truthily
equivalent.

For building, the name returned indicates building was possible, but it's
not truthily equivalent to none.

On Aug 27, 2017 12:08 AM, "C Anthony Risinger" <c...@anthonyrisinger.com>
wrote:

> On Sat, Aug 26, 2017 at 11:18 PM, xoviat <xov...@gmail.com> wrote:
>
>> > and they do not appear to receive the source or wheel directories.
>>
>> The source directory is the current directory, if I am not mistaken.
>>
>
> Oh right right, for some reason I was reading the `build_*` hooks as
> receiving the input directory rather than the output directory. I don't
> see it stated anywhere that the PWD will be the source directory however.
> Might be a good addition.
>
> In this case, `get_requires_for_*` hooks should also be able to depend on
> PWD being the applicable source directory (which of course might be an
> unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel
> pipeline).
>
>
>> > This lets `build_*` focus purely on building things straight away.
>> There is a difference between "no more reqs are needed to do X" and "no
>> possible req will achieve X" even though both add zero requirements. Why
>> not let this hook relay it's decision more completely?
>>
>> Not trying to speak on behalf of flit here, but if I understand
>> correctly, flit requires git to build a source distribution. Flit knows its
>> build requirements, so it can just return them when get_requires is called.
>> However, it needs to attempt to invoke git to find out whether it can build
>> a source distribution, which if I understand correctly, is a lengthy
>> operation (whether that's actually true is not actually relevant because we
>> are discussion potentially any backend/operation). It's more efficient if
>> git is only invoked once, and if a failure occurs, then under the proposal
>> "None" would be returned from build_sdist rather than the name of the built
>> source distribution.
>>
>
> What is the benefit in making `build_sdist` handle this assertion?
> Requirements checking is only performed once, earlier than building.
> Somewhere, somehow, something must look to see if git exists. Is it not
> more appropriate to check early, while asserting python requirements, vs.
> actually being told to build an sdist? That is already the expectation on
> the wheel side of things since it would also fail if necessary system
> binaries are missing. Checking for git is conflating system reqs and python
> reqs as "requirements", but I don't see why the get_requires_* hooks can't
> do both if they so choose.
>
>
>>  def get_requires_for_build_sdist(source_dir, ...):
>> # I have no interest in sdists and I never will.
>> # GO AWAY.
>> return None
>>
>> I have to say that the above example is not a good idea. Perhaps I was a
>> bit muddled on that point earlier: the reason that "return None" is a good
>> idea for build sdist is, well:
>>
>> def build_sdist(...):  # Indicates success
>> return "x.tar.gz"
>>
>> def build_sdist(...):  # Indicates failure
>> return None
>>
>>
>> Those are not truthily equivalent, which is important because it means
>> that someone is unlikely to make a mistake on that matter.
>>
>
> What is the difference? Both equally inform the frontend that building an
> sdist is not going to happen.
>
> At the end of the day, I'm not too partial to any of the solutions I've
> read thus far, and do not have any desire to derail progress. I just don't
> like the variation in `build_sdist` vs. `build_wheel` because it feels like
> a kludge to me. I thought perhaps defaulting sdists to "unsupported" and
> requiring a backend to opt-in was both more elegant and more explicit,
> allowing tools like flit to better express their conditional sdist support.
>
> --
>
> C Anthony
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> and they do not appear to receive the source or wheel directories.

The source directory is the current directory, if I am not mistaken.

> This lets `build_*` focus purely on building things straight away. There
is a difference between "no more reqs are needed to do X" and "no possible
req will achieve X" even though both add zero requirements. Why not let
this hook relay it's decision more completely?

Not trying to speak on behalf of flit here, but if I understand correctly,
flit requires git to build a source distribution. Flit knows its build
requirements, so it can just return them when get_requires is called.
However, it needs to attempt to invoke git to find out whether it can build
a source distribution, which if I understand correctly, is a lengthy
operation (whether that's actually true is not actually relevant because we
are discussion potentially any backend/operation). It's more efficient if
git is only invoked once, and if a failure occurs, then under the proposal
"None" would be returned from build_sdist rather than the name of the built
source distribution.

 def get_requires_for_build_sdist(source_dir, ...):
# I have no interest in sdists and I never will.
# GO AWAY.
return None

I have to say that the above example is not a good idea. Perhaps I was a
bit muddled on that point earlier: the reason that "return None" is a good
idea for build sdist is, well:

def build_sdist(...):  # Indicates success
return "x.tar.gz"

def build_sdist(...):  # Indicates failure
return None


Those are not truthily equivalent, which is important because it means that
someone is unlikely to make a mistake on that matter.

2017-08-26 23:05 GMT-05:00 C Anthony Risinger :

> On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith  wrote:
>
>> On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger
>>  wrote:
>> > On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:
>> >
>> > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
>> >  wrote:
>> >
>> > Sure sure, I understand all that, and why we think we need some special
>> > error signal from `build_sdist`, as currently written.
>> >
>> > What I'm suggesting, is maybe calling `build_sdist` without knowing if
>> it
>> > can succeed is already a mistake.
>> >
>> > Consider instead, if we make the following small changes:
>> >
>> > 1. `get_requires_for_build_*` is passed the sdist and wheel directories,
>> > just like `build_*`, giving them the chance to actually look at tree
>> before
>> > deciding what other reqs might be necessary.
>>
>> That's not a change, that's how it works :-).
>>
>
> Is that a change I missed from this thread? I'm reading here:
>
> https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301
> https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254
> https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist
> https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel
>
> and they do not appear to receive the source or wheel directories.
>
>
>> > 2. `get_requires_for_build_*` returns None to signal `build_*` is
>> > unsupported (superceded by static reqs defined in TOML) and [...] to
>> signal
>> > support (can be empty).
>> >
>> > 3. `get_requires_for_build_*` assumed to return None if missing (so
>> optional
>> > and implies no support).
>>
>> This is what I originally proposed, except you use None where I use
>> NotImplemented, which has the disadvantages I noted earlier. Also,
>> people didn't like the missing get_requires_for_build_* being treated
>> as no-support, which makes sense, since we expect that
>> get_requires_for_build_* won't be used very often. But one can switch
>> the default here without affecting much else. The reason we want to
>> let build_sdist report failure is just for convenience of backends who
>> don't have any other reason to implement get_requires_for_build_sdist.
>>
>
> Oh OK, good good. Well in that case I agree with you and missed the
> suggestion. I personally prefer NotImplemented as well here but None seemed
> mostly just as good and did not elicit as much pushback. It's not too big
> of deal either way.
>
> However, a missing `get_requires_for_build_wheel` technically signaling
> "unsupported" makes good sense to me because it's always supplemented by
> the static *and mandatory* `build-system.requires` list. There is no proper
> way (without breaking the spec) to signal "unsupported" for `build_wheel`
> since the backend itself (setuptools, wheel, flit) is specified here.
> "Unsupported" is only signaled when *both* the static and dynamic requires
> are None (or NotImplemented as mentioned). The kicker here in my offering,
> is that the presence of `build-system.requires` *does not in any way imply*
> `build_sdist` support. As written, there is no way to statically set the
> requirements for sdist support (though this could be changed of course with
> a 

Re: [Distutils] PEP 517 again

[xoviat]

Nathaniel:

Just to clarify, we're talking about returning none for the build function
not the get requirements function. The get requirements function is always
expected to succeed and is optional. If while the backend is discovering
requirements it finds that it cannot build, then it can return an empty
list. It cannot however signal in the get requirements function that it
cannot build. At least with what is being proposed.

The reason that the proposal works is that the build function can never
return none on success.

On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:

> On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
>  wrote:
> > On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:
> >>
> >> [removed Guido from CC]
> >>
> >> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
> >>
> >> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
> >> > In pretty much any other context, if you have an operation that
> returns
> >> > an
> >> > regular value or an error value, the error value should be None.
> >> > (Exceptions
> >> > include e.g. returning a non-negative int or -1 for errors, or True
> for
> >> > success and False for errors.)
> >>
> >> So, given that build_sdist returns the path of the newly built sdist,
> >> the correct way to signal "I didn't manage to build a sdist" would be
> >> to return None.
> >>
> >> Now that it's put this way, it seems glaringly obvious to me that this
> >> is the correct thing to do.
> >>
> >>
> >> Eh... I would really prefer something that's (a) more explicit about
> what
> >> specifically went wrong, and (b) harder to return by accident. It's not
> at
> >> all obvious that if the list of requirements is 'None' that means 'this
> >> build supports making sdists in general but cannot make them from this
> >> source tree but might still be able to make a wheel'. And if you forget
> to
> >> put in a return statement, then python returns None for you, which seems
> >> like it could lead to some super confusing error modes.
> >
> >
> > Why does the frontend need to know why an sdist was not created?
>
> This whole discussion is about handling a specific case: suppose you
> have a frontend like pip that when given a source directory and asked
> to build a wheel, wants to implement that as:
>   - build sdist
>   - unpack sdist
>   - build wheel from unpacked sdist
>
> And suppose you have a backend like flit, that can build sdists from
> some source directories (e.g. VCS checkouts) but not others (e.g.
> unpacked sdists). We need some way for pip and flit to negotiate that
> even though pip *normally* would implement its build-a-wheel operation
> by first building an sdist, in this case it's ok to silently fall back
> to some other strategy (like building the wheel directly in the source
> tree, or manually copying the source tree somewhere else and then
> building a wheel in it).
>
> But, we don't want this fallback behavior to hide real bugs. So if the
> backend says "look, I just can't do sdists here, and that's an
> expected thing, it's not something where the user needs to take any
> particular action like filing a bug report or fixing their system or
> anything like that, so if you have an alternative way to accomplish
> what you're trying to do then you should just silently discard this
> error and try that", ...cool. But if it doesn't explicitly say that,
> then we don't want to silently discard the error and do something
> else.
>
> It's taken a *lot* of back and forth to reach consensus that all we
> need here is some special error signal from the *_sdist operations.
> Let's focus on resolving that :-)
>
> > Frontend is asking the backend, given the current state of the world, to
> > either produce an sdist, or not. Sans ahead-of-time knowledge (see
> below), I
> > would expect build_sdist to make some sanity checks about the world, then
> > make a binary choice about whether sdist creation is a valid goal. If not
> > possible, return None or NotImplemented or False or dict-of-reasons or
> > whatever. Only if creation was *attempted*, and in the exceptional event
> it
> > then failed, would I expect an Exception. We don't have structured
> > exceptions sadly so they can't really carry much useful information from
> a
> > protocol perspective above and beyond a simple None or the like anyway.
> >
> > I'd personally like to see some parity between build_sdist and
> build_wheel
> > in this regard. Maybe the disconnect here is we have a way to specify
> hard
> > reqs for building a wheel, statically or dynamically, and build_wheel is
> > expected to never fail, but no way to specify hard reqs needed for
> > build_sdist, necessitating this optional signaling path?
>
> Not sure what you mean about hard reqs. The reason for the lack of
> parity is that we don't currently have any use cases where build_wheel
> is expected to fail, but this is expected in some sense (not sure what

Re: [Distutils] PEP 517 again

[xoviat]

With respect to this issue, if everyone told me that I was wrong then I
would say I'm obviously wrong. But some people are saying one thing and
other people are saying something else.

On Aug 26, 2017 5:03 PM, "xoviat" <xov...@gmail.com> wrote:

> As I said, I don't care what the particular solution is on this issue.
> However I'm simply trying to anticipate and resolve potential disagreements
> that could drag this out for a significant period of time. It is clear that
> Nick and Donald have a disagreement on this issue which is actually not in
> and of itself packaging related. The clear way to resolve this was to get a
> pronouncement.
>
> On Aug 26, 2017 4:56 PM, "Nathaniel Smith" <n...@pobox.com> wrote:
>
>> On Sat, Aug 26, 2017 at 2:06 PM, xoviat <xov...@gmail.com> wrote:
>> > I also think that Guido pretty much ruled out Notimplemented.
>>
>> As I've said, I don't think it matters a huge deal whether we use
>> NotImplemented or not. But please don't treat Guido as some kind of
>> pronouncement generating machine where you hurl out-of-context
>> questions at him and then use his response as a club to beat down
>> discussion. It's rude to Guido, it's rude to Nick and Donald (to whom
>> Guido has explicitly delegated his BDFL authority in packaging-related
>> matters), and it's rude to everyone trying to discuss proposals on
>> their merits.
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

As I said, I don't care what the particular solution is on this issue.
However I'm simply trying to anticipate and resolve potential disagreements
that could drag this out for a significant period of time. It is clear that
Nick and Donald have a disagreement on this issue which is actually not in
and of itself packaging related. The clear way to resolve this was to get a
pronouncement.

On Aug 26, 2017 4:56 PM, "Nathaniel Smith" <n...@pobox.com> wrote:

> On Sat, Aug 26, 2017 at 2:06 PM, xoviat <xov...@gmail.com> wrote:
> > I also think that Guido pretty much ruled out Notimplemented.
>
> As I've said, I don't think it matters a huge deal whether we use
> NotImplemented or not. But please don't treat Guido as some kind of
> pronouncement generating machine where you hurl out-of-context
> questions at him and then use his response as a club to beat down
> discussion. It's rude to Guido, it's rude to Nick and Donald (to whom
> Guido has explicitly delegated his BDFL authority in packaging-related
> matters), and it's rude to everyone trying to discuss proposals on
> their merits.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Nathaniel:

We're not talking about signaling failure in get_requires* we're talking
about signaling failure in build*.

On Aug 26, 2017 3:42 PM, "Nathaniel Smith"  wrote:

> On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore  wrote:
> > On 26 August 2017 at 20:17, Nathaniel Smith  wrote:
> >> Eh... I would really prefer something that's (a) more explicit about
> what
> >> specifically went wrong, and (b) harder to return by accident. It's not
> at
> >> all obvious that if the list of requirements is 'None' that means 'this
> >> build supports making sdists in general but cannot make them from this
> >> source tree but might still be able to make a wheel'. And if you forget
> to
> >> put in a return statement, then python returns None for you, which seems
> >> like it could lead to some super confusing error modes.
> >
> > Well, we've had an extensive discussion about how frontends need to
> > trust backends to get things right. I don't really see it as
> > reasonable to now argue that backends might "forget" to return the
> > right value - they might just as well "forget" to properly isolate
> > builds...
>
> It's not about division of responsibilities, it's about handling
> errors gracefully when they happen. There are three bins:
>
> - creating an sdist succeeded
> - creating an sdist failed for expected reasons, and a clever frontend
> might be able to handle the problem automatically if it understands
> what the problem is (sdist creation isn't supported in this case) and
> understands its goals (just trying to build a wheel really, so the
> sdist isn't crucial)
> - creating an sdist failed for unexpected reasons, that need a human
> to sort out (due to a broken system, or bugs – hey, they happen – or
> ...)
>
> The whole discussion has been about how we can most reliably
> distinguish between the second and third categories, and give good
> error messages for the third category. The argument for NotImplemented
> is that it avoids cases where some internal call raises
> NotImplementedError and it "leaks out" accidentally, causing a
> unexpected error to be incorrectly treated as expected error -- we
> don't want pip to be hiding real bugs in backend code. The argument
> for NotImplementedError is that it produces better error messages on
> buggy frontends. 'return None' is kind of the worst of both worlds, in
> that it's an easy thing to return accidentally, and it gives confusing
> error messages if the frontend fails to handle it properly. (Even more
> confusing, actually, because 'NoneType object has no attribute ...' is
> even harder to track down than 'NotImplementedType object has no
> attribute ...'.)
>
> > As regards an explicit description of what went wrong, why can't we
> > just use the same reporting methods that we will for any other build
> > issue (backends simply report the problem on stdout/stderr)? I don't
> > see why the backend has to package up its error information and send
> > it to the frontend to report, when we already have a perfectly
> > effective way for backends to report errors and/or warnings to the
> > user. If you're worried that the frontend might suppress the
> > information (maybe because it's planning on falling back to a direct
> > wheel build) then isn't that just the converse - backends need to
> > trust frontends to do the right thing?
>
> What I mean is more, if you're some random user and you see this in a
> build backend, what do you guess it means?
>
>   def get_requires_for_build_sdist(config_settings=None):
>   return None
>
> Now how about these?
>
>   def get_requires_for_build_sdist(config_settings=None):
>   return NotImplemented
>
>   def get_requires_for_build_sdist(config_settings=None):
>   raise NotImplementedError
>
>   def get_requires_for_build_sdist(config_settings=None):
>   raise SdistBuildNotSupported
>
> I mean, obviously return None will work. Basically anything that's
> different from "return a list or string" will work :-). That's what
> makes this a bikeshed topic, and I still think we're mostly just
> spinning our wheels here until Nick and Donald have a chance to hash
> something out that they both can agree on. But I really don't see any
> advantages to 'return None' compared to the other options that have
> been discussed
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

I also think that Guido pretty much ruled out Notimplemented.

On Aug 26, 2017 4:04 PM, "xoviat" <xov...@gmail.com> wrote:

> Why does the frontend need to know why an sdist was not created?
>
> I was of the opinion that such a distinction is not necessary because
> building a source distribution doesn't take that much time. However Donald
> thought that there needed to be a distinction because of the wasted time in
> attempting to build a wheel that was going to fail anyway. One of the
> things to consider is that site cythonizing takes time and maybe called for
> building source distribution. However since I think we're of the agreement
> that a source distribution should be as close to a checkout as possible,
> that may not be an issue because cythonizing may not be required to build
> the sdist.
>
> On Aug 26, 2017 3:47 PM, "C Anthony Risinger" <c...@anthonyrisinger.com>
> wrote:
>
>> On Aug 26, 2017 2:17 PM, "Nathaniel Smith" <n...@pobox.com> wrote:
>>
>>> [removed Guido from CC]
>>>
>>> On Aug 26, 2017 02:29, "Paul Moore" <p.f.mo...@gmail.com> wrote:
>>>
>>> On 26 August 2017 at 03:17, Guido van Rossum <gu...@python.org> wrote:
>>> > In pretty much any other context, if you have an operation that
>>> returns an
>>> > regular value or an error value, the error value should be None.
>>> (Exceptions
>>> > include e.g. returning a non-negative int or -1 for errors, or True for
>>> > success and False for errors.)
>>>
>>> So, given that build_sdist returns the path of the newly built sdist,
>>> the correct way to signal "I didn't manage to build a sdist" would be
>>> to return None.
>>>
>>> Now that it's put this way, it seems glaringly obvious to me that this
>>> is the correct thing to do.
>>>
>>>
>>> Eh... I would really prefer something that's (a) more explicit about
>>> what specifically went wrong, and (b) harder to return by accident. It's
>>> not at all obvious that if the list of requirements is 'None' that means
>>> 'this build supports making sdists in general but cannot make them from
>>> this source tree but might still be able to make a wheel'. And if you
>>> forget to put in a return statement, then python returns None for you,
>>> which seems like it could lead to some super confusing error modes.
>>>
>>
>> Why does the frontend need to know why an sdist was not created?
>>
>> Frontend is asking the backend, given the current state of the world, to
>> either produce an sdist, or not. Sans ahead-of-time knowledge (see below),
>> I would expect build_sdist to make some sanity checks about the world, then
>> make a binary choice about whether sdist creation is a valid goal. If not
>> possible, return None or NotImplemented or False or dict-of-reasons or
>> whatever. Only if creation was *attempted*, and in the exceptional event it
>> then failed, would I expect an Exception. We don't have structured
>> exceptions sadly so they can't really carry much useful information from a
>> protocol perspective above and beyond a simple None or the like anyway.
>>
>> I'd personally like to see some parity between build_sdist and
>> build_wheel in this regard. Maybe the disconnect here is we have a way to
>> specify hard reqs for building a wheel, statically or dynamically, and
>> build_wheel is expected to never fail, but no way to specify hard reqs
>> needed for build_sdist, necessitating this optional signaling path?
>>
>> If we had some definitive way for the frontend to know ahead of time if
>> build_sdist is even expected to work, it could be called with more
>> confidence.
>>
>> This could be a new sdist-related key in [build-system], a new table like
>> [sdist-system].requires, or making the get_requires_for_* less optional,
>> and defaulting to None instead of [ ].
>>
>> Frontend is responsible for prepping the world, so if it can't get a list
>> of reqs, somehow, for build_sdist, it knows it can't work. Same for
>> build_wheel, because you have to specify the backend itself, so there is at
>> least one requirement!
>>
>> Thus if you are a backend that can produce an sdist without additional
>> requirements beyond build reqs, you should explicitly return empty list
>> from get_requires_for_build_sdist.
>>
>> --
>>
>> C Anthony
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Why does the frontend need to know why an sdist was not created?

I was of the opinion that such a distinction is not necessary because
building a source distribution doesn't take that much time. However Donald
thought that there needed to be a distinction because of the wasted time in
attempting to build a wheel that was going to fail anyway. One of the
things to consider is that site cythonizing takes time and maybe called for
building source distribution. However since I think we're of the agreement
that a source distribution should be as close to a checkout as possible,
that may not be an issue because cythonizing may not be required to build
the sdist.

On Aug 26, 2017 3:47 PM, "C Anthony Risinger"  wrote:

> On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:
>
>> [removed Guido from CC]
>>
>> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
>>
>> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
>> > In pretty much any other context, if you have an operation that returns
>> an
>> > regular value or an error value, the error value should be None.
>> (Exceptions
>> > include e.g. returning a non-negative int or -1 for errors, or True for
>> > success and False for errors.)
>>
>> So, given that build_sdist returns the path of the newly built sdist,
>> the correct way to signal "I didn't manage to build a sdist" would be
>> to return None.
>>
>> Now that it's put this way, it seems glaringly obvious to me that this
>> is the correct thing to do.
>>
>>
>> Eh... I would really prefer something that's (a) more explicit about what
>> specifically went wrong, and (b) harder to return by accident. It's not at
>> all obvious that if the list of requirements is 'None' that means 'this
>> build supports making sdists in general but cannot make them from this
>> source tree but might still be able to make a wheel'. And if you forget to
>> put in a return statement, then python returns None for you, which seems
>> like it could lead to some super confusing error modes.
>>
>
> Why does the frontend need to know why an sdist was not created?
>
> Frontend is asking the backend, given the current state of the world, to
> either produce an sdist, or not. Sans ahead-of-time knowledge (see below),
> I would expect build_sdist to make some sanity checks about the world, then
> make a binary choice about whether sdist creation is a valid goal. If not
> possible, return None or NotImplemented or False or dict-of-reasons or
> whatever. Only if creation was *attempted*, and in the exceptional event it
> then failed, would I expect an Exception. We don't have structured
> exceptions sadly so they can't really carry much useful information from a
> protocol perspective above and beyond a simple None or the like anyway.
>
> I'd personally like to see some parity between build_sdist and build_wheel
> in this regard. Maybe the disconnect here is we have a way to specify hard
> reqs for building a wheel, statically or dynamically, and build_wheel is
> expected to never fail, but no way to specify hard reqs needed for
> build_sdist, necessitating this optional signaling path?
>
> If we had some definitive way for the frontend to know ahead of time if
> build_sdist is even expected to work, it could be called with more
> confidence.
>
> This could be a new sdist-related key in [build-system], a new table like
> [sdist-system].requires, or making the get_requires_for_* less optional,
> and defaulting to None instead of [ ].
>
> Frontend is responsible for prepping the world, so if it can't get a list
> of reqs, somehow, for build_sdist, it knows it can't work. Same for
> build_wheel, because you have to specify the backend itself, so there is at
> least one requirement!
>
> Thus if you are a backend that can produce an sdist without additional
> requirements beyond build reqs, you should explicitly return empty list
> from get_requires_for_build_sdist.
>
> --
>
> C Anthony
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

The current PEP requires that build_sdist return the basename of the sdist.
So forgetting a return statement is not an option unless people really
don't read the PEP.

On Aug 26, 2017 2:18 PM, "Nathaniel Smith"  wrote:

> [removed Guido from CC]
>
> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
>
> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
> > In pretty much any other context, if you have an operation that returns
> an
> > regular value or an error value, the error value should be None.
> (Exceptions
> > include e.g. returning a non-negative int or -1 for errors, or True for
> > success and False for errors.)
>
> So, given that build_sdist returns the path of the newly built sdist,
> the correct way to signal "I didn't manage to build a sdist" would be
> to return None.
>
> Now that it's put this way, it seems glaringly obvious to me that this
> is the correct thing to do.
>
>
> Eh... I would really prefer something that's (a) more explicit about what
> specifically went wrong, and (b) harder to return by accident. It's not at
> all obvious that if the list of requirements is 'None' that means 'this
> build supports making sdists in general but cannot make them from this
> source tree but might still be able to make a wheel'. And if you forget to
> put in a return statement, then python returns None for you, which seems
> like it could lead to some super confusing error modes.
>
> -n
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Should probably take Guido out of this email chain now. And does everyone
agree with what Nathaniel said on sys.path?

On Aug 26, 2017 11:00 AM, "Daniel Holth" <dho...@gmail.com> wrote:

I'm bored with that

On Sat, Aug 26, 2017, 11:59 xoviat <xov...@gmail.com> wrote:

> Is everyone on board with that?
>
> On Aug 26, 2017 4:29 AM, "Paul Moore" <p.f.mo...@gmail.com> wrote:
>
>> On 26 August 2017 at 03:17, Guido van Rossum <gu...@python.org> wrote:
>> > In pretty much any other context, if you have an operation that returns
>> an
>> > regular value or an error value, the error value should be None.
>> (Exceptions
>> > include e.g. returning a non-negative int or -1 for errors, or True for
>> > success and False for errors.)
>>
>> So, given that build_sdist returns the path of the newly built sdist,
>> the correct way to signal "I didn't manage to build a sdist" would be
>> to return None.
>>
>> Now that it's put this way, it seems glaringly obvious to me that this
>> is the correct thing to do.
>> Paul
>>
> ___
>
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Is everyone on board with that?

On Aug 26, 2017 4:29 AM, "Paul Moore"  wrote:

> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
> > In pretty much any other context, if you have an operation that returns
> an
> > regular value or an error value, the error value should be None.
> (Exceptions
> > include e.g. returning a non-negative int or -1 for errors, or True for
> > success and False for errors.)
>
> So, given that build_sdist returns the path of the newly built sdist,
> the correct way to signal "I didn't manage to build a sdist" would be
> to return None.
>
> Now that it's put this way, it seems glaringly obvious to me that this
> is the correct thing to do.
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Looks like Nick is on the losing side here, but we shall see.

On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ew...@canterbury.ac.nz> wrote:

> Oh, BDFL, will you settle whether we should raise an exception or return
> Notimplemented?
>
> On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ew...@canterbury.ac.nz> wrote:
>
> xoviat wrote:
>
>> I mean how is opening a file different than attempting to build an sdist?
>>
>
> 1. Opening a file is a very common operation.
>
> 2. Most file opens are expected to succeed, and if one doesn't,
> the appropriate place to deal with that is almost never at the
> site of the open call, but somewhere further up.
>
> In contrast, there's probably only about one place in any
> given frontend where the backend's build_sdist method gets
> invoked, and it's unlikely the ability to let a not-implemented
> exception bubble up from that point would be of great use.
>
>
> --
> Greg
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Oh, BDFL, will you settle whether we should raise an exception or return
Notimplemented?

On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ew...@canterbury.ac.nz> wrote:

xoviat wrote:

> I mean how is opening a file different than attempting to build an sdist?
>

1. Opening a file is a very common operation.

2. Most file opens are expected to succeed, and if one doesn't,
the appropriate place to deal with that is almost never at the
site of the open call, but somewhere further up.

In contrast, there's probably only about one place in any
given frontend where the backend's build_sdist method gets
invoked, and it's unlikely the ability to let a not-implemented
exception bubble up from that point would be of great use.


-- 
Greg

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

 > I don't agree that exceptions are the way to do "things"
in general. They're the way to so *some* things. The question
is whether the thing we're talking about is one of those
things.

I mean how is opening a file different than attempting to build an sdist?
open() could return true or false and not raise any exception. But it
doesn't. If you can't open the file, you get an exception. It's the same
idea except here we are asking for an sdist. We would greatly prefer to
have one. And if we don't have one, then we have to do something else to
handle the error.

This *is* Python. Sure, it may not be a good idea, though I think it's nice
most of the time. It may not even be accepted for this PEP. But, although I
don't have a strong opinion about whether it *should* be in the PEP, I do
have a strong opinion about whether it would be an exception to the norms
of Python, and my opinion is that it would. The fact that Nick, who I think
is the only core developer here, immediately jumped on this issue confirms
my suspicions here.

2017-08-25 20:00 GMT-05:00 Greg Ewing <greg.ew...@canterbury.ac.nz>:

> xoviat wrote:
>
>> I agree with Nick that exceptions are the way to do things in Python
>>
>
> I don't agree that exceptions are the way to do "things"
> in general. They're the way to so *some* things. The question
> is whether the thing we're talking about is one of those
> things.
>
> The reasoning behind this is that Python has in general adopted this
>> approach (Nick is right that they would have used NotImplementedError for
>> binary operations except for performance issues)
>>
>
> But only because NotImplemented is an out-of-band condition
> for operations in general. We're not talking about a general
> operation here, but a very specific one that only has a couple
> of possible results.
>
> --
> Greg
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Conditionless setup.py

[xoviat]

I personally do not understand the aversion to YAML. I mean yes, the
specification is more complicated, but it's also more popular and the YAML
files will not be complex enough for a C library to help that much. And
since it's more popular, people might even prefer specifying package
metadata in a pyproject.yaml. pip could even cache a wheel of the pyyaml
package between builds that could be imported at build time with a
zipimporter rather than vendoring the package. And as a plus it's not named
after an alleged sexist.

Honestly this is not an issue that interests me very much but this rant is
because I was surprised that toml was chosen when I first found out about
it.

2017-08-25 18:16 GMT-05:00 Nathaniel Smith :

> On Fri, Aug 25, 2017 at 1:00 PM, Jeremy Stanley  wrote:
> > (The
> > community around it is sensitive to gender diversity issues and
> > wants to avoid acquiring more of a "brogrammer" image, so some of us
> > worry that any conspicuous TOML files checked into revision control
> > repositories could be seen as a tacit endorsement of the author's
> > alleged behavior at GH a few years ago.)
>
> I was one of the folks championing TOML during the original
> discussions, and this is an issue that also worried me a lot. In case
> it's a useful data point: I actually contacted several of the main
> rust/cargo developers, since they were the major users of TOML and are
> also well known to be sensitive to these issues, to ask if they've had
> any issues with this, and they said that they haven't heard any
> complaints.
>
> Obviously there's a difference between "no-one complained" and "no-one
> was bothered", and I suspect the community's existing reputation may
> affect how this is interpreted as well, but... maybe useful as a data
> point.
>
> Between this and the way the TOML spec appears to have been abandoned
> at v0.4 (with the admonition "you should assume that is is unstable
> and act accordingly") I've wondered if we should fork it, rename it
> "the obvious minimal language", and release our own 1.0.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

For the record, I agree with the proposals made in Nathaniel's last two
emails.

2017-08-25 17:36 GMT-05:00 Nathaniel Smith <n...@pobox.com>:

> On Fri, Aug 25, 2017 at 2:26 PM, xoviat <xov...@gmail.com> wrote:
> >> I'm more or less persuaded by Nathaniel's argument that the source
> >> directory shouldn't be on sys.path
> >
> > I do too. There should be an option in pyproject.toml to disable this
> > behavior though so that numpy can build itself.
>
> My original proposal was to leave the srcdir off of sys.path, and then
> have a key in pyproject.toml like:
>
> [build-system]
> backend-python-path = ["."]
>
> Frontends would then do something like:
>
> os.chdir(srcdir)
> # This line is new to handle the above proposal:
> sys.path[:0] = [abspath(p) for p in
> config["build-system"].get("backend-python-path", [])]
> backend = resolve_entrypoint(config["build-system"]["build-backend"])
>
> I don't have a strong opinion on whether we put this into PEP 517
> (it's pretty light weight and doesn't interact with any other features
> AFAICT), or make it a followup PEP, or start out by deferring this
> option to a dedicated build backend, like:
>
> [build-system]
> requires = ["override_backend_path"]
> build-backend = "override_backend_path"
>
> [tool.override_backend_path]
> python-path = ["."]
> real-backend = "my_awesome:backend"
>
> These are all pretty similar.
>
> I think the big question for debate is: should sys.path be
> configurable, or not configurable? IIUC, the main argument for putting
> the source directory on the path was that extra configuration options
> are annoying, so we don't want one of those, but we do want to support
> in-tree backends (even though we expect that most projects won't use
> this), so we had better put the srcdir on sys.path.
>
> My feeling is that in practice, though, that the "no configuration,
> srcdir always on sys.path" approach is not going to hold up. So first,
> obviously, the hack above works just fine to make it configurable even
> if we don't put it in any PEP, so in fact it's configurable no
> matter what. Plus, sooner or later, someone's going to say "hey
> distutils-sig, I have a build backend package that I want to be able
> to bootstrap itself, AND I want to put my package inside src/ for
> reasons [1], wouldn't it be nice if I could put src/ on sys.path
> without jumping through hoops?". Or someone will release a new version
> of their build backend that adds a new dependency, or one of their
> transitive dependencies will release a new version that adds a new
> dependency, and it collides with some already-released package that
> uses that build-backend, and the project suffering the collision gets
> annoyed at being told they need to rearrange their source tree
> (retroactively, in already released versions!). And they'll come here
> and say "hey distutils-sig, can solve this problem once and for all?".
> And we'll be like... uh, fixing this would take what, <5 lines of new
> code in pip? Kinda hard to argue with that.
>
> So... it'll be configurable, one way or another.
>
> And if it's configurable... then the question is about whether the
> default configuration: is it srcdir on sys.path ("opt-out"), or srcdir
> not on sys.path ("opt-in"). And it seems to me that in this case, all
> the standard criteria say it should be opt-in.
>
> If it's opt-in, then everyone using build backends distributed on PyPI
> -- which we expect to be the vast majority of project -- never have to
> think about it and it does the right thing, with no risk of collisions
> or anything. In fact the only people who have to think about it are
> the ones implementing in-tree backends, and if you're already like,
> writing a whole backend and configuring your pyproject.toml to invoke
> it, then asking you to add one more line of configuration is really
> not a big deal.
>
> OTOH if it's opt-out, then it becomes Yet Another Bad Packaging
> Default, where conscientious package authors will fret about the risk
> of collisions and write blog posts about how every project needs to
> make sure to opt-out of this as a Best Practice, and I am so, so tired
> of those.
>
> -n
>
> [1] https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> I'm more or less persuaded by Nathaniel's argument that the source
directory shouldn't be on sys.path

I do too. There should be an option in pyproject.toml to disable this
behavior though so that numpy can build itself.

2017-08-25 16:21 GMT-05:00 Thomas Kluyver <tho...@kluyver.me.uk>:

> On Fri, Aug 25, 2017, at 10:02 PM, xoviat wrote:
>
> While we're waiting for others to respond, what are we going to do with
> respect to the sys.path issue? I don't think there has been discussion on
> that yet.
>
>
> I'm more or less persuaded by Nathaniel's argument that the source
> directory shouldn't be on sys.path, because:
>
> - Most projects won't need it to be
> - There are plausible ways it could go wrong
> - I think we could make a backend which loads the real hooks from the
> source dir, so there's a backdoor for projects that do want it.
>
> IIRC, however, Nick was still strongly in favour of including the source
> directory on sys.path, and he can usually persuade me of his position with
> a few emails. I do think that behaviour is more like the 'obvious thing'
> that Python programmers would expect.
>
> Leo:
>
> >  - removing srcdir from sys.path requires explicitly removing "" from
> the head of sys.path, which is not normally done.
>
> Not necessarily. If the hook process is launched with a script at the
> command line, it is the script directory, not the cwd, that is added to
> sys.path. My WIP hook-calling module works this way:
> https://github.com/takluyver/pep517/blob/ee43a9334c377d7c37badcc8527cb7
> a8500180f7/pep517/wrappers.py#L75
>
> With this code, we would actually need to explicitly *add* the source
> directory (CWD) to sys.path before importing the hook.
>
> > - it would break many setuptools based projects, probably forcing the
> setuptools backend to reinsert it, defeating the purpose of removing it in
> the first place.
>
> I don't think we should decide this based on what setuptools does; I
> expect it to need quite a bit of special handling in any case.
>
> Thomas
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Nathaniel:

What do you think of the proposal regarding DistutilsUnsupportedOperation?

2017-08-25 16:13 GMT-05:00 Nathaniel Smith :

> On Fri, Aug 25, 2017 at 9:49 AM, Thomas Kluyver 
> wrote:
> > Can I gently ask everyone involved to consider whether the
> > notimplemented/error discussion is verging into bikeshedding
> > (http://bikeshed.org/)?
> >
> > The technical arguments I have seen so far are:
> > - The exception can include a message
> > - The return value can't 'bubble up' from the internals of a hook like an
> > exception
>
> I believe Nick also feels that an important advantage of
> NotImplementedError is: if a frontend doesn't bother to properly
> implement the spec and special case NotImplemented, then you'll end up
> eventually getting some obscure error like "'NotImplementedType'
> object has no attribute ..." when it tries to treat it as a normal
> return value. With NotImplementedError, if the frontend doesn't treat
> it specially, the default is to treat it like other exceptions and
> show a proper traceback and error message. So lazy frontends give
> better UX for NotImplementedError than NotImplemented.
>
> Personally, I don't find the argument about lazy frontends terribly
> compelling because if we're assuming that we're hitting some buggy
> corner case with code not following the spec, then we should also
> consider the case of accidentally bubbled NotImplementedErrors.
> Between these two cases, an accidentally bubbled NotImplementedError
> causes even more confusing outcomes (the build frontend may silently
> start invoking other things! vs. a suboptimal error message), and it's
> harder to guard against (both designs require properly written
> frontends to contain a few lines of code special casing
> NotImplemented/NotImplementedError; only NotImplementedError also
> requires all careful *backends* to contain just-in-case try/except
> guards).
>
> Another minor point that's made me less happy with NotImplemented is:
> originally I thought we could make it a general fact about all the
> hooks that returning "NotImplemented" should be treated the same as if
> the hook were undefined. (Which is pretty much how it works for
> __binop__ methods.) On further consideration though I don't think this
> is a good idea. (Rationale: it's not really what we want for
> get_build_requires_for_sdist, & if we define future hooks that
> normally have no return value then there's a danger of buggy frontends
> missing it entirely, & it wouldn't have worked for Nick's suggestion
> that build_wheel(build_directory=foo) triggering a NotImplemented
> should fall back to build_wheel(build_directory=None), which is gone
> from the spec now but suggests that this could cause problems in the
> future.) So the bottom line of all this is that if we do go with
> NotImplemented, I now think it should only be a defined return value
> for get_requires_for_build_sdist and build_sdist, and should have
> special "sorry I can't do that Dave" semantics that are different from
> e.g. a missing get_requires_for_build_sdist hook. All of which will
> work fine, it's just less... aesthetically pleasing.
>
> Personally, I still have a weak preference for NotImplemented over
> NotImplementedError, but I don't care enough to have a big fight about
> it.
>
> It sounds like Nick and Donald are the only two folks who really have
> strong opinions here: can the two of you work something out? Should we
> flip a coin?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

While we're waiting for others to respond, what are we going to do with
respect to the sys.path issue? I don't think there has been discussion on
that yet.

2017-08-25 15:54 GMT-05:00 Nathaniel Smith <n...@pobox.com>:

> On Fri, Aug 25, 2017 at 1:51 PM, Thomas Kluyver <tho...@kluyver.me.uk>
> wrote:
> > On Fri, Aug 25, 2017, at 09:50 PM, xoviat wrote:
> >
> > > Genius!
> >
> >
> > 1% inspiration, 99% frustration :-P
>
> This joke is so clever that I fear we may be forced to implement the
> solution after all, just to punish Thomas.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Genius!

2017-08-25 15:47 GMT-05:00 Thomas Kluyver <tho...@kluyver.me.uk>:

> On Fri, Aug 25, 2017, at 09:41 PM, xoviat wrote:
>
> There's no need to return an instance of the exception class. As long as
> it is defined in the same name space as the hooks the front end will be
> able to access it.
>
>
> No, I meant NotImplementedError. As in:
>
> if i_cant_do_it:
>return NotImplementedError(msg)
>
> Thereby neatly annoying all sides of the debate at once.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

There's no need to return an instance of the exception class. As long as it
is defined in the same name space as the hooks the front end will be able
to access it.

On Aug 25, 2017 3:15 PM, "Thomas Kluyver" <tho...@kluyver.me.uk> wrote:

> What if hooks returned instances of the exception class? That way it
> doesn't bubble up from internal code, it can carry a message, and we can
> all agree that it's a semantically terrible idea that doesn't fit with any
> conventions.
>
> Even I'm not sure if this is a serious suggestion. ;-)
>
>
> On Fri, Aug 25, 2017, at 08:31 PM, xoviat wrote:
>
> Do you mean in addition to the two lines proposed here? I envision the
> typical use case as attempting to do something that is required to build an
> sdist deep inside the backend, discovering that it cannot be done, and then
> raising the proposed exception. That's (other than the two lines for the
> exception, which can put in Python later) shorter than catching some custom
> exception in the hook and then returning NotImplemented as a result. If you
> don't use exceptions, you'll probably need to unwind the stack manually
> with return values.
>
> 2017-08-25 14:25 GMT-05:00 Donald Stufft <don...@stufft.io>:
>
> It's silly to require every backend to write code to guard against
> possible issues when we can structure the API to prevent those issues.
>
> Sent from my iPhone
>
> On Aug 25, 2017, at 3:04 PM, xoviat <xov...@gmail.com> wrote:
>
> I agree with Nick that exceptions are the way to do things in Python and
> with Donald that in general, Python's use of exceptions has caused
> problems. But I don't think that this forum is the correct place to discuss
> Python conventions, and so I would ordinarily say that we should just
> accept that there will be problems and use the NotImplementedError,
> assuming that "solution" doesn't cause a firestorm that delays acceptance
> for a significant period of time. The reasoning behind this is that Python
> has in general adopted this approach (Nick is right that they would have
> used NotImplementedError for binary operations except for performance
> issues) even with its problems.
>
> However, another solution was dismissed without much thought: using the
> "UnsupportedOperation" exception. We could for now simply require an extra
> two lines in all backends:
>
> class DistutilsUnsupportedOperation(RuntimeError):
> pass
>
> and then put UnsupportedOperation in the "distutils" library in 3.7, with
> an eventual move to that. It's not pretty but there have been plenty of
> hacks for backwards compatibility so far (it's rare to see a project
> without some sort of compat.py file).
>
> 2017-08-25 13:34 GMT-05:00 Donald Stufft <don...@stufft.io>:
>
> The thing being bubbled up is a backend accidentally calling an API that
> has yet to be implemented (an error that should be reported) being bubbled
> up and erroneously handled as the backend reporting it doesn't support
> making a sdist (not an error, son no traceback).
>
> Sent from my iPhone
>
>
> > On Aug 25, 2017, at 2:23 PM, Paul Moore <p.f.mo...@gmail.com> wrote:
> >
> > There's little or no opportunity here for letting exceptions bubble up
> > to the user, or passing complex data back to the frontend. Ultimately,
> > it's pretty much immaterial which form of reporting is used.
>
> *___*
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Do you mean in addition to the two lines proposed here? I envision the
typical use case as attempting to do something that is required to build an
sdist deep inside the backend, discovering that it cannot be done, and then
raising the proposed exception. That's (other than the two lines for the
exception, which can put in Python later) shorter than catching some custom
exception in the hook and then returning NotImplemented as a result. If you
don't use exceptions, you'll probably need to unwind the stack manually
with return values.

2017-08-25 14:25 GMT-05:00 Donald Stufft <don...@stufft.io>:

> It's silly to require every backend to write code to guard against
> possible issues when we can structure the API to prevent those issues.
>
> Sent from my iPhone
>
> On Aug 25, 2017, at 3:04 PM, xoviat <xov...@gmail.com> wrote:
>
> I agree with Nick that exceptions are the way to do things in Python and
> with Donald that in general, Python's use of exceptions has caused
> problems. But I don't think that this forum is the correct place to discuss
> Python conventions, and so I would ordinarily say that we should just
> accept that there will be problems and use the NotImplementedError,
> assuming that "solution" doesn't cause a firestorm that delays acceptance
> for a significant period of time. The reasoning behind this is that Python
> has in general adopted this approach (Nick is right that they would have
> used NotImplementedError for binary operations except for performance
> issues) even with its problems.
>
> However, another solution was dismissed without much thought: using the
> "UnsupportedOperation" exception. We could for now simply require an extra
> two lines in all backends:
>
> class DistutilsUnsupportedOperation(RuntimeError):
> pass
>
> and then put UnsupportedOperation in the "distutils" library in 3.7, with
> an eventual move to that. It's not pretty but there have been plenty of
> hacks for backwards compatibility so far (it's rare to see a project
> without some sort of compat.py file).
>
> 2017-08-25 13:34 GMT-05:00 Donald Stufft <don...@stufft.io>:
>
>> The thing being bubbled up is a backend accidentally calling an API that
>> has yet to be implemented (an error that should be reported) being bubbled
>> up and erroneously handled as the backend reporting it doesn't support
>> making a sdist (not an error, son no traceback).
>>
>> Sent from my iPhone
>>
>> > On Aug 25, 2017, at 2:23 PM, Paul Moore <p.f.mo...@gmail.com> wrote:
>> >
>> > There's little or no opportunity here for letting exceptions bubble up
>> > to the user, or passing complex data back to the frontend. Ultimately,
>> > it's pretty much immaterial which form of reporting is used.
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

I agree with Nick that exceptions are the way to do things in Python and
with Donald that in general, Python's use of exceptions has caused
problems. But I don't think that this forum is the correct place to discuss
Python conventions, and so I would ordinarily say that we should just
accept that there will be problems and use the NotImplementedError,
assuming that "solution" doesn't cause a firestorm that delays acceptance
for a significant period of time. The reasoning behind this is that Python
has in general adopted this approach (Nick is right that they would have
used NotImplementedError for binary operations except for performance
issues) even with its problems.

However, another solution was dismissed without much thought: using the
"UnsupportedOperation" exception. We could for now simply require an extra
two lines in all backends:

class DistutilsUnsupportedOperation(RuntimeError):
pass

and then put UnsupportedOperation in the "distutils" library in 3.7, with
an eventual move to that. It's not pretty but there have been plenty of
hacks for backwards compatibility so far (it's rare to see a project
without some sort of compat.py file).

2017-08-25 13:34 GMT-05:00 Donald Stufft :

> The thing being bubbled up is a backend accidentally calling an API that
> has yet to be implemented (an error that should be reported) being bubbled
> up and erroneously handled as the backend reporting it doesn't support
> making a sdist (not an error, son no traceback).
>
> Sent from my iPhone
>
> > On Aug 25, 2017, at 2:23 PM, Paul Moore  wrote:
> >
> > There's little or no opportunity here for letting exceptions bubble up
> > to the user, or passing complex data back to the frontend. Ultimately,
> > it's pretty much immaterial which form of reporting is used.
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

The issue is about exceptions bubbling up to the interface level, not about
crashing pip. For example, if some class raises notimplemented error and
pip interprets that to mean it should call build_wheel.

On Aug 25, 2017 1:23 PM, "Paul Moore" <p.f.mo...@gmail.com> wrote:

> On 25 August 2017 at 18:06, xoviat <xov...@gmail.com> wrote:
> > Is pip going to fall back to building a wheel directly if any other
> error is
> > raised? That's what happens with setup.py install. If so, then it may be
> > fine if unexpected exceptions bubble up.
>
> Given that hooks need to be called in a subprocess (see
> https://www.python.org/dev/peps/pep-0517/#id15, "Frontends should call
> each hook in a fresh subprocess, so that backends are free to change
> process global state") there's no "bubbling up" involved at all. The
> frontend code would be something along the lines of
>
> hook_stub = '''
> import backend
> try:
> backend.build_sdist(...)
> except NotImplementedError:
> sys.exit(1)
> sys.exit(0)
> '''
> # Or...
> hook_stub = '''
> import backend
> if backend.build_sdist(...) == NotImplemented:
> sys.exit(1)
> sys.exit(0)
> '''
>
> if subprocess.call([sys.executable, "-c", hook_stub]) != 0:
> # We didn't build a sdist
>
> There's little or no opportunity here for letting exceptions bubble up
> to the user, or passing complex data back to the frontend. Ultimately,
> it's pretty much immaterial which form of reporting is used.
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Is pip going to fall back to building a wheel directly if any other error
is raised? That's what happens with setup.py install. If so, then it may be
fine if unexpected exceptions bubble up.

On Aug 25, 2017 11:57 AM, "Donald Stufft"  wrote:

>
> On Aug 25, 2017, at 12:49 PM, Thomas Kluyver  wrote:
>
> Can I gently ask everyone involved to consider whether the
> notimplemented/error discussion is verging into bikeshedding (
> http://bikeshed.org/)?
>
> The technical arguments I have seen so far are:
> - The exception can include a message
> - The return value can't 'bubble up' from the internals of a hook like an
> exception
>
> I don't think the discussion of semantics is going to go anywhere: they
> are both reasonable ways for the backend to reply "sorry, Dave, I can't do
> that".
>
>
>
> I don’t think they are both reasonable ways any more than it’s reasonable
> to do ``raise Return(value)`` instead of ``return value``. The semantics
> here are important because using exceptions for non-exceptional, non
> erroneous cases has *always*, in my experience, lead to weirdness [1].
>
> [1] Like for example, StopIteration which was deemed so bad as to need to
> break backwards compatibility and break consistency with all other uses of
> exceptions just to handle the weirdness in a saner way. Unfortunately we
> can’t modify the Python interpreter to fix our weirdness that is going to
> happen.
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

We could support both options on the frontend side. It's only a tiny bit of
duplication in pip.

On Aug 25, 2017 11:50 AM, "Thomas Kluyver" <tho...@kluyver.me.uk> wrote:

> Can I gently ask everyone involved to consider whether the
> notimplemented/error discussion is verging into bikeshedding (
> http://bikeshed.org/)?
>
> The technical arguments I have seen so far are:
> - The exception can include a message
> - The return value can't 'bubble up' from the internals of a hook like an
> exception
>
> I don't think the discussion of semantics is going to go anywhere: they
> are both reasonable ways for the backend to reply "sorry, Dave, I can't do
> that".
>
> On Fri, Aug 25, 2017, at 05:38 PM, xoviat wrote:
>
> According to the documentation, NotImplemented isn't appropriate either,
> as is for binary operations only. There is no one value that's taylor made
> for this situation, but an exception may be more appropriate as the
> underlying cause is probably an error.
>
> On Aug 25, 2017 11:11 AM, "Donald Stufft" <don...@stufft.io> wrote:
>
>
> On Aug 24, 2017, at 10:52 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
>
> Aye, I do, and it should be "raise NotImplementedError('Explanation of
> the failed check')"
>
> Rationale:
>
> - Python isn't C or Go, so we indicate failures with exceptions, not
> error codes (NotImplemented is an necessary performance hack for
> operand coercion in tight loops, not an example to be emulated in
> other APIs)
> - allows the backend to provide information on what went wrong
> - means an unhandled backend error results in a traceback pointing to
> where the build failed, not some later point in the frontend code
> - if a backend developer is sufficiently worried about accidentally
> propagating NotImplementedError that they want to pretend they're not
> writing Python any more, they can use this idiom:
>
>def public_hook_api(*args, **kwds):
>try:
>result, error_msg = _internal_hook_implementation(*args,
> **kwds)
>except NotImplementedError as exc:
>raise RuntimeError("Unexpected NotImplementedError") from exc
>if result is NotImplemented:
>raise NotImplementedError(error_msg)
>return result
>
> That provides the backend with all the same assurances against
> accidentally letting NotImplementedError escape that a return code
> based public API would, without frontends even needing to be aware of
> the backend developer's aversion to reporting errors as exceptions.
>
>
>
> I’m not really a fan of using NotImplementedError instead of
> NotImplemented. We’re not going to implement it by showing a traceback to
> where the NotImplementedError happened because it’s not an error case. And
> really that’s the important bit here, this is not an error case (as far as
> the API is concerned), this is just one of the possible return values that
> this function can produce.
>
> A front end may *choose* to make this an error case of course, but that is
> at a different layer than this API is operating.
>
> It’s arguably not even the correct usage of NotImplementedError, since
> that is (and I quote from the Python docs): "In user defined base classes,
> abstract methods should raise this exception when they require derived
> classes to override the method, or while the class is being developed to
> indicate that the real implementation still needs to be added.”
>
> This is not a case of some real implementation not having yet been added
> or some stub code getting called before it’s ready. This use case more
> closely resembles NotImplemented.
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
> *___*
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

By the way, what is the status on sys.path?

On Aug 25, 2017 11:45 AM, "Donald Stufft" <don...@stufft.io> wrote:

>
> On Aug 25, 2017, at 12:38 PM, xoviat <xov...@gmail.com> wrote:
>
> According to the documentation, NotImplemented isn't appropriate either,
> as is for binary operations only. There is no one value that's taylor made
> for this situation, but an exception may be more appropriate as the
> underlying cause is probably an error.
>
>
>
> The underlying cause is *not* an error just the same as it’s not an error
> for a __eq__ to not know how to test equality against a specific `other`.
> The underlying cause is explicitly “I do not want to or know how to handle
> this case” not “I’ve accidentally called some code that wasn’t implemented
> yet”. The use case is almost exactly the same as the binop use case.
>
>
> —
> Donald Stufft
>
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

According to the documentation, NotImplemented isn't appropriate either, as
is for binary operations only. There is no one value that's taylor made for
this situation, but an exception may be more appropriate as the underlying
cause is probably an error.

On Aug 25, 2017 11:11 AM, "Donald Stufft"  wrote:

>
> On Aug 24, 2017, at 10:52 AM, Nick Coghlan  wrote:
>
> Aye, I do, and it should be "raise NotImplementedError('Explanation of
> the failed check')"
>
> Rationale:
>
> - Python isn't C or Go, so we indicate failures with exceptions, not
> error codes (NotImplemented is an necessary performance hack for
> operand coercion in tight loops, not an example to be emulated in
> other APIs)
> - allows the backend to provide information on what went wrong
> - means an unhandled backend error results in a traceback pointing to
> where the build failed, not some later point in the frontend code
> - if a backend developer is sufficiently worried about accidentally
> propagating NotImplementedError that they want to pretend they're not
> writing Python any more, they can use this idiom:
>
>def public_hook_api(*args, **kwds):
>try:
>result, error_msg = _internal_hook_implementation(*args,
> **kwds)
>except NotImplementedError as exc:
>raise RuntimeError("Unexpected NotImplementedError") from exc
>if result is NotImplemented:
>raise NotImplementedError(error_msg)
>return result
>
> That provides the backend with all the same assurances against
> accidentally letting NotImplementedError escape that a return code
> based public API would, without frontends even needing to be aware of
> the backend developer's aversion to reporting errors as exceptions.
>
>
>
> I’m not really a fan of using NotImplementedError instead of
> NotImplemented. We’re not going to implement it by showing a traceback to
> where the NotImplementedError happened because it’s not an error case. And
> really that’s the important bit here, this is not an error case (as far as
> the API is concerned), this is just one of the possible return values that
> this function can produce.
>
> A front end may *choose* to make this an error case of course, but that is
> at a different layer than this API is operating.
>
> It’s arguably not even the correct usage of NotImplementedError, since
> that is (and I quote from the Python docs): "In user defined base classes,
> abstract methods should raise this exception when they require derived
> classes to override the method, or while the class is being developed to
> indicate that the real implementation still needs to be added.”
>
> This is not a case of some real implementation not having yet been added
> or some stub code getting called before it’s ready. This use case more
> closely resembles NotImplemented.
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Okay, but I would like to hear Paul's response to that first.

On Aug 25, 2017 12:54 AM, "Nick Coghlan" <ncogh...@gmail.com> wrote:

> On 25 August 2017 at 14:33, xoviat <xov...@gmail.com> wrote:
> > Just this morning, Paul said the following:
> >
> > That step's the problem. If the frontend does that it can potentially
> > be copying a lot of unneeded stuff (VCS history, for example). We
> > tried that with pip and it was a major issue. That problem is the
> > *whole point* of all the discussions about the various proposals that
> > ended up with build_tree.
> >
> > I took that to mean that we were trusting the backend to do the right
> thing.
> > And most people agreed with that. I don't personally care but there does
> > seem to be some miscommunication here.
>
> Folks are wanting the "right thing" to be defined in the spec, but
> that's the wrong place for it, since "the right thing" will be use
> case dependent. So I'm now saying "stop trying to define what the
> right thing is (beyond producing standards compliant artifacts), and
> instead just define a usable thing, run with it, and let consensus on
> 'the right thing' emerge over time".
>
> Donald and Paul originally didn't like that notion, but then Paul
> realised that pip had never actually tried implementing the "always
> build an sdist first" approach. That means pip's notion of "the right
> thing" can be adequately captured as "source tree -> sdist -> wheel"
> producing the same results as "source tree -> wheel", and pip can
> strongly encourage that behaviour in backend implementations by always
> trying build_sdist first, and only falling back to calling build_wheel
> directly if build_sdist raises NotImplementedError.
>
> Particular *frontends* (including pip) may further refine their
> definition of the "right thing" to categorically disallow source tree
> changes, but that's something they can do over time based on the bug
> reports they actually receive, rather than needing to be mandated a
> priori in the interface specification.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

And one more thing: it would probably be prescient to require a
get_backend_info hook that returns the name and url of the backend so that
pip can direct people where to file bugs when there are problems.

On Aug 24, 2017 11:52 PM, "xoviat" <xov...@gmail.com> wrote:

> Actually after reading the email that Nathaniel referenced, I disagree
> with Nick's position on in-tree builds. It's absolutely necessary because
> the tree may be read only and it's enforceable by pip through a simple
> recursive list.
>
> On Aug 24, 2017 11:33 PM, "xoviat" <xov...@gmail.com> wrote:
>
>> Just this morning, Paul said the following:
>>
>> That step's the problem. If the frontend does that it can potentially
>> be copying a lot of unneeded stuff (VCS history, for example). We
>> tried that with pip and it was a major issue. That problem is the
>> *whole point* of all the discussions about the various proposals that
>> ended up with build_tree.
>>
>> I took that to mean that we were trusting the backend to do the right
>> thing. And most people agreed with that. I don't personally care but there
>> does seem to be some miscommunication here.
>>
>> On Aug 24, 2017 11:24 PM, "Nathaniel Smith" <n...@pobox.com> wrote:
>>
>>> On Thu, Aug 24, 2017 at 9:17 PM, xoviat <xov...@gmail.com> wrote:
>>> > > I'm *not* OK with banning in-tree builds in the spec, since that's
>>> > > both unnecessary and unenforceable
>>> >
>>> > Well then either we can trust the backend or we cannot. If we can,
>>> then this
>>> > is both necessary and enforceable. If not, then we're back to pip
>>> copying
>>> > files. You can't make and argument that it's okay to trust build_sdist
>>> but
>>> > not build_wheel.
>>>
>>> I think at this point everyone has made their peace with the pip
>>> developers' decision that they want to keep copying files -- at least
>>> for now -- and that's just how it's going to be. This email has a more
>>> detailed discussion of the options, their "threat model", and the
>>> tradeoffs:
>>>
>>> https://mail.python.org/pipermail/distutils-sig/2017-July/03
>>> 1020.html
>>>
>>> I can see an argument for adding language saying that build_sdist
>>> SHOULD avoid modifying the source tree if possible, and MAY write
>>> scratch files to the sdist_directory.
>>>
>>> -n
>>>
>>> --
>>> Nathaniel J. Smith -- https://vorpus.org
>>>
>>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Actually after reading the email that Nathaniel referenced, I disagree with
Nick's position on in-tree builds. It's absolutely necessary because the
tree may be read only and it's enforceable by pip through a simple
recursive list.

On Aug 24, 2017 11:33 PM, "xoviat" <xov...@gmail.com> wrote:

> Just this morning, Paul said the following:
>
> That step's the problem. If the frontend does that it can potentially
> be copying a lot of unneeded stuff (VCS history, for example). We
> tried that with pip and it was a major issue. That problem is the
> *whole point* of all the discussions about the various proposals that
> ended up with build_tree.
>
> I took that to mean that we were trusting the backend to do the right
> thing. And most people agreed with that. I don't personally care but there
> does seem to be some miscommunication here.
>
> On Aug 24, 2017 11:24 PM, "Nathaniel Smith" <n...@pobox.com> wrote:
>
>> On Thu, Aug 24, 2017 at 9:17 PM, xoviat <xov...@gmail.com> wrote:
>> > > I'm *not* OK with banning in-tree builds in the spec, since that's
>> > > both unnecessary and unenforceable
>> >
>> > Well then either we can trust the backend or we cannot. If we can, then
>> this
>> > is both necessary and enforceable. If not, then we're back to pip
>> copying
>> > files. You can't make and argument that it's okay to trust build_sdist
>> but
>> > not build_wheel.
>>
>> I think at this point everyone has made their peace with the pip
>> developers' decision that they want to keep copying files -- at least
>> for now -- and that's just how it's going to be. This email has a more
>> detailed discussion of the options, their "threat model", and the
>> tradeoffs:
>>
>> https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html
>>
>> I can see an argument for adding language saying that build_sdist
>> SHOULD avoid modifying the source tree if possible, and MAY write
>> scratch files to the sdist_directory.
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Just this morning, Paul said the following:

That step's the problem. If the frontend does that it can potentially
be copying a lot of unneeded stuff (VCS history, for example). We
tried that with pip and it was a major issue. That problem is the
*whole point* of all the discussions about the various proposals that
ended up with build_tree.

I took that to mean that we were trusting the backend to do the right
thing. And most people agreed with that. I don't personally care but there
does seem to be some miscommunication here.

On Aug 24, 2017 11:24 PM, "Nathaniel Smith" <n...@pobox.com> wrote:

> On Thu, Aug 24, 2017 at 9:17 PM, xoviat <xov...@gmail.com> wrote:
> > > I'm *not* OK with banning in-tree builds in the spec, since that's
> > > both unnecessary and unenforceable
> >
> > Well then either we can trust the backend or we cannot. If we can, then
> this
> > is both necessary and enforceable. If not, then we're back to pip copying
> > files. You can't make and argument that it's okay to trust build_sdist
> but
> > not build_wheel.
>
> I think at this point everyone has made their peace with the pip
> developers' decision that they want to keep copying files -- at least
> for now -- and that's just how it's going to be. This email has a more
> detailed discussion of the options, their "threat model", and the
> tradeoffs:
>
> https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html
>
> I can see an argument for adding language saying that build_sdist
> SHOULD avoid modifying the source tree if possible, and MAY write
> scratch files to the sdist_directory.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

I'm *not* OK with banning in-tree builds in the spec, since that's
both unnecessary and unenforceable

Well then either we can trust the backend or we cannot. If we can, then
this is both necessary and enforceable. If not, then we're back to pip
copying files. You can't make and argument that it's okay to trust
build_sdist but not build_wheel.

On Aug 24, 2017 10:14 PM, "Nick Coghlan"  wrote:

> On 25 August 2017 at 10:08, Daniel Holth  wrote:
> > It would be simpler if the only requirement was to produce a correct
> wheel.
>
> +1
>
> To be clear on my current position:
>
> * due to the introduction of NotImplementedError, I'm now OK with
> explicitly out-of-tree builds being deferred to a later iteration of
> the API
> * I'm *not* OK with banning in-tree builds in the spec, since that's
> both unnecessary and unenforceable
>
> If a frontend really wants to ensure sdist/wheel consistency, then the
> way for it to do so is to build the sdist from the source tree, and
> then build the wheel from the sdist. Building a wheel directly from
> the source tree should ideally give the same results as building the
> sdist first, but it's explicitly OK for there to be cases where they
> end up being different (such as when build_sdist would raise
> NotImplementedError).
>
> As far as build_directory goes, the spec should just leave the
> question of artifact caching (or the lack thereof) up to the backend
> implementations for now - if/when we decide to add explicit artifact
> caching support to the API specification we can make a more informed
> decision at a future date based on what the initial generation of
> backends are actually doing.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

In case anyone is interested, here is a mostly correct implementation of
PEP 517 based on the behavior discussed here:

https://github.com/pypa/setuptools/pull/1039/files#diff-522bd9826e33902f7f58bd003c6a370c


As I said, there are a few items to be worked on still, but I really don't
think that it's too complicated.

2017-08-24 19:35 GMT-05:00 Nathaniel Smith :

> On Thu, Aug 24, 2017 at 6:11 AM, Thomas Kluyver 
> wrote:
> > Nathaniel seems to be busy with other things at the moment, so I hope he
> > won't mind me passing on this list of things he'd like to resolve with
> > the draft PEP. I'll quote his comments and put my responses inline.
>
> More like taking a break for mental health reasons, really, but I've
> been meaning to get back to it -- thanks for the nudge and I don't
> mind your posting it at all.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Nathaniel:

Are you okay with what has been proposed?

On Aug 24, 2017 7:29 PM, "Nathaniel Smith"  wrote:

> On Thu, Aug 24, 2017 at 7:52 AM, Nick Coghlan  wrote:
> >>> - I don't understand how out-of-tree builds and prepare_build_metadata
> >>> are supposed to interact.
> >
> > They don't, since the backend should only implement
> > prepare_build_metadata if it can generate the metadata without
> > actually triggering a full build of all the binary artifacts.
>
> The reason I had this on the list is that in many build systems, these
> two steps are intimately linked. E.g. in autoconf, "figure out which
> dependencies I'm going to use" (aka prepare_build_metadata) and "set
> up an out-of-tree build directory" are the same operation.
> Alternatively "sniff the system and config files etc. to figure out
> what what dependencies we're going to use" could be just another build
> target -- but then it still requires that the build directory be set
> up first.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

The current proposal is to remove build_dir and leave it up to the backend.

On Aug 24, 2017 7:07 PM, "Greg Ewing"  wrote:

> Paul Moore wrote:
>
>> 4. The point of all this is that the definition of build_directory
>> says "If build_directory is None, the backend may do an 'in place'
>> build which modifies the source directory and may produce different
>> results from those that would be obtained by exporting an sdist
>> first".
>>
>
> Is it really necessary for pip to specify exactly *where* to do
> an out-of-place build? Why should it care, as long as the end
> result is a clean wheel?
>
> --
> Greg
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Actually I submitted a compliant PR 3 months ago when the PEP was first
marked as accepted. Then it was marked as draft and I thought that I would
just wait for distutils-sig to handle it. And then I came over here to find
what was going on.

On Aug 24, 2017 5:33 PM, "xoviat" <xov...@gmail.com> wrote:

> I'm handling the setuptools backend. I will make sure that it complies
> with the clean directory.
>
> On Aug 24, 2017 5:30 PM, "Paul Moore" <p.f.mo...@gmail.com> wrote:
>
>> On 24 August 2017 at 20:00, Thomas Kluyver <tho...@kluyver.me.uk> wrote:
>> > So for the time being, I'd prefer to say that the build_wheel() hook
>> should
>> > not create or modify files in the source directory. Backends may keep
>> their
>> > own caches elsewhere to speed up repeated builds, but any more
>> provisions
>> > for developer use cases should be left for a future PEP. And developers
>> can
>> > invoke enscons/flit/... directly in the meantime.
>>
>> That means a PEP 517 compatible interface to setuptools is more
>> complex, as it would *have* to specify a temporary build directory.
>> And I'm not sure whether there are any *other* files it dumps in the
>> source directory (egg info comes to mind, and PKG-INFO) and if so
>> whether it's easy to get setuptools to place them elsewhere. That's
>> fine, as far as I'm concerned, but it should be made explicit, IMO.
>> I'm not yet sure who (if anyone) is proposing to write a PEP 517
>> wrapper for setuptools (or if we'll need to special-case setuptools in
>> pip for "legacy" support) but we should make it as easy as possible
>> for anyone who does to get the details right, as it's likely to be a
>> very widely used backend.
>>
>> It would be a lot easier to handle the edge cases here if there were
>> someone writing a setuptools wrapper. Any takers? (I'm not sure I can
>> make the time to do so, and I *am* sure that if I did, I don't have
>> the time to support and maintain it going forward...)
>>
>> Paul
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

I'm handling the setuptools backend. I will make sure that it complies with
the clean directory.

On Aug 24, 2017 5:30 PM, "Paul Moore"  wrote:

> On 24 August 2017 at 20:00, Thomas Kluyver  wrote:
> > So for the time being, I'd prefer to say that the build_wheel() hook
> should
> > not create or modify files in the source directory. Backends may keep
> their
> > own caches elsewhere to speed up repeated builds, but any more provisions
> > for developer use cases should be left for a future PEP. And developers
> can
> > invoke enscons/flit/... directly in the meantime.
>
> That means a PEP 517 compatible interface to setuptools is more
> complex, as it would *have* to specify a temporary build directory.
> And I'm not sure whether there are any *other* files it dumps in the
> source directory (egg info comes to mind, and PKG-INFO) and if so
> whether it's easy to get setuptools to place them elsewhere. That's
> fine, as far as I'm concerned, but it should be made explicit, IMO.
> I'm not yet sure who (if anyone) is proposing to write a PEP 517
> wrapper for setuptools (or if we'll need to special-case setuptools in
> pip for "legacy" support) but we should make it as easy as possible
> for anyone who does to get the details right, as it's likely to be a
> very widely used backend.
>
> It would be a lot easier to handle the edge cases here if there were
> someone writing a setuptools wrapper. Any takers? (I'm not sure I can
> make the time to do so, and I *am* sure that if I did, I don't have
> the time to support and maintain it going forward...)
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Fwd: Fwd: Re: PEP 517 again

[xoviat]

-- Forwarded message --
From: xoviat <xov...@gmail.com>
Date: 2017-08-24 14:26 GMT-05:00
Subject: Re: [Distutils] Fwd: Re: PEP 517 again
To: Thomas Kluyver <tho...@kluyver.me.uk>


And also, are Nick and Paul okay with this?

2017-08-24 14:24 GMT-05:00 xoviat <xov...@gmail.com>:

> So in summary:
> - NotImplementedError
> - Remove build_dir and specify that no modifications can be made to the
> source directory
>
> On the third issue:
>
> I understand where Nathaniel is coming from in regards to sys.path. But
> there should at least be an option for this because of for example, numpy:
> numpy builds itself with it's own build system that is defined in the
> source directory. If you remove '' from sys.path, then numpy cannot build
> itself because there is no other numpy.distutils package.
>
> 2017-08-24 14:00 GMT-05:00 Thomas Kluyver <tho...@kluyver.me.uk>:
>
>> On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote:
>>
>> It looks like a lot of trouble for a feature that is designed to solve a
>> problem for a very thin intersection of the Venn diagram of people who:
>>
>>  a. wants to quickly iterate while hacking a package
>>
>>  b. doesn't want to bother learning how to trigger the incremental build
>> system of that package or how to install it in editable mode.
>>
>> For better or worse, editable installs exist and don't need to be defined
>> by PEP 517 just to handle this case. I'm not convinced it's worth the
>> additional complexity of the spec just for that.
>>
>>
>> This is a good point. Given that developers can use the build tool
>> directly for development purposes, rather than going through the PEP 517
>> interface, we don't need to prioritise that use case for the spec.
>>
>> So for the time being, I'd prefer to say that the build_wheel() hook
>> should not create or modify files in the source directory. Backends may
>> keep their own caches elsewhere to speed up repeated builds, but any more
>> provisions for developer use cases should be left for a future PEP. And
>> developers can invoke enscons/flit/... directly in the meantime.
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

So in summary:
- NotImplementedError
- Remove build_dir and specify that no modifications can be made to the
source directory

On the third issue:

I understand where Nathaniel is coming from in regards to sys.path. But
there should at least be an option for this because of for example, numpy:
numpy builds itself with it's own build system that is defined in the
source directory. If you remove '' from sys.path, then numpy cannot build
itself because there is no other numpy.distutils package.

2017-08-24 14:00 GMT-05:00 Thomas Kluyver :

> On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote:
>
> It looks like a lot of trouble for a feature that is designed to solve a
> problem for a very thin intersection of the Venn diagram of people who:
>
>  a. wants to quickly iterate while hacking a package
>
>  b. doesn't want to bother learning how to trigger the incremental build
> system of that package or how to install it in editable mode.
>
> For better or worse, editable installs exist and don't need to be defined
> by PEP 517 just to handle this case. I'm not convinced it's worth the
> additional complexity of the spec just for that.
>
>
> This is a good point. Given that developers can use the build tool
> directly for development purposes, rather than going through the PEP 517
> interface, we don't need to prioritise that use case for the spec.
>
> So for the time being, I'd prefer to say that the build_wheel() hook
> should not create or modify files in the source directory. Backends may
> keep their own caches elsewhere to speed up repeated builds, but any more
> provisions for developer use cases should be left for a future PEP. And
> developers can invoke enscons/flit/... directly in the meantime.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

> It looks like a lot of trouble for a feature that is designed to solve a
problem for a very thin intersection of the Venn diagram of people who:

For the record, I don't agree at all that build_wheel_incremental should be
specified here. The suggestion is simply a compromise that I can tolerate
because it's optional.

> For better or worse, editable installs exist and don't need to be defined
by PEP 517 just to handle this case. I'm not convinced it's worth the
additional complexity of the spec just for that.

The editable installs in my opinion should have been specified in this PEP
because people are clearly trying to work around a lack of consensus on
that issue so that they don't have to wait for the next PEP and waste their
time waiting for uncached builds. I can't say I disagree with this but it's
trying to fit a square peg into a round hole.

> Is there any backend doing this "caching elsewhere" today? Is any backend
planning to?

Yes. Distutils caches in the build folder. It's very useful when working on
complex projects with Extensions.

> Can we arrive at some more concrete examples of how that would work? Like
how would pip be invoked, how would the backend react.

The reason that I'm willing to live with a separate optional function is
that I don't think it will ever be implemented. It will be a note in Python
history until install_editable is specified.

2017-08-24 13:30 GMT-05:00 Leonardo Rochael Almeida :

>
> On 24 August 2017 at 15:13, Thomas Kluyver  wrote:
>
>> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote:
>>
>> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
>> wrote:
>>
>> Is there a reason to ask for an 'unclean' build, though? There may be a
>> performance optimisation from reusing cached data,
>>
>>
>> For the same reason you would ever ask for incremental builds, to more
>> quickly iterate while hacking, imagining that you are using the PEP 517
>> interface to develop, perhaps to have a uniform interface to patch
>> something when you are not familiar with exactly the build system it uses.
>>
>> It looks like a lot of trouble for a feature that is designed to solve a
> problem for a very thin intersection of the Venn diagram of people who:
>
>  a. wants to quickly iterate while hacking a package
>
>  b. doesn't want to bother learning how to trigger the incremental build
> system of that package or how to install it in editable mode.
>
> For better or worse, editable installs exist and don't need to be defined
> by PEP 517 just to handle this case. I'm not convinced it's worth the
> additional complexity of the spec just for that.
>
> Yup, this is what I meant about using cached data for performance. But I
>> don't think this requires builds to modify the source directory. The build
>> system can cache data elsewhere on the filesystem, whether or not it's
>> given an explicit build_directory.
>>
>
> Is there any backend doing this "caching elsewhere" today? Is any backend
> planning to?
>
>
>> I think we can specify one kind of build that is both 'clean' (doesn't
>> modify the source directory) and incremental (can use cached data to speed
>> up the build). If that's workable, it seems like it would save a lot of
>> headaches rather than trying to specify them as two different options.
>>
>> We have explicitly excluded editable installs (i.e. inplace builds) from
>> this PEP, though we can add a hook for that in a later PEP. I agree with
>> you that this PEP should allow for fast incremental (but non-inplace)
>> builds if possible, though.
>>
>
> Can we arrive at some more concrete examples of how that would work? Like
> how would pip be invoked, how would the backend react. And how (and for
> whom) this would be better than either doing an editable install or
> straight clean install?
>
> I fear we're complicating the spec for something that will be hard to
> happen in practice.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

To be clear, is everyone okay with the following?

build_wheel(wheel_directory, config_settings=None,
metadata_directory=None): [REQUIRED]
build_wheel_incremental(wheel_directory, config_settings=None,
metadata_directory=None): [OPTIONAL]


I still think that the frontend should not provide a build_directory, but
if someone *really* wants the build_directory in the incremental function,
then I will "disagree and commit."

2017-08-24 13:13 GMT-05:00 Thomas Kluyver :

> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote:
>
> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
> wrote:
>
> Is there a reason to ask for an 'unclean' build, though? There may be a
> performance optimisation from reusing cached data,
>
>
> For the same reason you would ever ask for incremental builds, to more
> quickly iterate while hacking, imagining that you are using the PEP 517
> interface to develop, perhaps to have a uniform interface to patch
> something when you are not familiar with exactly the build system it uses.
>
>
> Yup, this is what I meant about using cached data for performance. But I
> don't think this requires builds to modify the source directory. The build
> system can cache data elsewhere on the filesystem, whether or not it's
> given an explicit build_directory.
>
> I think we can specify one kind of build that is both 'clean' (doesn't
> modify the source directory) and incremental (can use cached data to speed
> up the build). If that's workable, it seems like it would save a lot of
> headaches rather than trying to specify them as two different options.
>
> We have explicitly excluded editable installs (i.e. inplace builds) from
> this PEP, though we can add a hook for that in a later PEP. I agree with
> you that this PEP should allow for fast incremental (but non-inplace)
> builds if possible, though.
>
> Thomas
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

> For the same reason you would ever ask for incremental builds, to more
quickly iterate while hacking, imagining that you are using the PEP 517
interface to develop, perhaps to have a uniform interface to patch
something when you are not familiar with exactly the build system it uses.

And so I reiterate:

> I understand that, but what I disagree with is modifying build_wheel to
make up for a lack of consensus on editable installs.

I don't think the build_wheel hook should be used for iteration or
development, because the intended purpose is for production. And I don't
think we should even consider *any* specific deficiencies of setuptools or
wheel when designing a specification.

2017-08-24 12:20 GMT-05:00 Daniel Holth :

> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
> wrote:
>
>> On Thu, Aug 24, 2017, at 05:26 PM, Daniel Holth wrote:
>>
>> by including the build_dir parameter, a nearly universal build system
>> concept, pip gets an elegant way to ask for either a clean or unclean build.
>>
>>
>> Is there a reason to ask for an 'unclean' build, though? There may be a
>> performance optimisation from reusing cached data,
>>
>
> For the same reason you would ever ask for incremental builds, to more
> quickly iterate while hacking, imagining that you are using the PEP 517
> interface to develop, perhaps to have a uniform interface to patch
> something when you are not familiar with exactly the build system it uses.
>
>
>> but building in a separate directory doesn't preclude caching
>> intermediates somewhere else.
>>
>
> The argument is necessarily a hint to the build system? If it works it can
> do whatever.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

If the entire idea of copying out-of-tree is to work around setuptools
deficiencies, then perhaps it would be a better idea to push this onto the
setuptools build backend rather than bring these problems into PEP 517?

2017-08-24 10:32 GMT-05:00 xoviat <xov...@gmail.com>:

> May I ask what is wrong *in principle* with the setuptools "build" folder
> (other than the fact that it does not contain all tree changes)?
>
> 2017-08-24 10:27 GMT-05:00 xoviat <xov...@gmail.com>:
>
>> That is actually the first time that build_dir makes sense to me now.
>> Thank you.
>>
>> 2017-08-24 10:24 GMT-05:00 Paul Moore <p.f.mo...@gmail.com>:
>>
>>> On 24 August 2017 at 16:20, xoviat <xov...@gmail.com> wrote:
>>> >>  I *do* care about telling backends we don't want "different
>>> > results from those that would be obtained by exporting an sdist
>>> > first".
>>> >
>>> > I completely agree with this statement, but I don't believe that it
>>> can or
>>> > should be accomplished with this parameter. Let me just quote the
>>> process
>>> > that I proposed:
>>> >
>>> >
>>> >> Proposed process:
>>> > - Frontend copies source tree to temporary directory
>>>
>>> That step's the problem. If the frontend does that it can potentially
>>> be copying a lot of unneeded stuff (VCS history, for example). We
>>> tried that with pip and it was a major issue. That problem is the
>>> *whole point* of all the discussions about the various proposals that
>>> ended up with build_tree.
>>>
>>> > - Frontend invokes build-sdist to build an sdist
>>> > - Frontend extracts sdist to new temporary directory
>>> > - Frontend reloads backend from sdist directory and invokes build-wheel
>>>
>>> Paul
>>>
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

May I ask what is wrong *in principle* with the setuptools "build" folder
(other than the fact that it does not contain all tree changes)?

2017-08-24 10:27 GMT-05:00 xoviat <xov...@gmail.com>:

> That is actually the first time that build_dir makes sense to me now.
> Thank you.
>
> 2017-08-24 10:24 GMT-05:00 Paul Moore <p.f.mo...@gmail.com>:
>
>> On 24 August 2017 at 16:20, xoviat <xov...@gmail.com> wrote:
>> >>  I *do* care about telling backends we don't want "different
>> > results from those that would be obtained by exporting an sdist
>> > first".
>> >
>> > I completely agree with this statement, but I don't believe that it can
>> or
>> > should be accomplished with this parameter. Let me just quote the
>> process
>> > that I proposed:
>> >
>> >
>> >> Proposed process:
>> > - Frontend copies source tree to temporary directory
>>
>> That step's the problem. If the frontend does that it can potentially
>> be copying a lot of unneeded stuff (VCS history, for example). We
>> tried that with pip and it was a major issue. That problem is the
>> *whole point* of all the discussions about the various proposals that
>> ended up with build_tree.
>>
>> > - Frontend invokes build-sdist to build an sdist
>> > - Frontend extracts sdist to new temporary directory
>> > - Frontend reloads backend from sdist directory and invokes build-wheel
>>
>> Paul
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

That is actually the first time that build_dir makes sense to me now. Thank
you.

2017-08-24 10:24 GMT-05:00 Paul Moore <p.f.mo...@gmail.com>:

> On 24 August 2017 at 16:20, xoviat <xov...@gmail.com> wrote:
> >>  I *do* care about telling backends we don't want "different
> > results from those that would be obtained by exporting an sdist
> > first".
> >
> > I completely agree with this statement, but I don't believe that it can
> or
> > should be accomplished with this parameter. Let me just quote the process
> > that I proposed:
> >
> >
> >> Proposed process:
> > - Frontend copies source tree to temporary directory
>
> That step's the problem. If the frontend does that it can potentially
> be copying a lot of unneeded stuff (VCS history, for example). We
> tried that with pip and it was a major issue. That problem is the
> *whole point* of all the discussions about the various proposals that
> ended up with build_tree.
>
> > - Frontend invokes build-sdist to build an sdist
> > - Frontend extracts sdist to new temporary directory
> > - Frontend reloads backend from sdist directory and invokes build-wheel
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

> I know Daniel has been involved in the discussion and I don't think
he's raised any such objection about the PEP (and he developed
enscons, so he has direct experience of writing backends).

Daniel raised that specific objection.

2017-08-24 10:21 GMT-05:00 Paul Moore <p.f.mo...@gmail.com>:

> On 24 August 2017 at 16:15, xoviat <xov...@gmail.com> wrote:
> >> 2. I'm not completely clear on how pip's implementation will work - I
> > think the intention is to always build a sdist and build a wheel from
> > that, unless the backend reports it can't build a sdist, in which case
> > we ask it to build a wheel directly.
> >
> > This was the exact process that I proposed, but was told that:
> >
> >
> >> I agree that the way you want to do packaging is fundamentally
> >> incompatible
> > with build systems that do not resemble distutils. Perhaps since this is
> > distutils sig some here are too used to distutils as the only model of
> how
> > packaging might work. Beware that, when fighting monsters, you yourself
> do
> > not become a monster... for when you gaze long into the abyss. The abyss
> > gazes also into you.
>
> That wasn't me that said that.
>
> > There seems to be some miscommunication about the actual process being
> > proposed.
>
> Possibly - or there's some confusion about what you're proposing. I
> certainly have found your suggestions difficult to follow, and I don't
> recall you having said "we do sdist->wheel then fall back to
> requesting wheels directly". I also don't see how that's incompatible
> with systems that don't resemble distutils, so I suspect that whoever
> you quoted (which I think might have been Daniel) understood whatever
> you did say to mean something different.
>
> I know Daniel has been involved in the discussion and I don't think
> he's raised any such objection about the PEP (and he developed
> enscons, so he has direct experience of writing backends).
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

>  I don't recall you having said "we do sdist->wheel then fall back to
requesting wheels directly"

You're correct. I did not say that because build_sdist is required.

2017-08-24 10:21 GMT-05:00 Paul Moore <p.f.mo...@gmail.com>:

> On 24 August 2017 at 16:15, xoviat <xov...@gmail.com> wrote:
> >> 2. I'm not completely clear on how pip's implementation will work - I
> > think the intention is to always build a sdist and build a wheel from
> > that, unless the backend reports it can't build a sdist, in which case
> > we ask it to build a wheel directly.
> >
> > This was the exact process that I proposed, but was told that:
> >
> >
> >> I agree that the way you want to do packaging is fundamentally
> >> incompatible
> > with build systems that do not resemble distutils. Perhaps since this is
> > distutils sig some here are too used to distutils as the only model of
> how
> > packaging might work. Beware that, when fighting monsters, you yourself
> do
> > not become a monster... for when you gaze long into the abyss. The abyss
> > gazes also into you.
>
> That wasn't me that said that.
>
> > There seems to be some miscommunication about the actual process being
> > proposed.
>
> Possibly - or there's some confusion about what you're proposing. I
> certainly have found your suggestions difficult to follow, and I don't
> recall you having said "we do sdist->wheel then fall back to
> requesting wheels directly". I also don't see how that's incompatible
> with systems that don't resemble distutils, so I suspect that whoever
> you quoted (which I think might have been Daniel) understood whatever
> you did say to mean something different.
>
> I know Daniel has been involved in the discussion and I don't think
> he's raised any such objection about the PEP (and he developed
> enscons, so he has direct experience of writing backends).
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

>  I *do* care about telling backends we don't want "different
results from those that would be obtained by exporting an sdist
first".

I completely agree with this statement, but I don't believe that it can or
should be accomplished with this parameter. Let me just quote the process
that I proposed:


> Proposed process:
- Frontend copies source tree to temporary directory
- Frontend invokes build-sdist to build an sdist
- Frontend extracts sdist to new temporary directory
- Frontend reloads backend from sdist directory and invokes build-wheel

This process makes the most sense to me because there will be a limited
number of frontends AND the frontends will enforce correctness. I don't
think we should be too cautious about burdening the frontends will more
responsibility, because it will only be implemented once or twice. And to
clarify, the build_sdist is required.

2017-08-24 9:51 GMT-05:00 Paul Moore :

> On 24 August 2017 at 15:26, Thomas Kluyver  wrote:
> > I particularly want to hear from Nick, Daniel, Donald and Paul on their
> > understanding of the build_directory parameter. There were reasons why it
> > got added, and even if those reasons no longer seem so convincing to me,
> > it's only going to be removed again if we can reach a consensus on
> working
> > without it.
>
> OK. I've not had much time recently (RL issues) and so my apologies if
> I've forgotten context here. But my feelings are:
>
> 1. I've been pretty much persuaded that including features in the
> interface to defend against badly written backends is not worthwhile.
> (I still worry that it may be hard to provide good error reporting for
> backend errors, but that's a lesser issue).
> 2. I'm not completely clear on how pip's implementation will work - I
> think the intention is to always build a sdist and build a wheel from
> that, unless the backend reports it can't build a sdist, in which case
> we ask it to build a wheel directly.
> 3. I'm not particularly comfortable with backends saying that they can
> only produce a wheel and not a sdist, but accept that might have to be
> a reality in some cases. I would be *really* bothered by a backend
> that wasn't even willing to accept that sdist->wheel and direct wheel
> builds should be equivalent - but I don't know if that's something
> that is likely to happen (we can of course state that expectation in
> the PEP). The PEP seems to think this is possible, though[1] (see my
> next point).
> 4. The point of all this is that the definition of build_directory
> says "If build_directory is None, the backend may do an 'in place'
> build which modifies the source directory and may produce different
> results from those that would be obtained by exporting an sdist
> first". That's the bit that bothers me a lot - if we lose
> build_directory, we lose the option for pip to request that a backend
> produce a wheel "equivalent to producing an sdist first". And that
> capability is something I think pip should require[2]. I don't
> particularly care who specifies the actual directory (as per your
> comment, I thought someone else wanted this, for caching or something)
> but I *do* care about telling backends we don't want "different
> results from those that would be obtained by exporting an sdist
> first".
>
> Paul
>
> [1] The only example I can think of is flit where the user doesn't
> have git installed. But then I'd expect that a wheel generated
> directly in that case would be the same as if the user installed git
> then did sdist->wheel, so I'd still consider that to be maintaining
> the invariant I care about.
> [2] I'm not likely to be the one implementing pip's PEP 517 support,
> so I could be overridden on this, of course.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

> 2. I'm not completely clear on how pip's implementation will work - I
think the intention is to always build a sdist and build a wheel from
that, unless the backend reports it can't build a sdist, in which case
we ask it to build a wheel directly.

This was the exact process that I proposed, but was told that:


> I agree that the way you want to do packaging is fundamentally
incompatible
with build systems that do not resemble distutils. Perhaps since this is
distutils sig some here are too used to distutils as the only model of how
packaging might work. Beware that, when fighting monsters, you yourself do
not become a monster... for when you gaze long into the abyss. The abyss
gazes also into you.

There seems to be some miscommunication about the actual process being
proposed.


2017-08-24 10:07 GMT-05:00 Nick Coghlan :

> On 25 August 2017 at 00:51, Paul Moore  wrote:
> > 4. The point of all this is that the definition of build_directory
> > says "If build_directory is None, the backend may do an 'in place'
> > build which modifies the source directory and may produce different
> > results from those that would be obtained by exporting an sdist
> > first". That's the bit that bothers me a lot - if we lose
> > build_directory, we lose the option for pip to request that a backend
> > produce a wheel "equivalent to producing an sdist first". And that
> > capability is something I think pip should require[2]. I don't
> > particularly care who specifies the actual directory (as per your
> > comment, I thought someone else wanted this, for caching or something)
> > but I *do* care about telling backends we don't want "different
> > results from those that would be obtained by exporting an sdist
> > first".
>
> That's a phrasing error in the PEP - ideally, backends should always
> produce consistent results regardless of whether or not the frontend
> builds an sdist first. What that phrasing was attempting to
> acknowledge is that actually ensuring sdist/wheel consistency may have
> some extra requirements (like the git CLI) that may not apply to the
> "just give me a usable wheel, I don't care whether it's publication
> quality or not" case.
>
> The key thing that changed my mind on whether or not we needed to
> include the build_directory parameter from the start was actually the
> NotImplementedError discussion: as soon as we permitted backends to
> raise NotImplementedError for any non-None build_directory setting,
> we'd effectively made supporting the feature optional.
>
> And if supporting the feature's going to be optional anyway, then the
> *whole point* of defining the backend API in Python rather than as a
> CLI was to make it easy for us to add new optional methods later,
> which means we can start without build_directory support, and if we
> find we miss it for some specific reason, then we can add it back as a
> suitably named optional method like "build_wheel_variant" (if the
> use-case is cross-compiling for multiple ABIs without clobbering
> intermediate artifacts), or "build_wheel_with_artifact_cache" (if the
> use case is to allow the frontend to ask that the backend cache
> intermediate values in a particular place).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Thomas,

I did not receive the last message.

On Aug 24, 2017 9:31 AM, "Thomas Kluyver" <tho...@kluyver.me.uk> wrote:

> On Thu, Aug 24, 2017, at 03:29 PM, xoviat wrote:
>
> I understand that, but what I disagree with is modifying build_wheel to
> make up for a lack of consensus on editable installs.
>
>
> The reasoning for the build_directory parameter was not based around
> editable installs. This is the confusion I was referring to about the
> multiple meanings of 'out-of-tree builds'.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

I understand that, but what I disagree with is modifying build_wheel to
make up for a lack of consensus on editable installs.

On Aug 24, 2017 9:27 AM, "Thomas Kluyver" <tho...@kluyver.me.uk> wrote:

> On Thu, Aug 24, 2017, at 03:05 PM, xoviat wrote:
>
> During development, you want to use an editable install, which does modify
> the source tree.
>
>
> I think it was before you joined the discussion, but editable installs
> came up before and have deliberately been left out of PEP 517 because of
> the complexities. I am very keen that we do not go back into that
> discussion.
>
> I particularly want to hear from Nick, Daniel, Donald and Paul on their
> understanding of the build_directory parameter. There were reasons why it
> got added, and even if those reasons no longer seem so convincing to me,
> it's only going to be removed again if we can reach a consensus on working
> without it.
>
> Thomas
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

WRT the return value, I don't really care. But I am strongly in favor of
removing the build_dir option as I think that the entire premise is wrong.
During development, you want to use an editable install, which does modify
the source tree. And if you're just installing, then you don't care about
cache. And there is also nothing preventing pip from caching the temporary
directory, which may contain build artifacts.

On Aug 24, 2017 8:41 AM, "xoviat" <xov...@gmail.com> wrote:

> -- Forwarded message ------
> From: "xoviat" <xov...@gmail.com>
> Date: Aug 24, 2017 8:39 AM
> Subject: Re: [Distutils] PEP 517 again
> To: "Thomas Kluyver" <tho...@kluyver.me.uk>
> Cc:
>
> That's actually the general argument against exceptions and why golang
> doesn't have them. I have not seen notimplemented used in the wild ever
> though.
>
> Also, regarding the build_dir argument, it won't be used in pip. The pip
> devs may have said that but it's too much work to support now and most of
> them seem to have a shortage of time. Pip will be copying to a temporary
> directory for a long time.
>
> On Aug 24, 2017 8:28 AM, "Thomas Kluyver" <tho...@kluyver.me.uk> wrote:
>
>> On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote:
>>
>> I mean is this golang or Python? In Python, you raise notimplementederror.
>>
>>
>> But there's a NotImplemented singleton in Python as well. The argument
>> for using a return value is that the hook code has to deliberately return
>> that, whereas a NotImplementedError could bubble up from some internal
>> call, in which case it should really be registered as an error.
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Fwd: Re: PEP 517 again

[xoviat]

-- Forwarded message --
From: "xoviat" <xov...@gmail.com>
Date: Aug 24, 2017 8:39 AM
Subject: Re: [Distutils] PEP 517 again
To: "Thomas Kluyver" <tho...@kluyver.me.uk>
Cc:

That's actually the general argument against exceptions and why golang
doesn't have them. I have not seen notimplemented used in the wild ever
though.

Also, regarding the build_dir argument, it won't be used in pip. The pip
devs may have said that but it's too much work to support now and most of
them seem to have a shortage of time. Pip will be copying to a temporary
directory for a long time.

On Aug 24, 2017 8:28 AM, "Thomas Kluyver" <tho...@kluyver.me.uk> wrote:

> On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote:
>
> I mean is this golang or Python? In Python, you raise notimplementederror.
>
>
> But there's a NotImplemented singleton in Python as well. The argument for
> using a return value is that the hook code has to deliberately return that,
> whereas a NotImplementedError could bubble up from some internal call, in
> which case it should really be registered as an error.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

I mean is this golang or Python? In Python, you raise notimplementederror.

On Aug 24, 2017 8:17 AM, "Thomas Kluyver"  wrote:

> Nathaniel seems to be busy with other things at the moment, so I hope he
> won't mind me passing on this list of things he'd like to resolve with
> the draft PEP. I'll quote his comments and put my responses inline.
>
> > - needs some sort of NotImplemented(Error) handling specified
>
> We've gone round return-value vs exception a few times, but I don't
> think that debate is particularly heated, so it probably just needs
> someone to say "this is what we're doing". I can be that someone if
> needs be. But does anyone feel strongly about it?
>
> > - The out-of-tree build example (that makes an sdist and then builds
> > it) seems like exactly the kind of implementation that Donald says he
> > doesn't want to see? OTOH I think Nick said he wants to see a more
> > elaborated specification of out-of-tree build strategies with this
> > specifically as one of them.
> > - Nick has suggested that the to-be-defined NotImplemented(Error)
> > handling can be used by build_wheel to indicate that it can't do
> > out-of-tree builds, so maybe the frontend should try an in-tree build
> > instead. I don't know how to convert this into a real spec, though --
> > like in general, if I'm a frontend and I call `hook(*args, **kwargs)`
> > and it says "can't do that", then how do I know what the problem is
> > and what I should try instead?
> > - What happens if prepare_build_metadata returns NotImplemented /
> > raises NotImplementedError?
> > - I don't understand how out-of-tree builds and prepare_build_metadata
> > are supposed to interact.
> > - Also, AFAICT the out-of-tree build feature has no motivation
> > anymore. The only mentioned use case is to support incremental build
> > features in automatic build pipelines like Travis, but there are a
> > number of these build pipelines, and lots of existing build systems
> > that support out-of-tree builds, and AFAICT no-one actually uses them
> > together like this. (I think it's because most build systems'
> > out-of-tree build features are designed on the assumption that there's
> > a human running the show who will deal with any edge cases.)
>
> I believe the out-of-tree build option Nathaniel is referring to is the
> build_directory parameter to build_wheel(). His proposed APIs remove
> that parameter, and elsewhere in his email he describes that no-one
> seems to want it: everyone thinks someone else wants it.
>
> By my understanding, the reasons for including the build_directory
> parameter are:
>
> 1. Without an explicit build directory, the developers of pip are
> concerned that build backends will modify the source tree and cause
> issues which users report as bugs to pip.
> 2. A frontend-controlled build directory could be used for caching
> intermediate build artifacts. This was a secondary argument after we had
> the idea, and we've never really fleshed out how we expect it to work.
> There's also a concern that if the first round of frontends always use
> an empty tempdir, backends will end up relying on that. I think this
> second argument is a weak one unless we spend the time to figure out the
> details.
>
> Do other people see the situation in a similar way? How might we move
> forwards on this?
>
> > - needs some sort of conclusion to the srcdir-on-sys.path issue.
>
> While Nathaniel is in the minority regarding srcdir-on-sys.path, he
> argues that most of us are assuming a default position without really
> thinking through it, which is certainly true of me. I don't feel
> strongly about this topic, but I do want to come to a conclusion. As
> Nathaniel does feel strongly about it, does anyone object to either:
>
> A. Leaving the source dir off sys.path when loading the hooks, and
> making a special backend which exposes hooks from the CWD.
> B. Adding another key in the TOML file to control whether hooks can be
> loaded from the source dir.
>
> I can live with either, but I prefer A, because it means fewer options
> to describe in the spec.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Bootstrapping setuptools

[xoviat]

The OP was about "bootstrapping setuptools." If, as Leonardo said, we only
isolate on the first level (requirements from install_requires or user will
be isolated but requirements from build_requires will share the same
environment as the invoker of build_requires if they are not wheels), then
that opens up some interesting possibilities. However, I don't necessarily
think that this is a good idea because it opens up situations that could be
nasty to debug.

I still think the simpler method would be as I have said previously, to
have a very minimal backend that doesn't depend on anything and simply
archives the files and info. We can even put this backend into a separate
package that requires and depends on nothing, so that future build systems
don't have to re-invent the wheel here.

The question that I have is: how does flit address this issue? Does flit
require external dependencies to bootstrap itself?

2017-08-22 13:50 GMT-05:00 Jim Fulton <j...@jimfulton.info>:

> Oh, gawd. From now on, someone will have to say buildout 3 times before I
> appear.
>
> On Tue, Aug 22, 2017 at 2:06 PM, xoviat <xov...@gmail.com> wrote:
>
>> So are we going for a "primarily isolation" approach then where builds
>> are only isolated on the first level, but subsequent levels share the same
>> build environment?
>>
>
> I'm a huge fan of isolation, but isolation seems to me to be orthogonal to
> this discussion.
>
> I'm done with this thread. :)
>
> Jim
>
>
>>
>> 2017-08-22 10:23 GMT-05:00 Jim Fulton <j...@jimfulton.info>:
>>
>>> I didn't mention (nor do I recall anyone mentioning) venvs.
>>>
>>> Jim
>>>
>>>
>>> On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce <m...@nycresistor.com>
>>> wrote:
>>>
>>>> venvs within venvs... terrifying concept.
>>>>
>>>> On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton <j...@jimfulton.info>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth <dho...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Isn't this a special case of needing . on sys.path when building a
>>>>>> package? Then the same version of setuptools that is being packaged 
>>>>>> builds
>>>>>> itself.
>>>>>>
>>>>> No. The issue for us it wasn't setuptools itself, but it's
>>>>> dependencies and those dependencies conflicted with dependencies of of
>>>>> packages we were installing *and* those packages importing these
>>>>> dependences (indirectly) in their setup scripts.  Setup scripts that 
>>>>> import
>>>>> the thing they're about to install, generally to get the version :(, is
>>>>> something I'd love to see go away.
>>>>>
>>>>> Jim
>>>>>
>>>>> --
>>>>> Jim Fulton
>>>>> http://jimfulton.info
>>>>>
>>>>> ___
>>>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Jim Fulton
>>> http://jimfulton.info
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>
>
>
> --
> Jim Fulton
> http://jimfulton.info
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Bootstrapping setuptools

[xoviat]

So are we going for a "primarily isolation" approach then where builds are
only isolated on the first level, but subsequent levels share the same
build environment?

2017-08-22 10:23 GMT-05:00 Jim Fulton :

> I didn't mention (nor do I recall anyone mentioning) venvs.
>
> Jim
>
>
> On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce  wrote:
>
>> venvs within venvs... terrifying concept.
>>
>> On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton  wrote:
>>
>>>
>>>
>>> On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth  wrote:
>>>
 Isn't this a special case of needing . on sys.path when building a
 package? Then the same version of setuptools that is being packaged builds
 itself.

>>> No. The issue for us it wasn't setuptools itself, but it's dependencies
>>> and those dependencies conflicted with dependencies of of packages we were
>>> installing *and* those packages importing these dependences (indirectly) in
>>> their setup scripts.  Setup scripts that import the thing they're about to
>>> install, generally to get the version :(, is something I'd love to see go
>>> away.
>>>
>>> Jim
>>>
>>> --
>>> Jim Fulton
>>> http://jimfulton.info
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>
>
>
> --
> Jim Fulton
> http://jimfulton.info
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Bootstrapping setuptools

[xoviat]

This statement comes from something that Donald said:

> The unvendoring means that setuptools and the project code are now
competing over who gets to define what an acceptable version is for these
libraries to be installed with.

As if this isn't going to be true for any other build system, and
setuptools won't at all be in any way special in a PEP 517 world. So you
are always going to have complaints about "well build system X required a
newer version but my project requires an older version" when the real
problem is that people are requiring an older version/ projects do not have
a stable API.

2017-08-21 16:27 GMT-05:00 Jim Fulton <j...@jimfulton.info>:

>
>
> On Mon, Aug 21, 2017 at 5:17 PM, xoviat <xov...@gmail.com> wrote:
>
>> Of course, to be frank, the principle failure with this plan is that
>> third-party tools (buildout, os packagers) will not be compliant with PEP
>> 517 even after it is adopted, and will then complain about having to update
>> their build systems.
>>
>
> I don't understand this statement. If buildout needs to be compliant, it
> will be (eventually).  I can imagine a future where everything is
> distributed with wheels and buildout wouldn't need to build anything.
>
> Jim
>
>
>>
>> 2017-08-21 16:05 GMT-05:00 xoviat <xov...@gmail.com>:
>>
>>> Previously, the attempt to move setuptools off of vendored dependencies
>>> failed because it was not done correctly: install_requires was set to the
>>> vendored dependencies but not setup_requires, which would have been
>>> required to correctly specify dependencies. However, setup_requires would
>>> have introduced circular dependencies that are impossible to correctly
>>> resolve, so that would have simply shifted an impossible problem onto pip.
>>>
>>> The principle issue then, is that setuptools requires setuptools to
>>> build itself. And although, this problem was previously not worth solving
>>> because of 'setup.py install', it is now not a difficult problem to solve
>>> if we rethink slightly what should be included in the setuptools
>>> respository:
>>> - Rather than re-generating egg_info on each setup.py invocation, we can
>>> cache dist-info in the setuptools respository.
>>> - We can implement a simple entry_points script that updates dist-info
>>> based on the contents of setup.py, which is only runnable of course after
>>> setuptools is installed
>>> - We can implement the reference build backend for setuptools: simply
>>> copy the files and dist-info into a new compliant zip archive
>>>
>>> Viola! Setuptools is no longer required to build setuptools, and the
>>> installation process is fully compliant with Python PEPs.
>>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>
> --
> Jim Fulton
> http://jimfulton.info
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig