Re: [Distutils] PEP 517 again
Hi, On Tue, Aug 29, 2017 at 8:30 PM, xoviat wrote: >> For me, using NotImplemented is a misuse of the singleton since I know >> what it's meant to be used for (and so I cringe every time I hear it brought >> up as a solution). > > I hate to reiterate too much, but I think that you are correct here: > NotImplemented is not Pythonic. It's a complete hack that I would think > people would like to quarantine to the limited use case of binary > operations. Even with the poorly contextualized email that Guido received, I > think it was clear that he thought it wasn't appropriate outside of that use > case either (I don't think better context would have changed that). > > Personally, I was willing to overlook that for the sake of a speedy > resolution. But reading the email chain, it seems that only Thomas and > Daniel are currently in favor of using NotImplemented, which is a shift from > the earlier situation. So now it seems that NotImplemented is not likely to > be used. Just for the record, I thought Nathaniel's NotImplemented suggestion was a good one, and his explanation of why, was clear and convincing, but he's already accepted the alternative, of a custom error, so I don't think there's much point on going back over that. Also just for the record, I'd like to warmly thank y'all for your great patience in this long discussion. Cheers, Matthew ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
> For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution). I hate to reiterate too much, but I think that you are correct here: NotImplemented is not Pythonic. It's a complete hack that I would think people would like to quarantine to the limited use case of binary operations. Even with the poorly contextualized email that Guido received, I think it was clear that he thought it wasn't appropriate outside of that use case either (I don't think better context would have changed that). Personally, I was willing to overlook that for the sake of a speedy resolution. But reading the email chain, it seems that only Thomas and Daniel are currently in favor of using NotImplemented, which is a shift from the earlier situation. So now it seems that NotImplemented is not likely to be used. 2017-08-29 14:07 GMT-05:00 Paul Moore : > On 29 August 2017 at 19:45, Chris Barker wrote: > > But it would be nice if packages didn't NEED to be pip installed -- but I > > think that's an issue with the wheel spec (that may well be solved). > > Wheels don't need pip to install them - the spec has always been clear > on how to "manually" install a wheel (unzip it and move a few things > around), and the wheel package itself has a simple "wheel install" > command. > > If you want to create wrapper scripts for entry points, you need to do > a bit more work, but distlib offers that functionality (that's what > pip uses). > > One point about PEP 517 is that it provides a standard way to ask > *any* build system to make a wheel. So it's in theory possible to > install any package without pip (of course, if you try to handle all > the various corner cases, extra quirks, etc that come up in the real > world, you end up reinventing pip ;-)). > > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On 29 August 2017 at 19:45, Chris Barker wrote: > But it would be nice if packages didn't NEED to be pip installed -- but I > think that's an issue with the wheel spec (that may well be solved). Wheels don't need pip to install them - the spec has always been clear on how to "manually" install a wheel (unzip it and move a few things around), and the wheel package itself has a simple "wheel install" command. If you want to create wrapper scripts for entry points, you need to do a bit more work, but distlib offers that functionality (that's what pip uses). One point about PEP 517 is that it provides a standard way to ask *any* build system to make a wheel. So it's in theory possible to install any package without pip (of course, if you try to handle all the various corner cases, extra quirks, etc that come up in the real world, you end up reinventing pip ;-)). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
> conda-build doesn't produce or consume wheels. It works by creating a > clean > > conda environment and running a shell script to install the python > package > > into that environment To be really clear -- conda build doesn't directly use ANY build/install system. All files produced by the install > > script are then put into a tar file, which is conda's version of a wheel. > right -- conda-build simply runs the script it is told to run (by the recipe), and then looks and sees what files were installed. In theory, one could build a conda package directly out of a binary wheel -- after all, as mentioned, they are actually pretty similar, but most folks would probably just use pip inside their conda-build recipe. In fact, that is already being done -- most python packages are built with "setup.py install", but some are built by using pip to install a wheel. But I didn't think that the goal of PEP 517 was to ensconce pip as the only way to install a package in Python. So if a build system can install (like setuptools currently does), then conda recipes may well call it directly. I don't think this really has any bearing on the the PEP, though -- As I understand it, the goal of the PEP is to formalize the interface between build systems and package managers -- and that can only be a good thing for other package managers. NOTE: For the most part, folks want to use a pip-compatible install in conda-build, because users tend to also directly use pip to install stuff in conda environments e.g. if there is not a conda package available. If a conda package were built with, say, plain distutils, then pip won't find the meta data it's looking for, and might falsely see an unresolved dependency. Personally, I've always avoided pip installing packages in conda -- preferring to make conda packages for all the deps I need, but conda-pip compatibility has come a long way, so it does work fine for many users. But it would be nice if packages didn't NEED to be pip installed -- but I think that's an issue with the wheel spec (that may well be solved). -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On 29 August 2017 at 17:22, Phil Austin wrote: > On 2017-08-29 01:20, Paul Moore wrote: >> >> On 29 August 2017 at 01:13, Chris Barker wrote: >>> If the build API is clean and supported enough, I can see conda using it. >>> >>> just sayin' >> >> >> Thanks - that's really good to know. So *is* the build API clean >> enough for you? Specifically: >> >> 1. build_sdist can be missing, or can report back (somehow) that it >> failed to produce a sdist, and you should fall back to build_wheel. >> 2. build_wheel must always succeed and must always exist. >> 3. All hooks must be run in a separate subprocess >> >> There's also error handling, which I don't recall the details of, but >> I think boils down to the backend code can write what it likes to >> stdout/stderr and may raise an exception indicating "something blew >> up". >> >> Those are basically the key points at the moment. Add to that, does >> conda mind if build_wheel might result in a different wheel than >> build_sdist->build_wheel would produce? > > conda-build doesn't produce or consume wheels. It works by creating a clean > conda environment and running a shell script to install the python package > into that environment (including its own versions of /usr/lib /usr/etc > /usr/bin /urs/include /sbin -- which is why it can cope with non-python > packages like g++, julia, R, git etc.). All files produced by the install > script are then put into a tar file, which is conda's version of a wheel. OK, so I don't see how conda would use PEP 517 the way Chris suggested. But never mind - if it doesn't help conda that's fine. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On 2017-08-29 01:20, Paul Moore wrote: On 29 August 2017 at 01:13, Chris Barker wrote: On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore wrote: Me too. At the moment, I only expect two backends of any substance - your flit backend and xoviat's setuptools one. But I only know of one frontend, namely pip - and talk of projects like tox or twine acting as frontends never seems to get any traction. scikit-build also expressed some interest: https://www.mail-archive.com/distutils-sig@python.org/msg26553.html although I don't see the pull request. If the build API is clean and supported enough, I can see conda using it. just sayin' Thanks - that's really good to know. So *is* the build API clean enough for you? Specifically: 1. build_sdist can be missing, or can report back (somehow) that it failed to produce a sdist, and you should fall back to build_wheel. 2. build_wheel must always succeed and must always exist. 3. All hooks must be run in a separate subprocess There's also error handling, which I don't recall the details of, but I think boils down to the backend code can write what it likes to stdout/stderr and may raise an exception indicating "something blew up". Those are basically the key points at the moment. Add to that, does conda mind if build_wheel might result in a different wheel than build_sdist->build_wheel would produce? conda-build doesn't produce or consume wheels. It works by creating a clean conda environment and running a shell script to install the python package into that environment (including its own versions of /usr/lib /usr/etc /usr/bin /urs/include /sbin -- which is why it can cope with non-python packages like g++, julia, R, git etc.). All files produced by the install script are then put into a tar file, which is conda's version of a wheel. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On Mon, 28 Aug 2017 at 16:29 Nathaniel Smith wrote: > On Mon, Aug 28, 2017 at 1:27 PM, Thomas Kluyver > wrote: > > On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote: > > > > > Then end the debate by letting the PEP authors decide the return type, > and > > > write a paragraph explaining why the other options were rejected. It > is not > > > going to make a big difference. > > > > > > Will that work now? Are we all so tired of this endless war that people > will > > sign a peace treaty written by the people whose names are on the PEP > > (Nathaniel & me)? > > > > If so, let the trumpets sound, and the heralds declare that "return > > NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's > > preference right ;-) > > Fine with me, though if it turns out Donald and Nick prefer the > version where the backend has to export an exception class then I'm > fine with that too. (I'm basing this on -- Donald has said he likes > it, and Nick hasn't commented yet but AFAICT it does address his > concerns with NotImplemented, so it seem like a plausible outcome.) > I loathe to weigh in on this and add yet another voice in this discussion, but the exported exception seems like the best solution for everyone involved from my lurking perspective. For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution). And I was fine with NotImplementedError but if people want something more specific and None is out due to worries of accidental bare returns, then the exported exception comes the closest to making everyone happy (it does tick the EIBTI box :) . ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On 29 August 2017 at 01:13, Chris Barker wrote: > On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore wrote: >> >> Me too. At the moment, I only expect two backends of any substance - >> your flit backend and xoviat's setuptools one. But I only know of one >> frontend, namely pip - and talk of projects like tox or twine acting >> as frontends never seems to get any traction. > > > If the build API is clean and supported enough, I can see conda using it. > > just sayin' Thanks - that's really good to know. So *is* the build API clean enough for you? Specifically: 1. build_sdist can be missing, or can report back (somehow) that it failed to produce a sdist, and you should fall back to build_wheel. 2. build_wheel must always succeed and must always exist. 3. All hooks must be run in a separate subprocess There's also error handling, which I don't recall the details of, but I think boils down to the backend code can write what it likes to stdout/stderr and may raise an exception indicating "something blew up". Those are basically the key points at the moment. Add to that, does conda mind if build_wheel might result in a different wheel than build_sdist->build_wheel would produce? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 - a plan of action
On 27 August 2017 at 21:03, Thomas Kluyver wrote: > Thanks Paul for providing us with a way to move forwards. I'm in favour > of what you propose. > > On Sun, Aug 27, 2017, at 11:37 AM, Paul Moore wrote: >> 2. Thomas to complete and publish his library of helper functions for >> PEP 517 consumers (optional - but I think he said he had such a thing >> in progress). > > I do. It's not entirely up-to-date with the spec, but it can be found > here: > https://github.com/takluyver/pep517 > > My plan is to provide two levels of API: > 1. Lower-level machinery to easily call PEP 517 hooks in a subprocess > (this is what I've been writing so far) > 2. A higher-level 'give me a wheel' interface which identifies the build > dependencies, installs them into a build environment, and calls the > build_wheel hook. Alternatively, this could be a separate package. > > I'd like to transfer this repo to PyPA once it's a bit more ready - > assuming PyPA wants it. > >> 3. Build PEP 517 interfaces for flit and setuptools. > > The work for flit is well out of date, but it can be found here: > https://github.com/takluyver/flit/tree/toml-config > >> 1. On any remaining questions for the PEP, let's just go for the >> simplest solution for now. Let the PEP authors decide what that is, >> with Nick as BDFL arbitrating if there's disagreement. > > I believe that the remaining questions are: > - srcdir-on-sys.path > - Notimplemented/Error (/None) > > Unfortunately, it's not clear to me what's the 'simplest' solution on > either, and we can't easily change whichever decision we make. Nathaniel > & Nick still disagree on srcdir-on-sys.path, I believe; I'm not sure how > we resolve that. I've actually found the arguments in favour of leaving the source directory off sys.path by default fairly persuasive, as: 1. We have enough problems with Python's default sys.path[0] handling that I'm entirely sympathetic to the desire to omit it 2. If omitting it is genuinely a problem, we'll likely find out soon enough as part of implementing a setup.py backend 3. The `tools` section in pyproject.toml already offers a way to enable bootstrapping of a meta-backend that does `sys.path.insert(0, os.getcwd())` before loading a bundled backend So I think we can deem this one resolved in favour of "Frontends must ensure the current directory is *NOT* on sys.path before importing the designated backend", as starting there will mean we maximise our chances of learning something new as part of the initial rollout of the provisionally accepted API. > I still think the NotImplemented/Error issue is a prime example of > bikeshedding. There clearly isn't one 'right' way to do it, because > intelligent people continue to disagree about it. I'd like somebody to > make the decision and end the argument, but I'm not sure who is 'above > the fray' enough to be able to do that. --- Note: TL;DR of the below recap is that I'd accept a PEP that used any of the following three designs: - backends optionally define an AnticipatedBuildFailure exception for frontends to catch (preferred) - anticipated build failures are indicated by raising NotImplementedError - anticipated build failures are indicated by returning None I *wouldn't* accept a return value based design that used NotImplemented as the special return value So the choice amongst those 3 options then devolves to the folks actually planning to do the work - while it's my least preferred of the three, an implemented None-return based API is nevertheless immeasurably more useful than a purely theoretical exception based API. --- Taking another go at recapping this particular design question from my perspective: - the main point of calling the PEP 517 artifact creation hooks is for their side effect of actually creating the artifact, *not* their Python return value - as such, if they fail to create the requested artifact, they should raise an exception, just like any other state mutation operation - however, it's possible for artifact creation to fail for anticipated reasons, which can be signaled by raising a specific kind of exception describing that reason - we don't want to couple the API specification to any particular library, so we either need to choose an existing builtin exception, or else let the backend declare the exception it will use In other words, if I call an API to say "create this artifact", and I don't get an artifact out the other end, then that operation failed. It may have an anticipated reason for failing, and I may want to handle those anticipated failures differently from unexpected ones (so it's worth defining a protocol for distinguishing them), but it still failed. In the standard library, probably the best exemplar of this API design pattern would be the use of UnsupportedOperation in the `io` module base classes: https://docs.python.org/3/library/io.html#io.UnsupportedOperation - from the point of view of a running application, what matters about UnsupportedOperation i
