Re: [Distutils] PEP 517 again
On 26 August 2017 at 02:17, xoviat wrote: > The fact that Nick, who I think is > the only core developer here, immediately jumped on this issue confirms my > suspicions here. *ahem* Donald and myself are both core devs too. And possibly others, I honestly don't know. (Not that I think it makes much difference - arguments should be taken on their merit, not on who made them). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On 26 August 2017 at 03:17, Guido van Rossum wrote: > In pretty much any other context, if you have an operation that returns an > regular value or an error value, the error value should be None. (Exceptions > include e.g. returning a non-negative int or -1 for errors, or True for > success and False for errors.) So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None. Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Conditionless setup.py
Me too. YAML is *so much* more widely used, and the complicated edge cases can simply be ignored for this requirement. Maybe it's just because I've never heard of whatever improper behavior the author engaged in, but I don't think a data format needs to answer to the actions of its creator(s). If G. Klyne or C. Newman were to do something dreadful I wouldn't want dates to stop following ISO-8601.[*] [*] I don't even know the 8601 authors' first names, and assume they are good and honorable people. Just making an analogy. On Aug 25, 2017 5:46 PM, "xoviat" wrote: > I personally do not understand the aversion to YAML. I mean yes, the > specification is more complicated, but it's also more popular and the YAML > files will not be complex enough for a C library to help that much. And > since it's more popular, people might even prefer specifying package > metadata in a pyproject.yaml. pip could even cache a wheel of the pyyaml > package between builds that could be imported at build time with a > zipimporter rather than vendoring the package. And as a plus it's not named > after an alleged sexist. > > Honestly this is not an issue that interests me very much but this rant is > because I was surprised that toml was chosen when I first found out about > it. > > 2017-08-25 18:16 GMT-05:00 Nathaniel Smith : > >> On Fri, Aug 25, 2017 at 1:00 PM, Jeremy Stanley >> wrote: >> > (The >> > community around it is sensitive to gender diversity issues and >> > wants to avoid acquiring more of a "brogrammer" image, so some of us >> > worry that any conspicuous TOML files checked into revision control >> > repositories could be seen as a tacit endorsement of the author's >> > alleged behavior at GH a few years ago.) >> >> I was one of the folks championing TOML during the original >> discussions, and this is an issue that also worried me a lot. In case >> it's a useful data point: I actually contacted several of the main >> rust/cargo developers, since they were the major users of TOML and are >> also well known to be sensitive to these issues, to ask if they've had >> any issues with this, and they said that they haven't heard any >> complaints. >> >> Obviously there's a difference between "no-one complained" and "no-one >> was bothered", and I suspect the community's existing reputation may >> affect how this is interpreted as well, but... maybe useful as a data >> point. >> >> Between this and the way the TOML spec appears to have been abandoned >> at v0.4 (with the admonition "you should assume that is is unstable >> and act accordingly") I've wondered if we should fork it, rename it >> "the obvious minimal language", and release our own 1.0. >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
Is everyone on board with that? On Aug 26, 2017 4:29 AM, "Paul Moore" wrote: > On 26 August 2017 at 03:17, Guido van Rossum wrote: > > In pretty much any other context, if you have an operation that returns > an > > regular value or an error value, the error value should be None. > (Exceptions > > include e.g. returning a non-negative int or -1 for errors, or True for > > success and False for errors.) > > So, given that build_sdist returns the path of the newly built sdist, > the correct way to signal "I didn't manage to build a sdist" would be > to return None. > > Now that it's put this way, it seems glaringly obvious to me that this > is the correct thing to do. > Paul > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
I'm bored with that On Sat, Aug 26, 2017, 11:59 xoviat wrote: > Is everyone on board with that? > > On Aug 26, 2017 4:29 AM, "Paul Moore" wrote: > >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> an >> > regular value or an error value, the error value should be None. >> (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> Paul >> > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
Should probably take Guido out of this email chain now. And does everyone agree with what Nathaniel said on sys.path? On Aug 26, 2017 11:00 AM, "Daniel Holth" wrote: I'm bored with that On Sat, Aug 26, 2017, 11:59 xoviat wrote: > Is everyone on board with that? > > On Aug 26, 2017 4:29 AM, "Paul Moore" wrote: > >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> an >> > regular value or an error value, the error value should be None. >> (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> Paul >> > ___ > > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
[removed Guido from CC] On Aug 26, 2017 02:29, "Paul Moore" wrote: On 26 August 2017 at 03:17, Guido van Rossum wrote: > In pretty much any other context, if you have an operation that returns an > regular value or an error value, the error value should be None. (Exceptions > include e.g. returning a non-negative int or -1 for errors, or True for > success and False for errors.) So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None. Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do. Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes. -n ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
The current PEP requires that build_sdist return the basename of the sdist. So forgetting a return statement is not an option unless people really don't read the PEP. On Aug 26, 2017 2:18 PM, "Nathaniel Smith" wrote: > [removed Guido from CC] > > On Aug 26, 2017 02:29, "Paul Moore" wrote: > > On 26 August 2017 at 03:17, Guido van Rossum wrote: > > In pretty much any other context, if you have an operation that returns > an > > regular value or an error value, the error value should be None. > (Exceptions > > include e.g. returning a non-negative int or -1 for errors, or True for > > success and False for errors.) > > So, given that build_sdist returns the path of the newly built sdist, > the correct way to signal "I didn't manage to build a sdist" would be > to return None. > > Now that it's put this way, it seems glaringly obvious to me that this > is the correct thing to do. > > > Eh... I would really prefer something that's (a) more explicit about what > specifically went wrong, and (b) harder to return by accident. It's not at > all obvious that if the list of requirements is 'None' that means 'this > build supports making sdists in general but cannot make them from this > source tree but might still be able to make a wheel'. And if you forget to > put in a return statement, then python returns None for you, which seems > like it could lead to some super confusing error modes. > > -n > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On 26 August 2017 at 20:17, Nathaniel Smith wrote: > Eh... I would really prefer something that's (a) more explicit about what > specifically went wrong, and (b) harder to return by accident. It's not at > all obvious that if the list of requirements is 'None' that means 'this > build supports making sdists in general but cannot make them from this > source tree but might still be able to make a wheel'. And if you forget to > put in a return statement, then python returns None for you, which seems > like it could lead to some super confusing error modes. Well, we've had an extensive discussion about how frontends need to trust backends to get things right. I don't really see it as reasonable to now argue that backends might "forget" to return the right value - they might just as well "forget" to properly isolate builds... As regards an explicit description of what went wrong, why can't we just use the same reporting methods that we will for any other build issue (backends simply report the problem on stdout/stderr)? I don't see why the backend has to package up its error information and send it to the frontend to report, when we already have a perfectly effective way for backends to report errors and/or warnings to the user. If you're worried that the frontend might suppress the information (maybe because it's planning on falling back to a direct wheel build) then isn't that just the converse - backends need to trust frontends to do the right thing? It's hard to come up with convincing counterarguments to hypothetical concerns. Do you have a specific scenario that worries you? I see "return the path, or None indicating you couldn't build the sdist" as a nice, simple solution. If there's a case we know of right now where it's *too* simple, then let's understand the specifics - otherwise, why not defer worrying about it for now? We can revise the PEP later if experience shows we missed something. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore wrote: > On 26 August 2017 at 20:17, Nathaniel Smith wrote: >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. > > Well, we've had an extensive discussion about how frontends need to > trust backends to get things right. I don't really see it as > reasonable to now argue that backends might "forget" to return the > right value - they might just as well "forget" to properly isolate > builds... It's not about division of responsibilities, it's about handling errors gracefully when they happen. There are three bins: - creating an sdist succeeded - creating an sdist failed for expected reasons, and a clever frontend might be able to handle the problem automatically if it understands what the problem is (sdist creation isn't supported in this case) and understands its goals (just trying to build a wheel really, so the sdist isn't crucial) - creating an sdist failed for unexpected reasons, that need a human to sort out (due to a broken system, or bugs – hey, they happen – or ...) The whole discussion has been about how we can most reliably distinguish between the second and third categories, and give good error messages for the third category. The argument for NotImplemented is that it avoids cases where some internal call raises NotImplementedError and it "leaks out" accidentally, causing a unexpected error to be incorrectly treated as expected error -- we don't want pip to be hiding real bugs in backend code. The argument for NotImplementedError is that it produces better error messages on buggy frontends. 'return None' is kind of the worst of both worlds, in that it's an easy thing to return accidentally, and it gives confusing error messages if the frontend fails to handle it properly. (Even more confusing, actually, because 'NoneType object has no attribute ...' is even harder to track down than 'NotImplementedType object has no attribute ...'.) > As regards an explicit description of what went wrong, why can't we > just use the same reporting methods that we will for any other build > issue (backends simply report the problem on stdout/stderr)? I don't > see why the backend has to package up its error information and send > it to the frontend to report, when we already have a perfectly > effective way for backends to report errors and/or warnings to the > user. If you're worried that the frontend might suppress the > information (maybe because it's planning on falling back to a direct > wheel build) then isn't that just the converse - backends need to > trust frontends to do the right thing? What I mean is more, if you're some random user and you see this in a build backend, what do you guess it means? def get_requires_for_build_sdist(config_settings=None): return None Now how about these? def get_requires_for_build_sdist(config_settings=None): return NotImplemented def get_requires_for_build_sdist(config_settings=None): raise NotImplementedError def get_requires_for_build_sdist(config_settings=None): raise SdistBuildNotSupported I mean, obviously return None will work. Basically anything that's different from "return a list or string" will work :-). That's what makes this a bikeshed topic, and I still think we're mostly just spinning our wheels here until Nick and Donald have a chance to hash something out that they both can agree on. But I really don't see any advantages to 'return None' compared to the other options that have been discussed -n -- Nathaniel J. Smith -- https://vorpus.org ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: > [removed Guido from CC] > > On Aug 26, 2017 02:29, "Paul Moore" wrote: > > On 26 August 2017 at 03:17, Guido van Rossum wrote: > > In pretty much any other context, if you have an operation that returns > an > > regular value or an error value, the error value should be None. > (Exceptions > > include e.g. returning a non-negative int or -1 for errors, or True for > > success and False for errors.) > > So, given that build_sdist returns the path of the newly built sdist, > the correct way to signal "I didn't manage to build a sdist" would be > to return None. > > Now that it's put this way, it seems glaringly obvious to me that this > is the correct thing to do. > > > Eh... I would really prefer something that's (a) more explicit about what > specifically went wrong, and (b) harder to return by accident. It's not at > all obvious that if the list of requirements is 'None' that means 'this > build supports making sdists in general but cannot make them from this > source tree but might still be able to make a wheel'. And if you forget to > put in a return statement, then python returns None for you, which seems > like it could lead to some super confusing error modes. > Why does the frontend need to know why an sdist was not created? Frontend is asking the backend, given the current state of the world, to either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I would expect build_sdist to make some sanity checks about the world, then make a binary choice about whether sdist creation is a valid goal. If not possible, return None or NotImplemented or False or dict-of-reasons or whatever. Only if creation was *attempted*, and in the exceptional event it then failed, would I expect an Exception. We don't have structured exceptions sadly so they can't really carry much useful information from a protocol perspective above and beyond a simple None or the like anyway. I'd personally like to see some parity between build_sdist and build_wheel in this regard. Maybe the disconnect here is we have a way to specify hard reqs for building a wheel, statically or dynamically, and build_wheel is expected to never fail, but no way to specify hard reqs needed for build_sdist, necessitating this optional signaling path? If we had some definitive way for the frontend to know ahead of time if build_sdist is even expected to work, it could be called with more confidence. This could be a new sdist-related key in [build-system], a new table like [sdist-system].requires, or making the get_requires_for_* less optional, and defaulting to None instead of [ ]. Frontend is responsible for prepping the world, so if it can't get a list of reqs, somehow, for build_sdist, it knows it can't work. Same for build_wheel, because you have to specify the backend itself, so there is at least one requirement! Thus if you are a backend that can produce an sdist without additional requirements beyond build reqs, you should explicitly return empty list from get_requires_for_build_sdist. -- C Anthony ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
Why does the frontend need to know why an sdist was not created? I was of the opinion that such a distinction is not necessary because building a source distribution doesn't take that much time. However Donald thought that there needed to be a distinction because of the wasted time in attempting to build a wheel that was going to fail anyway. One of the things to consider is that site cythonizing takes time and maybe called for building source distribution. However since I think we're of the agreement that a source distribution should be as close to a checkout as possible, that may not be an issue because cythonizing may not be required to build the sdist. On Aug 26, 2017 3:47 PM, "C Anthony Risinger" wrote: > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: > >> [removed Guido from CC] >> >> On Aug 26, 2017 02:29, "Paul Moore" wrote: >> >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> an >> > regular value or an error value, the error value should be None. >> (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> >> >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. >> > > Why does the frontend need to know why an sdist was not created? > > Frontend is asking the backend, given the current state of the world, to > either produce an sdist, or not. Sans ahead-of-time knowledge (see below), > I would expect build_sdist to make some sanity checks about the world, then > make a binary choice about whether sdist creation is a valid goal. If not > possible, return None or NotImplemented or False or dict-of-reasons or > whatever. Only if creation was *attempted*, and in the exceptional event it > then failed, would I expect an Exception. We don't have structured > exceptions sadly so they can't really carry much useful information from a > protocol perspective above and beyond a simple None or the like anyway. > > I'd personally like to see some parity between build_sdist and build_wheel > in this regard. Maybe the disconnect here is we have a way to specify hard > reqs for building a wheel, statically or dynamically, and build_wheel is > expected to never fail, but no way to specify hard reqs needed for > build_sdist, necessitating this optional signaling path? > > If we had some definitive way for the frontend to know ahead of time if > build_sdist is even expected to work, it could be called with more > confidence. > > This could be a new sdist-related key in [build-system], a new table like > [sdist-system].requires, or making the get_requires_for_* less optional, > and defaulting to None instead of [ ]. > > Frontend is responsible for prepping the world, so if it can't get a list > of reqs, somehow, for build_sdist, it knows it can't work. Same for > build_wheel, because you have to specify the backend itself, so there is at > least one requirement! > > Thus if you are a backend that can produce an sdist without additional > requirements beyond build reqs, you should explicitly return empty list > from get_requires_for_build_sdist. > > -- > > C Anthony > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
I also think that Guido pretty much ruled out Notimplemented. On Aug 26, 2017 4:04 PM, "xoviat" wrote: > Why does the frontend need to know why an sdist was not created? > > I was of the opinion that such a distinction is not necessary because > building a source distribution doesn't take that much time. However Donald > thought that there needed to be a distinction because of the wasted time in > attempting to build a wheel that was going to fail anyway. One of the > things to consider is that site cythonizing takes time and maybe called for > building source distribution. However since I think we're of the agreement > that a source distribution should be as close to a checkout as possible, > that may not be an issue because cythonizing may not be required to build > the sdist. > > On Aug 26, 2017 3:47 PM, "C Anthony Risinger" > wrote: > >> On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: >> >>> [removed Guido from CC] >>> >>> On Aug 26, 2017 02:29, "Paul Moore" wrote: >>> >>> On 26 August 2017 at 03:17, Guido van Rossum wrote: >>> > In pretty much any other context, if you have an operation that >>> returns an >>> > regular value or an error value, the error value should be None. >>> (Exceptions >>> > include e.g. returning a non-negative int or -1 for errors, or True for >>> > success and False for errors.) >>> >>> So, given that build_sdist returns the path of the newly built sdist, >>> the correct way to signal "I didn't manage to build a sdist" would be >>> to return None. >>> >>> Now that it's put this way, it seems glaringly obvious to me that this >>> is the correct thing to do. >>> >>> >>> Eh... I would really prefer something that's (a) more explicit about >>> what specifically went wrong, and (b) harder to return by accident. It's >>> not at all obvious that if the list of requirements is 'None' that means >>> 'this build supports making sdists in general but cannot make them from >>> this source tree but might still be able to make a wheel'. And if you >>> forget to put in a return statement, then python returns None for you, >>> which seems like it could lead to some super confusing error modes. >>> >> >> Why does the frontend need to know why an sdist was not created? >> >> Frontend is asking the backend, given the current state of the world, to >> either produce an sdist, or not. Sans ahead-of-time knowledge (see below), >> I would expect build_sdist to make some sanity checks about the world, then >> make a binary choice about whether sdist creation is a valid goal. If not >> possible, return None or NotImplemented or False or dict-of-reasons or >> whatever. Only if creation was *attempted*, and in the exceptional event it >> then failed, would I expect an Exception. We don't have structured >> exceptions sadly so they can't really carry much useful information from a >> protocol perspective above and beyond a simple None or the like anyway. >> >> I'd personally like to see some parity between build_sdist and >> build_wheel in this regard. Maybe the disconnect here is we have a way to >> specify hard reqs for building a wheel, statically or dynamically, and >> build_wheel is expected to never fail, but no way to specify hard reqs >> needed for build_sdist, necessitating this optional signaling path? >> >> If we had some definitive way for the frontend to know ahead of time if >> build_sdist is even expected to work, it could be called with more >> confidence. >> >> This could be a new sdist-related key in [build-system], a new table like >> [sdist-system].requires, or making the get_requires_for_* less optional, >> and defaulting to None instead of [ ]. >> >> Frontend is responsible for prepping the world, so if it can't get a list >> of reqs, somehow, for build_sdist, it knows it can't work. Same for >> build_wheel, because you have to specify the backend itself, so there is at >> least one requirement! >> >> Thus if you are a backend that can produce an sdist without additional >> requirements beyond build reqs, you should explicitly return empty list >> from get_requires_for_build_sdist. >> >> -- >> >> C Anthony >> >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
Nathaniel: We're not talking about signaling failure in get_requires* we're talking about signaling failure in build*. On Aug 26, 2017 3:42 PM, "Nathaniel Smith" wrote: > On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore wrote: > > On 26 August 2017 at 20:17, Nathaniel Smith wrote: > >> Eh... I would really prefer something that's (a) more explicit about > what > >> specifically went wrong, and (b) harder to return by accident. It's not > at > >> all obvious that if the list of requirements is 'None' that means 'this > >> build supports making sdists in general but cannot make them from this > >> source tree but might still be able to make a wheel'. And if you forget > to > >> put in a return statement, then python returns None for you, which seems > >> like it could lead to some super confusing error modes. > > > > Well, we've had an extensive discussion about how frontends need to > > trust backends to get things right. I don't really see it as > > reasonable to now argue that backends might "forget" to return the > > right value - they might just as well "forget" to properly isolate > > builds... > > It's not about division of responsibilities, it's about handling > errors gracefully when they happen. There are three bins: > > - creating an sdist succeeded > - creating an sdist failed for expected reasons, and a clever frontend > might be able to handle the problem automatically if it understands > what the problem is (sdist creation isn't supported in this case) and > understands its goals (just trying to build a wheel really, so the > sdist isn't crucial) > - creating an sdist failed for unexpected reasons, that need a human > to sort out (due to a broken system, or bugs – hey, they happen – or > ...) > > The whole discussion has been about how we can most reliably > distinguish between the second and third categories, and give good > error messages for the third category. The argument for NotImplemented > is that it avoids cases where some internal call raises > NotImplementedError and it "leaks out" accidentally, causing a > unexpected error to be incorrectly treated as expected error -- we > don't want pip to be hiding real bugs in backend code. The argument > for NotImplementedError is that it produces better error messages on > buggy frontends. 'return None' is kind of the worst of both worlds, in > that it's an easy thing to return accidentally, and it gives confusing > error messages if the frontend fails to handle it properly. (Even more > confusing, actually, because 'NoneType object has no attribute ...' is > even harder to track down than 'NotImplementedType object has no > attribute ...'.) > > > As regards an explicit description of what went wrong, why can't we > > just use the same reporting methods that we will for any other build > > issue (backends simply report the problem on stdout/stderr)? I don't > > see why the backend has to package up its error information and send > > it to the frontend to report, when we already have a perfectly > > effective way for backends to report errors and/or warnings to the > > user. If you're worried that the frontend might suppress the > > information (maybe because it's planning on falling back to a direct > > wheel build) then isn't that just the converse - backends need to > > trust frontends to do the right thing? > > What I mean is more, if you're some random user and you see this in a > build backend, what do you guess it means? > > def get_requires_for_build_sdist(config_settings=None): > return None > > Now how about these? > > def get_requires_for_build_sdist(config_settings=None): > return NotImplemented > > def get_requires_for_build_sdist(config_settings=None): > raise NotImplementedError > > def get_requires_for_build_sdist(config_settings=None): > raise SdistBuildNotSupported > > I mean, obviously return None will work. Basically anything that's > different from "return a list or string" will work :-). That's what > makes this a bikeshed topic, and I still think we're mostly just > spinning our wheels here until Nick and Donald have a chance to hash > something out that they both can agree on. But I really don't see any > advantages to 'return None' compared to the other options that have > been discussed > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On Sat, Aug 26, 2017 at 2:06 PM, xoviat wrote: > I also think that Guido pretty much ruled out Notimplemented. As I've said, I don't think it matters a huge deal whether we use NotImplemented or not. But please don't treat Guido as some kind of pronouncement generating machine where you hurl out-of-context questions at him and then use his response as a club to beat down discussion. It's rude to Guido, it's rude to Nick and Donald (to whom Guido has explicitly delegated his BDFL authority in packaging-related matters), and it's rude to everyone trying to discuss proposals on their merits. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
As I said, I don't care what the particular solution is on this issue. However I'm simply trying to anticipate and resolve potential disagreements that could drag this out for a significant period of time. It is clear that Nick and Donald have a disagreement on this issue which is actually not in and of itself packaging related. The clear way to resolve this was to get a pronouncement. On Aug 26, 2017 4:56 PM, "Nathaniel Smith" wrote: > On Sat, Aug 26, 2017 at 2:06 PM, xoviat wrote: > > I also think that Guido pretty much ruled out Notimplemented. > > As I've said, I don't think it matters a huge deal whether we use > NotImplemented or not. But please don't treat Guido as some kind of > pronouncement generating machine where you hurl out-of-context > questions at him and then use his response as a club to beat down > discussion. It's rude to Guido, it's rude to Nick and Donald (to whom > Guido has explicitly delegated his BDFL authority in packaging-related > matters), and it's rude to everyone trying to discuss proposals on > their merits. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
With respect to this issue, if everyone told me that I was wrong then I would say I'm obviously wrong. But some people are saying one thing and other people are saying something else. On Aug 26, 2017 5:03 PM, "xoviat" wrote: > As I said, I don't care what the particular solution is on this issue. > However I'm simply trying to anticipate and resolve potential disagreements > that could drag this out for a significant period of time. It is clear that > Nick and Donald have a disagreement on this issue which is actually not in > and of itself packaging related. The clear way to resolve this was to get a > pronouncement. > > On Aug 26, 2017 4:56 PM, "Nathaniel Smith" wrote: > >> On Sat, Aug 26, 2017 at 2:06 PM, xoviat wrote: >> > I also think that Guido pretty much ruled out Notimplemented. >> >> As I've said, I don't think it matters a huge deal whether we use >> NotImplemented or not. But please don't treat Guido as some kind of >> pronouncement generating machine where you hurl out-of-context >> questions at him and then use his response as a club to beat down >> discussion. It's rude to Guido, it's rude to Nick and Donald (to whom >> Guido has explicitly delegated his BDFL authority in packaging-related >> matters), and it's rude to everyone trying to discuss proposals on >> their merits. >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger wrote: > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: >> >> [removed Guido from CC] >> >> On Aug 26, 2017 02:29, "Paul Moore" wrote: >> >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> > an >> > regular value or an error value, the error value should be None. >> > (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> >> >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. > > > Why does the frontend need to know why an sdist was not created? This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-) > Frontend is asking the backend, given the current state of the world, to > either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I > would expect build_sdist to make some sanity checks about the world, then > make a binary choice about whether sdist creation is a valid goal. If not > possible, return None or NotImplemented or False or dict-of-reasons or > whatever. Only if creation was *attempted*, and in the exceptional event it > then failed, would I expect an Exception. We don't have structured > exceptions sadly so they can't really carry much useful information from a > protocol perspective above and beyond a simple None or the like anyway. > > I'd personally like to see some parity between build_sdist and build_wheel > in this regard. Maybe the disconnect here is we have a way to specify hard > reqs for building a wheel, statically or dynamically, and build_wheel is > expected to never fail, but no way to specify hard reqs needed for > build_sdist, necessitating this optional signaling path? Not sure what you mean about hard reqs. The reason for the lack of parity is that we don't currently have any use cases where build_wheel is expected to fail, but this is expected in some sense (not sure what that would even mean), and there's some fallback that the frontend may want to invoke instead. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
Nathaniel: Just to clarify, we're talking about returning none for the build function not the get requirements function. The get requirements function is always expected to succeed and is optional. If while the backend is discovering requirements it finds that it cannot build, then it can return an empty list. It cannot however signal in the get requirements function that it cannot build. At least with what is being proposed. The reason that the proposal works is that the build function can never return none on success. On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger > wrote: > > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: > >> > >> [removed Guido from CC] > >> > >> On Aug 26, 2017 02:29, "Paul Moore" wrote: > >> > >> On 26 August 2017 at 03:17, Guido van Rossum wrote: > >> > In pretty much any other context, if you have an operation that > returns > >> > an > >> > regular value or an error value, the error value should be None. > >> > (Exceptions > >> > include e.g. returning a non-negative int or -1 for errors, or True > for > >> > success and False for errors.) > >> > >> So, given that build_sdist returns the path of the newly built sdist, > >> the correct way to signal "I didn't manage to build a sdist" would be > >> to return None. > >> > >> Now that it's put this way, it seems glaringly obvious to me that this > >> is the correct thing to do. > >> > >> > >> Eh... I would really prefer something that's (a) more explicit about > what > >> specifically went wrong, and (b) harder to return by accident. It's not > at > >> all obvious that if the list of requirements is 'None' that means 'this > >> build supports making sdists in general but cannot make them from this > >> source tree but might still be able to make a wheel'. And if you forget > to > >> put in a return statement, then python returns None for you, which seems > >> like it could lead to some super confusing error modes. > > > > > > Why does the frontend need to know why an sdist was not created? > > This whole discussion is about handling a specific case: suppose you > have a frontend like pip that when given a source directory and asked > to build a wheel, wants to implement that as: > - build sdist > - unpack sdist > - build wheel from unpacked sdist > > And suppose you have a backend like flit, that can build sdists from > some source directories (e.g. VCS checkouts) but not others (e.g. > unpacked sdists). We need some way for pip and flit to negotiate that > even though pip *normally* would implement its build-a-wheel operation > by first building an sdist, in this case it's ok to silently fall back > to some other strategy (like building the wheel directly in the source > tree, or manually copying the source tree somewhere else and then > building a wheel in it). > > But, we don't want this fallback behavior to hide real bugs. So if the > backend says "look, I just can't do sdists here, and that's an > expected thing, it's not something where the user needs to take any > particular action like filing a bug report or fixing their system or > anything like that, so if you have an alternative way to accomplish > what you're trying to do then you should just silently discard this > error and try that", ...cool. But if it doesn't explicitly say that, > then we don't want to silently discard the error and do something > else. > > It's taken a *lot* of back and forth to reach consensus that all we > need here is some special error signal from the *_sdist operations. > Let's focus on resolving that :-) > > > Frontend is asking the backend, given the current state of the world, to > > either produce an sdist, or not. Sans ahead-of-time knowledge (see > below), I > > would expect build_sdist to make some sanity checks about the world, then > > make a binary choice about whether sdist creation is a valid goal. If not > > possible, return None or NotImplemented or False or dict-of-reasons or > > whatever. Only if creation was *attempted*, and in the exceptional event > it > > then failed, would I expect an Exception. We don't have structured > > exceptions sadly so they can't really carry much useful information from > a > > protocol perspective above and beyond a simple None or the like anyway. > > > > I'd personally like to see some parity between build_sdist and > build_wheel > > in this regard. Maybe the disconnect here is we have a way to specify > hard > > reqs for building a wheel, statically or dynamically, and build_wheel is > > expected to never fail, but no way to specify hard reqs needed for > > build_sdist, necessitating this optional signaling path? > > Not sure what you mean about hard reqs. The reason for the lack of > parity is that we don't currently have any use cases where build_wheel > is expected to fail, but this is expected in some sense (not sure what > that would even mean), and there's some fallback that the frontend may > want to invoke instead
Re: [Distutils] PEP 517 again
On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger wrote: > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: >> >> [removed Guido from CC] >> >> On Aug 26, 2017 02:29, "Paul Moore" wrote: >> >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> > an >> > regular value or an error value, the error value should be None. >> > (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> >> >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. > > > Why does the frontend need to know why an sdist was not created? This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-) Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written. What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake. Consider instead, if we make the following small changes: 1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary. 2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty). 3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support). 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static) 6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported. 7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself). This arrangement allows pip to know ahead-of-time if `build_sdist` is appropriate. If no sdist reqs are explicitly acknowledged, then no sdist can be created. Full stop. Example usages: * Backeds that only support wheel creation do not implement `get_requires_for_build_sdist` at all. * Backeds that conditionally support sdist creation implement `get_requires_for_build_sdist` and return None if unsupported. This is where flit signals "impossible" from an unpacked sdist and [...] from VCS. * Backeds that always support sdist creation (setuptools) implement `get_requires_for_build_sdist` and return [...] unconditionally. So for pip's sdist -> unpack -> wheel path, it knows right away if it should build
Re: [Distutils] PEP 517 again
On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger wrote: > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: >> >> [removed Guido from CC] >> >> On Aug 26, 2017 02:29, "Paul Moore" wrote: >> >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> > an >> > regular value or an error value, the error value should be None. >> > (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> >> >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. > > > Why does the frontend need to know why an sdist was not created? This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-) Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written. What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake. Consider instead, if we make the following small changes: 1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary. 2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty). 3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support). 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static) 6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported. 7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself). This arrangement allows pip to know ahead-of-time if `build_sdist` is appropriate. If no sdist reqs are explicitly acknowledged, then no sdist can be created. Full stop. Example usages: * Backeds that only support wheel creation do not implement `get_requires_for_build_sdist` at all. * Backeds that conditionally support sdist creation implement `get_requires_for_build_sdist` and return None if unsupported. This is where flit signals "impossible" from an unpacked sdist and [...] from VCS. * Backeds that always support sdist creation (setuptools) implement `get_requires_for_build_sdist` and return [...] unconditionally. So for pip's sdist -> unpack -> wheel path, it knows right away if it should build a
Re: [Distutils] PEP 517 again
On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger wrote: > On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: > > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger > wrote: > > Sure sure, I understand all that, and why we think we need some special > error signal from `build_sdist`, as currently written. > > What I'm suggesting, is maybe calling `build_sdist` without knowing if it > can succeed is already a mistake. > > Consider instead, if we make the following small changes: > > 1. `get_requires_for_build_*` is passed the sdist and wheel directories, > just like `build_*`, giving them the chance to actually look at tree before > deciding what other reqs might be necessary. That's not a change, that's how it works :-). > 2. `get_requires_for_build_*` returns None to signal `build_*` is > unsupported (superceded by static reqs defined in TOML) and [...] to signal > support (can be empty). > > 3. `get_requires_for_build_*` assumed to return None if missing (so optional > and implies no support). This is what I originally proposed, except you use None where I use NotImplemented, which has the disadvantages I noted earlier. Also, people didn't like the missing get_requires_for_build_* being treated as no-support, which makes sense, since we expect that get_requires_for_build_* won't be used very often. But one can switch the default here without affecting much else. The reason we want to let build_sdist report failure is just for convenience of backends who don't have any other reason to implement get_requires_for_build_sdist. > 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) > > 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + > `build-system.requires` (static) build-system.requires contains the requirements that are always installed before we even try importing the backend, so they're available to all backend hooks equally. > 6. If no reqs are found for sdist (no declared reqs in TOML and > `get_requires_for_build_sdist` is missing or returns None), then > `build_sdist` is unsupported. > > 7. If no reqs are found for wheel (no declared reqs in TOML and > `get_requires_for_build_wheel` is missing or returns None), then > `build_wheel` is unsupported. This one is a spec violation because at least > one req is expected here (the backed itself). The TOML requires aren't really useful as a signal about whether sdist specifically is supported. Plus I think we probably want to leave no-requires-in-TOML as a valid option for saying "I don't need anything installed" (maybe because the backend is shipped inside the source tree) rather than overloading it to have extra meanings. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith wrote: > On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger > wrote: > > On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: > > > > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger > > wrote: > > > > Sure sure, I understand all that, and why we think we need some special > > error signal from `build_sdist`, as currently written. > > > > What I'm suggesting, is maybe calling `build_sdist` without knowing if it > > can succeed is already a mistake. > > > > Consider instead, if we make the following small changes: > > > > 1. `get_requires_for_build_*` is passed the sdist and wheel directories, > > just like `build_*`, giving them the chance to actually look at tree > before > > deciding what other reqs might be necessary. > > That's not a change, that's how it works :-). > Is that a change I missed from this thread? I'm reading here: https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel and they do not appear to receive the source or wheel directories. > > 2. `get_requires_for_build_*` returns None to signal `build_*` is > > unsupported (superceded by static reqs defined in TOML) and [...] to > signal > > support (can be empty). > > > > 3. `get_requires_for_build_*` assumed to return None if missing (so > optional > > and implies no support). > > This is what I originally proposed, except you use None where I use > NotImplemented, which has the disadvantages I noted earlier. Also, > people didn't like the missing get_requires_for_build_* being treated > as no-support, which makes sense, since we expect that > get_requires_for_build_* won't be used very often. But one can switch > the default here without affecting much else. The reason we want to > let build_sdist report failure is just for convenience of backends who > don't have any other reason to implement get_requires_for_build_sdist. > Oh OK, good good. Well in that case I agree with you and missed the suggestion. I personally prefer NotImplemented as well here but None seemed mostly just as good and did not elicit as much pushback. It's not too big of deal either way. However, a missing `get_requires_for_build_wheel` technically signaling "unsupported" makes good sense to me because it's always supplemented by the static *and mandatory* `build-system.requires` list. There is no proper way (without breaking the spec) to signal "unsupported" for `build_wheel` since the backend itself (setuptools, wheel, flit) is specified here. "Unsupported" is only signaled when *both* the static and dynamic requires are None (or NotImplemented as mentioned). The kicker here in my offering, is that the presence of `build-system.requires` *does not in any way imply* `build_sdist` support. As written, there is no way to statically set the requirements for sdist support (though this could be changed of course with a new TOML key/table), so you must explicitly signal it with something like: def get_requires_for_build_sdist(*args, **kwds): return [] This means, by default, `build_wheel` is "supported" and `build_sdist` is "unsupported", and both are no fail operations. If called, any exception is fatal to the entire process. If a backend goes through the work of supporting sdist creation, is it really a problem to relay this support with a 3 line function definition? Wheel (could also define nothing at all): def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None Flit: def get_requires_for_build_sdist(source_dir, ...): # I can output an sdist if the source directory is a VCS checkout I understand. requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_dir) return requires Setuptools: def get_requires_for_build_sdist(source_dir, ...): # I'm going to successfully create an sdist or die trying! # There is literally no directory I can't handle so I don't even look. return [] This seems pretty straightforward to me and avoids overloading `build_sdist`, keeping it no fail like `build_wheel`. Semantically, it's really the job of the requirements discovery mechanism to decide one of: a) Zero or more requirements are needed to transform target source_dir into an sdist. b) No requirement enables my backend to transform target source_dir into an sdist. This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely? > > 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) > > > > 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + > > `build-system.requires` (st
Re: [Distutils] PEP 517 again
> and they do not appear to receive the source or wheel directories. The source directory is the current directory, if I am not mistaken. > This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely? Not trying to speak on behalf of flit here, but if I understand correctly, flit requires git to build a source distribution. Flit knows its build requirements, so it can just return them when get_requires is called. However, it needs to attempt to invoke git to find out whether it can build a source distribution, which if I understand correctly, is a lengthy operation (whether that's actually true is not actually relevant because we are discussion potentially any backend/operation). It's more efficient if git is only invoked once, and if a failure occurs, then under the proposal "None" would be returned from build_sdist rather than the name of the built source distribution. def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None I have to say that the above example is not a good idea. Perhaps I was a bit muddled on that point earlier: the reason that "return None" is a good idea for build sdist is, well: def build_sdist(...): # Indicates success return "x.tar.gz" def build_sdist(...): # Indicates failure return None Those are not truthily equivalent, which is important because it means that someone is unlikely to make a mistake on that matter. 2017-08-26 23:05 GMT-05:00 C Anthony Risinger : > On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith wrote: > >> On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger >> wrote: >> > On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: >> > >> > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger >> > wrote: >> > >> > Sure sure, I understand all that, and why we think we need some special >> > error signal from `build_sdist`, as currently written. >> > >> > What I'm suggesting, is maybe calling `build_sdist` without knowing if >> it >> > can succeed is already a mistake. >> > >> > Consider instead, if we make the following small changes: >> > >> > 1. `get_requires_for_build_*` is passed the sdist and wheel directories, >> > just like `build_*`, giving them the chance to actually look at tree >> before >> > deciding what other reqs might be necessary. >> >> That's not a change, that's how it works :-). >> > > Is that a change I missed from this thread? I'm reading here: > > https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 > https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 > https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist > https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel > > and they do not appear to receive the source or wheel directories. > > >> > 2. `get_requires_for_build_*` returns None to signal `build_*` is >> > unsupported (superceded by static reqs defined in TOML) and [...] to >> signal >> > support (can be empty). >> > >> > 3. `get_requires_for_build_*` assumed to return None if missing (so >> optional >> > and implies no support). >> >> This is what I originally proposed, except you use None where I use >> NotImplemented, which has the disadvantages I noted earlier. Also, >> people didn't like the missing get_requires_for_build_* being treated >> as no-support, which makes sense, since we expect that >> get_requires_for_build_* won't be used very often. But one can switch >> the default here without affecting much else. The reason we want to >> let build_sdist report failure is just for convenience of backends who >> don't have any other reason to implement get_requires_for_build_sdist. >> > > Oh OK, good good. Well in that case I agree with you and missed the > suggestion. I personally prefer NotImplemented as well here but None seemed > mostly just as good and did not elicit as much pushback. It's not too big > of deal either way. > > However, a missing `get_requires_for_build_wheel` technically signaling > "unsupported" makes good sense to me because it's always supplemented by > the static *and mandatory* `build-system.requires` list. There is no proper > way (without breaking the spec) to signal "unsupported" for `build_wheel` > since the backend itself (setuptools, wheel, flit) is specified here. > "Unsupported" is only signaled when *both* the static and dynamic requires > are None (or NotImplemented as mentioned). The kicker here in my offering, > is that the presence of `build-system.requires` *does not in any way imply* > `build_sdist` support. As written, there is no way to statically set the > requirements for sdist support (though this could be changed of course with > a new TOML key/table), so you must explicitly signal it with something like: > > def get_requires_for_build_sdis
Re: [Distutils] PEP 517 again
On Sat, Aug 26, 2017 at 11:05 PM, C Anthony Risinger wrote: > On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith wrote: > >> On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger >> wrote: >> > On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: >> > >> > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger >> > wrote: >> > >> > Sure sure, I understand all that, and why we think we need some special >> > error signal from `build_sdist`, as currently written. >> > >> > What I'm suggesting, is maybe calling `build_sdist` without knowing if >> it >> > can succeed is already a mistake. >> > >> > Consider instead, if we make the following small changes: >> > >> > 1. `get_requires_for_build_*` is passed the sdist and wheel directories, >> > just like `build_*`, giving them the chance to actually look at tree >> before >> > deciding what other reqs might be necessary. >> >> That's not a change, that's how it works :-). >> > > Is that a change I missed from this thread? I'm reading here: > > https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 > https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 > https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist > https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel > > and they do not appear to receive the source or wheel directories. > > >> > 2. `get_requires_for_build_*` returns None to signal `build_*` is >> > unsupported (superceded by static reqs defined in TOML) and [...] to >> signal >> > support (can be empty). >> > >> > 3. `get_requires_for_build_*` assumed to return None if missing (so >> optional >> > and implies no support). >> >> This is what I originally proposed, except you use None where I use >> NotImplemented, which has the disadvantages I noted earlier. Also, >> people didn't like the missing get_requires_for_build_* being treated >> as no-support, which makes sense, since we expect that >> get_requires_for_build_* won't be used very often. But one can switch >> the default here without affecting much else. The reason we want to >> let build_sdist report failure is just for convenience of backends who >> don't have any other reason to implement get_requires_for_build_sdist. >> > > Oh OK, good good. Well in that case I agree with you and missed the > suggestion. I personally prefer NotImplemented as well here but None seemed > mostly just as good and did not elicit as much pushback. It's not too big > of deal either way. > > However, a missing `get_requires_for_build_wheel` technically signaling > "unsupported" makes good sense to me because it's always supplemented by > the static *and mandatory* `build-system.requires` list. There is no proper > way (without breaking the spec) to signal "unsupported" for `build_wheel` > since the backend itself (setuptools, wheel, flit) is specified here. > "Unsupported" is only signaled when *both* the static and dynamic requires > are None (or NotImplemented as mentioned). The kicker here in my offering, > is that the presence of `build-system.requires` *does not in any way imply* > `build_sdist` support. As written, there is no way to statically set the > requirements for sdist support (though this could be changed of course with > a new TOML key/table), so you must explicitly signal it with something like: > > def get_requires_for_build_sdist(*args, **kwds): return [] > > This means, by default, `build_wheel` is "supported" and `build_sdist` is > "unsupported", and both are no fail operations. If called, any exception is > fatal to the entire process. If a backend goes through the work of > supporting sdist creation, is it really a problem to relay this support > with a 3 line function definition? > > Wheel (could also define nothing at all): > > def get_requires_for_build_sdist(source_dir, ...): > # I have no interest in sdists and I never will. > # GO AWAY. > return None > > Flit: > > def get_requires_for_build_sdist(source_dir, ...): > # I can output an sdist if the source directory is a VCS checkout I > understand. > requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_ > dir) > return requires > > Setuptools: > > def get_requires_for_build_sdist(source_dir, ...): > # I'm going to successfully create an sdist or die trying! > # There is literally no directory I can't handle so I don't even look. > return [] > > This seems pretty straightforward to me and avoids overloading > `build_sdist`, keeping it no fail like `build_wheel`. > > Semantically, it's really the job of the requirements discovery mechanism > to decide one of: > > a) Zero or more requirements are needed to transform target source_dir > into an sdist. > b) No requirement enables my backend to transform target source_dir into > an sdist. > > This lets `build_*` focus purely on building things straight away. There > is a difference between "no more reqs are needed to do X" and "no possible > req will achieve X" even though both add zero requirement
Re: [Distutils] PEP 517 again
On Sat, Aug 26, 2017 at 11:18 PM, xoviat wrote: > > and they do not appear to receive the source or wheel directories. > > The source directory is the current directory, if I am not mistaken. > Oh right right, for some reason I was reading the `build_*` hooks as receiving the input directory rather than the output directory. I don't see it stated anywhere that the PWD will be the source directory however. Might be a good addition. In this case, `get_requires_for_*` hooks should also be able to depend on PWD being the applicable source directory (which of course might be an unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel pipeline). > > This lets `build_*` focus purely on building things straight away. > There is a difference between "no more reqs are needed to do X" and "no > possible req will achieve X" even though both add zero requirements. Why > not let this hook relay it's decision more completely? > > Not trying to speak on behalf of flit here, but if I understand correctly, > flit requires git to build a source distribution. Flit knows its build > requirements, so it can just return them when get_requires is called. > However, it needs to attempt to invoke git to find out whether it can build > a source distribution, which if I understand correctly, is a lengthy > operation (whether that's actually true is not actually relevant because we > are discussion potentially any backend/operation). It's more efficient if > git is only invoked once, and if a failure occurs, then under the proposal > "None" would be returned from build_sdist rather than the name of the built > source distribution. > What is the benefit in making `build_sdist` handle this assertion? Requirements checking is only performed once, earlier than building. Somewhere, somehow, something must look to see if git exists. Is it not more appropriate to check early, while asserting python requirements, vs. actually being told to build an sdist? That is already the expectation on the wheel side of things since it would also fail if necessary system binaries are missing. Checking for git is conflating system reqs and python reqs as "requirements", but I don't see why the get_requires_* hooks can't do both if they so choose. > def get_requires_for_build_sdist(source_dir, ...): > # I have no interest in sdists and I never will. > # GO AWAY. > return None > > I have to say that the above example is not a good idea. Perhaps I was a > bit muddled on that point earlier: the reason that "return None" is a good > idea for build sdist is, well: > > def build_sdist(...): # Indicates success > return "x.tar.gz" > > def build_sdist(...): # Indicates failure > return None > > > Those are not truthily equivalent, which is important because it means > that someone is unlikely to make a mistake on that matter. > What is the difference? Both equally inform the frontend that building an sdist is not going to happen. At the end of the day, I'm not too partial to any of the solutions I've read thus far, and do not have any desire to derail progress. I just don't like the variation in `build_sdist` vs. `build_wheel` because it feels like a kludge to me. I thought perhaps defaulting sdists to "unsupported" and requiring a backend to opt-in was both more elegant and more explicit, allowing tools like flit to better express their conditional sdist support. -- C Anthony ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 again
The difference is that an empty list for requires would indicate that building is possible but none would not. Those two values are truthily equivalent. For building, the name returned indicates building was possible, but it's not truthily equivalent to none. On Aug 27, 2017 12:08 AM, "C Anthony Risinger" wrote: > On Sat, Aug 26, 2017 at 11:18 PM, xoviat wrote: > >> > and they do not appear to receive the source or wheel directories. >> >> The source directory is the current directory, if I am not mistaken. >> > > Oh right right, for some reason I was reading the `build_*` hooks as > receiving the input directory rather than the output directory. I don't > see it stated anywhere that the PWD will be the source directory however. > Might be a good addition. > > In this case, `get_requires_for_*` hooks should also be able to depend on > PWD being the applicable source directory (which of course might be an > unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel > pipeline). > > >> > This lets `build_*` focus purely on building things straight away. >> There is a difference between "no more reqs are needed to do X" and "no >> possible req will achieve X" even though both add zero requirements. Why >> not let this hook relay it's decision more completely? >> >> Not trying to speak on behalf of flit here, but if I understand >> correctly, flit requires git to build a source distribution. Flit knows its >> build requirements, so it can just return them when get_requires is called. >> However, it needs to attempt to invoke git to find out whether it can build >> a source distribution, which if I understand correctly, is a lengthy >> operation (whether that's actually true is not actually relevant because we >> are discussion potentially any backend/operation). It's more efficient if >> git is only invoked once, and if a failure occurs, then under the proposal >> "None" would be returned from build_sdist rather than the name of the built >> source distribution. >> > > What is the benefit in making `build_sdist` handle this assertion? > Requirements checking is only performed once, earlier than building. > Somewhere, somehow, something must look to see if git exists. Is it not > more appropriate to check early, while asserting python requirements, vs. > actually being told to build an sdist? That is already the expectation on > the wheel side of things since it would also fail if necessary system > binaries are missing. Checking for git is conflating system reqs and python > reqs as "requirements", but I don't see why the get_requires_* hooks can't > do both if they so choose. > > >> def get_requires_for_build_sdist(source_dir, ...): >> # I have no interest in sdists and I never will. >> # GO AWAY. >> return None >> >> I have to say that the above example is not a good idea. Perhaps I was a >> bit muddled on that point earlier: the reason that "return None" is a good >> idea for build sdist is, well: >> >> def build_sdist(...): # Indicates success >> return "x.tar.gz" >> >> def build_sdist(...): # Indicates failure >> return None >> >> >> Those are not truthily equivalent, which is important because it means >> that someone is unlikely to make a mistake on that matter. >> > > What is the difference? Both equally inform the frontend that building an > sdist is not going to happen. > > At the end of the day, I'm not too partial to any of the solutions I've > read thus far, and do not have any desire to derail progress. I just don't > like the variation in `build_sdist` vs. `build_wheel` because it feels like > a kludge to me. I thought perhaps defaulting sdists to "unsupported" and > requiring a backend to opt-in was both more elegant and more explicit, > allowing tools like flit to better express their conditional sdist support. > > -- > > C Anthony > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
