Re: [Distutils] PEP 517 - specifying build system in pyproject.toml
On 20 May 2017 at 00:18, Thomas Kluyver wrote: > Hi, > > I'd like to make another push for PEP 517, which would make it possible > to build wheels from a source tree with other build tools, without > needing setup.py. > > https://www.python.org/dev/peps/pep-0517/ > > Last time this was discussed we made a couple of minor changes to the > PEP, but we didn't want to accept another packaging related PEP until > PEP 518 was implemented in pip. I'm pleased to say that that > implementation has just been merged: > https://github.com/pypa/pip/pull/4144 . Huzzah, and congratulations! :) Regarding the encoding question, I agree with your recommendation with one key amendment to account for the 16-bit console APIs on Windows: * on platforms with 8-bit standard streams (e.g. Linux, Mac OS X), build systems SHOULD emit UTF-8 encoded output * on platforms with 16-bit standard streams (e.g. Windows), build systems SHOULD emit UTF-16-LE encoded output * on platforms that offer both, build systems SHOULD use the 16-bit streams to match the default behaviour of CPython 3.6+ * install tools MUST NOT fail the build solely due to improperly encoded output, but are otherwise free to handle the situation as they see fit Folks on Python 3.5 and earlier on Windows may still have problems given that guidance (since that uses the 8-bit stream interfaces with the Windows native encodings by default), but that's also a large part of why CPython's behaviour on Windows was changed in 3.6 :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 - specifying build system in pyproject.toml
On Fri, May 19, 2017, at 05:17 PM, Paul Moore wrote: > On 19 May 2017 at 16:53, Daniel Holth wrote: > > Congrats on getting 518 in. > > Agreed, by the way. That's a big step! Thanks both. It does feel like an achievement. :-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 - specifying build system in pyproject.toml
On 19 May 2017 at 16:53, Daniel Holth wrote: > Congrats on getting 518 in. Agreed, by the way. That's a big step! Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 - specifying build system in pyproject.toml
Congrats on getting 518 in. On Fri, May 19, 2017, 11:37 Thomas Kluyver wrote: > On Fri, May 19, 2017, at 04:31 PM, Paul Moore wrote: > > For flit, would having the install tool set PYTHONIOENCODING help? > > If install tools were meant to set PYTHONIOENCODING, I probably wouldn't > do anything else in flit's code. Python should then take care of > ensuring that any output is UTF-8 encoded, and flit doesn't currently > invoke any separate processes to do the build. > > Thomas > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 - specifying build system in pyproject.toml
On Fri, May 19, 2017, at 04:31 PM, Paul Moore wrote: > For flit, would having the install tool set PYTHONIOENCODING help? If install tools were meant to set PYTHONIOENCODING, I probably wouldn't do anything else in flit's code. Python should then take care of ensuring that any output is UTF-8 encoded, and flit doesn't currently invoke any separate processes to do the build. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 - specifying build system in pyproject.toml
On 19 May 2017 at 16:16, Thomas Kluyver wrote: > On Fri, May 19, 2017, at 03:41 PM, Paul Moore wrote: >> Can we specify what encoding the informational text must be written >> in? > > Sure, that makes sense. What about: > > All hooks are run with working directory set to the root of the source > tree, and MAY print arbitrary informational text on stdout and stderr. > This text SHOULD be UTF-8 encoded, but as building may invoke other > processes, install tools MUST NOT fail if the data they receive is not > valid UTF-8; though in this case the display of the output may be > corrupted. Looks good, although whether UTF-8 is viable on Windows is something I'll have to think about. > Do we also want to recommend that install tools set > PYTHONIOENCODING=utf-8 when invoking build tools? Or leave this up to > the build tools? Good question. At the moment, the only 2 cases I know of are setuptools/distutils and flit. For setuptools, I'm pretty sure there's no handling of subprocesses, it just fires them off and lets them write to the console - so there's nothing to even ensure a consistent encoding. We may have to allow for special casing with setuptools, as I doubt anyone's going to put in the effort to add a transcoding layer in there. For flit, would having the install tool set PYTHONIOENCODING help? I don't know immediately what I'd do if I were designing a brand new build tool that called out to a 3rd party compiler. Let me think about it. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 - specifying build system in pyproject.toml
On Fri, May 19, 2017, at 03:41 PM, Paul Moore wrote: > Can we specify what encoding the informational text must be written > in? Sure, that makes sense. What about: All hooks are run with working directory set to the root of the source tree, and MAY print arbitrary informational text on stdout and stderr. This text SHOULD be UTF-8 encoded, but as building may invoke other processes, install tools MUST NOT fail if the data they receive is not valid UTF-8; though in this case the display of the output may be corrupted. Do we also want to recommend that install tools set PYTHONIOENCODING=utf-8 when invoking build tools? Or leave this up to the build tools? Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517 - specifying build system in pyproject.toml
On 19 May 2017 at 15:18, Thomas Kluyver wrote: > Hi, > > I'd like to make another push for PEP 517, which would make it possible > to build wheels from a source tree with other build tools, without > needing setup.py. A point that came up recently while dealing with a pip issue. """ All hooks are run with working directory set to the root of the source tree, and MAY print arbitrary informational text on stdout and stderr. """ Can we specify what encoding the informational text must be written in? At the moment pip has problems dealing with non-ASCII locales because it captures the build output and then displays it on error. This involves a decode/encode step (on Python 3) or printing arbitrary bytes to stdout (on Python 2). And at the moment we get UnicodeErrors if there's a mismatch. I've patched it to use errors=replace, but we still risk mojibake. Ideally, we should specify an encoding that hooks will use for output - but that's somewhat difficult as many build tools will want to do things like run compilers which could do arbitrarily silly things. I believe this is less of a problem on Unix (where there's a well-managed convention), but on Windows there's an "OEM" codepage for console programs, and an "ANSI" codepage for Windows programs - but not all programs use the same one - some console programs such as Mingw, I think, and Python itself if stdout is redirected (see https://docs.python.org/3.6/library/sys.html#sys.stdout) use the ANSI codepage. So we may have to fudge the situation a bit. (Maybe something like "Install tools MAY assume a specific encoding for the output, and MAY produce corrupted output if the build tool does not use that encoding, but install tools MUST NOT fail with an encoding error just because the encodings don't match"). But I don't think we should leave the situation completely unspecified. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP 517 - specifying build system in pyproject.toml
Hi, I'd like to make another push for PEP 517, which would make it possible to build wheels from a source tree with other build tools, without needing setup.py. https://www.python.org/dev/peps/pep-0517/ Last time this was discussed we made a couple of minor changes to the PEP, but we didn't want to accept another packaging related PEP until PEP 518 was implemented in pip. I'm pleased to say that that implementation has just been merged: https://github.com/pypa/pip/pull/4144 . I have an implementation of a build backend according to the current PEP 517 spec in a pull request on flit: https://github.com/takluyver/flit/pull/77/files#diff-6df5630df9568322a453f03159fc8fe8 Thanks, Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
