[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Roundup Robot added the comment: New changeset b26c8104e54f by Steve Dower in branch '3.6': Closes #27781: Removes special cases for the experimental aspect of PEP 529 https://hg.python.org/cpython/rev/b26c8104e54f New changeset b8233c779ff7 by Steve Dower in branch 'default': Closes #27781:

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: Before 3.6.0 beta 4 I need to make this change permanent. From memory, it's just an exception message that needs changing (and PEP 529 becomes final), but I'll review the changeset to be sure. -- nosy: +ned.deily priority: normal -> release blocker

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Mark Dickinson added the comment: That seems to have done the trick. Thanks! -- ___ Python tracker ___ ___

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Roundup Robot added the comment: New changeset 801634d3c105 by Steve Dower in branch 'default': Issue #27781: Fixes uninitialized fd when !MS_WINDOWS and !HAVE_OPENAT https://hg.python.org/cpython/rev/801634d3c105 -- ___ Python tracker

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Mark Dickinson added the comment: It looks as though this change in posixmodule.c is the cause: #ifdef MS_WINDOWS -if (path->wide) -fd = _wopen(path->wide, flags, mode); -else +fd = _wopen(path->wide, flags, mode); #endif #ifdef HAVE_OPENAT if

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Mark Dickinson added the comment: It looks as though this change might have broken the compile on OS X. On my OS X 10.9 machine, building from a clean Git checkout of the master branch fails; the tail of the failed build looks like this: ./python.exe -E -S -m sysconfig --generate-posix-vars

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Roundup Robot added the comment: New changeset faca0730270b by Steve Dower in branch 'default': Fixes tests broken by issue #27781. https://hg.python.org/cpython/rev/faca0730270b -- ___ Python tracker

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Roundup Robot added the comment: New changeset e20c7d8a8187 by Steve Dower in branch 'default': Issue #27781: Change file system encoding on Windows to UTF-8 (PEP 529) https://hg.python.org/cpython/rev/e20c7d8a8187 -- nosy: +python-dev ___ Python

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: This is pushed now - let the bug fixing begin :) -- stage: patch review -> commit review ___ Python tracker ___

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: Thanks for that review, Eryk, but I'm going to defer those to other issues (specifically issue27998 for scandir and we should file a new issue for the symlink concerns). I've got some more doc updates to do though, and then I'll check in if there are no other

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: One minor change - I removed the unused definition of Py_FileSystemDefaultDecodeErrors. -- ___ Python tracker ___

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: PEP 529 has been accepted, so this really needs a review now. But since it's experimental and all the tests pass, I'll be committing it shortly anyway and will be tidying up issues during beta. -- ___ Python tracker

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: Also see PEP 529 for the latest updates there. This is likely to be accepted as experimental for 3.6.0b1-3, and we'll commit to either the new default or a compatible default for b4. -- Added file: http://bugs.python.org/file44414/27781_1.patch

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Nick Coghlan added the comment: I belatedly remembered I've had this new test case hanging around for a while, and never got around to getting it into shape for inclusion in the standard library. With the prospect of reasonable cross-platform consistency in this area, it could be a good

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Changes by Brett Cannon : -- nosy: +brett.cannon ___ Python tracker ___ ___

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: Ah I see, if we end up sticking with MBCS and offering a switch to enable UTF-8. In that case, we'll definitely ensure the flag is the same (but I'm hopeful we will just make the reliable behavior on Windows the default, so it won't matter). --

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

STINNER Victor added the comment: Steve Dower added the comment: > By portable, do you mean not using an environment variable? I mean that "python3 -X utf8" should force sys.getfilesystemencoding() to UTF-8 on UNIX/BSD, it would ignore the current locale setting. --

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: By portable, do you mean not using an environment variable? Command line parsing is potentially affected by this on Windows - I'd have to look deeper - as command lines are provided as UTF-16. But we may not ever expose them as bytes. I don't even know that

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

STINNER Victor added the comment: > I suspect we'll have to go to Guido to get a ruling on the default, but I'll > add an environment variable to switch. If you go in this direction, I would like to follow you for the UNIX/BSD side to make the switch portable. I was thinking about "-X utf8"

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

STINNER Victor added the comment: > Is there a surrogatepass option? I'm talking about error handlers of Python codecs: text.encode('utf8', 'surrogatepass') -- ___ Python tracker

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Steve Dower added the comment: Thanks for the regen. I don't think git format is the problem as most of my patches are fine, it's probably because it was in a patch queue and so the parent isn't actually a known commit. I haven't tested whether this works without my other console patches but

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Changes by Chi Hsuan Yen : -- nosy: +Chi Hsuan Yen ___ Python tracker ___ ___

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

STINNER Victor added the comment: Steve Dower: Please don't use git format for diff, or the bug tracker is unable to create reviews. I regenerated the patch. By the way, you introduced a bug in posix_do_stat(): you added a new "else" in the !MS_WINDOWS path which leads to a compilation error.

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

STINNER Victor added the comment: Would it be acceptable for you to add a new option to switch to UTF-8 in Python 3.6, and discuss later if it's ok to enable it by default? In the python-ideas threed, you wrote that Windows allow surrogate characters in filenames, but not the UTF-8/strict

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Changes by Jeremy Kloth : -- nosy: +jkloth ___ Python tracker ___ ___

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

Decorater added the comment: I personally hate ansi myself so +1 to UTF-8/UTF-16. -- nosy: +Decorater ___ Python tracker ___

[issue27781] Change sys.getfilesystemencoding() on Windows to UTF-8

New submission from Steve Dower: I've attached my first pass at a patch to change the file system encoding on Windows to UTF-8 and remove use of the *A APIs. It would be trivial to change the encoding from UTF-8 back to CP_ACP and change the error mode if that's what we decide is better, but