[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: It seems like this change: def test_forced_io_encoding(self): # Checks forced configuration of embedded interpreter IO streams -out, err = self.run_embedded_interpreter("forced_io_encoding") -if support.verbose: +env =

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: Ronald Oussoren: > The macOS failures are at least partially caused by test assumptions that > aren't true on macOS (...) Nick is working on a fix for macOS: https://github.com/python/cpython/pull/2130 -- ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: > FreeBSD 10.x: if locale coercion succeeds, we then fail on get_codeset() > (perhaps because that doesn't recognise LC_CTYPE=UTF-8?) I created bpo-30647 to track this one. -- ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

Changes by Nick Coghlan : -- pull_requests: +2184 ___ Python tracker ___ ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

Ronald Oussoren added the comment: The macOS failures are at least partially caused by test assumptions that aren't true on macOS: in particular the filesystem encoding defaults to UTF-8 on macOS (because HFS+ and the recent APFS filesystem store unicode data and not pure byte strings).

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Initial look at the failures on the stable buildbots: FreeBSD 10.x: if locale coercion succeeds, we then fail on get_codeset() (perhaps because that doesn't recognise LC_CTYPE=UTF-8?) FreeBSD CURRENT: if locale coercion fails (due to no suitable locale), lots of

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Ah, it would have been too easy for all the other *nix variants to be close enough to Fedora & Ubuntu for everything to work first time :) -- ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: Tests fail on many buildbots. -- resolution: fixed -> status: closed -> open ___ Python tracker ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: And merged! Thanks to all involved in the process of getting this change through to implementation :) -- resolution: -> fixed stage: -> resolved status: open -> closed ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: New changeset 6ea4186de32d65b1f1dc1533b6312b798d300466 by Nick Coghlan in branch 'master': bpo-28180: Implementation for PEP 538 (#659) https://github.com/python/cpython/commit/6ea4186de32d65b1f1dc1533b6312b798d300466 --

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: The PEP 538 PR is mostly complete now, but I created https://bugs.python.org/issue30565 to track making a follow-up decision on whether or not we really want to emit a warning on *successful* implicit locale coercion. The pre-release What's New entry for PEP

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: OK, the PEP 538 reference implementation has reached the point where I was willing to create a PR for it: https://github.com/python/cpython/pull/659 That PR/branch also includes the necessary changes to always force the C.UTF-8 locale on Android rather than

[issue28180] sys.getfilesystemencoding() should default to utf-8

Changes by Nick Coghlan : -- pull_requests: +540 ___ Python tracker ___ ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: An updated reference implementation has been pushed to the pep538-coerce-c-locale branch in my GitHub fork: https://github.com/python/cpython/compare/master...ncoghlan:pep538-coerce-c-locale (That doesn't include Xavier's Android fixes yet, though) --

[issue28180] sys.getfilesystemencoding() should default to utf-8

Xavier de Gaye added the comment: > On Android, setlocale(CATEGORY, "") does not look for the locale environment > variables (LANG, ...) but sets the 'C' locale instead FWIW the source code of setlocale() on bionic (Android libc) is at

[issue28180] sys.getfilesystemencoding() should default to utf-8

Xavier de Gaye added the comment: pep538_coerce_legacy_c_locale_v3.diff fixes issue 28997 on Android (api 21 and 24). This issue is raised because there is an inconsistency between Python on Android that considers the locale encoding to be always UTF-8 and GNU Readline that does not accept

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Uploaded one last version of the patch implementing the previous PEP 538 design. This refactors the test cases so they systematically cover 4 cases that we expect to be reported as "the C locale": - LC_ALL, LC_CTYPE, and LANG all empty - one of them set to "C",

[issue28180] sys.getfilesystemencoding() should default to utf-8

Sworddragon added the comment: > $ cat badfilename.py > badfn = "こんにちは".encode('euc-jp').decode('utf-8', 'surrogateescape') > print("bad filename:", badfn) > > $ PYTHONIOENCODING=utf-8:backslashreplace python3 badfilename.py > bad filename: \udca4\udcb3\udca4\udcf3\udca4ˤ\udcc1\udca4\udccf > >

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: While the attached PEP 538 patches include their own tests, the uploaded pep538-check-click.sh script is the one I've been using to check that the changes have the desired effect of letting click "just work", even when the nominal locale is cleared, explicitly

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: I just pushed an update to PEP 538 based on PEP 540 and the feedback in the linux-sig discussion: https://github.com/python/peps/commit/221099d8765125bbd798e869846b005bcca84b47 I'll be starting a thread for that on python-ideas shortly, but in the context of

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: >> stderr is used to log errors. Getting a new error when trying to log >> an error is kind of annoying. > > Hm, what bad surprise/error could appear that would not appear with > backslashescape? $ cat badfilename.py badfn =

[issue28180] sys.getfilesystemencoding() should default to utf-8

Sworddragon added the comment: > What do you mean by "make the C locale"? I was pointing to the Platform Support Changes of PEP 538. > I'm not sure of the name of each mode yet. > > After having written the "Use Cases" section and especially the > Mojibake column of results, I consider the

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: Sworddragon added the comment: > (for me and maybe others that is explicitly preferred but maybe this depends > on each individual) That's why the PEP 540 has options to enable to disable its UTF-8 mode(s). > If I'm not wrong PEP 538 improves this for the

[issue28180] sys.getfilesystemencoding() should default to utf-8

Sworddragon added the comment: On looking into PEP 538 and PEP 540 I think PEP 540 is the way to go. It provides an option for a stronger encapsulation for the de-/encoding logic between the interpreter and the developer. Instead of caring about error handling the developer has now to care

[issue28180] sys.getfilesystemencoding() should default to utf-8

Jan Niklas Hasse added the comment: > Can you please tell me if these variables are set and if yes, give me their > value? None of these variables are set (with `docker run -it fedora:25 /bin/bash`). -- ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: And by PEP 528, I actually mean PEP 538 :) -- ___ Python tracker ___ ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Docker containers don't have a locale set by default - the approach proposed in PEP 528 actually comes from the way I configure Docker images (which in turn comes from Armin Ronacher's recommendations in click for Python 3 locale handling). In the Dockerfile

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: > Working with Docker I often end up with an environment where the locale isn't > correctly set. The locale encoding is controlled by 3 environment variables: LC_ALL, LC_CTYPE and LANG.

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: > That way each PEP can argue as strongly as it can for the respective authors > preferred approach to tackling the default C locale problem, even if they > point to a common background section in one of the PEPs (similar to the way > PEPs 522 and 524 shared

[issue28180] sys.getfilesystemencoding() should default to utf-8

Barry A. Warsaw added the comment: On Jan 05, 2017, at 11:11 AM, STINNER Victor wrote: >I'm sure that many Linux, UNIX and BSD systems don't have the "C.UTF-8" >locale. For example, HP-UX has "C.utf8" which is not exactly "C.UTF-8". > >I'm not sure that it's ok in 2017 to always force the UTF-8

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: The trade-offs here are incredibly complex (and are mainly a matter of deciding whose code and configurations we want to break in 3.7+), so I think competing PEPs are going to be better than attempting to create a combined PEP that tries to cover all the

[issue28180] sys.getfilesystemencoding() should default to utf-8

Marc-Andre Lemburg added the comment: While going for the full locale setting may be a good option, perhaps just focusing on the FS encoding for now is a better way forward (and also more in line with the ticket title). So essentially go for the PEP 529 approach on Unix as well (except that we

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: Sorry, I still didn't have enough time to read carefully the PEP 538. But since the discussion already started on this issue, I will add my comments: * I'm sure that many Linux, UNIX and BSD systems don't have the "C.UTF-8" locale. For example, HP-UX has

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: The PEP already explains how other runtimes achieve UTF-8 and UTF-18-LE everywhere: by ignoring the C/C++ locale entirely. While this breaks integration with other C/C++ components, the developers of those languages and runtimes simply don't care, as they never

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: No, requesting a locale that doesn't exist doesn't error out, because we don't check the return code - it just keeps working the same way it does now (i.e. falling back to the legacy C locale). However, it would be entirely reasonable to put together a

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: Why I want to add configure option to ignore locale is: 1. C.UTF-8 is not supported by RHEL7 (https://bugzilla.redhat.com/show_bug.cgi?id=1361965) RHEL7 will be used for a long time. And many people uses new Python instead of distro's Python, via pyenv or

[issue28180] sys.getfilesystemencoding() should default to utf-8

Marc-Andre Lemburg added the comment: On 05.01.2017 10:26, Nick Coghlan wrote: > > Anything purely on the Python side of things doesn't work in a traditional C > environment - CPython relies on the C lib to do conversions during startup, > so we need the C locale to be set correctly. We can

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: > Anything purely on the Python side of things doesn't work in a traditional C > environment - CPython relies on the C lib to do conversions during startup, > so we need the C locale to be set correctly. What I propose is non't use mbstowcs, like __ANDROID__

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Anything purely on the Python side of things doesn't work in a traditional C environment - CPython relies on the C lib to do conversions during startup, so we need the C locale to be set correctly. We can do things differently on Mac OS X and iOS because Apple

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: On Linux, I think most people wants UTF-8:surrogateescape by default, without fighting against locale and environment variables. There are already `#if defined(__APPLE__) || defined(__ANDROID__)` path for it. How about adding configure option to use same logic?

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: > The default encoding in the C/POSIX locale is ASCII (which is the entire > source of the problem). The reality is more complex than that :-) It depends on the OS. Some OS uses Latin1 for the POSIX locale. Some OS announces to use Latin1 for the POSIX

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: The default encoding in the C/POSIX locale is ASCII (which is the entire source of the problem). The initial verison of the PEP I uploaded didn't explain that background, but I added a section about it in the update earlier this week:

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: I'm sorry. I must search old discussion about why we can't simply use utf-8 for fsencoding when C locale, instead of asking here. -- ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: > That isn't the case on other *nix systems - there, we need CPython to be > consistent with the configured C/C++ locale, *and* we need it to be using > something other than ASCII as the default encoding. Isn't using UTF-8 as filesystem encoding and stdin/stdout

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: On Mac OS X, the XCode libc already ignores the locale settings and just uses UTF-8 as the default text encoding, so the hardcoding in CPython aligns with that behaviour. That isn't the case on other *nix systems - there, we need CPython to be consistent with

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: I read PEP 538 but I can't understand why just using UTF-8 when locale is C like macOS is bad idea. -- ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

Changes by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Updated patch adds some tests showing that this change should also help with cases where SSH environment forwarding results in an unknown locale being requested in the server environment. -- Added file:

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: If nothing is configured (i.e. none of LC_ALL, LC_CTYPE or LANG are set in the environment), then C reports the locale as "C". It's probably worthwhile for me to add a Background section to the PEP that explains the behaviour of ``setlocale`` at the C level, as

[issue28180] sys.getfilesystemencoding() should default to utf-8

Jan Niklas Hasse added the comment: Only important case for me: What when LANG is unset? -- ___ Python tracker ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: I've now written this up as a PEP: https://github.com/python/peps/blob/master/pep-0538.txt The latest attached patch implements the specific design proposed in the PEP. Relative to the last Fedora specific patch, this tweaks the warning message wording

[issue28180] sys.getfilesystemencoding() should default to utf-8

Changes by Sworddragon : -- nosy: +Sworddragon ___ Python tracker ___ ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

Changes by Akira Li <4kir4...@gmail.com>: -- nosy: +akira ___ Python tracker ___ ___ Python-bugs-list mailing

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: Previous related work: changeset: 89836:bc06f67234d0 user:Victor Stinner date:Tue Mar 18 01:18:21 2014 +0100 files: Doc/whatsnew/3.5.rst Lib/test/test_sys.py Misc/NEWS Python/pythonru description: Issue #19977:

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: For folks not following the Fedora BZ issue directly, I've also attached the latest draft downstream patch here, which gives the following behaviour: == $ ./python -c "import sys; print(sys.getfilesystemencoding())" utf-8 $ LANG=C.UTF-8

[issue28180] sys.getfilesystemencoding() should default to utf-8

Changes by Nick Coghlan : Added file: http://bugs.python.org/file45951/fedora-cpython-PYTHONALLOWCLOCALE.diff ___ Python tracker ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

Jan Niklas Hasse added the comment: > Usually, when a new option is added to Python, we add a command line option > (-X utf8) but also an environment variable: I propose PYTHONUTF8=1. > > Use your favorite method to define the env var "system wide" in your docker > containers. This doesn't

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: On 17 December 2016 at 20:15, Marc-Andre Lemburg wrote: > Another use case to consider is embedding the Python > interpreter in another application. In such situations, > the C locale will usually already be set by the main > application

[issue28180] sys.getfilesystemencoding() should default to utf-8

Marc-Andre Lemburg added the comment: On 17.12.2016 08:56, Nick Coghlan wrote: > > Making an explicit note of this so I remember to mention it in the draft PEP: > one of the biggest problems that arises in any attempt at a Python-only > solution to overriding the locale is that we can end up

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Making an explicit note of this so I remember to mention it in the draft PEP: one of the biggest problems that arises in any attempt at a Python-only solution to overriding the locale is that we can end up disagreeing with C/C++ extensions, and this is

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: We've been discussing this further downstream in the Fedora Python SIG, and we have a draft approach that we're pretty sure will work for us (based in turn on the approach Armin Ronacher came up with for click), and we think it should work for other distros as

[issue28180] sys.getfilesystemencoding() should default to utf-8

Changes by Chi Hsuan Yen : -- nosy: +Chi Hsuan Yen ___ Python tracker ___ ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: > I believe Victor put quite a bit of time into trying to get more selective > approaches to work reliably and eventually gave up. Yeah, it just doesn't work to use more than one encoding per process. You should use the same encoding for the whole lifetime of

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: Victor>> I proposed to add "-X utf8" command line option for UNIX to force utf8 encoding. Would it work for you? Jan Niklas Hasse> Unfortunately no, as this would mean I'll have to change all my python invocations in my scripts and it wouldn't work for

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Downstream Fedora issue proposing the above idea for F26: https://bugzilla.redhat.com/show_bug.cgi?id=1404918 I've also attached the patch from that issue here. -- keywords: +patch Added file:

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: The challenge that arises in being selective about this is that "sys.getfilesystemencoding()" is actually a misnomer, and some of the things we use it for (like decoding command line arguments and environment variables) necessarily happen *really* early in the

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: Sorry for confusing. I didn't meant defaulting LANG=C.UTF-8. I meant use UTF-8 as default fsencoding, stdioencoding regardless locale, and locale.getpreferredencoding() returns 'utf-8' when LC_CTYPE is ascii. -- ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

Marc-Andre Lemburg added the comment: If we just restrict this to the file system encoding (and not the whole LANG setting), how about: * default the file system encoding to 'utf-8' and use the surrogate escape handler as default error handler * add a PYTHONFSENCODING env var to set the

[issue28180] sys.getfilesystemencoding() should default to utf-8

Jan Niklas Hasse added the comment: https://sourceware.org/glibc/wiki/Proposals/C.UTF-8#Defaults mentions that C.UTF-8 should be glibc's default. This bug report also mentions Python: https://sourceware.org/bugzilla/show_bug.cgi?id=17318 It hasn't been fixed yet, though :/ --

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: >From CPython's point of view, glibc behaves the same way (i.e. reporting >`ascii` as the preferred encoding for operating system interfaces) regardless >of whether the cause is the locale not being set at all, or due to it being >explicitly set to the legacy

[issue28180] sys.getfilesystemencoding() should default to utf-8

Jan Niklas Hasse added the comment: Actually in a new Docker container, the LANG variable isn't set at all. Defaulting to UTF-8 in that case should be easier to reason about, shouldn't it? -- ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: Note also that if we say we're going to do this for 3.7, *and* go ahead and implement it, then distros may be more inclined to incorporate the same behavioural changes into distro-provided releases of 3.6, providing real world testing of the concept before we

[issue28180] sys.getfilesystemencoding() should default to utf-8

Nick Coghlan added the comment: I think we're genuinely getting to the point now where the majority of "LANG=C" cases are misconfigurations rather than intended behaviour. We're also to the point where: - on Mac OS X, binary system interfaces have been handled as UTF-8 by default since 3.0 -

[issue28180] sys.getfilesystemencoding() should default to utf-8

INADA Naoki added the comment: I want locale free Python which behaves like on C.UTF-8 locale. (stdio encoding, preferred encoding, weekday in _strptime._strptime, and more maybe) But Python 3.6 is feature freeze already >_<;; -- nosy: +inada.naoki

[issue28180] sys.getfilesystemencoding() should default to utf-8

Jan Niklas Hasse added the comment: Why not? -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: > is this someday already?) Not yet :-) -- ___ Python tracker ___ ___

[issue28180] sys.getfilesystemencoding() should default to utf-8

R. David Murray added the comment: I thought we "fixed" this by using surrogate escape when the locale was ASCII? We certainly have discussed changing the default and posix and so far have decided not to (someday that will change...is this someday already?) -- nosy: +r.david.murray

[issue28180] sys.getfilesystemencoding() should default to utf-8

Jan Niklas Hasse added the comment: Unfortunately no, as this would mean I'll have to change all my python invocations in my scripts and it wouldn't work for executable files with #!/usr/bin/env python3 would it? -- ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

STINNER Victor added the comment: > This is a duplicate of issue27781. issue27781 is specific to Windows. I'm not sure that it's the base in this issue. So I reopen the issue. @Jan Niklas Hasse: What is your OS? I proposed to add "-X utf8" command line option for UNIX to force utf8

[issue28180] sys.getfilesystemencoding() should default to utf-8

Emanuel Barry added the comment: This is a duplicate of issue27781. -- nosy: +ebarry resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> Change sys.getfilesystemencoding() on Windows to UTF-8 ___ Python tracker

[issue28180] sys.getfilesystemencoding() should default to utf-8

New submission from Jan Niklas Hasse: Working with Docker I often end up with an environment where the locale isn't correctly set. In these cases it would be great if sys.getfilesystemencoding() could default to 'utf-8' instead of 'ascii', as it's the encoding of the future and ascii is a