[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ned Deily added the comment: Ronald's PR 14738 LGTM. I merged it to master and backported for 3.8.0b4 and 3.7.5. Thanks, everyone! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 3.8, Python 3.9 -Python 2.7, Python 3.4, Python 3.5,

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

miss-islington added the comment: New changeset e471a543a4f7c52a8d0081ec5142adab3416d8fb by Miss Islington (bot) in branch '3.8': bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename (GH-14736)

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

miss-islington added the comment: New changeset 554143ebc2546e0b8b722dfafe397c0316f29980 by Miss Islington (bot) in branch '3.7': bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename (GH-14736)

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Change by miss-islington : -- pull_requests: +15245 pull_request: https://github.com/python/cpython/pull/15569 ___ Python tracker ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Change by miss-islington : -- pull_requests: +15246 pull_request: https://github.com/python/cpython/pull/15570 ___ Python tracker ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ned Deily added the comment: New changeset b0caf329815120acf50287e29858093d328b0e3c by Ned Deily (Ronald Oussoren) in branch 'master': bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename (GH-14736)

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Change by Mickaël Schoentgen : -- nosy: +Tiger-222 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: As promised there is now a pull request. I'd love a review (and a change to approve the pull request when reviewers are happy, I'm trying to get back into actively contributing). --- I now understand why locale.getdefaultlocale() fails even when LC_CTYPE

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Change by Ronald Oussoren : -- pull_requests: +14530 pull_request: https://github.com/python/cpython/pull/14736 ___ Python tracker ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: LC_CTYPE=UTF-8 is a valid configuration on macOS, and is in the default environment when you install a fresh system. This includes the beta's for macOS 10.15 and is therefore unlikely to change anytime soon. Interestingly enough I get this error even when

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ricardo F added the comment: I still have this issue on MacOS Mojave 10.14 Python 3.7.2 (default, Dec 27 2018, 07:35:06) [Clang 10.0.0 (clang-1000.11.45.5)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import locale >>> locale.getdefaultlocale()

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Matthew Woodcraft added the comment: That alias (C.UTF-8 to en_US.UTF-8) is surely a bug in itself nowadays. I've filed #30755 . -- nosy: +mattheww ___ Python tracker

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Marc-Andre Lemburg added the comment: On 13.01.2017 04:47, Nick Coghlan wrote: > Accepting "UTF-8" and interpreting it as functionally equivalent to C.UTF-8 > will mean that this setting will at least work as desired on servers that > offer C.UTF-8. I don't think that's within the scope of

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Changes by STINNER Victor : -- nosy: -haypo ___ Python tracker ___ ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Nick Coghlan added the comment: SSH environment forwarding will propagate this "LC_CTYPE=UTF-8" setting from Mac OS X clients to Linux servers. At present, that breaks in multiple ways, as CPython will interpret it as being the "C" locale (since Linux servers don't offer a "UTF-8" locale,

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Marc-Andre Lemburg added the comment: I think Ronald's patch issue18378-2015-07-25-py36.txt with added darwin check would be the best way forward. In the current form, it would allow using 'UTF-8' as locale string on all platforms - which is not such a good idea. -- nosy: +lemburg

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Wolfgang Maier added the comment: I think PEP 538 extended to the UTF-8 locale *would* help here. Specifically, it would coerce only LC_CTYPE to en_US.UTF-8 (unless OS X has C.UTF-8), which I guess is good enough for the purpose here. I do agree that it is not the kind of problem that PEP 538

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Nick Coghlan added the comment: PEP 538 wouldn't help here, as there's nothing wrong with CPython's assumptions about the text encoding to use for operating system interfaces - it's assuming UTF-8 (because it's Mac OS X) and that assumption is correct (because it's Mac OS X). The problem

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Wolfgang Maier added the comment: To me this issue seems quite related to PEP 538. Maybe the LC_CTYPE coercion proposed in the PEP could be extended to cover the case of LC_CTYPE=UTF-8? -- nosy: +ncoghlan versions: +Python 3.7 ___ Python tracker

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

László Károlyi added the comment: OSX Sierra + Python, the bug still exists. subscribing -- nosy: +karolyi ___ Python tracker ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Changes by Lawrence Velázquez : -- nosy: +larryv ___ Python tracker ___ ___ Python-bugs-list

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Changes by alexander.sturm : -- nosy: +alexander.sturm ___ Python tracker ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ilya Kulakov added the comment: Could someone provide a patch for Python 3.5? -- nosy: +Ilya.Kulakov ___ Python tracker ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Changes by Tommy Sparber : -- nosy: +tsparber ___ Python tracker ___ ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Wolfgang Maier added the comment: ping? Just ran into this issue on OS X El Capitan with Region set to Germany and Language to English. Just as Ned pointed out 2 years ago, this results in LC_CTYPE set to 'UTF-8' in the terminal and docutils still can't cope with it. -- nosy: +wolma

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Serhiy Storchaka added the comment: Yes, I were testing on a Linux machine and forgot that results are OS depending. I agree, that test should less depend on implementation details. As far as _locale._getdefaultlocale is defined only on Windows and UTF-8 is not valid locale on Windows, I

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: ping... I think the current behavior is a bug in Python and should be fixed in 2.7, 3.4, 3.5 and default (using Dmitry's patch). I'd like to commit the patch, but would like someone else's review of the patch before doing so. --

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Serhiy Storchaka added the comment: Needed tests. With the patch: $ LC_CTYPE=UTF-8 ./python import locale locale.getdefaultlocale() (None, 'UTF-8') locale.getpreferredencoding() 'ANSI_X3.4-1968' locale.getlocale() (None, None) $ LC_CTYPE=en_US_UTF-8 ./python import locale

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +serhiy.storchaka versions: +Python 3.5, Python 3.6 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18378 ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Serhiy Storchaka added the comment: Perhaps the better way to solve this issue is to use aliases table. What is the LC_CTYPE environment variable set when the default language set to non-English? How different native MacOS X command-line programs behave when set LC_CTYPE to other encoding

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: The alias mechanism cannot be used because LC_CTYPE=UTF-8 as the locale doesn't imply anything about languages. In Linux terms it is more or less equal to C.UTF-8 or POSIX.UTF-8, except that those two aren't valid locales on OSX. --

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: Testing this is interesting to say the least due to the dynamic way the module interface is built. Serhiy: are you testing on a Linux machine? On my machine getpreferredencoding() returns 'UTF-8' because it hits the CODESET path (which ends up calling

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: The only locale that doesn't include language information is the UTF-8 one, there is no locale named US-ASCII. See /usr/share/locale on an OSX system. PS. The more I look at locale.py the more problems I find with it. The code makes a unwarranted

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: I've attached a patch with more tests, but I'm not to happy about the new test because it too much of a white box test and is therefore fairly fragile w.r.t. the actual implementation of the module. -- Added file:

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: Dimitry's patch looks good, I added my patch before checking if there already is patch. The only thing that might be cause discussion is when to accept 'UTF-8' as a valid locale name. My patch only accepts in on OSX, while Dimitry's patch accepts it

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Changes by Ronald Oussoren ronaldousso...@mac.com: Added file: http://bugs.python.org/file39384/issue-18378-py27.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18378 ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Changes by Ronald Oussoren ronaldousso...@mac.com: Added file: http://bugs.python.org/file39385/issue-18378-py35.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18378 ___

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: 1) I agree with Ned that the OSX behavior is not broken, it is different but within spec. Python makes assumption about the format of locale names that aren't universally valid. 2) We should be careful in using CFLocale. Those APIs are part of

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Barry Alan Scott added the comment: Mac OS X use the __CF_USER_TEXT_ENCODING env var to setup the locale in for native libraries. I found that for GUI python code I needed to convert the value in __CF_USER_TEXT_ENCODING into a suitable call to setlocale(). The code I use is attached to

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ned Deily added the comment: I've looked at this a bit, primarily on OS X 10.9 Mavericks, although I expect mostly similar behavior on older recent releases of OS X. On 10.9, the setting of locale variables is done by whatever program is used to launch a shell. I looked at the behavior of

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: I didn't get this on my previous system (which was basically a 10.4 system updated through 10.5, 10.7, ..., to 10.9), but did get it on my current system, which has a fresh 10.9 install where I did not use the migration assistent to migrate settings. Thus

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: With the following C code: #include locale.h #include stdio.h int main(void) { char* res = setlocale(LC_CTYPE, UTF-8); printf(Result: %s\n, res); res = setlocale(LC_CTYPE, UTF-9); printf(Result: %s\n, res); return 0; }

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

R. David Murray added the comment: That is seriously broken on Apple's part. But I guess we have no choice but to emulate their bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18378

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ned Deily added the comment: Ronald or Dmitry, can you elaborate under what conditions you start your login shell on 10.9? I cannot reproduce the behavior you observe. With 10.9 Terminal.app and the default language settings in System Preferences and with the default Terminal.app

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: I just ran into this problem myself. On fresh installs of OSX 10.9 LC_CTYPE is set to UTF-8 (at least for english language users), and now sphinx won't work :-( Is Dimitrys patch acceptable (either as is, or with my suggestion of checking for sys.platform

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Martin v. Löwis added the comment: Why do you need the getdefaultlocale function in the first place? I'd advise against using it, precisely because it can trigger problems like this one. -- nosy: +loewis ___ Python tracker rep...@bugs.python.org

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Dmitry Jemerov added the comment: I personally don't, but the function is used by Sphinx, which is what I was trying to get to work when I ran into this problem. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18378

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

R. David Murray added the comment: Regardless of the resolution here, the use of getdefaultlocale could be reported as a bug on the sphinx tacker. -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18378

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Martin v. Löwis added the comment: FWIW, I couldn't find any use of getdefaultlocale in any of the hg revisions (using hg grep) in https://bitbucket.org/birkenfeld/sphinx/ Instead, it's (probably) docutils, which has this code: locale_encoding = locale.getlocale()[1] or

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

New submission from Dmitry Jemerov: On Mac OS X 10.8 with the default language set to English (System Preferences | Language and Text), the default terminal application sets the LC_CTYPE environment variable to UTF-8. If you run Python from the terminal and try to use

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: Strange, I have LANG=en_US.UTF-8 in my environment and no LC_CTYPE. A clean test account does have the same behavior as you are seeing. -- nosy: +ronaldoussoren ___ Python tracker rep...@bugs.python.org

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18378 ___ ___ Python-bugs-list

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: The UTF-8 value seems suspect to me, but is actually supported by the system, changing it to a nonsense value results in failure in the C function setlocale. As for the patch: I'd add this workaround only to the OSX platform (that is, test for sys.platform

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Dmitry Jemerov added the comment: Judging from the results of Googling for the error message, I'm far from the only one seeing this problem. What exactly would be the benefit of adding the code to check for the platform? -- ___ Python tracker

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Ronald Oussoren added the comment: The test for darwin is needed because other platforms don't support UTF-8 as a valid LC_CTYPE name, on a recent linux box: locale.setlocale(locale.LC_CTYPE, UTF-8) Traceback (most recent call last): File stdin, line 1, in module File

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Dmitry Jemerov added the comment: Why exactly does this matter? UTF-8 not being a valid LC_CTYPE value simply means that no one running Linux will ever have LC_CTYPE set to UTF-8, and the branch will never be hit. OTOH, adding the check will make the code harder to test and simply larger (no

[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English

Dmitry Jemerov added the comment: A related issue (with a patch that touches the same locale parsing code) is http://bugs.python.org/issue5815 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18378