[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Inada Naoki added the comment: Thank you for finding/fixing. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

miss-islington added the comment: New changeset f07448bef48d645c8cee862b1f25a99003a6140e by Miss Skeleton (bot) in branch '3.9': bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466) https://github.com/python/cpython/commit/f07448bef48d645c8cee862b1f25a99003a6140e

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

miss-islington added the comment: New changeset 47ca6799725bb4c40953bb26ebcd726d1d766361 by Miss Skeleton (bot) in branch '3.8': bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466) https://github.com/python/cpython/commit/47ca6799725bb4c40953bb26ebcd726d1d766361

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Change by miss-islington : -- pull_requests: +21675 pull_request: https://github.com/python/cpython/pull/22705 ___ Python tracker ___

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Inada Naoki added the comment: New changeset 2d2af320d94afc6561e8f8adf174c9d3fd9065bc by Kevin Adler in branch 'master': bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466) https://github.com/python/cpython/commit/2d2af320d94afc6561e8f8adf174c9d3fd9065bc --

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Change by miss-islington : -- nosy: +miss-islington nosy_count: 3.0 -> 4.0 pull_requests: +21674 pull_request: https://github.com/python/cpython/pull/22704 ___ Python tracker

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Inada Naoki added the comment: Yes, please. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Kevin added the comment: Ok, so should I switch the PR back from PyUnicode_DecodeFSDefault? -- ___ Python tracker ___ ___

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Inada Naoki added the comment: OK. Let's use PyUnicode_DecodeLocale() with surrogateescape for consistency. -- ___ Python tracker ___

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Serhiy Storchaka added the comment: In os.strerror() and PyErr_SetFromErrnoWithFilenameObjects() we use PyUnicode_DecodeLocale(s, "surrogateescape") for decoding the result of strerror(). -- ___ Python tracker

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Inada Naoki added the comment: > So the main problem is: should we allow surrogateescape in error message? Note that error message may be written to file, stream, structured log (JSON). They may be UTF-8:strict. We can not write surrogateescape-d string to them. --

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Inada Naoki added the comment: > I think that it is more correct to use the locale encoding. If error messages > are translated for readability, we should not ruin this by outputting \xXX. * PyUnicode_DecodeLocale() doesn't support "backslashescape" error handler. * Error message is usually

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Serhiy Storchaka added the comment: I think that it is more correct to use the locale encoding. If error messages are translated for readability, we should not ruin this by outputting \xXX. -- nosy: +serhiy.storchaka ___ Python tracker

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Inada Naoki added the comment: > I have since changed the PR to use PyUnicode_DecodeFSDefault based on review > feedback. I was going to say that you will have to fight it out with @methane > on GH, but I see that that's you. :D Would have been nice if you would have > left the updated

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Kevin added the comment: Glad you were able to reproduce on Linux. I have since changed the PR to use PyUnicode_DecodeFSDefault based on review feedback. I was going to say that you will have to fight it out with @methane on GH, but I see that that's you. :D Would have been nice if you would

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Inada Naoki added the comment: I succeeded to reproduce it on Ubuntu 20.04. $ sudo vi /var/lib/locales/supported.d/ja # add "ja_JP.EUC-JP EUC-JP" $ sudo locale-gen ja_JP.EUC-JP Generating locales (this might take a while)... ja_JP.EUC-JP... done Generation complete. $

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Change by Tal Einat : -- versions: -Python 3.5, Python 3.6, Python 3.7 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

Change by Kevin : -- keywords: +patch pull_requests: +21490 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22466 ___ Python tracker ___

[issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

New submission from Kevin : If a native module fails to load, the dynload code will call PyUnicode_FromString on the error message to give back to the user. This can cause a UnicodeDecodeError if the locale is not a UTF-8 locale and the error message contains non-ASCII code points. While