[issue29240] PEP 540: Add a new UTF-8 mode

2022-02-08 Thread Chih-Hsuan Yen 


Change by Chih-Hsuan Yen :


--
nosy:  -yan12125

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2022-02-06 Thread STINNER Victor 


STINNER Victor  added the comment:

> New changeset 91106cd9ff2f321c0f60fbaa09fd46c80aa5c266 by Victor Stinner in 
> branch 'master':
> bpo-29240: PEP 540: Add a new UTF-8 Mode (#855)
> https://github.com/python/cpython/commit/91106cd9ff2f321c0f60fbaa09fd46c80aa5c266

Oh, this change broke the mbcs alias on Windows and the test_codecs and 
test_site tests (2 tests!) missed the bug :-( I fixed it in:

New changeset 04dd60e50cd3da48fd19cdab4c0e4cc600d6af30 by Victor Stinner in 
branch 'main':
bpo-46659: Update the test on the mbcs codec alias (GH-31168)
https://github.com/python/cpython/commit/04dd60e50cd3da48fd19cdab4c0e4cc600d6af30

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-22 Thread STINNER Victor 

STINNER Victor  added the comment:

I partially reverted the commit 7ed7aead9503102d2ed316175f198104e0cd674c: on 
Android, UTF-8 is now always used, again. Paul Peny (aka pmpp) confirmed me 
that my commit broke Python on Android, at least with API 19 (locales don't 
work properly before API 21).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-22 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset 9089a265918754d95e105a7c4c409ac9352c87bb by Victor Stinner in 
branch 'master':
bpo-29240: PyUnicode_DecodeLocale() uses UTF-8 on Android (#5272)
https://github.com/python/cpython/commit/9089a265918754d95e105a7c4c409ac9352c87bb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-22 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +5116

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-17 Thread STINNER Victor 

STINNER Victor  added the comment:

test_readline pass again on all buildbots, especially on FreeBSD 3.6 and 3.x 
buildbots.

There are no more known issues, the implementation of the PEP 540 (UTF-8 Mode) 
is now complete!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-16 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset c2740e8a263e76427a8102a89f4b491a3089b2a1 by Victor Stinner (Miss 
Islington (bot)) in branch '3.6':
Skip test_readline.test_nonascii() on C locale (GH-5203) (#5204)
https://github.com/python/cpython/commit/c2740e8a263e76427a8102a89f4b491a3089b2a1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-16 Thread Roundup Robot 

Change by Roundup Robot :


--
pull_requests: +5058

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-16 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset c495e799ed376af91ae2ddf6c4bcc592490fe294 by Victor Stinner in 
branch 'master':
Skip test_readline.test_nonascii() on C locale (#5203)
https://github.com/python/cpython/commit/c495e799ed376af91ae2ddf6c4bcc592490fe294


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-16 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +5057

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-15 Thread STINNER Victor 

STINNER Victor  added the comment:

> New changeset 7ed7aead9503102d2ed316175f198104e0cd674c by Victor Stinner in 
> branch 'master':
> bpo-29240: Fix locale encodings in UTF-8 Mode (#5170)

Oh, this change broke test_nonascii() of test_readline() on FreeBSD.

Previsously, readline used ASCII/surrogateescape encoding for the POSIX locale. 
Now, mbstowcs() / wcstombs() is called directly, with the surrogateescape error 
handler.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-15 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +5043

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-15 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset 7ed7aead9503102d2ed316175f198104e0cd674c by Victor Stinner in 
branch 'master':
bpo-29240: Fix locale encodings in UTF-8 Mode (#5170)
https://github.com/python/cpython/commit/7ed7aead9503102d2ed316175f198104e0cd674c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-15 Thread STINNER Victor 

STINNER Victor  added the comment:

Attached test_all_locales.py is a test suite for locale functions: 
os.strerror(), locale.localeconv(), time.strftime(). I tested it on Linux 
Fedora 27, FreeBSD 11.0 and macOS 10.13.2.

The test should always pass on Python 2.7. On Python 3.6 and the master branch 
with PR 5170, 2 tests on numeric localeconv() fail because Python uses the 
wrong encoding: see bpo-31900. master with PR 5170 now has less encoding bugs 
than Python 3.6.

--
Added file: https://bugs.python.org/file47385/test_all_locales.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-12 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +5024

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-11 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset cb3ae5588bd7733e76dc09277bb7626652d9bb64 by Victor Stinner in 
branch 'master':
bpo-29240: Ignore UTF-8 Mode in time module (#5148)
https://github.com/python/cpython/commit/cb3ae5588bd7733e76dc09277bb7626652d9bb64


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-10 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +5005

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-10 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset 2cba6b85797ba60d67389126f184aad5c9e02ff3 by Victor Stinner in 
branch 'master':
bpo-29240: readline now ignores the UTF-8 Mode (#5145)
https://github.com/python/cpython/commit/2cba6b85797ba60d67389126f184aad5c9e02ff3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2018-01-10 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +5003

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-21 Thread STINNER Victor 

STINNER Victor  added the comment:

IHMO test_readline should be fixed by ignoring the UTF-8 mode in 
Py_EncodeLocale/Py_DecodeLocale, but only when called from the Python readline 
module. We need maybe new functions, something like: 
Py_EncodeCurrentLocale/Py_DecodeCurrentLocale.

I will work on a patch when I will be back from holiday. In the meanwhile, I 
skipped the test to repair FreeBSD 3.x buildbots.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-21 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset 424315fa865b43f67e36a40647107379adf031da by Victor Stinner in 
branch 'master':
bpo-29240: Skip test_readline.test_nonascii() (#4968)
https://github.com/python/cpython/commit/424315fa865b43f67e36a40647107379adf031da


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-21 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +4860

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-15 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset 9454060e84a669dde63824d9e2fcaf295e34f687 by Victor Stinner in 
branch 'master':
bpo-29240, bpo-32030: Py_Main() re-reads config if encoding changes (#4899)
https://github.com/python/cpython/commit/9454060e84a669dde63824d9e2fcaf295e34f687


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-15 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +4793

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-15 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset d2b02310acbfe6c978a8ad3cd3ac8b3f12927442 by Victor Stinner in 
branch 'master':
bpo-29240: Don't define decode_locale() on macOS (#4895)
https://github.com/python/cpython/commit/d2b02310acbfe6c978a8ad3cd3ac8b3f12927442


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-15 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +4787

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-13 Thread STINNER Victor 

STINNER Victor  added the comment:

test_readline failed. It seems to be related to my commit:

http://buildbot.python.org/all/#/builders/87/builds/360

==
FAIL: test_nonascii (test.test_readline.TestReadline)
--
Traceback (most recent call last):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_readline.py",
 line 219, in test_nonascii
self.assertIn(b"text 't\\xeb'\r\n", output)
AssertionError: b"text 't\\xeb'\r\n" not found in 
bytearray(b"^A^B^B^B^B^B^B^B\t\tx\t\r\n[\\303\\257nserted]|t\x07\x08\x08\x08\x08\x08\x08\x08\x07\x07xrted]|t\x08\x08\x08\x08\x08\x08\x08\x07\r\nresult
 \'[\\xefnsexrted]|t\'\r\nhistory \'[\\xefnsexrted]|t\'\r\n")

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-13 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset d5dda98fa80405db82e2eb36ac48671b4c8c0983 by Victor Stinner in 
branch 'master':
pymain_set_sys_argv() now copies argv (#4838)
https://github.com/python/cpython/commit/d5dda98fa80405db82e2eb36ac48671b4c8c0983


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-13 Thread STINNER Victor 

Change by STINNER Victor :


--
pull_requests: +4727
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-13 Thread STINNER Victor 

STINNER Victor  added the comment:


New changeset 91106cd9ff2f321c0f60fbaa09fd46c80aa5c266 by Victor Stinner in 
branch 'master':
bpo-29240: PEP 540: Add a new UTF-8 Mode (#855)
https://github.com/python/cpython/commit/91106cd9ff2f321c0f60fbaa09fd46c80aa5c266


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-12 Thread STINNER Victor 

STINNER Victor  added the comment:

Oh, PYTHONCOERCECLOCALE env var is read very early in main() by 
_Py_CoerceLegacyLocale(), it ignores -E command line option.

 * Ignoring -E and -I is safe from a security perspective, as we only use
 * the setting to turn *off* the implicit locale coercion, and anyone with
 * access to the process environment already has the ability to set
 * `LC_ALL=C` to override the C level locale settings anyway.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-12 Thread STINNER Victor 

STINNER Victor  added the comment:

The PEP 538 has two open issues: bpo-30672 and bpo-32238.

I recently refactored the Py_Main() code so it should be simpler to implement 
the PEP 540: see bpo-32030.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

STINNER Victor  added the comment:

I removed old patches in favor of the now up to date PR 855.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

Change by STINNER Victor :


Removed file: https://bugs.python.org/file46274/encodings.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

Change by STINNER Victor :


--
title: [WIP] Implementation of the PEP 540: Add a new UTF-8 mode -> PEP 540: 
Add a new UTF-8 mode

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

Change by STINNER Victor :


Removed file: https://bugs.python.org/file46263/pep540-3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

Change by STINNER Victor :


Removed file: https://bugs.python.org/file46258/pep540.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

Change by STINNER Victor :


Removed file: https://bugs.python.org/file46257/pep540_cli.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

STINNER Victor  added the comment:

I rebased my PR on master.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

Change by STINNER Victor :


Removed file: https://bugs.python.org/file46270/pep540-4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

2017-12-05 Thread STINNER Victor 

Change by STINNER Victor :


Removed file: https://bugs.python.org/file46262/pep540-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com