[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

I tested my 3.7 and master fixes: they fix test_c_locale_coercion on the 
FreeBSD CURRENT buildbot.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:


New changeset f6e323ce322cf54b1a9e9252b13f93ebc28b5c24 by Victor Stinner in 
branch '3.7':
bpo-34523: Fix C locale coercion on FreeBSD CURRENT (GH-10672) (GH-10673)
https://github.com/python/cpython/commit/f6e323ce322cf54b1a9e9252b13f93ebc28b5c24


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9929

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 353933e712b6c7f7ba9a9a50bd5bd472db7c35d0 by Victor Stinner in 
branch 'master':
bpo-34523: Fix C locale coercion on FreeBSD CURRENT (GH-10672)
https://github.com/python/cpython/commit/353933e712b6c7f7ba9a9a50bd5bd472db7c35d0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

The bug is that sys.getfilesystemencoding() should be the locale encoding. On 
FreeBSD CURRENT, Python uses ASCII for the filesystem encoding, whereas the 
locale encoding is UTF-8.

CURRENT-amd64% env -i ./python -X utf8=0 -c 'import locale, sys; 
print(sys.getfilesystemencoding(), locale.getpreferredencoding())'
ascii UTF-8

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

Oh, I also modified Python 3.7 to respect the "force ASCII" mode in bpo-34403, 
commit 21220bbe65108f5a763ead24a6b572f80d84c9e2. Hopefully, this change is not 
part of the latest Python 3.7.1 bugfix release.

By the way, the bug only impacts FreeBSD CURRENT users (after Nov 4, after 
C.UTF-8 has been added) who have no locale set (LC_CTYPE is "C", ex: python3 
running in an empty environment) and explicitly disable the UTF-8 mode... Well, 
I guess that very few users (I mean no one) are impacted :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

Ok, I found the bug: it is related to the commit 
905f1ace5f7424e314ca7bed997868a2a3044839 of bpo-34523. Python now uses ASCII 
for the filesystem encoding if the internal "force ASCII" mode is enabled in 
Py_DecodeLocale/Py_EncodeLocale.

FreeBSD CURRENT introduced a new C.UTF-8 locale which is used by C locale 
coercion (PEP 538).

_PyCoreConfig_Read() now uses ASCII for the filesystem encoding if the "force 
ASCII" mode is enable (to ensure that encodings are consistent everywhere 
Python).

Problem: The LC_CTYPE now uses UTF-8 (thanks to C locale coercion) whereas 
Python uses ASCII for the filesystem encoding since the "force ASCII" is still 
enabled.

I wrote PR 10672 to reset the "force ASCII" mode. I tested manually my fix on 
the FreeBSD CURRENT buildbot.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9927

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

> I'm wondering if we may need to make test_c_locale_coercion less opinionated, 
> where for most platforms it's just a "don't segfault" test, and it's only for 
> ones where we're certain about how we expect it to behave (i.e. Linux, Mac OS 
> X, AIX, maybe FreeBSD) that we try to check the specifics.

For test_embed, I tried to be smart to guess the encodings (ex: hardcode some 
encodings depending on  the platform) and... I gave up. It's very hard when you 
consider Linux, Windows, macOS, FreeBSD, but also HP-UX and AIX. Each platform 
has weird cases about encodings... Now the test spawns a Python subprocess to 
get the encoding. See get_stdio_encoding() and get_filesystem_encoding() in 
Lib/test/test_embed.py.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[Kubilay Kocak]

Change by Kubilay Kocak :


--
nosy: +koobs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[Nick Coghlan]

Nick Coghlan  added the comment:

Ah, interesting. https://bugs.python.org/issue30672 covered the fact that this 
test was already weird on FreeBSD, but the current status is that it was 
expecting BSD variants to act somewhat like Mac OS X, not like Linux.

I'm wondering if we may need to make test_c_locale_coercion less opinionated, 
where for most platforms it's just a "don't segfault" test, and it's only for 
ones where we're certain about how we expect it to behave (i.e. Linux, Mac OS 
X, AIX, maybe FreeBSD) that we try to check the specifics.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35290] [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

More FreeBSD commits, "Teach man(1) about C.UTF-8":
https://svnweb.freebsd.org/base?view=revision=340128

---


EXPECTED_C_LOCALE_STREAM_ENCODING = 'ascii'
EXPECTED_C_LOCALE_FS_ENCODING = 'ascii'

I guess that the problem comes from that. But I'm not sure of what is the FS 
encoding for LC_ALL=C or LC_ALL=C.UTF-8.

--
title: [FreeBSD] test_c_locale_coercion: fsencoding is ASCII instead of UTF-8 
on FreeBSD CURRENT buildbot -> [FreeBSD] test_c_locale_coercion doesn't support 
new C.UTF-8 locale of FreeBSD CURRENT

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com