[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Changes by STINNER Victor : -- versions: +Python 3.2, Python 3.3 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Roundup Robot added the comment: New changeset 41658a4fb3cc by Victor Stinner in branch '3.2': Issue #16218, #16414, #16444: Backport FS_NONASCII, TESTFN_UNDECODABLE, http://hg.python.org/cpython/rev/41658a4fb3cc New changeset 4d40c1ce8566 by Victor Stinner in branch '3.3': (Merge 3.2) Issue #16

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

STINNER Victor added the comment: Ooook, all remaining issues about undecodable bytes should now be fixed (until someone opens a new one? :-)) -- resolution: -> fixed status: open -> closed ___ Python tracker ___

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Roundup Robot added the comment: New changeset ed0ff4b3d1c4 by Victor Stinner in branch 'default': Issue #16444: test more bytes in support.TESTFN_UNDECODABLE to support more Windows code pages http://hg.python.org/cpython/rev/ed0ff4b3d1c4 -- ___ Pyt

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Serhiy Storchaka added the comment: Ping. -- stage: -> patch review type: -> enhancement ___ Python tracker ___ ___ Python-bugs-list

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Changes by Andrew Svetlov : -- nosy: +asvetlov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Serhiy Storchaka added the comment: > You can uses cpXXX encodings explictly to read or write a file, but these > encodings are not used for sys.getfilesystemencoding() (or > sys.stdout.encoding). At least CP1251 has been used for many cyrillic locales in before-UTF8 age (I use it sometimes sti

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Roundup Robot added the comment: New changeset 6017f09ead53 by Victor Stinner in branch '3.3': Issue #16218, #16444: Backport improvment on tests for non-ASCII characters http://hg.python.org/cpython/rev/6017f09ead53 -- ___ Python tracker

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

STINNER Victor added the comment: > These encodings used not only on Windows. You can uses cpXXX encodings explictly to read or write a file, but these encodings are not used for sys.getfilesystemencoding() (or sys.stdout.encoding). -- ___ Python tr

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

STINNER Victor added the comment: > I suppose you noticed you broke a bunch of buildbots :) Failures occur on FreeBSD, OpenIndiana and some other buildbots which don't set a locale and so use the "C" locale. main() decodes command line arguments from the locale encoding using _Py_char2wchar().

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Serhiy Storchaka added the comment: These encodings used not only on Windows. -- ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

STINNER Victor added the comment: > TESTFN_UNDECODABLE is not detected for cp1250, cp1251, cp1252, cp1254, cp1257 > and cp1258. The Python encoding and the real codec used by Windows are different: Python fails to decode bytes 0x80-0x9f, but Windows does decode them. I prefer to avoid these b

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Roundup Robot added the comment: New changeset 398f8770bf0d by Victor Stinner in branch 'default': Issue #16444: disable undecodable characters in test_non_ascii() test until http://hg.python.org/cpython/rev/398f8770bf0d -- ___ Python tracker

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Antoine Pitrou added the comment: I suppose you noticed you broke a bunch of buildbots :) -- nosy: +pitrou ___ Python tracker ___ ___

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Serhiy Storchaka added the comment: TESTFN_UNDECODABLE is not detected for cp1250, cp1251, cp1252, cp1254, cp1257 and cp1258. Just add b'\x81\x98\xae\xd5\xff', at leas one of this bytes undecodable in some encoding which has any undecodable bytes. -- _

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Roundup Robot added the comment: New changeset 6b8a8bc6ba9c by Victor Stinner in branch 'default': Issue #16444, #16218: Use TESTFN_UNDECODABLE on UNIX http://hg.python.org/cpython/rev/6b8a8bc6ba9c -- nosy: +python-dev ___ Python tracker

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

STINNER Victor added the comment: > The matter is not only in the fact that tests passed. Right, but I don't want to introduce a regression :-) > They should fail if the original bug occurs again. Have you tried to restore > the bugs? test_cmd_line_script.test_non_ascii() comes from the issu

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Serhiy Storchaka added the comment: > The full test suite pass on: The matter is not only in the fact that tests passed. They should fail if the original bug occurs again. Have you tried to restore the bugs? -- ___ Python tracker

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Serhiy Storchaka added the comment: Try b'\xed\xb2\x80' and b'\xed\xb4\x80' for UTF-8 (on Unix and Mac OS X). b'\xed\xb2\x80' is b'\x80'.decode('utf-8', 'surrogateescape').encode('utf-8', 'surrogatepass'). b'\xed\xb4\x80' is '\udd00'.encode('utf-8', 'surrogatepass') and '\udd00' can't be encod

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

Serhiy Storchaka added the comment: Try b'\x81', b'\x98', b'\xae', b'\xd5', b'\xff'. They are undecodable in all 1-byte encodings. b'\x81' : shift_jis_2004 shift_jis shift_jisx0213 cp869 cp874 cp932 cp1250 cp1252 cp1253 cp1254 cp1255 cp1257 cp1258 b'\x98' : shift_jis_2004 shift_jis shift_jisx0

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

STINNER Victor added the comment: > Please test the patch on UNIX, Windows and Mac OS X. The full test suite pass on: * Linux with UTF-8 locale encoding * Linux with ASCII locale encoding * Windows with cp932 ANSI code page * Mac OS 10.8 with ASCII locale encoding (and utf-8/surrogateescape

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

STINNER Victor added the comment: > We may also use support.TESTFN_UNDECODABLE > in test_cmd_line_script.test_non_ascii() on Windows Oh, subprocess doesn't support passing bytes arguments to a program anymore (since Python 3.0). http://bugs.python.org/issue4036#msg100376 So it's better to use

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

STINNER Victor added the comment: The patch contains two print to help debugging the patch itself, these print statements must be removed later. +print("TESTFN_UNDECODABLE = %a" % TESTFN_UNDECODABLE) +print("TESTFN_NONASCII = %a" % TESTFN_NONASCII) --

[issue16444] Use support.TESTFN_UNDECODABLE on UNIX

New submission from STINNER Victor: Attached patch changes how support.TESTFN_UNDECODABLE is computed on UNIX: use the filesystem encoding in *strict* mode, not using the surrogateescape error handler. So we can use support.TESTFN_UNDECODABLE to check if a function uses correctly the surrogat