[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: PEP 538 is implemented now in the master branch. Closing this issue as out of date for the following reasons: a) PR 4334 adds locale coercion for Android to the implementation of PEP 538 (and test_nonascii succeeds with this PR). b) With

[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: > I think we should wait for the resolution of PEP 538 and of issue 28684 s/issue 28684/issue 28180/ -- ___ Python tracker

[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: PEP 538 [1] coerces the C locale to UTF-8 by setting the locale environment variables (LC_ALL, LANG). The PEP has an implementation at issue 28180 as pep538_coerce_legacy_c_locale_v3.diff, and the patch fixes test_nonascii when run on the Android emulators

[issue28997] test_readline.test_nonascii fails on Android

Martin Panter added the comment: So the problem seems to be that Python assumes Readline’s encoding is UTF-8, but Readline actually uses ASCII (depending on locale variables). The code at the start of the test is supposed to catch when add_history() calls PyUnicode_EncodeLocale() and fails.

[issue28997] test_readline.test_nonascii fails on Android

Changes by Xavier de Gaye : -- assignee: xdegaye -> ___ Python tracker ___ ___

[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: Trying to track the 'char * text' pointer that is returned to Python by readline in rlhandler() and using gdb 'set follow-fork-mode child' and 'set detach-on-fork off' to stop (with a breakpoint in rlhandler) in the child spawned by run_pty() in

[issue28997] test_readline.test_nonascii fails on Android

Martin Panter added the comment: . Thanks for the explanation. It sounds like the Readline library assumes an ASCII-only locale and sets its “convert-meta” variable to “on”. But Python assumes UTF-8 and inputs b"\xC3\xAB" to the terminal. Readline converts the input to two escape sequences:

[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: With your patch Martin, test_nonascii is skipped on Android when LANG is not set and the issue is fixed. > Just to clarify, is the problem that Python (correctly) assumes UTF-8 > encoding on Android, but Readline does not unless you tweak the environment >

[issue28997] test_readline.test_nonascii fails on Android

Martin Panter added the comment: The basic idea of your patch may be reasonable, but something is not right. Imagine the locale is something other than UTF-8. The input code will now contain mojibake print("\xC3\xAB"), although the decode() call will translate the result back to the expected

[issue28997] test_readline.test_nonascii fails on Android

Changes by Xavier de Gaye : -- stage: needs patch -> patch review ___ Python tracker ___

[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: The test is ok on Android when LANG=en_US.UTF-8. When LANG is not set, the check made by the first statement in test_nonascii() should skip the test but fails to skip it on Android as Py_EncodeLocale() always encode to utf8 whatever the locale (same as with

[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: Checking that the the problem is indeed an encoding problem on Android with readline.set_pre_input_hook(): with tracing_the_completer_2.patch that replaces '\xEB' with 'A', the test now succeeds and prints: test_nonascii (test.test_readline.TestReadline) ...

[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: FWIW on Android we have: >>> readline.__doc__ and "libedit" in readline.__doc__ False >>> getattr(readline, "set_pre_input_hook", None) -- ___ Python tracker

[issue28997] test_readline.test_nonascii fails on Android

Xavier de Gaye added the comment: > For me, it prints this: (includes “te” with an umlaut) I got this same result when testing on Android with the interactive interpreter. See the content of the 'completer' list below, when run with tracing_the_completer.patch: On linux: test_nonascii

[issue28997] test_readline.test_nonascii fails on Android

Martin Panter added the comment: This is a breakdown of running the test script on my Linux setup (UTF-8 locale): ^A^B^B^B^B^B^B^B\t\tx\t\r\n Input echoed back (before Readline disables echo) [\xc3\xafnserted] Inserted by pre_input_hook() |t\xc3\xab[after] Inserted by the Ctrl+A

[issue28997] test_readline.test_nonascii fails on Android

New submission from Xavier de Gaye: test_nonascii has been implemented in issue 16182. The error message: == FAIL: test_nonascii (test.test_readline.TestReadline)