[issue17620] Python interactive console doesn't use sys.stdin for input

Change by Eryk Sun : -- versions: +Python 3.10, Python 3.8, Python 3.9 -Python 3.6 ___ Python tracker ___ ___ Python-bugs-list

[issue17620] Python interactive console doesn't use sys.stdin for input

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue17620] Python interactive console doesn't use sys.stdin for input

Steve Dower added the comment: Unassigning this. I meant to close it with another fix, but that would be wrong as we really ought to keep this open until we solve it properly. All I've done is make it use the right APIs on Windows, but we still don't handle it properly when we change stdin.

[issue17620] Python interactive console doesn't use sys.stdin for input

Adam Bartoš added the comment: > Unfortunately, it looks like detecting when a readline hook has been added is > going to involve significant changes to the tokenizer, which I really don't > want to do. We don't need to detect the presence of readline hook, it may be so that there is always

[issue17620] Python interactive console doesn't use sys.stdin for input

Steve Dower added the comment: I'm working on this as part of my fix for issue1602. Not yet sure how this will come out - compatibility with GNU readline seems to be the biggest issue, as if we want to keep that then we can't allow embedded '\0' in the encoded text (i.e. UTF-16 cannot be

[issue17620] Python interactive console doesn't use sys.stdin for input

Adam Bartoš added the comment: I've formulated a proposal regarding this issue: https://mail.python.org/pipermail/python-dev/2015-November/142246.html . Does it make sense? -- ___ Python tracker

[issue17620] Python interactive console doesn't use sys.stdin for input

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: -haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___ ___ Python-bugs-list

[issue17620] Python interactive console doesn't use sys.stdin for input

Changes by Paul Moore p.f.mo...@gmail.com: -- nosy: +paul.moore ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___ ___ Python-bugs-list

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: Note that under status quo PyOS_Readline is called from two places: the tokenizer during an interactive session and the builtin function input. The tokenizer passes promptstring encoded in sys.stdin.encoding while input() passes promtstring encoded in

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: Unfortunately, I have little or no experience with Python C code and I even don't have a C compiler installed so I cannot experiment. I'll just put my ideas how to solve this here. • Add sys.__readlinehook__ attribute, which can be set to a function taking a prompt

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: I have found another problem. PyOS_Readline can be called from two different places – from Parser/tokenizer.c:tok_nextc (by REPL), which uses sys.stdin.encoding to encode prompt argument, and from Python/bltinmodule.c:builtin_input_impl (by input() function), which

[issue17620] Python interactive console doesn't use sys.stdin for input

STINNER Victor added the comment: The Python parser works well with UTF8. If you know the encoding, decode from your encoding and encode to UTF8. You should pass the UTF8 flag to the parser. -- ___ Python tracker rep...@bugs.python.org

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: Antoine Pitrou: I understand. It would be nice to have that new Python string based readline hook. Its default implementation could be to call PyOS_Readline and decode the bytes using sys.stdin.encoding (as the tokenizer currently does). Tokenizer then woudn't need

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: I realized that the behavior I want can be achieved by setting PyOS_ReadlineFunctionPointer to a function calling sys.stdin.readline(). However I found another problem: Python REPL just doesn't work, when sys.stdin.encoding is UTF-16-LE. The tokenizer

[issue17620] Python interactive console doesn't use sys.stdin for input

Antoine Pitrou added the comment: without breaking obvious applications without breaking *existing* applications ;-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___

[issue17620] Python interactive console doesn't use sys.stdin for input

Antoine Pitrou added the comment: Why does PyOS_Readline return *char, rather than Python string object? For historical reasons and now for compatibility: we can't change the hook's signature without breaking obvious applications, obviously. If necessary, we could add a new hook that would

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: I have found another example of where the current interaction between readline and Python core lead to confussion. It started with following report on my package: https://github.com/Drekin/win-unicode-console/issues/2 . Basically, IPython interactive console on

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: I looked to the sourcecode and found the following. First, the codepath of how interactive loop gets its input follows: Python/pythonrun.c:PyRun_InteractiveLoopFlags Python/pythonrun.c:PyRun_InteractiveOneObject Python/pythonrun.c:PyParser_ASTFromFileObject

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: Sorry for formating in the previous message. Repeating… I looked to the sourcecode and found the following. First, the codepath of how interactive loop gets its input follows: Python/pythonrun.c:PyRun_InteractiveLoopFlags Python/pythonrun.c:PyRun_InteractiveOneObject

[issue17620] Python interactive console doesn't use sys.stdin for input

Guido van Rossum added the comment: Is this at all related to the use of GNU readline? -- nosy: +gvanrossum ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___

[issue17620] Python interactive console doesn't use sys.stdin for input

Antoine Pitrou added the comment: Yes, it is. GNU readline will use a FILE*. Apparently, one can customize this behaviour, see http://cnswww.cns.cwru.edu/php/chet/readline/readline.html#SEC25 Variable: rl_getc_func_t * rl_getc_function If non-zero, Readline will call indirectly through

[issue17620] Python interactive console doesn't use sys.stdin for input

Guido van Rossum added the comment: I propose not to mess with GNU readline. But that doesn't mean we can't try to fix this issue by detecting that sys.stdin has changed and use it if it isn't referring to the original process stdin. It will be tricky however to make sure nothing breaks.

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: My naive picture of ideal situation looks like this: When the interactive loop wants input, it just calls sys.stdin.readline, which delegates to sys.stdin.buffer.raw.readinto or .read, these can use GNU readline if available to get the data. May I ask, what's wrong

[issue17620] Python interactive console doesn't use sys.stdin for input

Guido van Rossum added the comment: sys.stdin.readline() never delegates to GNU readline. The REPL calls GNU readline directly. There's clearly some condition that determines whether to call GNU readline or sys.stdin.readline, but it may not correspond to what you want (e.g. it may just

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: There is still the serious inconsistency that the `sys.stdin` is not used for input by interactive loop but its encoding is. So if I replace `sys.stdin` with a custom object with its own `encoding` attribute, the standard interactive loop tries to use this encoding

[issue17620] Python interactive console doesn't use sys.stdin for input

Nick Coghlan added the comment: Steve, another one to look at in the context of improving the Unicode handling situation at the Windows command prompt. -- nosy: +steve.dower ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620

[issue17620] Python interactive console doesn't use sys.stdin for input

Steve Dower added the comment: Thanks Nick, but this has a pretty clear scope that may help the Unicode situation in cmd but doesn't directly relate to it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: Is there any chance the API will be added and used by python.exe? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___

[issue17620] Python interactive console doesn't use sys.stdin for input

Antoine Pitrou added the comment: • interactive console doesn't use sys.stdin for input, why? Modules/main.c calls PyRun_AnyFileFlags(stdin, stdin, ...). At this point, sys.stdin *is* the same as C stdin by construction, so I'm not sure how you came to encounter the issue. However, it's

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: I encountered it when I changed sys.stdin at runtime (I thought it was a supported feature) to affect the interactive console, see http://bugs.python.org/issue1602 . -- versions: -Python 2.7 ___ Python tracker

[issue17620] Python interactive console doesn't use sys.stdin for input

Antoine Pitrou added the comment: Ok, I guess it would need a new API (PyRun_Stdio()?) to run the interactive loop from sys.stdin, rather than from a fixed FILE*. -- stage: - needs patch type: behavior - enhancement versions: -Python 3.3 ___

[issue17620] Python interactive console doesn't use sys.stdin for input

Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___ ___ Python-bugs-list

[issue17620] Python interactive console doesn't use sys.stdin for input

Drekin added the comment: Sorry for typos. • interactive console doesn't use sys.stdin for input, why? • it uses sys.stdin.encoding, shouldn't it rather use sys.__stdin__.encoding if anything? • input() and hence code.interact() uses sys.stdin --

[issue17620] Python interactive console doesn't use sys.stdin for input

Changes by Georg Brandl ge...@python.org: -- assignee: - pitrou nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___ ___

[issue17620] Python interactive console doesn't use sys.stdin for input

Changes by Antoine Pitrou pit...@free.fr: -- assignee: pitrou - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___ ___ Python-bugs-list

[issue17620] Python interactive console doesn't use sys.stdin for input

Changes by Tshepang Lekhonkhobe tshep...@gmail.com: -- nosy: +tshepang ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___ ___

[issue17620] Python interactive console doesn't use sys.stdin for input

New submission from Drekin: The Python interactive console actually doesn't use sys.stdin but standard C stdin for input. Is there any reason for this? Why it then uses its encoding attribute? (Assigning sys.stdin something, that doesn't have encoding attribute freezes the interpreter.) If

[issue17620] Python interactive console doesn't use sys.stdin for input

Changes by Éric Araujo mer...@netwok.org: -- nosy: +eric.araujo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17620 ___ ___ Python-bugs-list