[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[miss-islington]

miss-islington  added the comment:


New changeset 91e888904478271c27c52c773863b41f5a8f7f30 by Miss Islington (bot) 
in branch '3.10':
bpo-14916: use specified tokenizer fd for file input (GH-31006)
https://github.com/python/cpython/commit/91e888904478271c27c52c773863b41f5a8f7f30


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[miss-islington]

Change by miss-islington :


--
pull_requests: +29250
pull_request: https://github.com/python/cpython/pull/31065

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[miss-islington]

miss-islington  added the comment:


New changeset 89b13042fcfc95bae21a49806a205ef62f1cdd73 by Paul m. p. P in 
branch 'main':
bpo-14916: use specified tokenizer fd for file input (GH-31006)
https://github.com/python/cpython/commit/89b13042fcfc95bae21a49806a205ef62f1cdd73


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[pmp-p]

Change by pmp-p :


--
pull_requests: +29187
pull_request: https://github.com/python/cpython/pull/31006

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[pmp-p]

Change by pmp-p :


--
pull_requests: +21251
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/22190

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[pmp-p]

Change by pmp-p :


--
versions: +Python 3.6, Python 3.7, Python 3.8, Python 3.9 -Python 2.6, Python 
2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[pmp-p]

pmp-p  added the comment:

all PyRun_InteractiveOne* functions are also affected
it is really annoying when implementing repl behaviour when embedding( use 
cases :  pyodide, android, wasi )

But I think the correct patch is : 
-char *newtok = PyOS_Readline(stdin, stdout, tok->prompt);
+char *newtok = PyOS_Readline(tok->fp? tok->fp : stdin, stdout, 
tok->prompt);

--
nosy: +pmpp

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Yauheni Kaliuta]

Yauheni Kaliuta added the comment:

Any progress with the problem? I just wanted to use the feature, but it looks 
like the bug.sh is still reproduces the bug.

--
nosy: +Yauheni Kaliuta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

Kevin,
I now fully agree with you. Regarding points 2  3 I dismissed modifying 
sys.stdin/out in python out of hand because it still would not allow to have a 
proper behaviour with two concurrent consoles on the same interpreter. Anyway 
this is not a very meaningful use-case, so modifying sys.stdin/out looks like 
the best solution.
As for point 1, it can be worked around in a twisted and probably non-portable 
way (setting PyOS_ReadlineFunctionPointer to a custom function that forks a 
child which runs GNU readline or whatnot on its fds 0,1,2, and which 
communicates with the parent through pipes) but it's a pity that from 
PyOS_Readline() on, argument sys_stdin is correctly passed down, and that the 
chain has only this gap in tok_nextc() which dismisses the caller's argument 
and uses plain stdin.
If the user arguments are not used, why does PyRun_InteractiveLoop() take any 
arguments at all?
It would be nice to know the opinion of the python development team and to work 
out a complete fix (considering also stdout as you suggest) under their 
authority.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

emmanuel,

The Python interpreter isn't reentrant, so you could only run two interactive 
sessions connected to the same Python environment if you implemented your own 
REPL function that unlocked the GIL when waiting for input, then lock it just 
long enough to interpret the input.

Also, the readline module already does what you suggest (sets 
PyOS_ReadlineFunctionPointer to a GNU libreadline wrapper.) My readline hack 
(working.c) forces it to behave as it's supposed to. Rather, it *undoes* what 
PyRun_InteractiveLoop does every iteration, which is pass stdin/stdout for I/O, 
which libreadline in turn uses.

I agree that it would be nice to get a Python developer involved, mostly 
because I expect things to break when this problem is fixed. Bad things happen 
when you think you've tested a lot of different cases that actually turn out to 
be the exact same case.

Kevin Barry

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

emmanuel,

Regarding your points: All three can be taken care of with a combination of my 
patch and setting sys.stdin, sys.stdout, and sys.stderr to the pty. (That 
should really be done internally with another patch, since os.fdopen is 
OS-specific. Also, sys.stdin, sys.stdout, and sys.stderr should each have 
distinct underlying file descriptors, which I didn't do in working.c.) Those 
can safely be replaced since they're just the effective standard files, and 
sys.__stdin__ et al. refer to the actual C stdin et al. The remaining issue 
would be that the same descriptor shouldn't be used for both input and output 
in the interpreter loop, especially if the FILE* passed is only open for 
reading (since standard input technically doesn't have to be writable.)

Kevin Barry

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

Kevin,
I've read more carefully your messages and investigated some more.
It seems that there are several issues:
1/ To take input from a defined tty without interfering with standard file 
descriptors
2/ To have the result (object) of evaluation printed to a defined tty without 
interfering with standard file descriptors
3/ (optionally) To direct to the tty (or not) the output that is a side effect 
of the evaluation
Provided that no one messes with PyOS_ReadlineFunctionPointer (as with import 
readline) it should be possible to solve 1 without modifying Python, and 
approximately solve 2 (with 3 implied out of necessity).
On the other hand, modifying Python as you suggest could solve 1, but issues 2 
and 3 would still remain and probably require some other modifications.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

Kevin,

These are good points.

I had a cursory look at the python source code and observed the following:
- There may also be a concern with stderr (used to print the prompt in 
PyOS_Readline)
- PyOS_Readline has two different definitions in files pgenmain.c and 
myreadline.c
- There is this interesting comment in myreadline.c:
/* By initializing this function pointer, systems embedding Python can
   override the readline function.
   Note: Python expects in return a buffer allocated with PyMem_Malloc. */
char *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, char *);

This pointer is actually used (set it to (void*)1 and the interpreter crashes) 
so it could offer a means to redirect stdin as we want. For stdout/stderr 
further investigation is needed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

Changes by emmanuel garcia6.emman...@wanadoo.fr:


--
nosy: +emmanuel

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

run the attached shell script to observe the bug
./bug.sh 0 - shows the bug
./bug.sh 1 - shows the expected behaviour (using a workaround)
tested on linux with python 2.7

--
Added file: http://bugs.python.org/file29431/bug.sh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

Changes by emmanuel garcia6.emman...@wanadoo.fr:


Added file: http://bugs.python.org/file29432/bug.sh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

Changes by emmanuel garcia6.emman...@wanadoo.fr:


Removed file: http://bugs.python.org/file29431/bug.sh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

emmanuel,

Thanks for the suggestion. Your workaround is exactly the same as using dup2 
(in C) to replace stdin/stdout/stderr with the pty, however. If you added the 
following lines to your C code, it would have the same effect as the 
command-line redirection in the workaround:

dup2(fileno(file), STDIN_FILENO);
dup2(fileno(file), STDOUT_FILENO);
dup2(fileno(file), STDERR_FILENO);

In fact, that's exactly what bash does after forking, just before executing 
exe. In most cases, developers who use PyRun_InteractiveLoop in a pty 
probably also do exactly that, which is why I'm the only one who's reported 
this as a bug. For applications like mine, however, where the interactive 
Python session needs to be an unobtrusive add-on to an otherwise-complete 
program, this solution won't work. The standard file descriptors aren't 
disposable in most of the programs I work on.

Thanks again!

Kevin Barry

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

Kevin,

Indeed the code I submitted can be written entirely in C using pipe fork execl 
dup2 etc. as you suggest. The only purpose of mixing bash and C is to have a 
short self-contained file showing the problem.

Anyway, whether in C or bash the workaround is less than satisfying in that it 
uses up fds 0 1 2, and I have exactly the same goal and constraints as you.

Finally, thanks for identifying the limitation in the python implementation and 
submitting a patch. Like you I hope it will be eventually applied.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

One additional issue, which my patch doesn't address, is that 
PyRun_InteractiveLoop should really take *two* FILE* arguments, with the second 
one being optional. This is because on Linux (and presumably on other *nixes) 
if a read operation is blocked on a file descriptor then write operations (from 
other threads) to the same file descriptor will also block. That doesn't happen 
in the current Python implementations because PyOS_Readline is always called 
with two FILE* objects, anyway (stdin and stdout.) I would, however, expect 
such a problem to appear if a user created a Python thread in the interactive 
session that periodically printed to the terminal, then read input from the 
terminal. In that case, I would expect to see no output from the thread while 
the read operations were blocked, but I haven't tested it. (I don't remember if 
this came up after I applied my patch locally.)

I actually considered this when I created the patch; however, I didn't feel 
like going to all the trouble of adding a member to tok and propagating the 
change throughout the entire core. I had hoped this bug would get more 
attention and I'd be able to discuss it with a developer involved in the Python 
project, but ultimately that didn't happen and I ended up forgetting about it.

Kevin Barry

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

I still see the potential cause addressed by my patch in the 2.7, 3.3, and 
default branches, so I'm assuming that all versions from 2.6 on have this 
problem.

I also see that I can elect to change the Status and Resolution of this 
report. Does that mean I need to do something besides wait for someone involved 
in the project to look at my patch?

Kevin Barry

--
versions: +Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 
3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry ta0k...@gmail.com added the comment:

The patch from before needed a slight modification for when Python actually 
defaults to an interactive session on stdin. Since I rebuild this for my 
current distro (Slackware64 13.37,) I switched to the Python 2.6.6 source. This 
might not be the proper way to handle the default case (e.g. 'Py_Main'), but 
it's a start.


The patch (also attached):

--- ./Parser/tokenizer.c.orig   2012-07-23 22:24:56.513992301 -0400
+++ ./Parser/tokenizer.c2012-07-23 22:23:24.329992167 -0400
@@ -805,7 +805,7 @@
 return Py_CHARMASK(*tok-cur++);
 }
 if (tok-prompt != NULL) {
-char *newtok = PyOS_Readline(stdin, stdout, tok-prompt);
+char *newtok = PyOS_Readline(tok-fp? tok-fp : stdin, (tok-fp  
tok-fp != stdin)? tok-fp : stdout, tok-prompt);
 if (tok-nextprompt != NULL)
 tok-prompt = tok-nextprompt;
 if (newtok == NULL)


Kevin Barry

--
Added file: 
http://bugs.python.org/file26501/Python-2.6.6-Run_Interactive-fix.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry ta0k...@gmail.com added the comment:

I've attached a simplified example program (working3.c) that demonstrates both 
the original problem and that the patch 
(Python-2.6.6-Run_Interactive-fix.patch) works. It eliminates the need for a 
pty, 'xterm', and redirection.


Compile the attached program with:

 gcc `python-config --cflags` working3.c -o working3 `python-config --ldflags`

and run it with (before and after patching):

 ./working3


Also, for verification, run 'python' with no arguments to show that default 
interactivity is preserved.

Kevin Barry

--
Added file: http://bugs.python.org/file26503/working3.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry ta0k...@gmail.com added the comment:

Here is a patch that corrects the problem (quoted below and attached.) This 
only corrects the problem when 'PyOS_ReadlineFunctionPointer' is set, e.g. you 
must 'import readline', otherwise Python will defer to stdin/stdout with 
'PyOS_StdioReadline'.


The patch:


--- Python-2.6.8/Parser/tokenizer.c 2012-04-10 11:32:11.0 -0400
+++ Python-2.6.8-patched/Parser/tokenizer.c 2012-07-23 19:56:39.645992101 
-0400
@@ -805,7 +805,7 @@
 return Py_CHARMASK(*tok-cur++);
 }
 if (tok-prompt != NULL) {
-char *newtok = PyOS_Readline(stdin, stdout, tok-prompt);
+char *newtok = PyOS_Readline(tok-fp? tok-fp : stdin, tok-fp? 
tok-fp : stdout, tok-prompt);
 if (tok-nextprompt != NULL)
 tok-prompt = tok-nextprompt;
 if (newtok == NULL)


Kevin Barry

--
keywords: +patch
status: open - pending
Added file: 
http://bugs.python.org/file26491/Python-2.6.8-Run_Interactive-fix.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry ta0k...@gmail.com added the comment:

I've attached a new example source file to demonstrate the fix.


Compile the attached program with (*after* patching and installing Python):

 gcc `python-config --cflags` working2.c -o working2 `python-config --ldflags`

and run it with:

 ./working2 xterm -S/0  /dev/null  /dev/null

(The redirection shows that it works when stdin/stdout aren't a tty.)


I looked at the most-recent revision of tokenizer.c 
(http://hg.python.org/cpython/file/52032b13243e/Parser/tokenizer.c) and see 
that the change in my patch above hasn't been made already.

Kevin Barry

--
status: pending - open
Added file: http://bugs.python.org/file26492/working2.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

New submission from Kevin Barry ta0k...@gmail.com:

I have been trying to get PyRun_InteractiveLoop to run on a pty (Linux) without 
replacing stdin and stdout with that pty; however, it seems like Python (2.6.6) 
is hard-coded to only run interactively on stdin and stdout.


Compile the attached program with:

 gcc `python-config --cflags` working.c -o working `python-config --ldflags`

and run it with:

 ./working xterm -S/0

and you should see that there is no interactivity in the xterm that's opened.


Compile the attached file with:

 gcc -DREADLINE_HACK `python-config --cflags` working.c -o working 
 `python-config --ldflags` -lreadline -lcurses

and run it with:

 ./working xterm -S/0

to see how it runs with my best attempt to get it to function properly with a 
readline hack. Additionally, try running:

 ./working xterm -S/0  /dev/null
 ./working xterm -S/0  /dev/null

both of which should cause interactivity in the xterm to fail, indicating that 
Python is checking stdin/stdout for tty status when determining if it should 
run interactively (i.e. it's not checking the tty status of the file passed to 
PyRun_InteractiveLoop.)


Am I somehow using this function wrong? I've been trying to work around this 
problem for a while, and I don't think I should be using readline hacks 
(especially since they don't port to other OSes with ptys, e.g. OS X.) I even 
tried to patch the call to PyOS_Readline in tok_nextc (Parser/tokenizer.c) to 
use tok-fp instead of stdin/stdout, which caused I/O to use the pty but it 
still failed to make interactivity work.


Thanks!

Kevin Barry

--
components: Interpreter Core, Library (Lib)
files: working.c
messages: 161586
nosy: Kevin.Barry
priority: normal
severity: normal
status: open
title: PyRun_InteractiveLoop fails to run interactively when using a Linux pty 
that's not tied to stdin/stdout
type: behavior
versions: Python 2.6
Added file: http://bugs.python.org/file25706/working.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com