[issue41222] Undocumented behaviour change of POpen.stdout.readine with bufsize=0 or =1
Yann Dirson added the comment: With upcoming 3.10 phasing out 2.7 compatibility I have to find a solution to this, so I'm back digging here. Even .read(1) on a subprocess pipe causes an underlying buffered read, so working around the problem by a loop of 1-byte reads has to do with os.read(), though its usage on file-like object is discouraged. It looks like one of those would be needed, depending on the expected semantics of `POpen`'s `bufsize` parameter: * use the provided bufsize for the underlying buffering * provide a dummy pipe fd through fileno(), feeding it data as long as a read() call leaves data in the underlying buffer (indeed a simple conditional 1-byte read or write to the pipe before returning to caller should provide the correct semantics) -- versions: +Python 3.9 ___ Python tracker <https://bugs.python.org/issue41222> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41222] Undocumented behaviour change of POpen.stdout.readine with bufsize=0 or =1
Yann Dirson added the comment: Relevant commits include this one from v3.1.4: commit 877766dee8e60c7971ed0cabba89fbe981c2ab1b Author: Antoine Pitrou Date: Sat Mar 19 17:00:37 2011 +0100 Issue #11459: A `bufsize` value of 0 in subprocess.Popen() really creates unbuffered pipes, such that select() works properly on them. I can't use that commit without cherry-picking this one from v3.2.2, though: commit e96ec6810184f5daacb2d47ab8801365c99bb206 Author: Antoine Pitrou Date: Sat Jul 23 21:46:35 2011 +0200 Issue #12591: Allow io.TextIOWrapper to work with raw IO objects (without a read1() method), and add an undocumented *write_through* parameter to mandate unbuffered writes. And my test script still shows the same behaviour, with poll.poll() or poll.select(). The fact that my stdout object has no read1() and needs the above patch looks like a good lead for further investigation? -- ___ Python tracker <https://bugs.python.org/issue41222> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41222] Undocumented behaviour change of POpen.stdout.readine with bufsize=0 or =1
Change by Yann Dirson : -- keywords: +patch pull_requests: +24542 stage: -> patch review pull_request: https://github.com/python/cpython/pull/25859 ___ Python tracker <https://bugs.python.org/issue41222> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41222] POpen bufsize=0 ignored with universal_newlines=True
Yann Dirson added the comment: > The fact that my stdout object has no read1() and needs the above patch looks > like a good lead for further investigation? That's linked to universal_newlines, the bug only shows when that flag is set. Testcases provided in https://github.com/python/cpython/pull/25859 -- title: Undocumented behaviour change of POpen.stdout.readine with bufsize=0 or =1 -> POpen bufsize=0 ignored with universal_newlines=True ___ Python tracker <https://bugs.python.org/issue41222> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41222] Undocumented behaviour change of POpen.stdout.readine with bufsize=0 or =1
New submission from Yann Dirson :
On a POpen object created with bufsize=0, stdout.readline() does a buffered
reading with python3, whereas in 2.7 it did char-by-char reading. See attached
example.
As a result, a poll including the stdout object suffers a behaviour change when
stdout is ready for writing and there is more than one line of data available.
In both cases we get notified by poll() that data is available on the fd and we
can stdout.readline() and get back to our polling loop. Then:
* with python2 poll() then returns immediately and stdout.readline() will then
return the next line
* with python3 poll() now blocks
Running the attached example under strace reveals the underlying difference:
write(4, "go\n", 3) = 3
poll([{fd=5, events=POLLIN|POLLERR|POLLHUP}], 1, -1) = 1 ([{fd=5,
revents=POLLIN}])
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "x", 1) = 1
-read(5, "\n", 1)= 1
-fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x2), ...}) = 0
+read(5, "\nyyy\naaa"..., 8192) = 74
write(1, ">\n", 14) = 14
We can see a buffered read, which explains the behaviour difference.
Changing to bufsize=1, strace does not show a difference here.
This is especially troubling, as the first note in
https://docs.python.org/3/library/io.html#class-hierarchy mentions that even in
buffered mode there is an unoptimized readline() implementation.
--
components: IO
files: testproc-unbuffered.py
messages: 373165
nosy: yann
priority: normal
severity: normal
status: open
title: Undocumented behaviour change of POpen.stdout.readine with bufsize=0 or
=1
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file49299/testproc-unbuffered.py
___
Python tracker
<https://bugs.python.org/issue41222>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
