[issue21332] subprocess bufsize=1 docs are misleading

STINNER Victor added the comment: New changeset a2670565d8f5c502388378aba1fe73023fd8c8d4 by Victor Stinner (Alexey Izbyshev) in branch 'master': bpo-32236: open() emits RuntimeWarning if buffering=1 for binary mode (GH-4842)

[issue21332] subprocess bufsize=1 docs are misleading

Change by Alexey Izbyshev : -- pull_requests: +4732 ___ Python tracker ___ ___

[issue21332] subprocess bufsize=1 docs are misleading

Changes by Akira Li 4kir4...@gmail.com: Added file: http://bugs.python.org/file36679/subprocess-line-buffering-issue21332-ps5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___

[issue21332] subprocess bufsize=1 docs are misleading

Changes by Antoine Pitrou pit...@free.fr: -- assignee: gregory.p.smith - stage: - commit review versions: -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___

[issue21332] subprocess bufsize=1 docs are misleading

Antoine Pitrou added the comment: The latest patch looks good to me. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___ ___

[issue21332] subprocess bufsize=1 docs are misleading

Roundup Robot added the comment: New changeset 38867f90f1d9 by Antoine Pitrou in branch '3.4': Issue #21332: Ensure that ``bufsize=1`` in subprocess.Popen() selects line buffering, rather than block buffering. https://hg.python.org/cpython/rev/38867f90f1d9 New changeset 763d565e5840 by Antoine

[issue21332] subprocess bufsize=1 docs are misleading

Antoine Pitrou added the comment: Pushed! Thank you! -- resolution: - fixed stage: commit review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___

[issue21332] subprocess bufsize=1 docs are misleading

raylu added the comment: I'm fairly sure this hasn't been fixed in tip so I think we're still waiting on patch review. Is there an update here? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332

[issue21332] subprocess bufsize=1 docs are misleading

akira added the comment: The short answer is: no, you don't have to make you thread thread safe, as long as it can reliably run even in the presence of background threads (like the tkinter threads Victor mentions). https://mail.python.org/pipermail/python-dev/2014-May/134541.html It seems

[issue21332] subprocess bufsize=1 docs are misleading

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___

[issue21332] subprocess bufsize=1 docs are misleading

akira added the comment: I've asked about thread-safety of tests on python-dev mailing list: https://mail.python.org/pipermail/python-dev/2014-May/134523.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332

[issue21332] subprocess bufsize=1 docs are misleading

akira added the comment: I've updated the patch to remove changes to test_universal_newlines test that was fixed in revision 37d0c41ed8ad that closes #21396 issue -- Added file: http://bugs.python.org/file35189/subprocess-line-buffering-issue21332-ps4.patch

[issue21332] subprocess bufsize=1 docs are misleading

Martin Panter added the comment: On second thoughts maybe the idea of closing the input is not such a good idea in practice. Once you call os.close() on the file descriptor, that descriptor becomes unallocated, and I can’t see any way to prevent p.stdin.close(), Popen() context, destructors,

[issue21332] subprocess bufsize=1 docs are misleading

akira added the comment: to be clear: the test itself doesn't use threads, `python -mtest -j0` runs tests using multiple *processes*, not threads. There is no issue. If the test were to run in the presence of multiple threads then the issue would be the *explicit* p.stdin.close() in the test

[issue21332] subprocess bufsize=1 docs are misleading

akira added the comment: I've changed test_newlines to work also with Python io implementation. I've updated the patch. Note: tests use 10 seconds timeouts: I don't know how long it should take to read back a line from a subprocess so that the timeout would indicate a deadlock. --

[issue21332] subprocess bufsize=1 docs are misleading

Martin Panter added the comment: Perhaps you can avoid the 10 s deadlock timeout and threading in your test by closing the underlying input pipe file descriptor (or raw file object), without flushing it. Also, it seems to me that “line_buffering=True” is redundant with “write_through=True”.

[issue21332] subprocess bufsize=1 docs are misleading

Martin Panter added the comment: Sorry, it seems I was wrong on the second point. Looking closer, it seems write-through mode only flushes the TextIOWrapper layer, not the underlying binary file object, whereas line-buffering mode flushes everything to the OS, so the extra line_buffering=True

[issue21332] subprocess bufsize=1 docs are misleading

akira added the comment: yes, line_buffering=(bufsize == 1) is necessary to support the current Python io implementation or if C implementation is fixed to avoid buffer.flush() on every write with write_through=True -- otherwise bufsize is not respected in text mode (it would always mean

[issue21332] subprocess bufsize=1 docs are misleading

akira added the comment: It looks like a bug in the subprocess module e.g., if child process does: sys.stdout.write(sys.stdin.readline()) sys.stdout.flush() and the parent: p.stdin.write(line) #NOTE: no flush line = p.stdout.readline() then a deadlock may happen with bufsize=1

[issue21332] subprocess bufsize=1 docs are misleading

akira added the comment: Related issue #21396 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___ ___ Python-bugs-list mailing list

[issue21332] subprocess bufsize=1 docs are misleading

Changes by Ned Deily n...@acm.org: -- nosy: +gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___ ___ Python-bugs-list mailing

[issue21332] subprocess bufsize=1 docs are misleading

Gregory P. Smith added the comment: Thanks for the report, diagnosis and patch! Your change looks good to me. I'll commit it soon. -- assignee: - gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332

[issue21332] subprocess bufsize=1 docs are misleading

Changes by Jakub Wilk jw...@jwilk.net: -- nosy: +jwilk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___ ___ Python-bugs-list mailing list

[issue21332] subprocess bufsize=1 docs are misleading

Changes by akira 4kir4...@gmail.com: -- nosy: +akira ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21332 ___ ___ Python-bugs-list mailing list

[issue21332] subprocess bufsize=1 docs are misleading

New submission from raylu: https://docs.python.org/3.3/library/subprocess.html says bufsize will be supplied as the corresponding argument to the open() function when creating the stdin/stdout/stderr pipe file objects: ... 1 means line buffered but it calls io.open with 'wb' and 'rb':