[issue22443] read(1) blocks on unflushed output
Sworddragon added the comment: It works if -q 0 is given without the need of a workaround. So this was just a feature of apt that was causing this behavior. I think here is nothing more to do so I'm closing this ticket. -- resolution: - not a bug status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
eryksun added the comment: unbuffer works for me. It fakes a tty, so apt-get doesn't automatically enter quiet mode. import io import subprocess args = ['unbuffer', 'apt-get', 'download', 'firefox'] p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) out = io.TextIOWrapper(p.stdout, newline='') while True: c = out.read(1) if c == '': break print(c, end='', flush=True) p.wait() unbuffer isn't required if you disable quiet mode as follows: args = ['apt-get', '-q=0', 'download', 'firefox'] -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
New submission from Sworddragon: On reading the output of an application (for example apt-get download firefox) that dynamically changes a line (possibly with the terminal control character \r) I have noticed that read(1) does not read the output until it has finished with a newline. This happens even with disabled buffering. In the attachments is a testcase for this problem. Also here are 2 screenshots to compare the results: Direct call: http://picload.org/image/crldgri/normal.png Subprocess: http://picload.org/image/crldgrw/subprocess.png -- components: Library (Lib) files: test.py messages: 227097 nosy: Sworddragon priority: normal severity: normal status: open title: read(1) blocks on unflushed output type: behavior versions: Python 3.4 Added file: http://bugs.python.org/file36661/test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
STINNER Victor added the comment: On reading the output of an application (for example apt-get download firefox) that dynamically changes a line (possibly with the terminal control character \r) I have noticed that read(1) does not read the output until it has finished with a newline. The buffering of stdout and/or stderr of your application probably changes if the application runs in a terminal (TTY) or if the output is redirected to a pipe (not a TTY). Set the setvbuf() function. You can try my hack to disable buffering using LD_PRELOAD: https://bitbucket.org/haypo/misc/src/4d133ea3e46550808305b093557ee51d2de2ac9f/misc/nobuffer.c?at=default -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
Changes by Sworddragon sworddrag...@aol.com: Removed file: http://bugs.python.org/file36661/test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
Sworddragon added the comment: Edit: Updated testcase as I forgot to flush the output (in case somebody hints to it). -- Added file: http://bugs.python.org/file36662/test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
Sworddragon added the comment: The buffering of stdout and/or stderr of your application probably changes if the application runs in a terminal (TTY) or if the output is redirected to a pipe (not a TTY). Set the setvbuf() function. This means in the worst case there is currently no official way to get this output before it writes a newline? You can try my hack to disable buffering using LD_PRELOAD: https://bitbucket.org/haypo/misc/src/4d133ea3e46550808305b093557ee51d2de2ac9f/misc/nobuffer.c?at=default I will try later if I can successfully compile Python with this hack. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
STINNER Victor added the comment: This means in the worst case there is currently no official way to get this output before it writes a newline? The behaviour of stdout/stderr is defined in the C library, see setvbuf() manual for more information. I don't know a generic way to change the default buffering without modifying the application (or the LD_PRELOAD hack). I will try later if I can successfully compile Python with this hack. You don't need to compile Python. Just compile nobuffer.c to libnobuffer.so. See the documentation in nobuffer.c. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
Sworddragon added the comment: You don't need to compile Python. Just compile nobuffer.c to libnobuffer.so. See the documentation in nobuffer.c. Strictly following the documentation does not work: sworddragon@ubuntu:~/tmp$ gcc -shared -o nobuffer.so interceptor.c gcc: error: interceptor.c: No such file or directory gcc: fatal error: no input files compilation terminated. Trying to fix this results in this error: sworddragon@ubuntu:~/tmp$ gcc -shared -o nobuffer.so nobuffer.c /usr/bin/ld: /tmp/ccgArKHv.o: relocation R_X86_64_PC32 against undefined symbol `stdout@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: error: ld returned 1 exit status -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
STINNER Victor added the comment: Ah yes, try gcc -shared nobuffer.c -o libnobuffer.so -fPIC. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
Sworddragon added the comment: I was able to compile the library but after executing LD_PRELOAD=./libnobuffer.so ./test.py I'm seeing no difference. The unflushed output is still not being read with read(1). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
eryksun added the comment: stdbuf is the typical way to apply the LD_PRELOAD trick: https://www.gnu.org/software/coreutils/manual/html_node/stdbuf-invocation.html There's also the unbuffer expect script: http://expect.sourceforge.net/example/unbuffer.man.html -- nosy: +eryksun ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
Sworddragon added the comment: stdbuf -o 0 ./test.py and unbuffer ./test.py doesn't change the result too. Or is something wrong with my testcase? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22443] read(1) blocks on unflushed output
Akira Li added the comment: Related: http://stackoverflow.com/questions/25923901/last-unbuffered-line-cant-be-read Make sure you follow the links in the comments. -- nosy: +akira ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22443 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
