[issue4705] python3.0 -u: unbuffered stdout
Joe Borg added the comment: Can I confirm this is still in the trunk? I have 3.3.2 and am suffering from the fact that `-u` isn't setting stdin to unbuffered. I'm have to run a flush every command, which is awful. -- nosy: +Joe.Borg, georg.brandl versions: +Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Mark Dickinson dicki...@gmail.com added the comment: It's not about changing it, stdin has always been buffered in py3k. Sorry: I should have been clearer. It's the change from 2.x to 3.x that interests me. So 'python3.0 -u' has buffered stdin, while 'python2.6 -u' does not; I'm wondering: was this an intentional design change? Or was it just an accident/by-product of the rewritten io? Anyway, the patch looks good to me. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: So 'python3.0 -u' has buffered stdin, while 'python2.6 -u' does not; I'm wondering: was this an intentional design change? Or was it just an accident/by-product of the rewritten io? I'm not sure (I didn't write the new io in the first place) but I'd say it was simply overlooked. Otherwise 'python3.0 -u' would have had at least unbuffered stdout/stderr, which it didn't have. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Changes by Antoine Pitrou pit...@free.fr: -- assignee: - pitrou resolution: - accepted ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: Committed and applied a small fix to the test so that it passes in debug mode (r68977, r68981, r68982). Thanks! -- resolution: accepted - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Mark Dickinson dicki...@gmail.com added the comment: since I don't see how the behaviour can differ for a read-only non-seekable stream. Unless I'm misunderstanding you (quite likely), I think one *can* get different results with buffered and unbuffered stdin. For example, on my machine, if I create the following script: #!/usr/bin/python -u import sys print sys.stdin.readline() and name it test.py, I get the following result in an OS X Terminal running bash: dickinsm$ ls python_source/trunk/Objects/ | (./test.py; ./test.py) abstract.c boolobject.c Whereas if I remove the '-u' from the shebang line I just get: dickinsm$ ls python_source/trunk/Objects/ | (./test.py; ./test.py) abstract.c I'm not 100% sure that I understand exactly what's going on here, but it's something like the following: in the first (unbuffered) case, the stdin.readline call of the first ./test.py only reads one line from stdin, leaving the rest intact; so the second ./test.py also gets to output a line. In the second case some larger amount of stdin (1024 bytes?) is immediately slurped into the stdin buffer for the first Python process, so the second ./test.py doesn't get anything. -- nosy: +marketdickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: [...] I hadn't thought of such situations :-/ So the question is whether it is really useful to enforce unbuffered stdin with the '-u' option (or your example is simply too borderline). If so, the patch will have to be replaced with another one implementing read1() in the FileIO class. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: Thinking about it, TextIOWrapper has its own input buffering (the `decoded_chars` attribute), so your use case would probably not be satisfied. (and disabling TextIOWrapper's internal buffering would be a bad idea since it would make it horribly slow) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Mark Dickinson dicki...@gmail.com added the comment: So the question is whether it is really useful to enforce unbuffered stdin with the '-u' option (or your example is simply too borderline). Hard to say. It seems at least possible that there are Python users for whom stdin being unbuffered (with -u) matters, so if there's any reasonable way of avoiding changing this it should probably be considered. Though I have to admit that I'm not one of those users (I don't think I've *ever* used the -u option outside of testing...). ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: Hard to say. It seems at least possible that there are Python users for whom stdin being unbuffered (with -u) matters, so if there's any reasonable way of avoiding changing this it should probably be considered. It's not about changing it, stdin has always been buffered in py3k. My original commit actually attempted to change it, and it failed (I hadn't noticed it at first due to mis-testing on my part). The new patch is about putting it back in buffered mode even with '-u'. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file12475/unbufferedstdout.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file12479/test_stdin.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file12556/unbufferedstdout-4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file12657/unbufferedstdout-5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: Reopening, since sys.stdin is actually broken in unbuffered mode: $ ./python -u Python 3.1a0 (py3k:68756, Jan 19 2009, 01:17:26) [GCC 4.3.2] on linux2 Type help, copyright, credits or license for more information. import sys sys.stdin.read(1) Traceback (most recent call last): File stdin, line 1, in module File /home/antoine/py3k/__svn__/Lib/io.py, line 1739, in read eof = not self._read_chunk() File /home/antoine/py3k/__svn__/Lib/io.py, line 1565, in _read_chunk input_chunk = self.buffer.read1(self._CHUNK_SIZE) AttributeError: 'FileIO' object has no attribute 'read1' What I propose is that stdin be always opened in buffered mode (even with -u), since I don't see how the behaviour can differ for a read-only non-seekable stream. -- resolution: fixed - stage: commit review - committed/rejected status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: Here is a patch. Added file: http://bugs.python.org/file12798/unbuffered-stdin.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Changes by Antoine Pitrou pit...@free.fr: -- resolution: - accepted stage: patch review - commit review versions: +Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: Committed in r68451. Thanks! -- resolution: accepted - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: If `PyObject_SetAttrString(raw, _name, text)` fails, a reference to raw is leaked. Other than that, the patch looks good. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
STINNER Victor victor.stin...@haypocalc.com added the comment: Updated patch: clear raw on error + if (!Py_UnbufferedStdioFlag) + Py_XDECREF(raw); Question: Should we use line_buffering in unbuffered mode? Added file: http://bugs.python.org/file12657/unbufferedstdout-5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Benjamin Peterson benja...@python.org added the comment: Instead of importing IO each time in create_stdio, maybe you should just pass io.open to create_stdio. -- nosy: +benjamin.peterson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Changes by STINNER Victor victor.stin...@haypocalc.com: Removed file: http://bugs.python.org/file12477/unbufferedstdout-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
STINNER Victor victor.stin...@haypocalc.com added the comment: Instead of importing IO each time in create_stdio, maybe you should just pass io.open to create_stdio create_stdio() uses io.open() but also io.TextIOWrapper. Since io module is already imported in initstdio(), I updated the patch to just pass the pointer to the module to create_stdio(). Added file: http://bugs.python.org/file12556/unbufferedstdout-4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Changes by STINNER Victor victor.stin...@haypocalc.com: Removed file: http://bugs.python.org/file12478/unbufferedstdout-3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: pitrou's patch changes PyFile_FromFd() behaviour for a text file opened with buffering=0: /* As a convenience, when buffering == 0 on a text file, we open the underlying binary stream in unbuffered mode and wrap it with a text stream in line-buffered mode. */ Why changing PyFile_FromFd() and not io.open() directly? I must admit I'm a bit lazy, and changing io.open() means changing a fundamental public API, as Guido said on python-dev, so more discussion and some parts of the patches delayed to 3.1. If someone else wants to do it, please don't hesitate... Note: I prefer Py_UnbufferedStdoutFlag=1 instead of Py_UnbufferedStdoutFlag++ (for -u command line option). Well, I minimally changed the existing code. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
STINNER Victor victor.stin...@haypocalc.com added the comment: Why changing PyFile_FromFd() and not io.open() directly? I must admit I'm a bit lazy, and changing io.open() means changing a fundamental public API, as Guido said on python-dev, so more discussion and some parts of the patches delayed to 3.1. You're right, and PyFile_FromFd() is also a fundamental public API. Since TextIOWrapper doesn't support real unbuffered buffer (only pseudo line buffer: unbuffered raw buffer and line buffering for TextIOWrapper), I prefer to change only stdout/stderr instead of PyFile_FromFd(). My new patch only changes initstdio() using pitrou's code. Should we also change stdin? Added file: http://bugs.python.org/file12477/unbufferedstdout-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: Le dimanche 28 décembre 2008 à 12:19 +, STINNER Victor a écrit : STINNER Victor victor.stin...@haypocalc.com added the comment: Why changing PyFile_FromFd() and not io.open() directly? I must admit I'm a bit lazy, and changing io.open() means changing a fundamental public API, as Guido said on python-dev, so more discussion and some parts of the patches delayed to 3.1. You're right, and PyFile_FromFd() is also a fundamental public API. Well, open() is fundamental as in part of the built-ins and used pervasively. PyFile_FromFd(), on the other hand, is a relic of the 2.x C file handling API. Let's see what others have to say about this. Should we also change stdin? I don't know, but python -h only talks about stderr/stdout. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: It seems the name field of the TextIOWrapper object isn't set in create_stdio() (the char *name parameter isn't used). Otherwise, the patch looks good. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
STINNER Victor victor.stin...@haypocalc.com added the comment: Should we also change stdin? I don't know, but python -h only talks about stderr/stdout. The manpage of Python2 is clear: -u Force stdin, stdout and stderr to be totally unbuffered. stdin is also unbuffered. It seems the name field of the TextIOWrapper object isn't set in create_stdio() It used only used for buffered output. Without the patch, sys.stdout.name == sys.stdout.buffer.name == '1' :-/ New patch: - use create_stdio() to create unbuffered sys.stdin - rename Py_UnbufferedStdoutFlag to Py_UnbufferedStdioFlag - replace Py_UnbufferedStdioFlag++; by Py_UnbufferedStdioFlag = 1; - change create_stdio(): (...) Note: there is no test for unbuffered input because I don't know how to test this (even by manual tests) :-p Added file: http://bugs.python.org/file12478/unbufferedstdout-3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Antoine Pitrou pit...@free.fr added the comment: Here is a patch. -- keywords: +patch nosy: +pitrou priority: - high stage: - patch review type: - behavior Added file: http://bugs.python.org/file12475/unbufferedstdout.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
STINNER Victor victor.stin...@haypocalc.com added the comment: pitrou's patch changes PyFile_FromFd() behaviour for a text file opened with buffering=0: /* As a convenience, when buffering == 0 on a text file, we open the underlying binary stream in unbuffered mode and wrap it with a text stream in line-buffered mode. */ Why changing PyFile_FromFd() and not io.open() directly? Note: I prefer Py_UnbufferedStdoutFlag=1 instead of Py_UnbufferedStdoutFlag++ (for -u command line option). Except the minor comments, I like the patch (and it has unit tests!) ;-) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
New submission from STINNER Victor victor.stin...@haypocalc.com:
I like and I need an unbuffered standard output which was provided
by -u command line option (or PYTHONUNBUFFERED environment variable).
Current status of -u option in Python3: the option exists and change
the buffer size (disable buffering) of the stdin, stdout and stderr
file descriptors.
The problem is in initstdio() which creates files with buffering=-1
(default buffer) instead of buffering=0 (no buffering) or buffering=1
(line buffer). But open() enable line buffering of TextIOWrapper is
buffering=-1 and the raw file is a tty.
Example with py3k trunk:
$ ./python
import sys; sys.stdout.line_buffering
True
$ ./python |cat
import sys; sys.stdout.line_buffering
False
I would like line buffering when stdout is redirected to a pipe and -u
option is used. initstdio() have to be changed to choose buffering
option. So it's something like:
Index: Python/pythonrun.c
===
--- Python/pythonrun.c (révision 67870)
+++ Python/pythonrun.c (copie de travail)
@@ -810,7 +810,12 @@
#endif
}
else {
- if (!(std = PyFile_FromFd(fd, stdout, w, -1,
encoding,
+ int buffering;
+ if (1)
+ buffering = 1; /* line */
+ else
+ buffering = -1; /* default */
+ if (!(std = PyFile_FromFd(fd, stdout, w,
buffering, encoding,
errors, \n, 0))) {
goto error;
}
But if (1) have to be replaced if -u option is used :-) See
unbuffered variable of Modules/main.c.
--
messages: 78102
nosy: haypo
severity: normal
status: open
title: python3.0 -u: unbuffered stdout
versions: Python 3.0
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4705
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4705] python3.0 -u: unbuffered stdout
Fabio Zadrozny fab...@users.sourceforge.net added the comment: Just as a note, Pydev needs the unbuffered output (or it cannot get it). This has been brought up in the python-dev list: http://mail.python.org/pipermail/python-dev/2008-December/084436.html As a workaround for now I'm using: sys.stdout._line_buffering = True, but that doesn't seem right as it's accessing an internal attribute. -- nosy: +fabioz ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4705 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
