Re: Unbuffered stderr in Python 3

[srinivas devaki]

On Mon, Nov 2, 2015 at 1:22 PM, Steven D'Aprano
 wrote:
>
> So how come Python 3 has line buffered stderr? And more importantly, how can
> I turn buffering off?
>
> I don't want to use the -u unbuffered command line switch, because that
> effects stdout as well. I'm happy for stdout to remain buffered.
>
you can simply turn buffering off for stderr by redefining the print
function or declaring a new print function like


from functools import partial
print = partial(print, flush=True)
# or
from functools import partial
import sys
printerr = partial(print, flush=True, file=sys.stderr)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Terry Reedy]

On 11/3/2015 10:18 PM, Steven D'Aprano wrote:

On Wednesday 04 November 2015 09:25, Terry Reedy wrote:


On 11/3/2015 10:42 AM, Chris Angelico wrote:

On Wed, Nov 4, 2015 at 2:00 AM, Random832  wrote:

Nobody  writes:


It's probably related to the fact that std{in,out,err} are Unicode
streams.


There's no fundamental reason a Unicode stream should have to be line
buffered. If it's "related", it's only in that an oversight was made in
the course of making that change.


The current behavior is not an 'oversight'.  I was considered, decided,
and revisited in https://bugs.python.org/issue13601.  Guido:
"Line-buffering should be good enough since in practice errors messages
are always terminated by a newline."  If not, print(part_line,
file=sys.stderr, flush=True) works for the unusual case.


This is one of the offending line from our code base:

print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)

So that ought to be terminated by a newline.


Or include 'flush=True' if you really want that to be a partial line.

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Wolfgang Maier]

On 04.11.2015 11:43, Wolfgang Maier wrote:

On 04.11.2015 11:24, Steven D'Aprano wrote:

On Wed, 4 Nov 2015 07:19 pm, Wolfgang Maier wrote:


On 04.11.2015 04:18, Steven D'Aprano wrote:


This is one of the offending line from our code base:

print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)

So that ought to be terminated by a newline. And yet, the stderr output
doesn't show up until the program exits.



For me, that prints immediately. Where is your output going? Console,
file, ...?


Going to stderr, like the whole thread is about :-)



I see :), but seriously, what I meant was: is it going to the console
directly or is your code base redirecting sys.stderr to another buffered
object first?



Standard I/O streams are line-buffered only if interactive (i.e., 
connected to a console), but block-buffered otherwise.

What does sys.stderr.isatty() return?


--
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Random832]

Wolfgang Maier  writes:
> Standard I/O streams are line-buffered only if interactive (i.e.,
> connected to a console), but block-buffered otherwise.

That's not appropriate for stderr, nor is it justified by the argument
that Terry Reedy cited earlier. I had assumed he was making an implicit
claim that stderr is _always_ line-buffered, _even if_ it is not
interactive.

> What does sys.stderr.isatty() return?

There are many situations in which a nominally interactive process might
not have a "tty" for stderr on MS Windows. In an Emacs M-x shell buffer,
in a cygwin terminal (for a non-cygwin python), etc.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Dave Farrance]

Random832  wrote:

>The opposite of line buffering is not no buffering, but full
>(i.e. block) buffering, that doesn't get flushed until it runs
>out of space. TextIOWrapper has its own internal buffer, and its
>design apparently doesn't contemplate the possibility of using
>it with a raw FileIO object (which may mean that the posted code
>isn't guaranteed to work) or disabling buffering.

Hmmm. That even seems to cause trouble for sys.stderr.write
(in Python3 with a non-tty e.g. a piped output):

$ python2 -c 'import sys;sys.stderr.write("1\n");print("2")' 2>&1 | tee
1
2
$ python3 -c 'import sys;sys.stderr.write("1\n");print("2")' 2>&1 | tee
2
1
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Wolfgang Maier]

On 04.11.2015 04:18, Steven D'Aprano wrote:

On Wednesday 04 November 2015 09:25, Terry Reedy wrote:


On 11/3/2015 10:42 AM, Chris Angelico wrote:

On Wed, Nov 4, 2015 at 2:00 AM, Random832  wrote:

Nobody  writes:


It's probably related to the fact that std{in,out,err} are Unicode
streams.


There's no fundamental reason a Unicode stream should have to be line
buffered. If it's "related", it's only in that an oversight was made in
the course of making that change.


The current behavior is not an 'oversight'.  I was considered, decided,
and revisited in https://bugs.python.org/issue13601.  Guido:
"Line-buffering should be good enough since in practice errors messages
are always terminated by a newline."  If not, print(part_line,
file=sys.stderr, flush=True) works for the unusual case.


This is one of the offending line from our code base:

print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)

So that ought to be terminated by a newline. And yet, the stderr output
doesn't show up until the program exits.



For me, that prints immediately. Where is your output going? Console, 
file, ...?


--
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Steven D'Aprano]

On Wed, 4 Nov 2015 07:19 pm, Wolfgang Maier wrote:

> On 04.11.2015 04:18, Steven D'Aprano wrote:
>> On Wednesday 04 November 2015 09:25, Terry Reedy wrote:
>>
>>> On 11/3/2015 10:42 AM, Chris Angelico wrote:
 On Wed, Nov 4, 2015 at 2:00 AM, Random832 
 wrote:
> Nobody  writes:
>
>> It's probably related to the fact that std{in,out,err} are Unicode
>> streams.
>
> There's no fundamental reason a Unicode stream should have to be line
> buffered. If it's "related", it's only in that an oversight was made
> in the course of making that change.
>>>
>>> The current behavior is not an 'oversight'.  I was considered, decided,
>>> and revisited in https://bugs.python.org/issue13601.  Guido:
>>> "Line-buffering should be good enough since in practice errors messages
>>> are always terminated by a newline."  If not, print(part_line,
>>> file=sys.stderr, flush=True) works for the unusual case.
>>
>> This is one of the offending line from our code base:
>>
>> print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)
>>
>> So that ought to be terminated by a newline. And yet, the stderr output
>> doesn't show up until the program exits.
>>
> 
> For me, that prints immediately. Where is your output going? Console,
> file, ...?

Going to stderr, like the whole thread is about :-)


The output does eventually show up written to the console, but only after
Python exits.



-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Wolfgang Maier]

On 04.11.2015 11:24, Steven D'Aprano wrote:

On Wed, 4 Nov 2015 07:19 pm, Wolfgang Maier wrote:


On 04.11.2015 04:18, Steven D'Aprano wrote:


This is one of the offending line from our code base:

print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)

So that ought to be terminated by a newline. And yet, the stderr output
doesn't show up until the program exits.



For me, that prints immediately. Where is your output going? Console,
file, ...?


Going to stderr, like the whole thread is about :-)



I see :), but seriously, what I meant was: is it going to the console 
directly or is your code base redirecting sys.stderr to another buffered 
object first?




The output does eventually show up written to the console, but only after
Python exits.





--
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Random832]

Nobody  writes:

> It's probably related to the fact that std{in,out,err} are Unicode
> streams. 

There's no fundamental reason a Unicode stream should have to be line
buffered. If it's "related", it's only in that an oversight was made in
the course of making that change.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Chris Angelico]

On Wed, Nov 4, 2015 at 2:00 AM, Random832  wrote:
> Nobody  writes:
>
>> It's probably related to the fact that std{in,out,err} are Unicode
>> streams.
>
> There's no fundamental reason a Unicode stream should have to be line
> buffered. If it's "related", it's only in that an oversight was made in
> the course of making that change.

Yep. Unicode *input* streams need to be buffered, but *output* can
always be insta-flushed. The only significance of Unicode to output is
that a single character may cause multiple bytes to be output; and
since output can block for even a single byte, it should be no
different.

+1 for making sys.stderr unbuffered.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[George Trojan]

 Forwarded Message 
Subject:Re: Unbuffered stderr in Python 3
Date:   Tue, 03 Nov 2015 18:03:51 +
From:   George Trojan <george.tro...@noaa.gov>
To: python-list@python.org



On 11/03/2015 05:00 PM, python-list-requ...@python.org wrote:

On Mon, 02 Nov 2015 18:52:55 +1100, Steven D'Aprano wrote:


In Python 2, stderr is unbuffered.

In most other environments (the shell, C...) stderr is unbuffered.

It is usually considered a bad, bad thing for stderr to be buffered. What
happens if your application is killed before the buffer fills up? The
errors in the buffer will be lost.

So how come Python 3 has line buffered stderr? And more importantly, how
can I turn buffering off?

It's probably related to the fact that std{in,out,err} are Unicode
streams.

> type(sys.stderr)

> type(sys.stderr.buffer)

> type(sys.stderr.buffer.raw)


It appears that you can turn it off with:

sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
or:
sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())

This results in a sys.stderr which appears to work and whose
.line_buffering property is False.



This does set line buffering, but does not change the behaviour:

(devenv-3.4.1) dilbert@gtrojan> cat x.py
import sys
import time
if sys.version>'3':
import io
sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())
#sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
print(sys.stderr.line_buffering)
sys.stderr.write('a')
time.sleep(10)

This is python2.7.5. a is printed before ^C.

(devenv-3.4.1) dilbert@gtrojan> /bin/python x.py
a^CTraceback (most recent call last):

Here buffer is flushed on close, after typing ^C.//

(devenv-3.4.1) dilbert@gtrojan> python x.py
False
^CaTraceback (most recent call last):

George

Found it. write_through must be set to True.

(devenv-3.4.1) dilbert@gtrojan> cat x.py
import sys
import time
if sys.version>'3':
import io
sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach(), 
write_through=True)

#sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
print(sys.stderr.line_buffering)
sys.stderr.write('a')
time.sleep(10)
(devenv-3.4.1) dilbert@gtrojan> python x.py
False
a^CTraceback (most recent call last):

/

/
--
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[George Trojan]

On 11/03/2015 05:00 PM, python-list-requ...@python.org wrote:

On Mon, 02 Nov 2015 18:52:55 +1100, Steven D'Aprano wrote:


In Python 2, stderr is unbuffered.

In most other environments (the shell, C...) stderr is unbuffered.

It is usually considered a bad, bad thing for stderr to be buffered. What
happens if your application is killed before the buffer fills up? The
errors in the buffer will be lost.

So how come Python 3 has line buffered stderr? And more importantly, how
can I turn buffering off?

It's probably related to the fact that std{in,out,err} are Unicode
streams.

> type(sys.stderr)

> type(sys.stderr.buffer)

> type(sys.stderr.buffer.raw)


It appears that you can turn it off with:

sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
or:
sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())

This results in a sys.stderr which appears to work and whose
.line_buffering property is False.



This does set line buffering, but does not change the behaviour:

(devenv-3.4.1) dilbert@gtrojan> cat x.py
import sys
import time
if sys.version>'3':
import io
sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())
#sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
print(sys.stderr.line_buffering)
sys.stderr.write('a')
time.sleep(10)

This is python2.7.5. a is printed before ^C.

(devenv-3.4.1) dilbert@gtrojan> /bin/python x.py
a^CTraceback (most recent call last):

Here buffer is flushed on close, after typing ^C.

(devenv-3.4.1) dilbert@gtrojan> python x.py
False
^CaTraceback (most recent call last):

George
--
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Terry Reedy]

On 11/3/2015 10:42 AM, Chris Angelico wrote:

On Wed, Nov 4, 2015 at 2:00 AM, Random832  wrote:

Nobody  writes:


It's probably related to the fact that std{in,out,err} are Unicode
streams.


There's no fundamental reason a Unicode stream should have to be line
buffered. If it's "related", it's only in that an oversight was made in
the course of making that change.


The current behavior is not an 'oversight'.  I was considered, decided, 
and revisited in https://bugs.python.org/issue13601.  Guido: 
"Line-buffering should be good enough since in practice errors messages 
are always terminated by a newline."  If not, print(part_line, 
file=sys.stderr, flush=True) works for the unusual case.



Yep. Unicode *input* streams need to be buffered, but *output* can
always be insta-flushed. The only significance of Unicode to output is
that a single character may cause multiple bytes to be output; and
since output can block for even a single byte, it should be no
different.

+1 for making sys.stderr unbuffered.


-
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Random832]

George Trojan  writes:
> This does set line buffering, but does not change the behaviour:

The opposite of line buffering is not no buffering, but full
(i.e. block) buffering, that doesn't get flushed until it runs
out of space. TextIOWrapper has its own internal buffer, and its
design apparently doesn't contemplate the possibility of using
it with a raw FileIO object (which may mean that the posted code
isn't guaranteed to work) or disabling buffering.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Steven D'Aprano]

On Wednesday 04 November 2015 09:25, Terry Reedy wrote:

> On 11/3/2015 10:42 AM, Chris Angelico wrote:
>> On Wed, Nov 4, 2015 at 2:00 AM, Random832  wrote:
>>> Nobody  writes:
>>>
 It's probably related to the fact that std{in,out,err} are Unicode
 streams.
>>>
>>> There's no fundamental reason a Unicode stream should have to be line
>>> buffered. If it's "related", it's only in that an oversight was made in
>>> the course of making that change.
> 
> The current behavior is not an 'oversight'.  I was considered, decided,
> and revisited in https://bugs.python.org/issue13601.  Guido:
> "Line-buffering should be good enough since in practice errors messages
> are always terminated by a newline."  If not, print(part_line,
> file=sys.stderr, flush=True) works for the unusual case.

This is one of the offending line from our code base:

print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)

So that ought to be terminated by a newline. And yet, the stderr output 
doesn't show up until the program exits.




-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Nobody]

On Mon, 02 Nov 2015 18:52:55 +1100, Steven D'Aprano wrote:

> In Python 2, stderr is unbuffered.
> 
> In most other environments (the shell, C...) stderr is unbuffered.
> 
> It is usually considered a bad, bad thing for stderr to be buffered. What
> happens if your application is killed before the buffer fills up? The
> errors in the buffer will be lost.
> 
> So how come Python 3 has line buffered stderr? And more importantly, how
> can I turn buffering off?

It's probably related to the fact that std{in,out,err} are Unicode
streams. 

> type(sys.stderr)

> type(sys.stderr.buffer)

> type(sys.stderr.buffer.raw)


It appears that you can turn it off with:

sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
or:
sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())

This results in a sys.stderr which appears to work and whose
.line_buffering property is False.

-- 
https://mail.python.org/mailman/listinfo/python-list

Unbuffered stderr in Python 3

[Steven D'Aprano]

In Python 2, stderr is unbuffered.

In most other environments (the shell, C...) stderr is unbuffered.

It is usually considered a bad, bad thing for stderr to be buffered. What 
happens if your application is killed before the buffer fills up? The errors 
in the buffer will be lost.

So how come Python 3 has line buffered stderr? And more importantly, how can 
I turn buffering off?

I don't want to use the -u unbuffered command line switch, because that 
effects stdout as well. I'm happy for stdout to remain buffered.


Here is the function I'm using to test this in in the interactive 
interpreter:

import sys, time

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):
print(i, file=sys.stderr, end="")
time.sleep(2)
print("", file=sys.stderr)


Running it pauses for 20 seconds, then displays "0123456789" in one go. What 
I expect is to see the digits arrive individually.

I tried this:

py> sys.stderr.line_buffering = False
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: readonly attribute


Next I tried this:

py> sys.stderr.buffer.line_buffering = False


which succeeded, but has no effect on print: it still buffers.




-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Wolfgang Maier]

On 02.11.2015 11:48, Wolfgang Maier wrote:


Since Python3.3, the print function has a flush keyword argument that
accepts a boolean and lets you do just this. Rewrite your example as:

import sys, time

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):
_print(i, file=sys.stderr, end="", flush=True)
time.sleep(2)
print("", file=sys.stderr)

and it should do what you want.


sorry for this mess (copy pasted from the wrong source).
The code should be:

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):
print(i, file=sys.stderr, end="", flush=True)
time.sleep(2)
print("", file=sys.stderr)


--
https://mail.python.org/mailman/listinfo/python-list

Re: Unbuffered stderr in Python 3

[Wolfgang Maier]

On 02.11.2015 08:52, Steven D'Aprano wrote:

In Python 2, stderr is unbuffered.

In most other environments (the shell, C...) stderr is unbuffered.

It is usually considered a bad, bad thing for stderr to be buffered. What
happens if your application is killed before the buffer fills up? The errors
in the buffer will be lost.

So how come Python 3 has line buffered stderr? And more importantly, how can
I turn buffering off?



I cannot comment on your first question, but unbuffered writing to 
sys.stderr is possible just like for any other buffered file object. You 
just call its flush method after writing.


Since Python3.3, the print function has a flush keyword argument that 
accepts a boolean and lets you do just this. Rewrite your example as:


import sys, time

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):
_print(i, file=sys.stderr, end="", flush=True)
time.sleep(2)
print("", file=sys.stderr)

and it should do what you want.


Before Python 3.3, if I needed an unbuffered print function, I used a 
wrapper to call flush for me, so instead of:




import sys, time

def test():
 # Simulate a slow calculation that prints status and/or error
 # messages to stderr.
 for i in range(10):
 print(i, file=sys.stderr, end="")
 time.sleep(2)
 print("", file=sys.stderr)



I'd use (and still do this if I need compatibility with 3.2):

import sys, time

def _print (*args, **kwargs):
   file = kwargs.get('file', sys.stdout)
   print(*args, **kwargs)
   file.flush()

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):
_print(i, file=sys.stderr, end="")
time.sleep(2)
print("", file=sys.stderr)

Best,
Wolfgang

--
https://mail.python.org/mailman/listinfo/python-list