Re: Reading from sys.stdin reads the whole file in
Chris Angelico writes: > On Wed, Aug 27, 2014 at 4:37 PM, Steven D'Aprano wrote: >> On Wed, 27 Aug 2014 08:29:20 +0300, Marko Rauhamaa wrote: >> >>> Try flushing after each print. >> >> Doesn't help. > > It does, but insufficiently. If slurp.py is run under Py3, it works > fine; or take Naoki's suggestion (although without the parens): > > import sys > import time > > for line in iter(sys.stdin.readline, ''): > print "Time of input:", time.ctime(), line > sys.stdin.flush() > sys.stdout.flush() > > Then it works. > > ChrisA It looks like this bug http://bugs.python.org/issue3907 `python -u out.py | python -u slurp.py` could be used to avoid .flush() calls everywhere. Or reassign `sys.stdin = io.open(sys.stdin.fileno(), 'r', 1)` inside the script. http://stackoverflow.com/questions/107705/python-output-buffering -- Akira -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
Marko Rauhamaa wrote: > Peter Otten <__pete...@web.de>: > >> In addition to what already has been said: you can switch off output >> buffering of stdout/stderr with >> >> python -u out.py >> >> or by setting the PYTHONUNBUFFERED environment variable. > > Very often such externalities are not in the control of the application > developer. Sometimes it's possible to use a wrapper script rather than to sprinkle your code with flush(). Sometimes the "offending" python script is not even written and maintained by you, and setting an environment variable may be a price you are willing to pay to keep it that way. -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
Peter Otten <__pete...@web.de>: > In addition to what already has been said: you can switch off output > buffering of stdout/stderr with > > python -u out.py > > or by setting the PYTHONUNBUFFERED environment variable. Very often such externalities are not in the control of the application developer. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
Steven D'Aprano wrote:
> I'm trying to read from stdin. Here I simulate a process that slowly
> outputs data to stdout:
>
> steve@runes:~$ cat out.py
> import time
>
> print "Hello..."
> time.sleep(10)
> print "World!"
> time.sleep(10)
> print "Goodbye!"
In addition to what already has been said: you can switch off output
buffering of stdout/stderr with
python -u out.py
or by setting the PYTHONUNBUFFERED environment variable.
You still need the readline trick to get unbuffered input. Quoting the man-
page:
"""
-u Force stdin, stdout and stderr to be totally unbuffered. On
systems where it matters, also put stdin, stdout and stderr in
binary mode. Note that there is internal buffering in xread‐
lines(), readlines() and file-object iterators ("for line in
sys.stdin") which is not influenced by this option. To work
around this, you will want to use "sys.stdin.readline()"
inside a "while 1:" loop.
"""
--
https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
On Tue, 26 Aug 2014 23:07:36 -0700, Naoki INADA wrote: > for line in iter(sys.stdin.readline(), ''): Thanks for that. Removing the parens after readline seems to do the trick. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
On Wed, Aug 27, 2014 at 4:37 PM, Steven D'Aprano wrote: > On Wed, 27 Aug 2014 08:29:20 +0300, Marko Rauhamaa wrote: > >> Try flushing after each print. > > Doesn't help. It does, but insufficiently. If slurp.py is run under Py3, it works fine; or take Naoki's suggestion (although without the parens): import sys import time for line in iter(sys.stdin.readline, ''): print "Time of input:", time.ctime(), line sys.stdin.flush() sys.stdout.flush() Then it works. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
On Wed, 27 Aug 2014 08:29:20 +0300, Marko Rauhamaa wrote: > Steven D'Aprano : > >> When I pipe one to the other, I expect each line to be printed as they >> arrive, but instead they all queue up and happen at once: > > Try flushing after each print. Doesn't help. Here is an update that may make the problem more clear: steve@runes:~$ cat out.py import time import sys print "Time of output:", time.ctime() sys.stdout.flush() time.sleep(10) print "Time of output:", time.ctime() sys.stdout.flush() time.sleep(10) print "Time of output:", time.ctime() steve@runes:~$ cat slurp.py import sys import time for line in sys.stdin: print "Time of input:", time.ctime(), line sys.stdin.flush() sys.stdout.flush() And the results: steve@runes:~$ python out.py | python slurp.py Time of input: Wed Aug 27 16:35:48 2014 Time of output: Wed Aug 27 16:35:28 2014 Time of input: Wed Aug 27 16:35:48 2014 Time of output: Wed Aug 27 16:35:38 2014 Time of input: Wed Aug 27 16:35:48 2014 Time of output: Wed Aug 27 16:35:48 2014 -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
I recommend Python 3.
On Python 2, iterating lines without buffering is slow, tricky and ugly.
for line in iter(sys.stdin.readline(), ''):
print line
—
Sent from Mailbox
On Wed, Aug 27, 2014 at 3:03 PM, Chris Angelico wrote:
> On Wed, Aug 27, 2014 at 3:19 PM, Steven D'Aprano wrote:
>> When I pipe one to the other, I expect each line to be printed as they
>> arrive, but instead they all queue up and happen at once:
> You're seeing two different problems here. One is the flushing of
> stdout in out.py, as Marko mentioned, but it's easily proven that
> that's not the whole issue. Compare "python out.py" and "python
> out.py|cat" - the latter will demonstrate whether or not it's getting
> flushed properly (the former, where stdout is a tty, will always flush
> correctly).
> But even with that sorted, iterating over stdin has issues in Python
> 2. Here's a tweaked version of your files (note that I cut the sleeps
> to 2 seconds, but the effect is the same):
> rosuav@sikorsky:~$ cat out.py
> import time
> print("Hello...",flush=True)
> time.sleep(2)
> print("World!",flush=True)
> time.sleep(2)
> print("Goodbye!",flush=True)
> rosuav@sikorsky:~$ cat slurp.py
> from __future__ import print_function
> import sys
> import time
> for line in sys.stdin:
> print(time.ctime(), line)
> rosuav@sikorsky:~$ python3 out.py|python slurp.py
> Wed Aug 27 16:00:16 2014 Hello...
> Wed Aug 27 16:00:16 2014 World!
> Wed Aug 27 16:00:16 2014 Goodbye!
> rosuav@sikorsky:~$ python3 out.py|python3 slurp.py
> Wed Aug 27 16:00:19 2014 Hello...
> Wed Aug 27 16:00:21 2014 World!
> Wed Aug 27 16:00:23 2014 Goodbye!
> rosuav@sikorsky:~$
> With a Py2 consumer, there's still buffering happening. With a Py3
> consumer, it works correctly. How to control the Py2 buffering,
> though, I don't know.
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list--
https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
On Wed, Aug 27, 2014 at 3:19 PM, Steven D'Aprano wrote:
> When I pipe one to the other, I expect each line to be printed as they
> arrive, but instead they all queue up and happen at once:
You're seeing two different problems here. One is the flushing of
stdout in out.py, as Marko mentioned, but it's easily proven that
that's not the whole issue. Compare "python out.py" and "python
out.py|cat" - the latter will demonstrate whether or not it's getting
flushed properly (the former, where stdout is a tty, will always flush
correctly).
But even with that sorted, iterating over stdin has issues in Python
2. Here's a tweaked version of your files (note that I cut the sleeps
to 2 seconds, but the effect is the same):
rosuav@sikorsky:~$ cat out.py
import time
print("Hello...",flush=True)
time.sleep(2)
print("World!",flush=True)
time.sleep(2)
print("Goodbye!",flush=True)
rosuav@sikorsky:~$ cat slurp.py
from __future__ import print_function
import sys
import time
for line in sys.stdin:
print(time.ctime(), line)
rosuav@sikorsky:~$ python3 out.py|python slurp.py
Wed Aug 27 16:00:16 2014 Hello...
Wed Aug 27 16:00:16 2014 World!
Wed Aug 27 16:00:16 2014 Goodbye!
rosuav@sikorsky:~$ python3 out.py|python3 slurp.py
Wed Aug 27 16:00:19 2014 Hello...
Wed Aug 27 16:00:21 2014 World!
Wed Aug 27 16:00:23 2014 Goodbye!
rosuav@sikorsky:~$
With a Py2 consumer, there's still buffering happening. With a Py3
consumer, it works correctly. How to control the Py2 buffering,
though, I don't know.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
Marko Rauhamaa : > Try flushing after each print. http://stackoverflow.com/questions/230751/how-to-flush-ou tput-of-python-print> Since Python 3.3, there is no need to use sys.stdout.flush(): print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False) Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading from sys.stdin reads the whole file in
Steven D'Aprano : > When I pipe one to the other, I expect each line to be printed as they > arrive, but instead they all queue up and happen at once: Try flushing after each print. When sys.stdout is a pipe, flushing happens only when the internal buffer fills up. Marko -- https://mail.python.org/mailman/listinfo/python-list
Reading from sys.stdin reads the whole file in
I'm trying to read from stdin. Here I simulate a process that slowly outputs data to stdout: steve@runes:~$ cat out.py import time print "Hello..." time.sleep(10) print "World!" time.sleep(10) print "Goodbye!" and another process that reads from stdin: steve@runes:~$ cat slurp.py import sys import time for line in sys.stdin: print time.ctime(), line When I pipe one to the other, I expect each line to be printed as they arrive, but instead they all queue up and happen at once: steve@runes:~$ python out.py | python slurp.py Wed Aug 27 15:13:44 2014 Hello... Wed Aug 27 15:13:44 2014 World! Wed Aug 27 15:13:44 2014 Goodbye! (Note how the time stamps are all together, instead of ten seconds apart.) Why is this happening? How can I read from sys.stdin "on the fly", so to speak, without waiting for the first process to end? Is there established terminology for talking about this sort of thing? -- Steven -- https://mail.python.org/mailman/listinfo/python-list
