Re: Reducing cache/buffer for faster display
On 29/09/12 02:20:50, Rikishi42 wrote: On 2012-09-28, Dennis Lee Bieber wlfr...@ix.netcom.com wrote: On Thu, 27 Sep 2012 22:25:39 + (UTC), John Gordon gor...@panix.com declaimed the following in gmane.comp.python.general: Isn't terminal output line-buffered? I don't understand why there would be an output delay. (Unless the \r is messing things up...) It's the trailing , The \r is being used to reset to the beginning of the console line, but the comma says more output for /this/ line will be coming... So no output until explicitly flushed, or a new-line is issued. Well, the \r seems to be the problem, allright. But output was not completely blocked, just delayed a very long time. So perhaps flushing and a sending a newline aren't the only triggers for output. Perhaps there's a maximum delay or a maximum cumulated size, and the output is flushed when such a limit is reached. There's a maximum cumulated size; it's called the buffer size. Output goes into a buffer, and when the buffer is full, it's printed all at once. One way to avoid it, is to use an unbuffered stream. Another, more efficient, way to avoid it, is to invoke the stream's .flush() method after writing to it. Anyway, that's mainly academic. I doubt there will be a correction to that behaviour. It's an optimization. When it was invented, 40 years ago, it was a really necessary to do this, to get something resembling performance. The performance of a system without stream buffering would probably be tolerable on modern hardware. But the people maintaining Python are unlikely to cut out buffering, because few people would benefit (yours is pretty much the only use case where buffereing hurts) and some would suffer (those who write many short strings to a disk file). Hope this helps, -- HansM -- http://mail.python.org/mailman/listinfo/python-list
Re: Reducing cache/buffer for faster display
On Fri, Sep 28, 2012 at 10:05 AM, Rikishi42 skunkwo...@rikishi42.net wrote: The scripts in question only increase numbers. But should that not be the case, solutions are simple enough. The numbers can be formatted to have a fixed size. In the case of random line contents (a list of filesnames, say) it's enough to create an output function that is aware of the length of the previously printed line, and add enough spaces to the current one to wipe exess content. Yep, that's a pretty effective way to do it. One simple method to it is to format the whole string as a single whole, then left justify it in a field of (say) 79 characters, and output that: msg = Progress: %d%% (%d/%d)... %s % (done*100/total, done, total, current_file) print msg.ljust(79)+\r, sys.stdout.flush() ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Reducing cache/buffer for faster display
On 2012-09-28, Dennis Lee Bieber wlfr...@ix.netcom.com wrote: On Thu, 27 Sep 2012 22:25:39 + (UTC), John Gordon gor...@panix.com declaimed the following in gmane.comp.python.general: Isn't terminal output line-buffered? I don't understand why there would be an output delay. (Unless the \r is messing things up...) It's the trailing , The \r is being used to reset to the beginning of the console line, but the comma says more output for /this/ line will be coming... So no output until explicitly flushed, or a new-line is issued. Well, the \r seems to be the problem, allright. But output was not completely blocked, just delayed a very long time. So perhaps flushing and a sending a newline aren't the only triggers for output. Perhaps there's a maximum delay or a maximum cumulated size, and the output is flushed when such a limit is reached. Anyway, that's mainly academic. I doubt there will be a correction to that behaviour. -- When in doubt, use brute force. -- Ken Thompson -- http://mail.python.org/mailman/listinfo/python-list
Re: Reducing cache/buffer for faster display
On 2012-09-28, Chris Angelico ros...@gmail.com wrote: On Fri, Sep 28, 2012 at 10:05 AM, Rikishi42 skunkwo...@rikishi42.net wrote: The scripts in question only increase numbers. But should that not be the case, solutions are simple enough. The numbers can be formatted to have a fixed size. In the case of random line contents (a list of filesnames, say) it's enough to create an output function that is aware of the length of the previously printed line, and add enough spaces to the current one to wipe exess content. Yep, that's a pretty effective way to do it. One simple method to it is to format the whole string as a single whole, then left justify it in a field of (say) 79 characters, and output that: msg = Progress: %d%% (%d/%d)... %s % (done*100/total, done, total, current_file) print msg.ljust(79)+\r, sys.stdout.flush() Mmm, I allmost went for that. It's elegant, simple and clear. But there's one drawback: I usually reduce the terminal's window to take up less desktop surface during those long runs. So fixing it to 79 chars won't do. And I'm not even tempted to go for a detection of the width of the terminal from within the script. The idea is after all to keep the scripts simple (syntax) and light (execution). Well, good night everyone. -- When in doubt, use brute force. -- Ken Thompson -- http://mail.python.org/mailman/listinfo/python-list
Re: Reducing cache/buffer for faster display
On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 skunkwo...@rikishi42.net wrote: I have these 2 scripts that are very heavy on the file i/o, consume a very reasonable amount of cpu and output their counters at a - very - relaxed pace to the console. The output is very simply done using something like: print files:, nFiles, \r, Yet alltough there is no real reason for it, even a pace of a print every 10-30 secs will be cached, only to actually show an output update every 1-2 min or so. Yup! Just add a call to sys.stdout.flush() after each print. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Reducing cache/buffer for faster display
In mailman.1522.1348783807.27098.python-l...@python.org Chris Angelico ros...@gmail.com writes: On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 skunkwo...@rikishi42.net wrote: I have these 2 scripts that are very heavy on the file i/o, consume a very reasonable amount of cpu and output their counters at a - very - relaxed pace to the console. The output is very simply done using something like: print files:, nFiles, \r, Yet alltough there is no real reason for it, even a pace of a print every 10-30 secs will be cached, only to actually show an output update every 1-2 min or so. Yup! Just add a call to sys.stdout.flush() after each print. Isn't terminal output line-buffered? I don't understand why there would be an output delay. (Unless the \r is messing things up...) -- John Gordon A is for Amy, who fell down the stairs gor...@panix.com B is for Basil, assaulted by bears -- Edward Gorey, The Gashlycrumb Tinies -- http://mail.python.org/mailman/listinfo/python-list
Re: Reducing cache/buffer for faster display
On Fri, Sep 28, 2012 at 8:25 AM, John Gordon gor...@panix.com wrote: Isn't terminal output line-buffered? I don't understand why there would be an output delay. (Unless the \r is messing things up...) This is a classic progress-indication case, which does indeed mess up line-buffering. The carriage return (and no line feed, done in the Python 2 style of a trailing comma) puts the cursor back to the beginning of the line, ready to overwrite, and ripe for one of those old favorite incomplete overwrite errors - if nFiles monotonically increases, it's fine, but if it decreases, the display can get ugly. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Reducing cache/buffer for faster display
On 2012-09-27, Chris Angelico ros...@gmail.com wrote: On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 skunkwo...@rikishi42.net wrote: I have these 2 scripts that are very heavy on the file i/o, consume a very reasonable amount of cpu and output their counters at a - very - relaxed pace to the console. The output is very simply done using something like: print files:, nFiles, \r, Yet alltough there is no real reason for it, even a pace of a print every 10-30 secs will be cached, only to actually show an output update every 1-2 min or so. Yup! Just add a call to sys.stdout.flush() after each print. Update: tried it, ran it, I love it. Thanks ! -- When in doubt, use brute force. -- Ken Thompson -- http://mail.python.org/mailman/listinfo/python-list
Re: Reducing cache/buffer for faster display
On 2012-09-27, Chris Angelico ros...@gmail.com wrote: On Fri, Sep 28, 2012 at 8:25 AM, John Gordon gor...@panix.com wrote: Isn't terminal output line-buffered? I don't understand why there would be an output delay. (Unless the \r is messing things up...) This is a classic progress-indication case, which does indeed mess up line-buffering. The carriage return (and no line feed, done in the Python 2 style of a trailing comma) puts the cursor back to the beginning of the line, ready to overwrite, and ripe for one of those old favorite incomplete overwrite errors - if nFiles monotonically increases, it's fine, but if it decreases, the display can get ugly. True, but that wasn't the problem here. The updates where. Thanks for the given answer, I'll try it. The scripts in question only increase numbers. But should that not be the case, solutions are simple enough. The numbers can be formatted to have a fixed size. In the case of random line contents (a list of filesnames, say) it's enough to create an output function that is aware of the length of the previously printed line, and add enough spaces to the current one to wipe exess content. Thanks again for the suggestion. -- When in doubt, use brute force. -- Ken Thompson -- http://mail.python.org/mailman/listinfo/python-list
