Re: Is print thread safe?
On Tue, Aug 12, 2014 at 2:31 PM, Cameron Simpson wrote: > I would expect file.write to be fast enough that the lock would usually be > free. Until the day when it becomes really REALLY slow, because your program's piped into 'less' and the user's paging through it. But even apart from that, writing to stdout can take a notable amount of time. Expecting the lock to usually be free will depend on the nature of the program - how much of it is spent in silent computation and how much in production of output. If that ratio is sufficiently skewed, then sure, the lock'll usually be free. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Is print thread safe?
On 12Aug2014 09:56, Steven D'Aprano wrote: Cameron Simpson wrote: On 12Aug2014 02:07, Steven D'Aprano wrote: Is this documented somewhere? In python/2.7.6/reference/simple_stmts.html#index-22, "print" is described in terms of a "write" for each object, and a "write" for the separators. There is no mention of locking. Ah, thanks! On that basis, I would find the interleaving described normal and expected. And certainly not "broken". I personally didn't describe it as "broken", Yes, sorry. but it is, despite the documentation. I just ran a couple of trials where I collected the output of sys.stdout while 50 threads blasted "Spam ABCD EFGH" (plus the implicit newline) to stdout as fast as possible using print. The result was that out of 248165 lines[1], 595 were mangled. Many of the mangled lines were the expected simple run-ons: Spam ABCD EFGHSpam ABCD EFGH\n\n which makes sense given the documentation, but there were lots of anomalies. Mysterious spaces appearing in the strings: Spam ABCD EFGH Spam ABCD EFGH\n\n Spam ABCD EFGH Spam ABCD EFGH\n Spam ABCD EFGH\n occasional collisions mid-string: Spam ABSpam ABCD EFGH\nCD EFGH\n letters disappearing: Spam AB\nD EFGH\n and at least one utterly perplexing (to me) block of ASCII NULs appearing in the middle of the output: \x00\x00\x00...\x00\x00\n This is with Python 2.7.2 on Linux. Sounds like print is not thread safe. Which it does not promise to be. But I would normally expect most file.write methods to be thread safe. Naively. Just use a lock! And rebind "print"! Or use the logging system! Personally, I believe that print ought to do its own locking. I don't, but I kind of believe "file"s should have thread safe write calls. Again, not guarrenteed AFAIR. And print is a statement, although in this case there's no need to support anything older than 2.6, so something like this ought to work: from __future__ import print_function _print = print _rlock = threading.RLock() def print(*args, **kwargs): with _rlock: _print(*args, **kwargs) Sadly, using print as a function alone isn't enough to fix this problem, but in my quick tests, using locking as above does fix it, and with no appreciable slowdown. I would expect file.write to be fast enough that the lock would usually be free. With no evidence, just personal expectation. Taking a free lock should be almost instant. [1] Even the number of lines of output demonstrates a bug. I had fifty threads printing 5000 times each, which makes 25 lines, not 248165. Sounds like the file internals are unsafe. Ugh. Cheers, Cameron Simpson If it ain't broken, keep playing with it. -- https://mail.python.org/mailman/listinfo/python-list
Re: Is print thread safe?
On 12Aug2014 08:01, Marko Rauhamaa wrote: Steven D'Aprano : Personally, I believe that print ought to do its own locking. And print is a statement, although in this case there's no need to support anything older than 2.6, so something like this ought to work: from __future__ import print_function _print = print _rlock = threading.RLock() def print(*args, **kwargs): with _rlock: _print(*args, **kwargs) Could this cause a deadlock if print were used in signal handlers? At the C level one tries to do as little as possible in q signal handler. Typically setting a flag or putting something on a queue for later work. In Python that may be a much smaller issue, since I imagine the handler runs in the ordinary course of interpretation, outside the C-level handler context. I personally wouldn't care if this might deadlock in a handler (lots of things might; avoid as many things as possible). Also, the code above uses an RLock; less prone to deadlock than a plain mutex Lock. Cheers, Cameron Simpson A host is a host from coast to coast & no one will talk to a host that's close Unless the host (that isn't close) is busy, hung or dead - David Lesher, wb8...@skybridge.scl.cwru.edu -- https://mail.python.org/mailman/listinfo/python-list
Re: Is print thread safe?
Steven D'Aprano : > Personally, I believe that print ought to do its own locking. And > print is a statement, although in this case there's no need to support > anything older than 2.6, so something like this ought to work: > > > from __future__ import print_function > > _print = print > _rlock = threading.RLock() > def print(*args, **kwargs): > with _rlock: > _print(*args, **kwargs) Could this cause a deadlock if print were used in signal handlers? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Is print thread safe?
On Tue, Aug 12, 2014 at 9:56 AM, Steven D'Aprano wrote: > from __future__ import print_function > > _print = print > _rlock = threading.RLock() > def print(*args, **kwargs): > with _rlock: > _print(*args, **kwargs) You're conflating print and stdout here. Do you know which one is the cause of your problems? Alternatively, can you be certain that you never use either without the other? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Is print thread safe?
Cameron Simpson wrote: > On 12Aug2014 02:07, Steven D'Aprano > wrote: >>INADA Naoki wrote: >> >>> On Python 3, print is thread safe. >>> But Python 2 has broken scenario: >> >>Is this documented somewhere? > > In python/2.7.6/reference/simple_stmts.html#index-22, "print" is described > in terms of a "write" for each object, and a "write" for the separators. > There is no mention of locking. Ah, thanks! > On that basis, I would find the interleaving described normal and > expected. And certainly not "broken". I personally didn't describe it as "broken", but it is, despite the documentation. I just ran a couple of trials where I collected the output of sys.stdout while 50 threads blasted "Spam ABCD EFGH" (plus the implicit newline) to stdout as fast as possible using print. The result was that out of 248165 lines[1], 595 were mangled. Many of the mangled lines were the expected simple run-ons: Spam ABCD EFGHSpam ABCD EFGH\n\n which makes sense given the documentation, but there were lots of anomalies. Mysterious spaces appearing in the strings: Spam ABCD EFGH Spam ABCD EFGH\n\n Spam ABCD EFGH Spam ABCD EFGH\n Spam ABCD EFGH\n occasional collisions mid-string: Spam ABSpam ABCD EFGH\nCD EFGH\n letters disappearing: Spam AB\nD EFGH\n and at least one utterly perplexing (to me) block of ASCII NULs appearing in the middle of the output: \x00\x00\x00...\x00\x00\n This is with Python 2.7.2 on Linux. > Just use a lock! And rebind "print"! Or use the logging system! Personally, I believe that print ought to do its own locking. And print is a statement, although in this case there's no need to support anything older than 2.6, so something like this ought to work: from __future__ import print_function _print = print _rlock = threading.RLock() def print(*args, **kwargs): with _rlock: _print(*args, **kwargs) Sadly, using print as a function alone isn't enough to fix this problem, but in my quick tests, using locking as above does fix it, and with no appreciable slowdown. [1] Even the number of lines of output demonstrates a bug. I had fifty threads printing 5000 times each, which makes 25 lines, not 248165. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Is print thread safe?
On 12Aug2014 02:07, Steven D'Aprano wrote: INADA Naoki wrote: On Python 3, print is thread safe. But Python 2 has broken scenario: Is this documented somewhere? In python/2.7.6/reference/simple_stmts.html#index-22, "print" is described in terms of a "write" for each object, and a "write" for the separators. There is no mention of locking. On that basis, I would find the interleaving described normal and expected. And certainly not "broken". Just use a lock! And rebind "print"! Or use the logging system! Cheers, Cameron Simpson Wow! Yet another place that I've been quoted... - Andy Beals -- https://mail.python.org/mailman/listinfo/python-list
Re: Is print thread safe?
INADA Naoki wrote: > On Python 3, print is thread safe. > > But Python 2 has broken scenario: Is this documented somewhere? -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Is print thread safe?
On Python 3, print is thread safe. But Python 2 has broken scenario: print "spam", "spam", "spam" # thread 1 print "eggs", "eggs", "eggs" # thread 2 In this case, 2 lines are mixed. In your case, "spam spam spam" and "eggs eggs eggs" are not mixed. But newline is mixed like: spam spam spameggs eggs eggs eggs eggs eggsspam spam spam eggs eggs eggs spam spam spam On Mon, Aug 11, 2014 at 4:44 PM, Steven D'Aprano wrote: > Specifically for Python 2.6 and 2.7, but answers for 3.x appreciated as > well. > > Is print thread safe? That is, if I have two threads that each call > print, say: > > print "spam spam spam" # thread 1 > print "eggs eggs eggs" # thread 2 > > I don't care which line prints first, but I do care if the two lines are > mixed in together, something like this: > > spam spaeggs eggs m seggspams > > > Does print perform its own locking to prevent this? > > > > -- > Steven > -- > https://mail.python.org/mailman/listinfo/python-list -- INADA Naoki -- https://mail.python.org/mailman/listinfo/python-list
Is print thread safe?
Specifically for Python 2.6 and 2.7, but answers for 3.x appreciated as well. Is print thread safe? That is, if I have two threads that each call print, say: print "spam spam spam" # thread 1 print "eggs eggs eggs" # thread 2 I don't care which line prints first, but I do care if the two lines are mixed in together, something like this: spam spaeggs eggs m seggspams Does print perform its own locking to prevent this? -- Steven -- https://mail.python.org/mailman/listinfo/python-list
