[issue40399] IO streams locking can be broken after fork() with threads

[Antoine Pitrou]

Antoine Pitrou  added the comment:

The TextIOWrapper uses an underlying BufferedWriter, which is thread-safe (and 
therefore has an internal lock).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40399] IO streams locking can be broken after fork() with threads

[Delgan]

Delgan  added the comment:

Yeah, I just wanted to illustrate the issue with a more realistic example. The 
thread is often abstracted away by a class or a library. Conclusion: do not 
abstract it away. :)

I've noticed that the mere fact of using "sys.stderr.write()", without even 
involving a queue, could cause the problem.

Out of curiosity: my understanding of "sys.stderr" being a "TextIOWrapper" 
implies it's not thread-safe. Therefore, do you have any idea of which lock is 
involved in this issue?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40399] IO streams locking can be broken after fork() with threads

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40399] IO streams locking can be broken after fork() with threads

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Well, as the documentation states, `QueueListener.start` """starts up a 
background thread to monitor the queue for LogRecords to process""" :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40399] IO streams locking can be broken after fork() with threads

[Delgan]

Delgan  added the comment:

Thank you for having looked into the problem.

To be more specific, I don't generally mix threads with multiprocessing, but 
it's a situation where there is one global and hidden consumer thread listening 
to a queue for non-blocking logging.

Actually, I think the problem is reproducible using the QueueListener provided 
in "logging.handlers". The following snippet is inspired by this part of the 
documentation: 
https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block

--

import logging
import multiprocessing
import queue
from logging.handlers import QueueHandler, QueueListener


if __name__ == "__main__":
que = multiprocessing.Queue()

queue_handler = QueueHandler(que)
handler = logging.StreamHandler()
listener = QueueListener(que, handler)
root = logging.getLogger()
root.addHandler(queue_handler)
listener.start()

for i in range(1):
root.warning('Look out!')
p = multiprocessing.Process(target=lambda: None)
p.start()
p.join()
print("Not hanging yet", i)


listener.stop()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40399] IO streams locking can be broken after fork() with threads

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I can reproduce on Ubuntu 18.04 with git master.

Here is a better example which clearly shows the issue:
https://gist.github.com/pitrou/d9784d5ec679059cd02fce4b38ea2fa6

After a few runs, you'll see that the child Process hangs when trying to flush 
the standard streams:
Timeout (0:00:01)!
Thread 0x7efbff6c0080 (most recent call first):
  File "/home/antoine/cpython/default/Lib/multiprocessing/util.py", line 435 in 
_flush_std_streams
  File "/home/antoine/cpython/default/Lib/multiprocessing/process.py", line 335 
in _bootstrap
  File "/home/antoine/cpython/default/Lib/multiprocessing/popen_fork.py", line 
71 in _launch
  File "/home/antoine/cpython/default/Lib/multiprocessing/popen_fork.py", line 
19 in __init__
  File "/home/antoine/cpython/default/Lib/multiprocessing/context.py", line 276 
in _Popen
  File "/home/antoine/cpython/default/Lib/multiprocessing/context.py", line 224 
in _Popen
  File "/home/antoine/cpython/default/Lib/multiprocessing/process.py", line 121 
in start
  File "/home/antoine/cpython/default/bpo40399.py", line 25 in 
Child process failed!


@Delgan, mixing processes and threads is problematic with the default settings. 
 See here:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

"""Note that safely forking a multithreaded process is problematic."""

If you call `multiprocessing.set_start_method("forkserver")` at the start of 
your program, the problem will disappear.

--
nosy: +pitrou, vstinner
title: Program hangs if process created right after adding object to a Queue -> 
IO streams locking can be broken after fork() with threads
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com