New submission from holger <holger...@freyther.de>: I intended to use the asyncio framework for building an end-to-end test for our software. In the test I would spawn somewhere between 5k to 10k processes and have the same number of sockets to manage.
When I built a prototype I ran into some scaling issues. Instead of launching our real software I tested it with calls to sleep 30. At some point started processes would finish, a SIGCHLD would be delivered to python and then it would fail: Exception ignored when trying to write to the signal wakeup fd: BlockingIOError: [Errno 11] Resource temporarily unavailable Using strace I saw something like: send(5, "\0", 1, 0) = -1 EAGAIN (Resource temporarily unavailable) waitpid(12218, 0xbf8592d8, WNOHANG) = 0 waitpid(12219, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG) = 12219 send(5, "\0", 1, 0) = -1 EAGAIN (Resource temporarily unavailable) waitpid(12220, 0xbf8592d8, WNOHANG) = 0 waitpid(12221, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG) = 12221 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12293, si_uid=1001, si_status=0, si_utime=0, si_stime= 0} --- getpid() = 11832 write(5, "\21", 1) = -1 EAGAIN (Resource temporarily unavailable) sigreturn({mask=[]}) = 12221 write(2, "Exception ignored when trying to"..., 64) = 64 write(2, "BlockingIOError: [Errno 11] Reso"..., 61) = 61 Looking at the code I see that si_pid of the signal will be ignored and instead wait(2) will be called for all processes. This doesn't seem to scale well enough for my intended use case. I think what could be done is one of the following: * Switch to signalfd for the event notification? * Take si_pid and instead of just notifying that work is there.. inform about the PID that exited? * Use wait(-1,... if there can be only one SIGCHLD handler to collect any dead child ---------- components: asyncio messages: 311692 nosy: asvetlov, holger+lp, yselivanov priority: normal severity: normal status: open title: asyncio SIGCHLD scalability problems _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32776> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com