commit: 0a383b99701ab7e9da2b9465c4e04de06a733075
Author: Florian Schmaus <flow <AT> gentoo <DOT> org>
AuthorDate: Thu Aug 28 09:41:59 2025 +0000
Commit: Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sun Sep 14 03:14:09 2025 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=0a383b99
process: do not poll join() in MultiprocessingProcesss if we can
Based on the initial analysis of Esteve Varela Colominas, the polling
on join() with the fixed 100ms delay incurs a significant performance
penalty, especially for short-lived processes. And since portage is
prone to spawning many of those, the penalty adds up easily.
Instead of pooling proc.join() with a fixed 100ms delay, we now use
the blocking variant of join() started in a thread which we await in
the coroutine. Unfortunately, we only can use this if the 'forkserver'
or 'spawn' start method is active for multiprocessing. Otherwise,
there is the risk of deadlocks due the interaction between threads and
the 'fork' start method.
Before this change:
( cd lib; python3.13 -m timeit 'import portage.process;
portage.process.spawn("true")' )
2 loops, best of 5: 104 msec per loop
After this change:
( cd lib; python3.14 -m timeit 'import portage.process;
portage.process.spawn("true")' )
1 loop, best of 5: 54.9 msec per loop
Note that the forkerserver in Python 3.14 seems to come with a
performance hit. While polling allowed spawn times ~5 ms, we can't seem
to get lower than 55 ms with forkserver.
Thanks to Zac Medico for input and assistance on this change.
Bug: https://bugs.gentoo.org/958635
Closes: https://bugs.gentoo.org/962721
Signed-off-by: Florian Schmaus <flow <AT> gentoo.org>
Part-of: https://github.com/gentoo/portage/pull/1463
Signed-off-by: Sam James <sam <AT> gentoo.org>
lib/portage/process.py | 31 ++++++++++++++++++++++++-------
1 file changed, 24 insertions(+), 7 deletions(-)
diff --git a/lib/portage/process.py b/lib/portage/process.py
index d252cc7c4a..e17eaf3c33 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -482,13 +482,30 @@ class MultiprocessingProcess(AbstractProcess):
except ValueError:
pass
- # Now that proc.sentinel is ready, poll until process exit
- # status has become available.
- while True:
- proc.join(0)
- if proc.exitcode is not None:
- break
- await asyncio.sleep(self._proc_join_interval, loop=loop)
+ # Now that proc.sentinel is ready, join on proc.
+
+ async def join_via_polling(proc):
+ while True:
+ proc.join(0)
+ if proc.exitcode is not None:
+ break
+ await asyncio.sleep(self._proc_join_interval, loop=loop)
+
+ # We can only safely create a new thread to await the join if
+ # we use 'forkserver' or 'spawn'.
+ if multiprocessing.get_start_method() in ("forkserver", "spawn"):
+ try:
+ await _asyncio.to_thread(proc.join)
+ except RuntimeError as exc:
+ # A RuntimeError may be thrown if this is invoked
+ # during shutdown, e.g., via run_coroutine_exitfuncs
+ # as in the Socks5ServerAtExistTestCase, hence we need
+ # to fall back to polling in this case.
+ if str(exc) != "cannot schedule new futures after shutdown":
+ raise
+ await join_via_polling(proc)
+ else:
+ await join_via_polling(proc)
def _proc_join_done(self, future):
# The join task should never be cancelled, so let it raise
