I was investigating speeding up Mercurial's test suite (it runs ~13,000 Python processes) and I believe I've identified CPython process/interpreter creation and destruction as sources of significant overhead and thus a concern for any CPython user.
Full details are at [1]. tl;dr 10-18% of CPU time in test suite execution was spent doing the equivalent of `python -c 1` and 30-38% of CPU time was spent doing the equivalent of `hg version` (I suspect this is mostly dominated by module importing - and Mercurial has lazy module importing). My measurements show CPython is significantly slower than Perl and Ruby when it comes to process/interpreter creation/destruction. Furthermore, Python 3 appears to be >50% slower than Python 2. Any system spawning many Python processes will likely feel this overhead. I'm also a contributor to Firefox's build and testing infrastructure. Those systems collectively spawn thousands of Python processes. While I haven't yet measured, I fear that CPython process overhead is also contributing to significant overhead there. While more science and knowledge should be added before definite conclusions are reached, I'd like to ring the figurative bell about this problem. This apparent inefficiency has me concerned about the use of CPython in scenarios where process spawning is a common operation and decent latency is important. This includes CLI tools, build systems, and testing. I'm curious if others feel this is a problem and what steps can be taken to mitigate or correct it. [1] http://www.selenic.com/pipermail/mercurial-devel/2014-May/058854.html Gregory Szorc gregory.sz...@gmail.com P.S. I'm not subscribed, so please CC me on responses. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com