https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup
Simplified version of another change proposal|This change was originally proposed for [[Releases/32|Fedora 32]] as [[Changes/PythonStaticSpeedup]], however based on [https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/NWPVQSKVWDKA75PDEIJNJIFL5C5SJXB2/ community feedback], it has been significantly reduced. == Summary == We add the <code>-fno-semantic-interposition</code> compiler/linker flag when building Python interpreters, as it provides significant performance improvement, up to 27% depending on the workload. Users will no longer be able to use LD_PRELOAD to override a symbol from libpython, which we consider a good trade off for the speedup. == Owner == * Name: [[User:Cstratak| Charalampos Stratakis]], [[User:Vstinner| Victor Stinner]], [[User:Churchyard| Miro Hrončok]] * Email: python-ma...@redhat.com * Shout-out: [[User:Jankratochvil|Jan Kratochvíl]] for first suggesting this instead of the original proposal, followed by [[User:Kkofler|Kevin Kofler]]. [[User:Fweimer|Florian Weimer]] for providing answers to our questions. David Gray for originally suggesting to link Python statically to gain performance. == Detailed Description == When we build the Python interpreter with the <code>-fno-semantic-interposition</code> compiler/linker flag, we can achieve a performance gain of 5% to 27% depending on the workload. Link time optimizations and profile guided optimizations also have a greater impact when python3 is built this way. As a negative side effect, it disables the LD_PRELOAD feature: it's no longer possible to override symbols in libpython with LD_PRELOAD. Interposition is enabled by default in compilers like GCC: function calls to a library goes through a "Procedure Linkage Table" (PLT). This indirection is required to allow a library loaded by LD_PRELOAD environment variable to override a function. The indirection puts more pressure on the CPU level 1 cache (instruction cache). In term of performance, the main drawback is that function calls from a library to the same library cannot be inlined, to respect the interposition semantics. Inlining is usually a big win in term of performance. Disabling interposition for libpython removes the overhead on function calls by avoiding the PLT indirection, and allows to inline more function calls. We're describing function calls from libpython to libpython, something which is very common in Python: almost all function calls are calls from libpython to libpython. If Fedora users need to use LD_PRELOAD to override symbols in libpython, the recommend way is to build a custom Python without <code>-fno-semantic-interposition</code>. It is still possible to use LD_PRELOAD to override symbols in other libraries (for example in glibc). === Affected Pythons === Primarily, we will change the interpreter in the {{package|python3}} package, that is Python 3.8 in Fedora 32 and any later version of Python in future Fedora releases. Impact on other Python packages (and generally software using Python) is not anticipated (other than the possible speedup). We will also change the [https://developer.fedoraproject.org/tech/languages/python/multiple-pythons.html alternate Python interpreters] where possible and useful, primarily the upstream supported versions of CPython, such as {{package|python39}} (if already packaged), {{package|python37}} and {{package|python36}}. === Affected Fedora releases === This is a Fedora 32 change and it will be implemented in Rawhide (Fedora 32) only. Any future versions of Fedora will inherit the change until it is reverted for some reason. If it turns out that there are absolutely no issues, we might consider backporting the speedup to already released Fedora versions (for example Fedora 31). Such action would be separately coordinated with [https://docs.fedoraproject.org/en-US/fesco/ FESCo]. == Benefit to Fedora == Python's performance will increase significantly depending on the workload. Since many core components of the OS also depend on Python this could lead to an increase in their performance as well, however individual benchmarks will need to be conducted to verify the performance gain for those components. [https://pyperformance.readthedocs.io/ pyperformance] results, ignoring differences smaller than 5%: (See change proposal) == Scope == * Proposal owners: ** Review and merge the [https://src.fedoraproject.org/rpms/python3/pull-request/151 pull request with the implementation]. ** Monitor Koschei for significant problems. ** Backport the change to alternate Python versions. * Other developers are encouraged to check if their package works as expected * Release engineering: N/A (not needed for this Change) -- this change does not require a mass rebuild nor any other special releng work * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) == Upgrade/compatibility impact == Python package maintainers should verify that their packages work as expected and the only impact the end users should see is a performance increase for workloads relying on Python. == How To Test == Test that everything Python related in Fedora works as usual. === Was the flag applied test === You can test whether the <code>-fno-semantic-interposition</code> flag was applied for your Python build: <pre> >>> import sysconfig >>> '-fno-semantic-interposition' in (sysconfig.get_config_var('PY_CFLAGS') + >>> sysconfig.get_config_var('PY_CFLAGS_NODIST')) True >>> '-fno-semantic-interposition' in (sysconfig.get_config_var('PY_LDFLAGS') + >>> sysconfig.get_config_var('PY_LDFLAGS_NODIST')) True </pre> Before the change, you would see <code>False</code>, <code>False</code>. === Performance test === The performance speedup can be measured using the official Python benchmark suite [https://pyperformance.readthedocs.io/ pyperformance]: see [https://pyperformance.readthedocs.io/usage.html#run-benchmarks Run benchmarks]. == User Experience == Python based workloads should see a performance gain of up to 27%. == Dependencies == This change is not dependent on anything else. == Contingency Plan == * Contingency mechanism: If issues appear that cannot be fixed in a timely manner the change can be easily reverted and will be considered again for the next fedora release. * Contingency deadline: Before the beta freeze of Fedora 32 (2020-02-25) * Blocks release? Yes * Blocks product? None == Documentation == This change proposal has all the documentation. See the [[Changes/PythonStaticSpeedup|previous change proposal]] and the [https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/NWPVQSKVWDKA75PDEIJNJIFL5C5SJXB2/ thread about it on the devel mailing list] for more relevant information about what we are not doing -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org
