[issue38980] Compile libpython with -fno-semantic-interposition

STINNER Victor added the comment: Since Fedora and RHEL build Python with -fno-semantic-interposition, we did not get any user bug report about the LD_PRELOAD use case. IMO we can safely consider that no user rely on LD_PRELOAD to override libpython symbols. Thanks for implementing the

[issue38980] Compile libpython with -fno-semantic-interposition

Change by Petr Viktorin : -- stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ Python-bugs-list

[issue38980] Compile libpython with -fno-semantic-interposition

Petr Viktorin added the comment: New changeset c6d7e82d19c091af698d4e4b3623648e259843e3 by Petr Viktorin in branch 'master': bpo-38980: Only apply -fno-semantic-interposition if available (GH-22892) https://github.com/python/cpython/commit/c6d7e82d19c091af698d4e4b3623648e259843e3

[issue38980] Compile libpython with -fno-semantic-interposition

Change by Petr Viktorin : -- pull_requests: +21824 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/22892 ___ Python tracker ___

[issue38980] Compile libpython with -fno-semantic-interposition

Petr Viktorin added the comment: I was too eager in reviewing this :( It turns out `-fno-semantic-interposition` is GCC 5.3, so [builds fail on older GCC](https://buildbot.python.org/all/#/builders/96/builds/216). I'm researching how to make this conditional in autotools. -- nosy:

[issue38980] Compile libpython with -fno-semantic-interposition

Pablo Galindo Salgado added the comment: New changeset b451b0e9a772f009f4161f7a46476190d0d17ac1 by Pablo Galindo in branch 'master': bpo-38980: Add -fno-semantic-interposition when building with optimizations (GH-22862)

[issue38980] Compile libpython with -fno-semantic-interposition

Change by Pablo Galindo Salgado : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue38980] Compile libpython with -fno-semantic-interposition

Pablo Galindo Salgado added the comment: Victor is on vacation for some weeks, so I am creating a PR to push this forward. -- ___ Python tracker ___

[issue38980] Compile libpython with -fno-semantic-interposition

Change by Pablo Galindo Salgado : -- keywords: +patch pull_requests: +21804 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22862 ___ Python tracker

[issue38980] Compile libpython with -fno-semantic-interposition

Inada Naoki added the comment: +1 -- versions: +Python 3.10 -Python 3.9 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue38980] Compile libpython with -fno-semantic-interposition

Ammar Askar added the comment: Hey Victor, should we try to land this in Python 3.10? Given that no one has brought up any big concerns aside from LD_PRELOAD based hacks and how clang has already had this as the default I think it's relatively safe to make a default for with-optimizations.

[issue38980] Compile libpython with -fno-semantic-interposition

Gregory P. Smith added the comment: Yes this should become part of --with-optimizations when building on a platform using a compiler that (a) supports it and (b) where it matters. If this is only relevant on --enable-shared builds (not the default), i'd assume also make it conditional on

[issue38980] Compile libpython with -fno-semantic-interposition

Gregory P. Smith added the comment: and to echo others: Do not worry about LD_PRELOAD users trying to override internals. That is not a supported use case. It is always a hack. anyone using it knows this. -- ___ Python tracker

[issue38980] Compile libpython with -fno-semantic-interposition

Change by Tianon : -- nosy: +tianon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38980] Compile libpython with -fno-semantic-interposition

Change by Dong-hee Na : -- nosy: +corona10 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38980] Compile libpython with -fno-semantic-interposition

Change by hai shi : -- nosy: +shihai1991 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38980] Compile libpython with -fno-semantic-interposition

STINNER Victor added the comment: We wrote an article about -fno-semantic-interposition flag that we use with GCC on RHEL8 and Fedora: https://developers.redhat.com/blog/2020/06/25/red-hat-enterprise-linux-8-2-brings-faster-python-3-8-run-speeds/ "Enabling this flag disables semantic

[issue38980] Compile libpython with -fno-semantic-interposition

Change by David Filiatrault : -- nosy: +David Filiatrault ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38980] Compile libpython with -fno-semantic-interposition

STINNER Victor added the comment: Pablo: > I have seen people using LD_PRELOAD (...) to interpose faster versions of > some functions or to collect metrics (although there are better ways). IMHO if someone has to go so far into "hacking" Python, they should recompile Python with specific

[issue38980] Compile libpython with -fno-semantic-interposition

STINNER Victor added the comment: > In case of malloc, every memory allocating code need to use > malloc/calloc/realloc. This is official and the only way to allocate a > memory. But we do not guarantee that Python core uses only public C API like > PyErr_Occurred(). It can use more

[issue38980] Compile libpython with -fno-semantic-interposition

Serhiy Storchaka added the comment: In case of malloc, every memory allocating code need to use malloc/calloc/realloc. This is official and the only way to allocate a memory. But we do not guarantee that Python core uses only public C API like PyErr_Occurred(). It can use more low-level and

[issue38980] Compile libpython with -fno-semantic-interposition

Pablo Galindo Salgado added the comment: > It seems to me that the performance gains here really outweigh any weird > usage of LD_PRELOAD. I am very convinced of this assertion, but other users could not be, I think the discussion is how to provide/activate the option in the less intrusive

[issue38980] Compile libpython with -fno-semantic-interposition

Ammar Askar added the comment: Just for a quick datapoint: llvm/clang do this by default and you need an explicit `-fsemantic-interposition` to disable it http://lists.llvm.org/pipermail/llvm-dev/2016-November/107625.html It seems to me that the performance gains here really outweigh any

[issue38980] Compile libpython with -fno-semantic-interposition

Pablo Galindo Salgado added the comment: > I think it will add to the complexity of the --with-optimizations flag which > already implies PGO and LTO. That is why I was suggesting it: --with-optimizations for me means "activate everything that you can to make python faster". --

[issue38980] Compile libpython with -fno-semantic-interposition

Charalampos Stratakis added the comment: > Maybe it should only be default when using --with-optimizations I think it will add to the complexity of the --with-optimizations flag which already implies PGO and LTO. Maybe an opt-in flag would be better IMHO. -- nosy: +cstratak

[issue38980] Compile libpython with -fno-semantic-interposition

Pablo Galindo Salgado added the comment: > Could these user use a "./configure --with-interposition --enable-shared" > build? Sure, but the problem is the default value, no? Maybe it should only be default when using --with-optimizations -- ___

[issue38980] Compile libpython with -fno-semantic-interposition

STINNER Victor added the comment: > If we do this by default, once functions will be inlined these use cases will > be broken. Could these user use a "./configure --with-interposition --enable-shared" build? -- ___ Python tracker

[issue38980] Compile libpython with -fno-semantic-interposition

Pablo Galindo Salgado added the comment: I have seen people using LD_PRELOAD to interpose some auditing functions that can modify the actual call into libpython, or to interpose faster versions of some functions or to collect metrics (although there are better ways). If we do this by

[issue38980] Compile libpython with -fno-semantic-interposition

Change by Miro Hrončok : -- nosy: +hroncok ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38980] Compile libpython with -fno-semantic-interposition

STINNER Victor added the comment: Maybe we need to offer a way to *opt out* from -fno-semantic-interposition. For example, ./configure --with-interposition. The default would be --without-interposition. -- ___ Python tracker

[issue38980] Compile libpython with -fno-semantic-interposition

New submission from STINNER Victor : The Fedora packaging has been modified to compile libpython with -fno-semantic-interposition flag: it makes Python up to 1.3x faster without having to touch any line of the C code! See pyperformance results: