Hi, I'm now convinced that C extensions must *not* be linked to libpython on Unix.
I wrote PR 12946: https://github.com/python/cpython/pull/12946 bpo-21536: C extensions are no longer linked to libpython On Unix, C extensions are no longer linked to libpython. It is now possible to load a C extension built using a shared library Python with a statically linked Python. When Python is embedded, libpython must not be loaded with RTLD_LOCAL, but RTDL_GLOBAL instead. Previously, using RTLD_LOCAL, it was already not possible to load C extensions which were not linked to libpython, like C extensions of the standard library built by the "*shared*" section of Modules/Setup. distutils, python-config and python-config.py have been modified. This PR allows to load a C extension built by a shared libpython with a statically linked Python: https://bugs.python.org/issue21536#msg340819 It also allows to load C extension built in release mode with a Python built in debug mode: https://bugs.python.org/issue21536#msg340821 Le jeu. 25 avr. 2019 à 08:31, Nathaniel Smith <n...@pobox.com> a écrit : > In principle, having extension modules link to libpython.so is a good thing. > Suppose that someone wants to dynamically load the python interpreter into > their program as some kind of plugin. (Examples: Apache's mod_python, > LibreOffice's support for writing macros in Python.) It would be nice to be > able to load python2 and python3 simultaneously into the same process as > distinct plugins. And this is totally doable in theory, *but* it means that > you can't assume that the interpreter's symbols will be automagically > injected into extension modules, so it's only possible if extension modules > link to libpython.so. I'm aware of 2 special use cases of libpython: (A) Embed Python using RTLD_LOCAL: dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW) Example of issues describing this use case: * 2003: https://bugs.python.org/issue832799 * 2006: https://bugs.python.org/issue1429775 * 2018: https://bugs.python.org/issue34814 and https://bugzilla.redhat.com/show_bug.cgi?id=1585201 Python started to link C extensions to libpython in 2006 for this use case. (B) Load "libpython2" (Python 2) and "libpython3" (Python 3). I heard this idea... but I never saw anyone doing it in practice. I don't understand how it could work in a single address space. Linking C extensions to libpython is causing different issues: (1) C extension built by a shared library Python cannot be loaded with a statically linked Python: https://bugs.python.org/issue21536 (2) C extension built in release mode cannot be loaded with Python built in debug mode. That's the issue discussed in this thread ;-) (3) C extension built by Python 3.6 cannot be loaded in Python 3.7, even if it has been compiled using the stable ABI (Py_LIMITED_API). (4) C extensions of the standard library built by "*shared*" of Modules/Setup are *not* linked to libpython. For example, _struct.so on Fedora is not linked to libpython, whereas . If libpython is loaded with RTLD_LOCAL (use case A), import _struct fails. The use case (A) (RTLD_LOCAL) is trivial to fix: replace RTLD_LOCAL with RTLD_GLOBAL. The use case (B) (libpython2 + libpython3) is also easy to workaround: just use 2 separated processes. Python 2 will reach its end of life at the end of the year, I'm not sure that we should worry too much about this use case. The issue (1) (statically/shared) is a very practical issue. Fedora/RHEL uses libpython whereas Debian/Ubuntu uses statically linked Python. C extension compiled on Fedora/RHEL is linked to libpython and so cannot be loaded on Debian/Ubuntu (their Python doesn't have libpython!). That's why manylinux forbid link to link to libpython: be able to load C extensions on all Linux distributions. IMHO issues (1), (2), (3), (4) are more valuable to be fixed than supporting use cases (A) and (B). Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
