Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On Mon, Apr 29, 2019 at 5:01 PM Neil Schemenauer wrote: > As far as I understand, we have a similar problem already for > gc.get_objects() because those static type objects don't have a > PyGC_Head. My 2-cent proposal for fixing things in the long term > would be to introduce a function like PyType_Ready that returns a > pointer to the new type. The argument to it would be what is the > current static type structure. The function would copy things from > the static type structure into a newly allocated type structure. I doubt you'll be able to get rid of static types entirely, due to the usual issues with C API breakage. And I'm guessing that static types make up such a tiny fraction of the address space that merely tweaking the percent up or down won't affect performance. But your proposed new API would make it *way* easier to migrate existing code to the stable ABI. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
You have my support is you work on removing static types :-) Here are my notes on the current C APIs to define a type: https://pythoncapi.readthedocs.io/type_object.html IMHO static types should go away in the long term. They are causing too many practical issues. Victor Le mar. 30 avr. 2019 à 02:01, Neil Schemenauer a écrit : > > On 2019-04-27, Nathaniel Smith wrote: > > For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce > > a list of all live objects on demand. If that's the goal, then static type > > objects aren't a huge deal. You can't add extra data into the type objects > > themselves, but since there's a fixed set of them and they're immortal, you > > can just build a static list of all of them in PyType_Ready. > > As far as I understand, we have a similar problem already for > gc.get_objects() because those static type objects don't have a > PyGC_Head. My 2-cent proposal for fixing things in the long term > would be to introduce a function like PyType_Ready that returns a > pointer to the new type. The argument to it would be what is the > current static type structure. The function would copy things from > the static type structure into a newly allocated type structure. > > We have a kind of solution already with PyType_FromSpec, etc. > However, I think it is harder to convert existing extension module > source code to use that API. We want to make it very easy for > people to fix source code. > > If we can remove static types, that would allow us to kill off > Py_TYPE(o)->tp_is_gc(o). I understand why that exists but I think > it is quite an ugly detail of the current GC implementation. I > wonder about the performance impact of it given current memory > latencies. When we do a full GC run, we call PyObject_IS_GC() on > many objects. I fear having to lookup and call tp_is_gc could be > quite expensive. > > I've been playing with the idea of using memory bitmaps rather then > the PyGC_Head. That idea seems to depend on removing static type > objects. Initially I was thinking of it as reducing the memory > overhead for GC types. Now I think the memory overhead doesn't > matter too much but perhaps the bitmaps would be much faster due to > memory latency. There is an interesting Youtube video that compares > vector traversals vs linked list traversals in C++. Linked lists on > modern machines are really terrible. > > Regards, > > Neil -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 2019-04-27, Nathaniel Smith wrote: > For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce > a list of all live objects on demand. If that's the goal, then static type > objects aren't a huge deal. You can't add extra data into the type objects > themselves, but since there's a fixed set of them and they're immortal, you > can just build a static list of all of them in PyType_Ready. As far as I understand, we have a similar problem already for gc.get_objects() because those static type objects don't have a PyGC_Head. My 2-cent proposal for fixing things in the long term would be to introduce a function like PyType_Ready that returns a pointer to the new type. The argument to it would be what is the current static type structure. The function would copy things from the static type structure into a newly allocated type structure. We have a kind of solution already with PyType_FromSpec, etc. However, I think it is harder to convert existing extension module source code to use that API. We want to make it very easy for people to fix source code. If we can remove static types, that would allow us to kill off Py_TYPE(o)->tp_is_gc(o). I understand why that exists but I think it is quite an ugly detail of the current GC implementation. I wonder about the performance impact of it given current memory latencies. When we do a full GC run, we call PyObject_IS_GC() on many objects. I fear having to lookup and call tp_is_gc could be quite expensive. I've been playing with the idea of using memory bitmaps rather then the PyGC_Head. That idea seems to depend on removing static type objects. Initially I was thinking of it as reducing the memory overhead for GC types. Now I think the memory overhead doesn't matter too much but perhaps the bitmaps would be much faster due to memory latency. There is an interesting Youtube video that compares vector traversals vs linked list traversals in C++. Linked lists on modern machines are really terrible. Regards, Neil ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
FYI I pushed my 3 changes to implement my idea. It is now possible to install some extensions in release mode and some others in debug mode. Python in debug mode prefers debug extensions. I documented changes here: https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build The library filename has to be different in debug mode, so it can be co-installable with release build of a C extension. Victor Le samedi 27 avril 2019, Stefan Behnel a écrit : > Matthias Klose schrieb am 25.04.19 um 13:48: >> Are there use cases where you only want to load *some* >> debug extensions, even if more are installed? > > Not sure if there are _important_ use cases (that could justify certain > design decisions), but I can certainly imagine using a non-debug (and > therefore faster) Pandas or NumPy for preparing some data that I need to > debug my own code. More generally, whenever I can avoid using a debug > version of a *dependency* that I don't need to include in my debug > analysis, it's probably a good idea to not use the debug version. > > Even given venvs and virtualisation techniques, it would probably be nice > if users could install debug+nondebug versions of libraries once and then > import the right one at need, rather than having to set up a new > environment (while they're on a train in the middle of nowhere without fast > access to PyPI). > > Stefan > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On Sat, Apr 27, 2019, 04:27 Armin Rigo wrote: > Hi Neil, > > On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer > wrote: > > Regarding the Py_TRACE_REFS fields, I think we can't do them without > > breaking the ABI because of the following. For GC objects, they are > > always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we > > can allocate the extra space needed for the GC linked list. For > > non-GC objects, that's not the case. Extensions can allocate using > > malloc() directly or their own allocator and then pass that memory > > to be initialized as a PyObject. > > > > I think that's a poor design and I think we should try to make slow > > progress in fixing it. > > Such progress needs to start with the global static PyTypeObjects that > all extensions define. This is going to be impossible to fix without > requiring a big fix in of *all* of them. (Unless of course you mean > to still allow them, but then Py_TRACE_REF can't be implemented in a > way that doesn't break the ABI.) > For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce a list of all live objects on demand. If that's the goal, then static type objects aren't a huge deal. You can't add extra data into the type objects themselves, but since there's a fixed set of them and they're immortal, you can just build a static list of all of them in PyType_Ready. -n > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Matthias Klose schrieb am 25.04.19 um 13:48: > Are there use cases where you only want to load *some* > debug extensions, even if more are installed? Not sure if there are _important_ use cases (that could justify certain design decisions), but I can certainly imagine using a non-debug (and therefore faster) Pandas or NumPy for preparing some data that I need to debug my own code. More generally, whenever I can avoid using a debug version of a *dependency* that I don't need to include in my debug analysis, it's probably a good idea to not use the debug version. Even given venvs and virtualisation techniques, it would probably be nice if users could install debug+nondebug versions of libraries once and then import the right one at need, rather than having to set up a new environment (while they're on a train in the middle of nowhere without fast access to PyPI). Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Hi Neil, On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer wrote: > Regarding the Py_TRACE_REFS fields, I think we can't do them without > breaking the ABI because of the following. For GC objects, they are > always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we > can allocate the extra space needed for the GC linked list. For > non-GC objects, that's not the case. Extensions can allocate using > malloc() directly or their own allocator and then pass that memory > to be initialized as a PyObject. > > I think that's a poor design and I think we should try to make slow > progress in fixing it. Such progress needs to start with the global static PyTypeObjects that all extensions define. This is going to be impossible to fix without requiring a big fix in of *all* of them. (Unless of course you mean to still allow them, but then Py_TRACE_REF can't be implemented in a way that doesn't break the ABI.) A bientôt, Armin. On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer wrote: > > On 2019-04-24, Victor Stinner wrote: > > The current blocker issue is that the Py_DEBUG define imply the > > Py_TRACE_REFS define > > I think your change to make Py_TRACE_REFS as separate configure flag > is fine. I've used the trace fields to debug occasionally but I > don't use it often enough to need it enabled by Py_DEBUG. > > > Being able to switch between Python in release mode and Python in > > debug mode is a first step. My long term plan would be to better > > separate "Python" from its "runtime". > > Regarding the Py_TRACE_REFS fields, I think we can't do them without > breaking the ABI because of the following. For GC objects, they are > always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we > can allocate the extra space needed for the GC linked list. For > non-GC objects, that's not the case. Extensions can allocate using > malloc() directly or their own allocator and then pass that memory > to be initialized as a PyObject. > > I think that's a poor design and I think we should try to make slow > progress in fixing it. I think non-GC objects should also get > allocated by a Python API. In that case, the Py_TRACE_REFS > functionality could be implemented in a way that doesn't break the > ABI. It also makes the CPython API more friendly for alternative > Python runtimes like PyPy, etc. > > Note that this change would not prevent an extension from allocating > memory with it's own allocator. It just means that memory can't > hold a PyObject. The extension PyObject would need to have a > pointer that points to this externally allocated memory. > > I can imagine there could be some situations when people really > want a PyObject to reside in a certain memory location. E.g. maybe > you have some kind of special shared memory area. In that case, I > think we could have specialized APIs to create PyObjects using a > specialized allocator. Those APIs would not be supported by > some runtimes (e.g. tracing/moving GC for PyObjects) and the APIs > would not be used by most extensions. > > Regards, > > Neil > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/armin.rigo%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 25.04.19 13:14, Victor Stinner wrote:
> Le jeu. 25 avr. 2019 à 09:34, Matthias Klose a écrit :
>> there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So
>> depending on the package maintainer, you already have that available, but it
>> is
>> extra maintenance cost. Simplifying that would be a good idea.
>
> Fedora provides "debuginfo" for all binarry packages (like numpy), but
> that's different from a debug build. Usually, C code of packages are
> optimized by gcc -O2 or even gcc -O3 which makes the debugging
> experience very painful: gdb fails to read C local variables and just
> say "". To debug internals, you want a debug build
> compiled by gcc -Og or (better IMHO) gcc -O0.
>
> If you want to inspect *Python* internals but you don't need to
> inspect numpy internals, being able to run a release numpy on a debug
> Python is convenient.
yes, the Debian/Ubuntu packages contain both the debug build, and the debug info
for they normal build, e.g.
/usr/lib/debug/.build-id/3a/8ea2ab6ee85ff68879a48170966873eb8da781.debug
/usr/lib/debug/.build-id/78/5ff95f8d2d06c5990ae4e03cdff99452ca0de9.debug
/usr/lib/debug/.build-id/92/e008cffa3f09106214bfb6b80b7fd02ceab74f.debug
/usr/lib/debug/.build-id/ab/33160518c41acc0488bbc3af878995ef74e07f.debug
/usr/lib/debug/.build-id/bd/65896626a4c6566e96ad008362922cf6a39cd6.debug
/usr/lib/debug/.build-id/f1/e83b14a76dd9564e962dcdd2f70202e6fdb2b1.debug
/usr/lib/debug/.build-id/ff/5eab5fd2d14f4bfa6a1ef2300358efdc7dd800.debug
/usr/lib/python3/dist-packages/lxml/_elementpath.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/builder.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/etree.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/html/clean.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/html/diff.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/objectify.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/sax.cpython-37dm-x86_64-linux-gnu.so
> With an additional change on SOABI (I will open a separated issue for
> that), my PR 12946 (no longer link C extensions to libpython) allows
> to load lxml built in release mode in a Python built in debug mode!
> That's *very* useful for debugging. I show an example of the gdb
> experience with a release Python vs debug Python:
>
> https://bugs.python.org/issue21536#msg340821
>
> With a release Python, the basic function "py-bt" works as expected,
> but inspecting Python internals doesn't work: most local C variables
> are "optimized out" :-(
>
> With a debug Python, the debugging experience is *much* better: it's
> possible to inspect Python internals!
>
>
>> However I still
>> would like to be able to have "debug" and "non-debug" builds co-installable
>> at
>> the same time.
>
> One option is to keep "d" flag in the SOABI so C extensions get a
> different SO filename (no change compared to Python 3.7):
> "NAME.cpython-38-x86_64-linux-gnu.so" for release vs
> "NAME.cpython-38d-x86_64-linux-gnu.so" for debug, debug gets "d"
> suffix ("cpython-38" vs "cpython-38d").
>
> *But* modify importlib when Python is compiled in debug mode to look
> also to SO without the "d" suffix: first try load
> "NAME.cpython-38d-x86_64-linux-gnu.so" (debug: "d" suffix). If there
> is no match, look for "NAME.cpython-38-x86_64-linux-gnu.so" (release:
> no suffix). Since the ABI is now compatible in Python 3.8, it should
> "just work" :-)
>
> From a Linux packager perspective, nothing changes ;-) We can still
> provide "apt install python3-numpy-dbg" (debug) which can is
> co-installable with "apt install python3-numpy" (release).
>
> The benefit is that it will be possible to load C extensions which are
> only available in the release flavor with a debug Python ;-)
yes, that sounds good. Are there use cases where you only want to load *some*
debug extensions, even if more are installed?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 25.04.19 13:26, Victor Stinner wrote: > I looked how fonforge gets compiler and linker flags to embed Python: > it seems like to "pkg-config --libs python-2.7" which returns > "-lpython2.7". My PR doesn't change Misc/python.pc. Should I modify > Misc/python.pc as well... or not? :-) I'm not used to pkg-config. I > don't know if it's common that C extensions are built using > pkg-config. I guess that distutils is more commonly used to build C > extensions. ... except for all the software which is doing some embedding (e.g. vim), or is building some bindings as part of the upstream software. So yes, there is some stuff ... The tendency seems to deprecate your own config helper in favor of pkgconfig. However I'm not sure how this would do with the current MacOS python-config python script. If we want to differentiate between embedding and extensions, then we need two different module names, maybe keeping the current one for extensions, and having a new one for embedding. Not sure about python-config, if we want a new helper for embedding, or add new options for the existing script. > Victor > > Le jeu. 25 avr. 2019 à 12:53, Victor Stinner a écrit : >> Le jeu. 25 avr. 2019 à 09:30, Matthias Klose a écrit : >>> the purpose of python-config here is not clear. Whether it's intended to be >>> used >>> for linking extensions, or embedded interpreters. Currently you are using >>> the >>> same for both use cases. >> >> My PR 12946 removes libpython from distutils, python-config and >> python-config.py: >> https://github.com/python/cpython/pull/12946 >> >> Do you mean that this change will break the build of applications >> embedding Python? If yes, what can done to fix that? >> >> Provide a different script to the specific case of embedded Python? Or >> add a new option to specify that you are embedding Python? >> >> In Python 3.7, the required linker flag is "-lpython3.7m". It's not >> trivial to guess the "m" suffix. FYI Python 3.8 it becames just >> "-lpython3.8": I removed the "m" suffix which was useless. >> >> Victor >> -- >> Night gathers, and now my watch begins. It shall not end until my death. > > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
I looked how fonforge gets compiler and linker flags to embed Python: it seems like to "pkg-config --libs python-2.7" which returns "-lpython2.7". My PR doesn't change Misc/python.pc. Should I modify Misc/python.pc as well... or not? :-) I'm not used to pkg-config. I don't know if it's common that C extensions are built using pkg-config. I guess that distutils is more commonly used to build C extensions. Victor Le jeu. 25 avr. 2019 à 12:53, Victor Stinner a écrit : > Le jeu. 25 avr. 2019 à 09:30, Matthias Klose a écrit : > > the purpose of python-config here is not clear. Whether it's intended to be > > used > > for linking extensions, or embedded interpreters. Currently you are using > > the > > same for both use cases. > > My PR 12946 removes libpython from distutils, python-config and > python-config.py: > https://github.com/python/cpython/pull/12946 > > Do you mean that this change will break the build of applications > embedding Python? If yes, what can done to fix that? > > Provide a different script to the specific case of embedded Python? Or > add a new option to specify that you are embedding Python? > > In Python 3.7, the required linker flag is "-lpython3.7m". It's not > trivial to guess the "m" suffix. FYI Python 3.8 it becames just > "-lpython3.8": I removed the "m" suffix which was useless. > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Le jeu. 25 avr. 2019 à 09:34, Matthias Klose a écrit :
> there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So
> depending on the package maintainer, you already have that available, but it
> is
> extra maintenance cost. Simplifying that would be a good idea.
Fedora provides "debuginfo" for all binarry packages (like numpy), but
that's different from a debug build. Usually, C code of packages are
optimized by gcc -O2 or even gcc -O3 which makes the debugging
experience very painful: gdb fails to read C local variables and just
say "". To debug internals, you want a debug build
compiled by gcc -Og or (better IMHO) gcc -O0.
If you want to inspect *Python* internals but you don't need to
inspect numpy internals, being able to run a release numpy on a debug
Python is convenient.
With an additional change on SOABI (I will open a separated issue for
that), my PR 12946 (no longer link C extensions to libpython) allows
to load lxml built in release mode in a Python built in debug mode!
That's *very* useful for debugging. I show an example of the gdb
experience with a release Python vs debug Python:
https://bugs.python.org/issue21536#msg340821
With a release Python, the basic function "py-bt" works as expected,
but inspecting Python internals doesn't work: most local C variables
are "optimized out" :-(
With a debug Python, the debugging experience is *much* better: it's
possible to inspect Python internals!
> However I still
> would like to be able to have "debug" and "non-debug" builds co-installable at
> the same time.
One option is to keep "d" flag in the SOABI so C extensions get a
different SO filename (no change compared to Python 3.7):
"NAME.cpython-38-x86_64-linux-gnu.so" for release vs
"NAME.cpython-38d-x86_64-linux-gnu.so" for debug, debug gets "d"
suffix ("cpython-38" vs "cpython-38d").
*But* modify importlib when Python is compiled in debug mode to look
also to SO without the "d" suffix: first try load
"NAME.cpython-38d-x86_64-linux-gnu.so" (debug: "d" suffix). If there
is no match, look for "NAME.cpython-38-x86_64-linux-gnu.so" (release:
no suffix). Since the ABI is now compatible in Python 3.8, it should
"just work" :-)
From a Linux packager perspective, nothing changes ;-) We can still
provide "apt install python3-numpy-dbg" (debug) which can is
co-installable with "apt install python3-numpy" (release).
The benefit is that it will be possible to load C extensions which are
only available in the release flavor with a debug Python ;-)
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Le jeu. 25 avr. 2019 à 09:30, Matthias Klose a écrit : > the purpose of python-config here is not clear. Whether it's intended to be > used > for linking extensions, or embedded interpreters. Currently you are using the > same for both use cases. My PR 12946 removes libpython from distutils, python-config and python-config.py: https://github.com/python/cpython/pull/12946 Do you mean that this change will break the build of applications embedding Python? If yes, what can done to fix that? Provide a different script to the specific case of embedded Python? Or add a new option to specify that you are embedding Python? In Python 3.7, the required linker flag is "-lpython3.7m". It's not trivial to guess the "m" suffix. FYI Python 3.8 it becames just "-lpython3.8": I removed the "m" suffix which was useless. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Hi,
I'm now convinced that C extensions must *not* be linked to libpython on Unix.
I wrote PR 12946:
https://github.com/python/cpython/pull/12946
bpo-21536: C extensions are no longer linked to libpython
On Unix, C extensions are no longer linked to libpython.
It is now possible to load a C extension built using a shared library
Python with a statically linked Python.
When Python is embedded, libpython must not be loaded with
RTLD_LOCAL, but RTDL_GLOBAL instead. Previously, using RTLD_LOCAL, it
was already not possible to load C extensions which were not linked
to libpython, like C extensions of the standard library built by the
"*shared*" section of Modules/Setup.
distutils, python-config and python-config.py have been modified.
This PR allows to load a C extension built by a shared libpython with
a statically linked Python:
https://bugs.python.org/issue21536#msg340819
It also allows to load C extension built in release mode with a Python
built in debug mode:
https://bugs.python.org/issue21536#msg340821
Le jeu. 25 avr. 2019 à 08:31, Nathaniel Smith a écrit :
> In principle, having extension modules link to libpython.so is a good thing.
> Suppose that someone wants to dynamically load the python interpreter into
> their program as some kind of plugin. (Examples: Apache's mod_python,
> LibreOffice's support for writing macros in Python.) It would be nice to be
> able to load python2 and python3 simultaneously into the same process as
> distinct plugins. And this is totally doable in theory, *but* it means that
> you can't assume that the interpreter's symbols will be automagically
> injected into extension modules, so it's only possible if extension modules
> link to libpython.so.
I'm aware of 2 special use cases of libpython:
(A) Embed Python using RTLD_LOCAL: dlopen("libpython2.7.so.1.0",
RTLD_LOCAL | RTLD_NOW)
Example of issues describing this use case:
* 2003: https://bugs.python.org/issue832799
* 2006: https://bugs.python.org/issue1429775
* 2018: https://bugs.python.org/issue34814 and
https://bugzilla.redhat.com/show_bug.cgi?id=1585201
Python started to link C extensions to libpython in 2006 for this use case.
(B) Load "libpython2" (Python 2) and "libpython3" (Python 3).
I heard this idea... but I never saw anyone doing it in practice. I
don't understand how it could work in a single address space.
Linking C extensions to libpython is causing different issues:
(1) C extension built by a shared library Python cannot be loaded with
a statically linked Python:
https://bugs.python.org/issue21536
(2) C extension built in release mode cannot be loaded with Python
built in debug mode. That's the issue discussed in this thread ;-)
(3) C extension built by Python 3.6 cannot be loaded in Python 3.7,
even if it has been compiled using the stable ABI (Py_LIMITED_API).
(4) C extensions of the standard library built by "*shared*" of
Modules/Setup are *not* linked to libpython. For example, _struct.so
on Fedora is not linked to libpython, whereas . If libpython is loaded
with RTLD_LOCAL (use case A), import _struct fails.
The use case (A) (RTLD_LOCAL) is trivial to fix: replace RTLD_LOCAL
with RTLD_GLOBAL.
The use case (B) (libpython2 + libpython3) is also easy to workaround:
just use 2 separated processes. Python 2 will reach its end of life at
the end of the year, I'm not sure that we should worry too much about
this use case.
The issue (1) (statically/shared) is a very practical issue.
Fedora/RHEL uses libpython whereas Debian/Ubuntu uses statically
linked Python. C extension compiled on Fedora/RHEL is linked to
libpython and so cannot be loaded on Debian/Ubuntu (their Python
doesn't have libpython!). That's why manylinux forbid link to link to
libpython: be able to load C extensions on all Linux distributions.
IMHO issues (1), (2), (3), (4) are more valuable to be fixed than
supporting use cases (A) and (B).
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 25.04.19 08:31, Nathaniel Smith wrote: > You don't necessarily need rpath actually. The Linux loader has a > bug/feature where once it has successfully loaded a library with a given > soname, then any future requests for that soname within the same process > will automatically return that same library, regardless of rpath settings > etc. So as long as the main interpreter has loaded libpython.whatever from > the correct directory, then extension modules will all get that same > version. The rpath won't matter at all. > > It is annoying in general that on Linux, we have these two different ways > to build extension modules. It definitely violates TOOWTDI :-). It would be > nice at some point to get rid of one of them. > > Note that we can't get rid of the two different ways entirely though – on > Windows, extension modules *must* link to libpython.dll, and on macOS, > extension modules *can't* link to libpython.dylib. So the best we can hope > for is to make Linux consistently do one of these, instead of supporting > both. > > In principle, having extension modules link to libpython.so is a good > thing. Suppose that someone wants to dynamically load the python > interpreter into their program as some kind of plugin. (Examples: Apache's > mod_python, LibreOffice's support for writing macros in Python.) It would > be nice to be able to load python2 and python3 simultaneously into the same > process as distinct plugins. And this is totally doable in theory, *but* it > means that you can't assume that the interpreter's symbols will be > automagically injected into extension modules, so it's only possible if > extension modules link to libpython.so. > > In practice, extension modules have never consistently linked to > libpython.so, so everybody who loads the interpreter as a plugin has > already worked around this. Specifically, they use RTLD_GLOBAL to dump all > the interpreter's symbols into the global namespace. This is why you can't > have python2 and python3 mod_python at the same time in the same Apache. > And since everyone is already working around this, linking to libpython.so > currently has zero benefit... in fact manylinux wheels are actually > forbidden to link to libpython.so, because this is the only way to get > wheels that work on every interpreter. extensions in Debian/Ubuntu packages are not linked against libpython.so, but the main reason here is that sometimes you have to extensions built in transition periods like for 3.6 and 3.7. And this is also the default when not configuring with --enable-shared. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 24.04.19 01:44, Victor Stinner wrote:
> Hi,
>
> Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by
> default in debug build", but I lost myself in details, I forgot the
> main purpose of my proposal...
>
> Let me retry from scratch with a more explicit title: I would like to
> be able to run C extensions compiled in release mode on a Python
> compiled in debug mode ("pydebug"). The use case is to debug bugs in C
> extensions thanks to additional runtime checks of a Python debug
> build, and more generally get a better debugging experiences on
> Python. Even for pure Python, a debug build is useful (to get the
> Pyhon traceback in gdb using "py-bt" command).
>
> Currently, using a Python compiled in debug mode means to have to
> recompile C extensions in debug mode. Compile a C extension requires a
> C compiler, header files, pull dependencies, etc. It can be very
> complicated in practical (and pollute your system with all these
> additional dependencies). On Linux, it's already hard, but on Windows
> it can be even harder.
>
> Just one concrete example: no debug build of numpy is provided at
> https://pypi.org/project/numpy/ Good luck to build numpy in debug mode
> manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.)
> :-)
there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So
depending on the package maintainer, you already have that available, but it is
extra maintenance cost. Simplifying that would be a good idea. However I still
would like to be able to have "debug" and "non-debug" builds co-installable at
the same time.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 24.04.19 18:02, Victor Stinner wrote: > Hum, I found issues with libpython: C extensions are explicitly linked > to libpython built in release mode. So a debug python loading a C > extension may load libpython in release mode, whereas libpython in > debug mode is already loaded. > > When Python is built with --enable-shared, the python3.7 program is > linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly > linked to libpython3.7m as well: > > $ python3.7-config --ldflags > ... -lpython3.7m ... > > Example with numpy: > > $ ldd > /usr/lib64/python3.7/site-packages/numpy/core/umath.cpython-37m-x86_64-linux-gnu.so > ... > libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) > ... > > When Python 3.7 is compiled in debug mode, libpython gets a "d" flag > for debug: libpython3.7dm.so.1.0. > > I see 2 solutions: > > (1) Use a different directory. If "libpython" gets the same filename > in release and debug mode, at least, they must be installed in > different directories. If libpython build in debug mode is installed > in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be > compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug > libpython. > > (2) If "libpython" gets a different filename in debug mode, C > extensions should not be linked to libpython explicitly but > *implicitly* to avoid picking the wrong libpython. For example, remove > "-lpython3.7m" from "python3.7-config --ldflags" output. > > The option (1) rely on rpath which is discouraged by Linux vendors and > may not be supported by all operating systems. > > The option (2) is simpler and likely more portable. > > Currently, C extensions of the standard library may or may not be > linked to libpython depending on they are built. In practice, both > work since python3.7 is already linked to libpython: so libpython is > already loaded in memory before C extensions are loaded. the purpose of python-config here is not clear. Whether it's intended to be used for linking extensions, or embedded interpreters. Currently you are using the same for both use cases. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
You don't necessarily need rpath actually. The Linux loader has a bug/feature where once it has successfully loaded a library with a given soname, then any future requests for that soname within the same process will automatically return that same library, regardless of rpath settings etc. So as long as the main interpreter has loaded libpython.whatever from the correct directory, then extension modules will all get that same version. The rpath won't matter at all. It is annoying in general that on Linux, we have these two different ways to build extension modules. It definitely violates TOOWTDI :-). It would be nice at some point to get rid of one of them. Note that we can't get rid of the two different ways entirely though – on Windows, extension modules *must* link to libpython.dll, and on macOS, extension modules *can't* link to libpython.dylib. So the best we can hope for is to make Linux consistently do one of these, instead of supporting both. In principle, having extension modules link to libpython.so is a good thing. Suppose that someone wants to dynamically load the python interpreter into their program as some kind of plugin. (Examples: Apache's mod_python, LibreOffice's support for writing macros in Python.) It would be nice to be able to load python2 and python3 simultaneously into the same process as distinct plugins. And this is totally doable in theory, *but* it means that you can't assume that the interpreter's symbols will be automagically injected into extension modules, so it's only possible if extension modules link to libpython.so. In practice, extension modules have never consistently linked to libpython.so, so everybody who loads the interpreter as a plugin has already worked around this. Specifically, they use RTLD_GLOBAL to dump all the interpreter's symbols into the global namespace. This is why you can't have python2 and python3 mod_python at the same time in the same Apache. And since everyone is already working around this, linking to libpython.so currently has zero benefit... in fact manylinux wheels are actually forbidden to link to libpython.so, because this is the only way to get wheels that work on every interpreter. -n On Wed, Apr 24, 2019, 09:54 Victor Stinner wrote: > Hum, I found issues with libpython: C extensions are explicitly linked > to libpython built in release mode. So a debug python loading a C > extension may load libpython in release mode, whereas libpython in > debug mode is already loaded. > > When Python is built with --enable-shared, the python3.7 program is > linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly > linked to libpython3.7m as well: > > $ python3.7-config --ldflags > ... -lpython3.7m ... > > Example with numpy: > > $ ldd /usr/lib64/python3.7/site-packages/numpy/core/ > umath.cpython-37m-x86_64-linux-gnu.so > ... > libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) > ... > > When Python 3.7 is compiled in debug mode, libpython gets a "d" flag > for debug: libpython3.7dm.so.1.0. > > I see 2 solutions: > > (1) Use a different directory. If "libpython" gets the same filename > in release and debug mode, at least, they must be installed in > different directories. If libpython build in debug mode is installed > in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be > compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug > libpython. > > (2) If "libpython" gets a different filename in debug mode, C > extensions should not be linked to libpython explicitly but > *implicitly* to avoid picking the wrong libpython. For example, remove > "-lpython3.7m" from "python3.7-config --ldflags" output. > > The option (1) rely on rpath which is discouraged by Linux vendors and > may not be supported by all operating systems. > > The option (2) is simpler and likely more portable. > > Currently, C extensions of the standard library may or may not be > linked to libpython depending on they are built. In practice, both > work since python3.7 is already linked to libpython: so libpython is > already loaded in memory before C extensions are loaded. > > I opened https://bugs.python.org/issue34814 to discuss how C > extensions of the standard library should be linked but I closed it > because we failed to find a consensus and the initial use case became > a non-issue. It seems like we should reopen the discussion :-) > > Victor > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com > On Wed, Apr 24, 2019, 09:54 Victor Stinner wrote: > Hum, I found issues with libpython: C extensions are explicitly linked > to libpython built in release mode. So a debug python loading a C > extension may load libpython in release mode, whereas libpython in > debug mode is already loaded. > > When Python is built with --enable-shared, the
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 2019-04-24, Victor Stinner wrote: > The current blocker issue is that the Py_DEBUG define imply the > Py_TRACE_REFS define I think your change to make Py_TRACE_REFS as separate configure flag is fine. I've used the trace fields to debug occasionally but I don't use it often enough to need it enabled by Py_DEBUG. > Being able to switch between Python in release mode and Python in > debug mode is a first step. My long term plan would be to better > separate "Python" from its "runtime". Regarding the Py_TRACE_REFS fields, I think we can't do them without breaking the ABI because of the following. For GC objects, they are always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we can allocate the extra space needed for the GC linked list. For non-GC objects, that's not the case. Extensions can allocate using malloc() directly or their own allocator and then pass that memory to be initialized as a PyObject. I think that's a poor design and I think we should try to make slow progress in fixing it. I think non-GC objects should also get allocated by a Python API. In that case, the Py_TRACE_REFS functionality could be implemented in a way that doesn't break the ABI. It also makes the CPython API more friendly for alternative Python runtimes like PyPy, etc. Note that this change would not prevent an extension from allocating memory with it's own allocator. It just means that memory can't hold a PyObject. The extension PyObject would need to have a pointer that points to this externally allocated memory. I can imagine there could be some situations when people really want a PyObject to reside in a certain memory location. E.g. maybe you have some kind of special shared memory area. In that case, I think we could have specialized APIs to create PyObjects using a specialized allocator. Those APIs would not be supported by some runtimes (e.g. tracing/moving GC for PyObjects) and the APIs would not be used by most extensions. Regards, Neil ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 24.04.2019 17:03, Antoine Pitrou wrote: On Wed, 24 Apr 2019 01:44:17 +0200 Victor Stinner wrote: The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 extra fields (_ob_prev and _ob_next) which change the offset of all attributes of all objects and makes the ABI completely incompatible. I propose to no longer imply Py_TRACE_REFS *by default* (but keep the code): https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615 +1 from me. The second issue is that library filenames are different for a debug build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then also look for "NAME.cpython-38m.so" (flags: "m"). Sounds fair (but only on Unix, I guess). Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode. But that's more for convenience, it's not really required, since it is easy to switch the Python runtime between release and debug build. Not sure what you mean by "easy to switch the Python runtime". As soon as I want to use pip, I have to use a release build, right? No, pip works with a debug Python just as well (python.bat -m ensurepip) and installs modules to `/site-packages` IIRC. But building extensions is broken in this case as per https://mail.python.org/pipermail/python-dev/2019-April/157180.html . Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Jeroen Demeyer schrieb am 24.04.19 um 09:24: > On 2019-04-24 01:44, Victor Stinner wrote: >> I would like to >> be able to run C extensions compiled in release mode on a Python >> compiled in debug mode > > That seems like a very good idea. I would certainly use the debug mode > while developing CPython or C extensions. +1 Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On Wed, 24 Apr 2019 18:02:18 +0200 Victor Stinner wrote: > > I see 2 solutions: > > (1) Use a different directory. If "libpython" gets the same filename > in release and debug mode, at least, they must be installed in > different directories. If libpython build in debug mode is installed > in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be > compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug > libpython. > > (2) If "libpython" gets a different filename in debug mode, C > extensions should not be linked to libpython explicitly but > *implicitly* to avoid picking the wrong libpython. For example, remove > "-lpython3.7m" from "python3.7-config --ldflags" output. > > The option (1) rely on rpath which is discouraged by Linux vendors and > may not be supported by all operating systems. > > The option (2) is simpler and likely more portable. > > Currently, C extensions of the standard library may or may not be > linked to libpython depending on they are built. In practice, both > work since python3.7 is already linked to libpython: so libpython is > already loaded in memory before C extensions are loaded. You can participate in https://bugs.python.org/issue21536 Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Hum, I found issues with libpython: C extensions are explicitly linked to libpython built in release mode. So a debug python loading a C extension may load libpython in release mode, whereas libpython in debug mode is already loaded. When Python is built with --enable-shared, the python3.7 program is linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly linked to libpython3.7m as well: $ python3.7-config --ldflags ... -lpython3.7m ... Example with numpy: $ ldd /usr/lib64/python3.7/site-packages/numpy/core/umath.cpython-37m-x86_64-linux-gnu.so ... libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) ... When Python 3.7 is compiled in debug mode, libpython gets a "d" flag for debug: libpython3.7dm.so.1.0. I see 2 solutions: (1) Use a different directory. If "libpython" gets the same filename in release and debug mode, at least, they must be installed in different directories. If libpython build in debug mode is installed in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug libpython. (2) If "libpython" gets a different filename in debug mode, C extensions should not be linked to libpython explicitly but *implicitly* to avoid picking the wrong libpython. For example, remove "-lpython3.7m" from "python3.7-config --ldflags" output. The option (1) rely on rpath which is discouraged by Linux vendors and may not be supported by all operating systems. The option (2) is simpler and likely more portable. Currently, C extensions of the standard library may or may not be linked to libpython depending on they are built. In practice, both work since python3.7 is already linked to libpython: so libpython is already loaded in memory before C extensions are loaded. I opened https://bugs.python.org/issue34814 to discuss how C extensions of the standard library should be linked but I closed it because we failed to find a consensus and the initial use case became a non-issue. It seems like we should reopen the discussion :-) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On Wed, 24 Apr 2019 01:44:17 +0200 Victor Stinner wrote: > > The first requirement for the use case is that a Python debug build > supports the ABI of a release build. The current blocker issue is that > the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 > extra fields (_ob_prev and _ob_next) which change the offset of all > attributes of all objects and makes the ABI completely incompatible. I > propose to no longer imply Py_TRACE_REFS *by default* (but keep the > code): > > https://bugs.python.org/issue36465 > https://github.com/python/cpython/pull/12615 +1 from me. > The second issue is that library filenames are different for a debug > build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build > should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then > also look for "NAME.cpython-38m.so" (flags: "m"). Sounds fair (but only on Unix, I guess). > Maybe pip could be enhanced to support installing C extensions > compiled in release mode when using a debug mode. But that's more for > convenience, it's not really required, since it is easy to switch the > Python runtime between release and debug build. Not sure what you mean by "easy to switch the Python runtime". As soon as I want to use pip, I have to use a release build, right? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 24.04.2019 3:50, Ivan Pozdeev via Python-Dev wrote:
On 24.04.2019 2:44, Victor Stinner wrote:
Hi,
Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by
default in debug build", but I lost myself in details, I forgot the
main purpose of my proposal...
Let me retry from scratch with a more explicit title: I would like to
be able to run C extensions compiled in release mode on a Python
compiled in debug mode ("pydebug").
This is going to be impossible because debug Python links against debug C runtime which is binary incompatible with the release one (at
least, in Windows).
To elaborate:
As per
https://stackoverflow.com/questions/37541210/whats-the-difference-in-usage-between-shared-libraries-built-in-debug-and-relea/37580323#37580323 ,
Problems will occur if you have two modules that 1. use different versions or binary representations of a type and 2. exchange objects of
that type
Now, I trust Victor has ensured no discrepancies in explicitly exchanged types.
But I'm not sure if Python and the extension still rely on implicitly sharing some C runtime entities. (In Py2, that would at least be
descriptor table that MSVCRT maintains privately but Py3 doesn't rely on it AFAIK).
The use case is to debug bugs in C
extensions thanks to additional runtime checks of a Python debug
build, and more generally get a better debugging experiences on
Python. Even for pure Python, a debug build is useful (to get the
Pyhon traceback in gdb using "py-bt" command).
That said, debug vs release extension compilation is currently bugged. It's impossible to make a debug build of an extension against a
release Python (linked against release runtime, so not fully debug, just without optimizations) and vice versa. pip fails to build
extensions for a debug Python for the same reason. I've no idea how (and if at all) people manage to diagnose problems in extensions.
https://bugs.python.org/issue33637
Currently, using a Python compiled in debug mode means to have to
recompile C extensions in debug mode. Compile a C extension requires a
C compiler, header files, pull dependencies, etc. It can be very
complicated in practical (and pollute your system with all these
additional dependencies). On Linux, it's already hard, but on Windows
it can be even harder.
Just one concrete example: no debug build of numpy is provided at
https://pypi.org/project/numpy/ Good luck to build numpy in debug mode
manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.)
:-)
The above paragraph is probably the reason ;-)
--
The first requirement for the use case is that a Python debug build
supports the ABI of a release build. The current blocker issue is that
the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2
extra fields (_ob_prev and _ob_next) which change the offset of all
attributes of all objects and makes the ABI completely incompatible. I
propose to no longer imply Py_TRACE_REFS *by default* (but keep the
code):
https://bugs.python.org/issue36465
https://github.com/python/cpython/pull/12615
(Py_TRACE_REFS would be a different ABI.)
The second issue is that library filenames are different for a debug
build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build
should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then
also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not
possible: a debug build contains many additional functions missing
from a release build.
For Windows, maybe we should provide a Python compiled in debug mode
with the same C Runtime than a Python compiled in release mode.
Otherwise, the debug C Runtime is causing another ABI issue.
Maybe pip could be enhanced to support installing C extensions
compiled in release mode when using a debug mode. But that's more for
convenience, it's not really required, since it is easy to switch the
Python runtime between release and debug build.
Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in
structures. I know that the COUNT_ALLOCS define changes the ABI, but
it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I
propose to do the same for Py_TRACE_REFS ;-))
Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory
leaks, only sys.gettotalrefcount().
--
Python debug build has many benefit. If you ignore C extensions, the
debug build is usually compiled with compiler optimization disabled
which makes debugging in gdb a much better experience. If you never
tried: on a release build, most (if not all) variables are "" and it's really painful to basic debug functions like displaying
the current Python frame.
Assertions are removed in release modes, whereas they can detect a
wide range of bugs way earlier: integer overflow, buffer under- and
overflow, exceptions ignored silently, etc. Nobody likes to see a bug
for the first time in production. For example, I modified Python 3.8
to now logs I/O errors when a file is closed implicitly, but only in
debug or
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 2019-04-24 01:44, Victor Stinner wrote: I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode That seems like a very good idea. I would certainly use the debug mode while developing CPython or C extensions. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Le mer. 24 avr. 2019 à 01:44, Victor Stinner a écrit : > The current blocker issue is that > the Py_DEBUG define imply the Py_TRACE_REFS define (...): > > https://bugs.python.org/issue36465 > https://github.com/python/cpython/pull/12615 I updated my PR: """ Release build and debug build are now ABI compatible: the Py_DEBUG define no longer implies Py_TRACE_REFS define which introduces the only ABI incompatibility. A new "./configure --with-trace-refs" build option is now required to get Py_TRACE_REFS define which adds sys.getobjects() function and PYTHONDUMPREFS environment variable. Changes: * Add ./configure --with-trace-refs * Py_DEBUG no longer implies Py_TRACE_REFS * The "d" flag of SOABI (sys.implementation.cache_tag) is now only added by --with-trace-refs. It is no longer added by --with-pydebug. """ > Maybe pip could be enhanced to support installing C extensions > compiled in release mode when using a debug mode. In fact, pip doesn't have to be modified. I "fixed" sys.implementation.cache_tag by removing "d" in debug mode instead ;-) By the way, the "m" ABI flag for pymalloc is outdated. I proposed the following change to simply remove it: https://bugs.python.org/issue36707 https://github.com/python/cpython/pull/12931/files With my PR 12931 and my PR 12615, the only remaining ABI flag which be "d" which would only be enabled by ./configure --with-trace-refs, whereas ./configure --with-pydebug has no more effect on SOABI (sys.implementation.cache_tag). Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
On 24.04.2019 2:44, Victor Stinner wrote:
Hi,
Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by
default in debug build", but I lost myself in details, I forgot the
main purpose of my proposal...
Let me retry from scratch with a more explicit title: I would like to
be able to run C extensions compiled in release mode on a Python
compiled in debug mode ("pydebug").
This is going to be impossible because debug Python links against debug C runtime which is binary incompatible with the release one (at
least, in Windows).
The use case is to debug bugs in C
extensions thanks to additional runtime checks of a Python debug
build, and more generally get a better debugging experiences on
Python. Even for pure Python, a debug build is useful (to get the
Pyhon traceback in gdb using "py-bt" command).
That said, debug vs release extension compilation is currently bugged. It's impossible to make a debug build of an extension against a
release Python (linked against release runtime, so not fully debug, just without optimizations) and vice versa. pip fails to build
extensions for a debug Python for the same reason. I've no idea how (and if at all) people manage to diagnose problems in extensions.
https://bugs.python.org/issue33637
Currently, using a Python compiled in debug mode means to have to
recompile C extensions in debug mode. Compile a C extension requires a
C compiler, header files, pull dependencies, etc. It can be very
complicated in practical (and pollute your system with all these
additional dependencies). On Linux, it's already hard, but on Windows
it can be even harder.
Just one concrete example: no debug build of numpy is provided at
https://pypi.org/project/numpy/ Good luck to build numpy in debug mode
manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.)
:-)
The above paragraph is probably the reason ;-)
--
The first requirement for the use case is that a Python debug build
supports the ABI of a release build. The current blocker issue is that
the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2
extra fields (_ob_prev and _ob_next) which change the offset of all
attributes of all objects and makes the ABI completely incompatible. I
propose to no longer imply Py_TRACE_REFS *by default* (but keep the
code):
https://bugs.python.org/issue36465
https://github.com/python/cpython/pull/12615
(Py_TRACE_REFS would be a different ABI.)
The second issue is that library filenames are different for a debug
build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build
should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then
also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not
possible: a debug build contains many additional functions missing
from a release build.
For Windows, maybe we should provide a Python compiled in debug mode
with the same C Runtime than a Python compiled in release mode.
Otherwise, the debug C Runtime is causing another ABI issue.
Maybe pip could be enhanced to support installing C extensions
compiled in release mode when using a debug mode. But that's more for
convenience, it's not really required, since it is easy to switch the
Python runtime between release and debug build.
Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in
structures. I know that the COUNT_ALLOCS define changes the ABI, but
it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I
propose to do the same for Py_TRACE_REFS ;-))
Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory
leaks, only sys.gettotalrefcount().
--
Python debug build has many benefit. If you ignore C extensions, the
debug build is usually compiled with compiler optimization disabled
which makes debugging in gdb a much better experience. If you never
tried: on a release build, most (if not all) variables are "" and it's really painful to basic debug functions like displaying
the current Python frame.
Assertions are removed in release modes, whereas they can detect a
wide range of bugs way earlier: integer overflow, buffer under- and
overflow, exceptions ignored silently, etc. Nobody likes to see a bug
for the first time in production. For example, I modified Python 3.8
to now logs I/O errors when a file is closed implicitly, but only in
debug or development mode. In release Python silently ignored EBADF
error on such case, whereas it can lead to very nasty bugs causing
Python to call abort() (which creates a coredump on Linux): see
https://bugs.python.org/issue18748 ...
DeprecationWarning and ResourceWarning are shown by default in debug mode :-)
There are too many different additional checks done at runtime: I
cannot list them all here.
--
Being able to switch between Python in release mode and Python in
debug mode is a first step. My long term plan would be to better
separate "Python" from its "runtime". CPython in release mode would be
one runtime, CPython in debug mode would be another runtime, PyPy can
Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Le mer. 24 avr. 2019 à 01:44, Victor Stinner a écrit :
> The first requirement for the use case is that a Python debug build
> supports the ABI of a release build. (...) I
> propose to no longer imply Py_TRACE_REFS (...)
>
> Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in
> structures. (...)
I tested manually: just by disabling Py_TRACE_REFS, the release ABI
looks *fully* compatible with a debug build!
I modified Python 3.7 to disable Py_TRACE_REFS and to omit "d" from
SOABI when build in debug mode. I built Python in debug mode.
I ran tests on numpy and lxml.etree: I can use .so libraries from
/usr/lib64/python3.7/site-packages (compiled in release mode), it just
works! I was very surprised of not getting any crash on such non
trivial C extension, so I checked manually that I was running a debug
build: yes, sys.gettotalrefcount is present :-)
I also wanted to test an even more complex application: I installed
gajim, a Jabber client written in Python 3 with PyGTK. It uses many C
extensions. Running Gajim with my debug build is slower, well, that's
not a surprise, but it works well! (no crash)
About the SOABI, maybe we should only keep "d" when Py_TRACE_REFS is
used, since technically, the ABI is same between release and debug
mode without Py_TRACE_REFS. In that case, pip doesn't need to be
modified ;-)
If you also want to try, use:
PYTHONPATH=/usr/lib64/python3.7/site-packages:/usr/lib/python3.7/site-packages
./python /usr/bin/gajim
On a Python compiled with "./configure --with-pydebug && make" and the
following patch:
diff --git a/Include/object.h b/Include/object.h
index bcf78afe6b..4c807981c4 100644
--- a/Include/object.h
+++ b/Include/object.h
@@ -51,13 +51,8 @@ A standard interface exists for objects that
contain an array of items
whose size is determined when the object is allocated.
*/
-/* Py_DEBUG implies Py_TRACE_REFS. */
-#if defined(Py_DEBUG) && !defined(Py_TRACE_REFS)
-#define Py_TRACE_REFS
-#endif
-
-/* Py_TRACE_REFS implies Py_REF_DEBUG. */
-#if defined(Py_TRACE_REFS) && !defined(Py_REF_DEBUG)
+/* Py_DEBUG implies Py_REF_DEBUG. */
+#if defined(Py_DEBUG) && !defined(Py_REF_DEBUG)
#define Py_REF_DEBUG
#endif
diff --git a/configure b/configure
index 2db11e6e86..7271e9de40 100755
--- a/configure
+++ b/configure
@@ -6365,7 +6365,6 @@ $as_echo "#define Py_DEBUG 1" >>confdefs.h
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
$as_echo "yes" >&6; };
Py_DEBUG='true'
- ABIFLAGS="${ABIFLAGS}d"
else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
$as_echo "no" >&6; }; Py_DEBUG='false'
fi
diff --git a/configure.ac b/configure.ac
index e5fb7e7b0b..fa4bb1944f 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1246,7 +1246,6 @@ then
[Define if you want to build an interpreter with many run-time checks.])
AC_MSG_RESULT(yes);
Py_DEBUG='true'
- ABIFLAGS="${ABIFLAGS}d"
else AC_MSG_RESULT(no); Py_DEBUG='false'
fi],
[AC_MSG_RESULT(no)])
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode
Hi,
Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by
default in debug build", but I lost myself in details, I forgot the
main purpose of my proposal...
Let me retry from scratch with a more explicit title: I would like to
be able to run C extensions compiled in release mode on a Python
compiled in debug mode ("pydebug"). The use case is to debug bugs in C
extensions thanks to additional runtime checks of a Python debug
build, and more generally get a better debugging experiences on
Python. Even for pure Python, a debug build is useful (to get the
Pyhon traceback in gdb using "py-bt" command).
Currently, using a Python compiled in debug mode means to have to
recompile C extensions in debug mode. Compile a C extension requires a
C compiler, header files, pull dependencies, etc. It can be very
complicated in practical (and pollute your system with all these
additional dependencies). On Linux, it's already hard, but on Windows
it can be even harder.
Just one concrete example: no debug build of numpy is provided at
https://pypi.org/project/numpy/ Good luck to build numpy in debug mode
manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.)
:-)
--
The first requirement for the use case is that a Python debug build
supports the ABI of a release build. The current blocker issue is that
the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2
extra fields (_ob_prev and _ob_next) which change the offset of all
attributes of all objects and makes the ABI completely incompatible. I
propose to no longer imply Py_TRACE_REFS *by default* (but keep the
code):
https://bugs.python.org/issue36465
https://github.com/python/cpython/pull/12615
(Py_TRACE_REFS would be a different ABI.)
The second issue is that library filenames are different for a debug
build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build
should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then
also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not
possible: a debug build contains many additional functions missing
from a release build.
For Windows, maybe we should provide a Python compiled in debug mode
with the same C Runtime than a Python compiled in release mode.
Otherwise, the debug C Runtime is causing another ABI issue.
Maybe pip could be enhanced to support installing C extensions
compiled in release mode when using a debug mode. But that's more for
convenience, it's not really required, since it is easy to switch the
Python runtime between release and debug build.
Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in
structures. I know that the COUNT_ALLOCS define changes the ABI, but
it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I
propose to do the same for Py_TRACE_REFS ;-))
Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory
leaks, only sys.gettotalrefcount().
--
Python debug build has many benefit. If you ignore C extensions, the
debug build is usually compiled with compiler optimization disabled
which makes debugging in gdb a much better experience. If you never
tried: on a release build, most (if not all) variables are "" and it's really painful to basic debug functions like displaying
the current Python frame.
Assertions are removed in release modes, whereas they can detect a
wide range of bugs way earlier: integer overflow, buffer under- and
overflow, exceptions ignored silently, etc. Nobody likes to see a bug
for the first time in production. For example, I modified Python 3.8
to now logs I/O errors when a file is closed implicitly, but only in
debug or development mode. In release Python silently ignored EBADF
error on such case, whereas it can lead to very nasty bugs causing
Python to call abort() (which creates a coredump on Linux): see
https://bugs.python.org/issue18748 ...
DeprecationWarning and ResourceWarning are shown by default in debug mode :-)
There are too many different additional checks done at runtime: I
cannot list them all here.
--
Being able to switch between Python in release mode and Python in
debug mode is a first step. My long term plan would be to better
separate "Python" from its "runtime". CPython in release mode would be
one runtime, CPython in debug mode would be another runtime, PyPy can
seeen as another runtime, etc. The more general idea is: "compile your
C extension once and use any Python runtime".
https://pythoncapi.readthedocs.io/runtimes.html#runtimes
If you opt-in for the stable ABI, you can already switch between
runtimes of different Python versions (ex: Python 3.6 or Python 3.8).
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
