[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg382882

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[雅雯]

雅雯  added the comment:

幹你娘

--
nosy: +zhtw1234

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Erik Bray]

Erik Bray  added the comment:

Thanks everyone. And FWIW I agree the original change is positive overall, if a 
bit presumptuous about different linkers' behaviors :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

> FWIW I checked the Android build after those last changes.

Thank you! I was overconfident when I merged the Cygwin which looked good to 
me, whereas it broke Android :-( Hopefully it was quickly fixed. I hope that 
everything will be alright for the the beta1 to start from a good milestone!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Xavier de Gaye]

Xavier de Gaye  added the comment:

FWIW I checked the Android build after those last changes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b1fc41784136a2eb9737a7f0c821db52ab8d4dee by Victor Stinner (E. M. 
Bray) in branch 'master':
bpo-21536: Fix configure.ac for LIBPYTHON on Android/Cygwin (GH-13552)
https://github.com/python/cpython/commit/b1fc41784136a2eb9737a7f0c821db52ab8d4dee


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Erik Bray]

Change by Erik Bray :


--
pull_requests: +13463

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

I merged E. M. Bray's PR to get in Python 3.8 beta 1. If anything goes wrong, 
we can fix it later ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:


New changeset c994c8fc196a167c57c8850e8abdee170d366eec by Victor Stinner (E. M. 
Bray) in branch 'master':
bpo-21536: On Cygwin, C extensions must be linked with libpython (GH-13549)
https://github.com/python/cpython/commit/c994c8fc196a167c57c8850e8abdee170d366eec


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Erik Bray]

Change by Erik Bray :


--
pull_requests: +13461

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

> This completely breaks building extension modules on Windows-based platforms 
> like Cygwin and MinGW, where it is necessary when building even shared 
> libraries for all global symbols to resolvable at link time.

C extensions are always linked to libpython on Android. I'm open to do 
something similar for Cygwin and/or MinGW if you can come up with a PR :-) See 
LIBPYTHON variable in configure.ac.

You can use Xavier's change as an example: commit 
254b309c801f82509597e3d7d4be56885ef94c11, PR 12989.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Erik Bray]

Erik Bray  added the comment:

I vaguely recall seeing some discussion about this on python-dev or elsewhere 
and wish I had chimed in sooner, as I didn't realize action was going to be 
taken on this so soon.

This completely breaks building extension modules on Windows-based platforms 
like Cygwin and MinGW, where it is necessary when building even shared 
libraries for all global symbols to resolvable at link time.

I'm not actually sure this use case can even be achieved on these platforms.  I 
tried several hacks but nothing works.  I'll open a follow-up issue...

--
nosy: +erik.bray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 4ebcd7e2980d650c711d21b93447af39a8604554 by Victor Stinner in 
branch 'master':
bpo-21536: Update What's New in Python 3.8 entry (GH-13242)
https://github.com/python/cpython/commit/4ebcd7e2980d650c711d21b93447af39a8604554


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13152

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[reimar]

reimar  added the comment:

As explained in Issue34814, this change not only breaks RTLD_LOCAL of 
libpython, but it breaks it in fact system-wide.
It seems a bit much for Python to enforce a system-wide policy to not use 
RTLD_LOCAL just because of its own technical challenges, and thus I'd ask to 
reconsider the original change.

--
nosy: +reimar

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 254b309c801f82509597e3d7d4be56885ef94c11 by Victor Stinner 
(xdegaye) in branch 'master':
 bpo-21536: On Android, C extensions are linked to libpython (GH-12989)
https://github.com/python/cpython/commit/254b309c801f82509597e3d7d4be56885ef94c11


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Xavier de Gaye]

Xavier de Gaye  added the comment:

PR 12989 fixes the Android issue and has been checked on the Android emulator 
at API 24.

Python is cross-compiled with '--enable-shared' and the python-config scripts 
give the expected result:

generic_x86_64:/data/local/tmp/python/bin $ python -c "from sysconfig import 
get_config_var; print(get_config_var('Py_ENABLE_SHARED'))"   
1
generic_x86_64:/data/local/tmp/python/bin $ sh python-config --libs 
   
-lpython3.8d -ldl  -lm -lm 
generic_x86_64:/data/local/tmp/python/bin $ python python-config.py --libs  
   
-lpython3.8d -ldl -lm -lm

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Xavier de Gaye]

Change by Xavier de Gaye :


--
pull_requests: +12912
stage: resolved -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

I reopen the issue for the Android case.

I am fine with having a different linking policy per platform. For example, 
Windows has a different policy than Linux. We can continue to link C extensions 
to libpython on Android. The ability to load release C extension in debug 
Python is a feature more for debugging. I am not sure that Android is the best 
platform for debugging anyway. The debug can be done on a desktop Linux for 
example, no?

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

Xavier: would you be interested to work on a fix for Android?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Thanks for fixing the regression Victor.

Here is another potential problem.

On Android API 24 built with NDK r19, symbol resolution fails in the _socket 
shared library after changeset 8c3ecc6bacc8d0cd534f2b5b53ed962dd1368c7b when 
python is built with '--enable-shared':

  generic_x86_64:/data/local/tmp/python $ python -c "import _socket"
  Traceback (most recent call last):
File "", line 1, in 
  ImportError: dlopen failed: cannot locate symbol "PyByteArray_Type" 
referenced by 
"/data/local/tmp/python/lib/python3.8/lib-dynload/_socket.cpython-38d.so"...

This does not happen when the build is configured without '--enable-shared'.

An NDK issue [1] reports the same problem and dimitry in this issue explains 
that on Android >= 23 the shared library must be linked against the shared 
library where the symbol is defined. Dimitry says that the Android loader was 
fixed in M (i.e. API 23) and it must refer to the changes listed in the 
"RTLD_LOCAL (Available in API level >= 23)" section of "Android changes for NDK 
developers" [2].

[1] https://github.com/android-ndk/ndk/issues/201
[2] 
https://android.googlesource.com/platform/bionic/+/android-n-mr2-preview-1/android-changes-for-ndk-developers.md

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 01f073f5e43329547471c846c37e23157255e30a by Victor Stinner in 
branch 'master':
bpo-21536: Revert Makefile change on python-config (GH-12971)
https://github.com/python/cpython/commit/01f073f5e43329547471c846c37e23157255e30a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

> Please revert the change in Makefile.pre.in made by changeset 
> 8c3ecc6bacc8d0cd534f2b5b53ed962dd1368c7b as it breaks builds made out of the 
> source tree (OST). The error message is:
>
> make: *** No rule to make target 
> '/path_to_build_dir/Misc/python-config.sh', needed by 'python-config'.  Stop.
> (...)

Oh. I tried to "fix" the Makefile but it seems like I misunderstood how 
Misc/python-config.sh is handled. This file is generated from 
Misc/python-config.sh.in.

I wrote 12971 to revert the change but also add a comment to explain the 
"magic" behind this file.

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

I tested manually and I confirm that my latest change fix the compilation out 
of the source tree. I close again the issue. Sorry for the regression, it's now 
fixed.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12898

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Please revert the change in Makefile.pre.in made by changeset 
8c3ecc6bacc8d0cd534f2b5b53ed962dd1368c7b as it breaks builds made out of the 
source tree (OST). The error message is:

make: *** No rule to make target 
'/path_to_build_dir/Misc/python-config.sh', needed by 'python-config'.  Stop.

When the source tree has been already built at least once and 'make clean' is 
run in the source tree (as required for building OST), the OST build does not 
fail but incorrectly uses the stale python-config.sh from the source tree as 
'make clean' does not remove Misc/python-config.sh.

--
nosy: +xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

Since this change is causing subtle issues like bpo-36721 "Add pkg-config 
python-3.8-embed", I don't think that it would be a good idea to backport this 
change to Python 2.7 or 3.7. I close the issue.

See also bpo-36722 "In debug build, load also C extensions compiled in release 
mode or compiled using the stable ABI".

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

> If an application is linked to libpython to embed Python, the application 
> build requires flags to link to libpython. (...)

I created bpo-36721 "Add pkg-config python-3.8-embed" to discuss this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8c3ecc6bacc8d0cd534f2b5b53ed962dd1368c7b by Victor Stinner in 
branch 'master':
bpo-21536: C extensions are no longer linked to libpython (GH-12946)
https://github.com/python/cpython/commit/8c3ecc6bacc8d0cd534f2b5b53ed962dd1368c7b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

If an application is linked to libpython to embed Python, the application build 
requires flags to link to libpython. Currently, there are:

$ python3-config --libs
-lpython3.7m -lcrypt -lpthread -ldl  -lutil -lm 

$ pkg-config --libs python-3.7 
-lpython3.7m 

My PR doens't change Misc/python.pc.

Discussion on python-dev:

* https://mail.python.org/pipermail/python-dev/2019-April/157196.html
* https://mail.python.org/pipermail/python-dev/2019-April/157201.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

With an additonal change on SOABI (I will open a separated issue for that), PR 
12946 allows to load lxml built in release mode in a Python built in debug 
mode! That's *very* useful for debugging: see my gdb example below.

---

I just modified the ABI of debug build so release and debug build now have the 
same ABI: bpo-36465.

I wrote a patch to use the same sys.implementation.cache_tag (SOABI) in release 
and debug mode:

diff --git a/configure b/configure
index b02d17c053..38eb7f1bd6 100755
--- a/configure
+++ b/configure
@@ -6325,7 +6325,6 @@ $as_echo "#define Py_DEBUG 1" >>confdefs.h
   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
 $as_echo "yes" >&6; };
   Py_DEBUG='true'
-  ABIFLAGS="${ABIFLAGS}d"
 else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
 $as_echo "no" >&6; }; Py_DEBUG='false'
 fi
diff --git a/configure.ac b/configure.ac
index 65d3f8e691..1b2cd3076c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1223,7 +1223,6 @@ then
   [Define if you want to build an interpreter with many run-time checks.])
   AC_MSG_RESULT(yes);
   Py_DEBUG='true'
-  ABIFLAGS="${ABIFLAGS}d"
 else AC_MSG_RESULT(no); Py_DEBUG='false'
 fi],
 [AC_MSG_RESULT(no)])


(That's a temporary patch, I will design a better solution later.)

Using this patch + PR 12946, it becomes possible to load a C extension compiled 
in release mode in a debug Python!


---

Example building lxml in release mode and then load it in debug mode.


Install Python in *release* mode into /opt/py38release (shared libpython):

git clean -fdx; ./configure --enable-shared --prefix /opt/py38release && make 
&& make install


Install Python in *debug* mode into /opt/py38debug (shared libpython):

git clean -fdx; ./configure CFLAGS="-O0" --enable-shared --prefix 
/opt/py38debug --with-pydebug && make && make install


Build lxml in release mode:

LD_LIBRARY_PATH=/opt/py38release/lib/ /opt/py38release/bin/python3.8 -m pip 
install lxml


By default, the debug Python doesn't have lxml:

$ LD_LIBRARY_PATH=/opt/py38debug/lib/ /opt/py38debug/bin/python3.8  -c 'import 
lxml'
Traceback (most recent call last):
  File "", line 1, in 
ModuleNotFoundError: No module named 'lxml'


Give access to C extensions compiled in release mode to the debug Python:

$ LD_LIBRARY_PATH=/opt/py38debug/lib/ 
PYTHONPATH=/opt/py38release/lib/python3.8/site-packages 
/opt/py38debug/bin/python3.8 
Python 3.8.0a3+ (heads/omit_libpython-dirty:8a03782387, Apr 25 2019, 02:52:01) 
>>> import lxml
>>> import lxml.etree
>>> lxml.etree

>>> import sys
>>> sys.gettotalrefcount()
108304

It works!

Explanation:

* This is a debug build: sys.gettotalrefcount() is present (and works)
* lxml has been compiled in release mode
* etree.cpython-38-x86_64-linux-gnu.so is not linked to libpython and so 
doesn't rely on the *release* /opt/py38release/lib/libpython3.8.so
* this lxml can be loaded in a Python compiled in debug mode!

---

Now let's have a look at the gdb experience of release vs debug Python build.

I put a breakpoint on lxml.etree.iterparse('example.xml'): the C function is 
called __pyx_tp_new_4lxml_5etree_iterparse.


Using the release build, gdb fails to read many C local variables:

$ LD_LIBRARY_PATH=/opt/py38release/lib/ gdb -args 
/opt/py38release/bin/python3.8 parse.py 

(gdb) source /home/vstinner/prog/python/master/python-gdb.py 
(gdb) b __pyx_tp_new_4lxml_5etree_iterparse
Function "__pyx_tp_new_4lxml_5etree_iterparse" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
(gdb) run

Breakpoint 1, __pyx_tp_new_4lxml_5etree_iterparse (t=0x7fffea724900 
<__pyx_type_4lxml_5etree_iterparse>, a=('example.xml',), k=0x0) at 
src/lxml/etree.c:218930
218930  src/lxml/etree.c: No such file or directory.

(gdb) py-bt
Traceback (most recent call first):
  File "parse.py", line 4, in func
context = etree.iterparse('example.xml')
  File "parse.py", line 12, in 
func("arg")

(gdb) frame 4
#4  _PyEval_EvalFrameDefault (f=, throwflag=) at 
Python/ceval.c:3268
3268res = call_function(&sp, oparg, NULL);
(gdb) p f
$1 = 
(gdb) p co
$2 = 
(gdb) p sp
$3 = 
(gdb) p oparg
$4 = 

The basic function "py-bt" works as expected, but inspecting Python internals 
doesn't work: most local C variables are "optimized out" :-(



New attempt using a debug build:

$ LD_LIBRARY_PATH=/opt/py38debug/lib/ 
PYTHONPATH=/opt/py38release/lib/python3.8/site-packages gdb -args 
/opt/py38debug/bin/python3.8 parse.py 

... same commands to load python-gdb.py and put a breakpoint ...

Breakpoint 1, __pyx_tp_new_4lxml_5etree_iterparse (t=0x7fffea606900 
<__pyx_type_4lxml_5etree_iterparse>, a=('example.xml',), k=0x0) at 
src/lxml/etree.c:218930
218930  src/lxml/etree.c: No such file or directory.

(gdb) py-bt
Traceback (most recent call first):
  File "parse.py", line 4, in func
context = etree.iterparse('example.xml')
  File "parse.py", line 12, in 
func("arg")

(gdb) frame 4
#4  0x77d6e4f8 in _PyE

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

I wrote PR 12946: "On Unix, C extensions are no longer linked to libpython".

Using PR 12946, the use case described in the initial message (msg218806) now 
works as expected. I can load a C extension built by a shared library Python 
with a statically linked Python:

$ LD_LIBRARY_PATH=/opt/py38shared/lib /opt/py38notshared/bin/python3.8
Python 3.8.0a3+ (heads/master:be0099719c, Apr 25 2019, 02:10:57) 
>>> import sys; sys.path.insert(0, '/opt/py38shared/lib/python3.8/lib-dynload')
>>> import _ssl
>>> _ssl


/opt/py38notshared/bin/python3.8 is statically linked, whereas 
/opt/py38shared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so 
comes from a shared library Python.

Install shared libray Python into /opt/py38shared:

git clean -fdx; ./configure CFLAGS="-O0" --enable-shared --prefix 
/opt/py38shared && make && make install

Install statically linked Python into /opt/py38notshared:

git clean -fdx; ./configure CFLAGS="-O0" --prefix /opt/py38notshared && make && 
make install

--

As Antoine said, the opposite already works. Just in case, I also tested and I 
confirm that it still works:

$ LD_LIBRARY_PATH=/opt/py38shared/lib /opt/py38shared/bin/python3.8
Python 3.8.0a3+ (heads/master:be0099719c, Apr 25 2019, 02:09:02) 
>>> import sys; sys.path.insert(0, 
>>> '/opt/py38notshared/lib/python3.8/lib-dynload')
>>> import _ssl
>>> _ssl


_ssl comes from statically linked Python (/opt/py38notshared) and is loaded in 
shared library Python (/opt/py38shared).

--
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

> bpo-34814 is linked to this use case: 
> https://bugzilla.redhat.com/show_bug.cgi?id=1585201 is an example of Python 
> embedded in C using dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW). 
> Problem: some C extensions of the standard library cannot be loaded in this 
> case, like _struct.

bpo-1429775 is another example of RTLD_LOCAL usage, see:
https://mail.python.org/pipermail/python-dev/2006-February/060533.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +12870
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

Antoine:
> Hmm, apparently the -l flag was added in #832799, for a rather complicated 
> case where the interpreter is linked with a library dlopened by an embedding 
> application (I suppose for some kind of plugin system). The OP there also 
> mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving 
> the desired effect).
> (also, the OP didn't mention why he used a shared library build, instead of 
> linking Python statically with the dlopened library)

bpo-34814 is linked to this use case: 
https://bugzilla.redhat.com/show_bug.cgi?id=1585201 is an example of Python 
embedded in C using dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW). 
Problem: some C extensions of the standard library cannot be loaded in this 
case, like _struct.

On Fedora and RHEL, some C extensions like _struct are built by the "*shared*" 
section of Modules/Setup. In this case, these C extensions are not explicitly 
linked to libpython.

IHMO it's a bad usage of dlopen(): libpython must always be loaded with 
RTLD_GLOBAL.

bpo-832799 has been fixed by the following commit which modify distutils to 
link C extensions to libpython:

commit 10acfd00b28a2aad7b73d35afdbc64b0baebea20
Author: Martin v. Löwis 
Date:   Mon Apr 10 12:39:36 2006 +

Patch #1429775: Link Python modules to libpython on linux if
--enable-shared. Fixes #832799.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-34814.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

Change by Antoine Pitrou :


--
versions: +Python 3.7, Python 3.8 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Piotr Dobrogost]

Changes by Piotr Dobrogost :


--
nosy: +piotr.dobrogost

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

Antoine Pitrou added the comment:

Martin, what do you think about the aforementioned use case?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

Antoine Pitrou added the comment:

Hmm, apparently the -l flag was added in #832799, for a rather complicated case 
where the interpreter is linked with a library dlopened by an embedding 
application (I suppose for some kind of plugin system). The OP there also 
mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving the 
desired effect).

(also, the OP didn't mention why he used a shared library build, instead of 
linking Python statically with the dlopened library)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

Antoine Pitrou added the comment:

> PHP might be close to our case: Debian includes a libphp5.so
> (in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so 
> link against it, and all the PHP modules (in 
> /usr/lib/php5/20100525+lfs/) don't link with the shared library -
> on Debian. I wonder how it is on systems that actually use the PHP 
> shared library.

Regardless of libphp5.so, the modules in /usr/lib/php5/20100525 don't link 
against /usr/lib/apache2/modules/libphp5.so, though the latter is obviously 
able to load those modules.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Martin v . Löwis]

Martin v. Löwis added the comment:

I think you are right. It would IMO be useful to research a few comparable 
systems. E.g. Apache modules don't link a shared library, but still refer to 
apr_ functions as undefined symbols - but then, there isn't an APR shared 
library in the first place (at least not on Debian - how about Redhat?)

PHP might be close to our case: Debian includes a libphp5.so (in 
/usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so link 
against it, and all the PHP modules (in /usr/lib/php5/20100525+lfs/) don't link 
with the shared library - on Debian. I wonder how it is on systems that 
actually use the PHP shared library.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

Antoine Pitrou added the comment:

Actually, it's not a -L flag but a -l flag. Removing the "-lpython3.5m" flag 
from the linker line works fine under Linux, and allows the resulting extension 
to be loaded with both a shared libary Python and a statically-linked Python.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21536] extension built with a shared python cannot be loaded with a static python

[Antoine Pitrou]

New submission from Antoine Pitrou:

When a C extension is built (using distutils) with a shared library Python, it 
cannot be loaded with an otherwise identical statically linked Python. The 
other way round works fine. Trivial example using the _ssl module:

>>> import sys
>>> sys.path.insert(0, 
>>> '/home/antoine/cpython/shared/build/lib.linux-x86_64-3.5/')
>>> import _ssl
Traceback (most recent call last):
  File "", line 1, in 
ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file 
or directory

This is probably because of an additional -L flag that is passed when linking a 
C extension with a shared library Python. I don't think the flag is useful 
under Linux (or perhaps under any other OS at all), since the relevant symbols 
are already loaded when the interpreter tries to load the C extension.

(AFAIK, systems notorious for providing shared library Pythons are RedHat-alike 
systems, while Debian/Ubuntu provide statically linked Pythons)

--
components: Library (Lib)
messages: 218806
nosy: dstufft, eric.araujo, loewis, ncoghlan, pitrou
priority: normal
severity: normal
status: open
title: extension built with a shared python cannot be loaded with a static 
python
type: enhancement
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com