Re: Finding .so files without setting LD_LIBRARY_PATH

[Lawrence D’Oliveiro]

On Thursday, May 12, 2016 at 9:51:02 AM UTC+12, Paul Smith wrote:
> ... here's the problem: because LD_LIBRARY_PATH is in
> Python's environment it is also passed down to programs invoked by
> Python.  That means if I (for example) invoke subprocess.call(['ssh',
> ...]) then it fails because the system ssh is looking for the system
> libcrypto.so, and when it finds the Python libcrypto.so instead
> (because of LD_LIBRARY_PATH) it fails.

That’s easy enough to fix: you can pass a custom environment down to 
subprocesses by using the “env” argument to the various subprocess calls 
.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Finding .so files without setting LD_LIBRARY_PATH

[Paul Smith]

On Thu, 2016-05-12 at 07:55 +0300, Jussi Piitulainen wrote:
> eryk sun writes:
> 
> > On Wed, May 11, 2016 at 10:39 PM, Paul Smith wrote:
> > > Hi all.  I have a locally-built version of Python (2.7.11) that I'm
> > > copying around to different systems, running all different versions of
> > > GNU/Linux.
> > ...
> > > What I'd like to do is have a way of setting the library path that
> > > Python uses when it tries to load .so files (for example in the ssl
> > > module which loads lib-dynload/_ssl.so which links to libssl.so and
> > > libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that
> > > Python passes to its children.
> > 
> > An ELF header can contain either an RPATH or a RUNPATH to extend the
> > library search path at load time.

Yes, I'm familiar with rpath.  However I don't know how to set it for
Python .so's that appear in the lib-dynload directory, including site
-packages, which is where it needs to be ... I tried to go through the
Modules/Setup etc. but to no avail :(.

> There's a tool (GPLv3+) called patchelf that can set these in an ELF
> binary:

Aha!  That's excellent!  The only one I was aware of was chrpath which
doesn't allow you to add an rpath if one doesn't exist, or add an rpath
which is longer than the one that already exists.

This will make things much simpler.

Cheers!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Finding .so files without setting LD_LIBRARY_PATH

[dieter]

Paul Smith  writes:
> ...
> That works fine, but here's the problem: because LD_LIBRARY_PATH is in
> Python's environment it is also passed down to programs invoked by
> Python.  That means if I (for example) invoke subprocess.call(['ssh',
> ...]) then it fails because the system ssh is looking for the system
> libcrypto.so, and when it finds the Python libcrypto.so instead
> (because of LD_LIBRARY_PATH) it fails.

I have had a similar problem - however with "PYTHONPATH" instead
of "LD_LIBRARY_PATH". I solved it by removing "PYTHONPATH" from
"os.environ" during Python startup. This way, the envvar
"PYTHONPATH" was effective to set up "sys.path" but
processes started from this python would not see it.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Finding .so files without setting LD_LIBRARY_PATH

[Jussi Piitulainen]

eryk sun writes:

> On Wed, May 11, 2016 at 10:39 PM, Paul Smith wrote:
>> Hi all.  I have a locally-built version of Python (2.7.11) that I'm
>> copying around to different systems, running all different versions of
>> GNU/Linux.
> ...
>> What I'd like to do is have a way of setting the library path that
>> Python uses when it tries to load .so files (for example in the ssl
>> module which loads lib-dynload/_ssl.so which links to libssl.so and
>> libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that
>> Python passes to its children.
>
> An ELF header can contain either an RPATH or a RUNPATH to extend the
> library search path at load time. RPATH is searched before
> LD_LIBRARY_PATH, while RUNPATH is searched after LD_LIBRARY_PATH. Note
> that the Linux loader only uses an embedded RUNPATH when searching for
> the immediate dependencies of an object. Use an RPATH instead if you
> need to extend the path that dlopen searches at runtime (e.g. for
> loading shared libraries via ctypes).
>
> Paths can be defined relative to the object using ${ORIGIN}. For
> example, the following configures a Python build to add a relative
> RPATH:
>
> $ ./configure LDFLAGS='-Wl,-rpath,\$${ORIGIN}/lib'
> $ make
>
> Or a RUNPATH:
>
> $ ./configure LDFLAGS='-Wl,-rpath,\$${ORIGIN}/lib,--enable-new-dtags'
> $ make
>
> Here's the resulting RPATH in the ELF header:
>
> $ readelf -d python | grep RPATH
>  0x000f (RPATH)  Library rpath: [${ORIGIN}/lib]

There's a tool (GPLv3+) called patchelf that can set these in an ELF
binary:

https://nixos.org/patchelf.html
https://github.com/NixOS/patchelf
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Finding .so files without setting LD_LIBRARY_PATH

[eryk sun]

On Wed, May 11, 2016 at 10:39 PM, Paul Smith  wrote:
> Hi all.  I have a locally-built version of Python (2.7.11) that I'm
> copying around to different systems, running all different versions of
> GNU/Linux.
...
> What I'd like to do is have a way of setting the library path that
> Python uses when it tries to load .so files (for example in the ssl
> module which loads lib-dynload/_ssl.so which links to libssl.so and
> libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that
> Python passes to its children.

An ELF header can contain either an RPATH or a RUNPATH to extend the
library search path at load time. RPATH is searched before
LD_LIBRARY_PATH, while RUNPATH is searched after LD_LIBRARY_PATH. Note
that the Linux loader only uses an embedded RUNPATH when searching for
the immediate dependencies of an object. Use an RPATH instead if you
need to extend the path that dlopen searches at runtime (e.g. for
loading shared libraries via ctypes).

Paths can be defined relative to the object using ${ORIGIN}. For
example, the following configures a Python build to add a relative
RPATH:

$ ./configure LDFLAGS='-Wl,-rpath,\$${ORIGIN}/lib'
$ make

Or a RUNPATH:

$ ./configure LDFLAGS='-Wl,-rpath,\$${ORIGIN}/lib,--enable-new-dtags'
$ make

Here's the resulting RPATH in the ELF header:

$ readelf -d python | grep RPATH
 0x000f (RPATH)  Library rpath: [${ORIGIN}/lib]
-- 
https://mail.python.org/mailman/listinfo/python-list

Finding .so files without setting LD_LIBRARY_PATH

[Paul Smith]

Hi all.  I have a locally-built version of Python (2.7.11) that I'm
copying around to different systems, running all different versions of
GNU/Linux.  Because I need this to work across systems I'm bundling
important .so's with my Python installation (libcrypto, libssl,
libreadline, libgmp) which are dynamically loaded when I do things like
"import ssl" etc.

I have a wrapper script around this Python that everyone runs, rather
than invoking it directly, and it is able to set some environment
variables etc.  So, I had been setting LD_LIBRARY_PATH before I invoke
Python so it can find the "right" versions of libcrypto.so etc.

That works fine, but here's the problem: because LD_LIBRARY_PATH is in
Python's environment it is also passed down to programs invoked by
Python.  That means if I (for example) invoke subprocess.call(['ssh',
...]) then it fails because the system ssh is looking for the system
libcrypto.so, and when it finds the Python libcrypto.so instead
(because of LD_LIBRARY_PATH) it fails.

What I'd like to do is have a way of setting the library path that
Python uses when it tries to load .so files (for example in the ssl
module which loads lib-dynload/_ssl.so which links to libssl.so and
libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that
Python passes to its children.

Is there any way to do this?

It seems like there must be some way to do it, because if after I start
python with LD_LIBRARY_PATH set, then I delete it from os.environ, then
I import ssl it works, so it's still set in Python's environment.  But
I don't want to have to modify all my scripts to unset LD_LIBRARY_PATH,
plus that's not the right thing to do in case someone needs to set
LD_LIBRARY_PATH for themselves.

What I've tried so far is writing a little Python script that unsets
LD_LIBRARY_PATH and setting PYTHONSETUP to the name of that script in
my wrapper, which does seem to work, but that takes away the ability
for users to use PYTHONSETUP themselves which is sub-optimal.

Does anyone have any other ideas?
-- 
https://mail.python.org/mailman/listinfo/python-list