bug#64309: Python dlopen()s musl libc
I've had experiences now with multiple Guix packages, including gajim (bug 60235) and now python-neovim-remote, which have an issue where Python tries to dlopen() libc, but finds the system libc instead of Guix's, resulting on Alpine Linux hosts in a crash with this message: ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory There are a variety of tracebacks that lead up to this, depending on the package in question. pgpud8qmC_AkJ.pgp Description: OpenPGP digital signature
bug#64309: Python dlopen()s musl libc
Hi, Athena Martin skribis: > I've had experiences now with multiple Guix packages, including gajim > (bug 60235) and now python-neovim-remote, which have an issue where > Python tries to dlopen() libc, but finds the system libc instead of > Guix's, resulting on Alpine Linux hosts in a crash with this message: > > ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such > file or directory > > There are a variety of tracebacks that lead up to this, depending on > the package in question. Could you provide a command to reproduce this? Thanks in advance, Ludo’.
bug#64309: Python dlopen()s musl libc
> > ImportError: libc.musl-x86_64.so.1: cannot open shared object file: > > No such file or directory > > Could you provide a command to reproduce this? On an Alpine Linux Edge host, I reproduce with: $ guix install python-neovim-remote $ nvr It's possible that there's some quirk of my specific environment. (I'm currently in the process of switching to Guix System so I may not be able to reproduce this forever.) pgpw5r_ldMtq8.pgp Description: OpenPGP digital signature
bug#64309: Python dlopen()s musl libc
Hi Athena, Athena Martin via Bug reports for GNU Guix writes: > On an Alpine Linux Edge host, I reproduce with: > > $ guix install python-neovim-remote > $ nvr > > It's possible that there's some quirk of my specific environment. > > (I'm currently in the process of switching to Guix System so I may not > be able to reproduce this forever.) Can you do `LD_DEBUG=libs nvr` so that we get a log of what ld's trying to load? Best, -- Josselin Poiret signature.asc Description: PGP signature
bug#64309: Python dlopen()s musl libc
> Can you do `LD_DEBUG=libs nvr` so that we get a log of what ld's > trying to load? I've attached the full log. The first mention of musl is on line 200 and everything seems to happen pretty fast, here's the most relevant portion: 16405: find library=libc.musl-x86_64.so.1 [0]; searching 16405: search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache 16405: search path=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib (system search path) 16405: trying file=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.musl-x86_64.so.1 16405: Traceback (most recent call last): File "/gnu/store/dsgxdqs620pp284bfm1drbsjqpb36i4n-python-neovim-remote-2.5.1/bin/.nvr-real", line 4, in import nvr.nvr as mod File "/home/alm/.local/lib/python3.10/site-packages/nvr/__init__.py", line 1, in from .nvr import main File "/home/alm/.local/lib/python3.10/site-packages/nvr/nvr.py", line 34, in import psutil File "/home/alm/.local/lib/python3.10/site-packages/psutil/__init__.py", line 102, in from . import _pslinux as _psplatform File "/home/alm/.local/lib/python3.10/site-packages/psutil/_pslinux.py", line 26, in from . import _psutil_linux as cext ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory After that it starts calling fini()s and terminates. 16405: find library=libc.so.6 [0]; searching 16405: search cache=/gnu/store/rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16/etc/ld.so.cache 16405: trying file=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6 16405: 16405: 16405: calling init: /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/ld-linux-x86-64.so.2 16405: 16405: 16405: calling init: /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6 16405: 16405: 16405: initialize program: /gnu/store/rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16/bin/bash 16405: 16405: 16405: transferring control: /gnu/store/rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16/bin/bash 16405: 16405: find library=libpython3.10.so.1.0 [0]; searching 16405: search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache 16405: trying file=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/lib/libpython3.10.so.1.0 16405: 16405: find library=libcrypt.so.1 [0]; searching 16405: search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache 16405: trying file=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libcrypt.so.1 16405: 16405: find library=libm.so.6 [0]; searching 16405: search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache 16405: trying file=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libm.so.6 16405: 16405: find library=libexpat.so.1 [0]; searching 16405: search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache 16405: trying file=/gnu/store/fw1wywd34vh33l4dq182ds5d7jdz45j5-expat-2.5.0/lib/libexpat.so.1 16405: 16405: find library=libgcc_s.so.1 [0]; searching 16405: search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache 16405: trying file=/gnu/store/930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib/lib/libgcc_s.so.1 16405: 16405: find library=libc.so.6 [0]; searching 16405: search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache 16405: trying file=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6 16405: 16405: 16405: calling init: /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/ld-linux-x86-64.so.2 16405: 16405: 16405: calling init: /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6 16405: 16405: 16405: calling init: /gnu/store/930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib/lib/libgcc_s.so.1 16405: 16405: 16405: calling init: /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libm.so.6 16405: 16405: 16405: calling init: /gnu/store/fw1wywd34vh33l4dq182ds5d7jdz45j5-expat-2.5.0/lib/libexpat.so.1 16405: 16405: 16405: calling init: /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libcrypt.so.1 16405: 16405: 16405: calling init: /gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/lib/libpython3.10.so.1.0 16405: 16405:
bug#64309: Python dlopen()s musl libc
Hi Athena, So it's not the LD_DEBUG output that hold a clue, but rather the Python traceback. Athena Martin writes: > File > "/gnu/store/dsgxdqs620pp284bfm1drbsjqpb36i4n-python-neovim-remote-2.5.1/bin/.nvr-real", > line 4, in > import nvr.nvr as mod > File "/home/alm/.local/lib/python3.10/site-packages/nvr/__init__.py", line > 1, in > from .nvr import main > File "/home/alm/.local/lib/python3.10/site-packages/nvr/nvr.py", line 34, > in > import psutil > File "/home/alm/.local/lib/python3.10/site-packages/psutil/__init__.py", > line 102, in > from . import _pslinux as _psplatform > File "/home/alm/.local/lib/python3.10/site-packages/psutil/_pslinux.py", > line 26, in > from . import _psutil_linux as cext > ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such > file or directory The nvr package in ~/.local seems to be used instead of a Guix package. That locally installed nvr package expects to use the host's libc, but since the python interpreter being used has a fixed RPATH and system search path it won't find it. .nvr-real should definitely be using the Python code inside the store, I wonder why that isn't being done. Maybe our sitecustomize.py is misbehaving? Can you do `guix shell python-neovim-remote python -- python3` then type `import sys.path; sys.path`? Best, -- Josselin Poiret signature.asc Description: PGP signature
bug#64309: Python dlopen()s musl libc
> So it's not the LD_DEBUG output that hold a clue, but rather the
> Python traceback.
> The nvr package in ~/.local seems to be used instead of a Guix
> package. That locally installed nvr package expects to use the host's
> libc, but since the python interpreter being used has a fixed RPATH
> and system search path it won't find it.
Ah, I see.
> .nvr-real should definitely be using the Python code inside the
> store, I wonder why that isn't being done. Maybe our
> sitecustomize.py is misbehaving? Can you do `guix shell
> python-neovim-remote python -- python3` then type `import sys.path;
> sys.path`?
$ guix shell python-neovim-remote python -- python3
>>> sys.path
['',
'/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip',
'/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10',
'/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload',
'/home/alm/.local/lib/python3.10/site-packages',
'/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages',
'/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
(It's not from the environment:)
$ guix shell python-neovim-remote python --pure -- python3
>>> sys.path
['',
'/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip',
'/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10',
'/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload',
'/home/alm/.local/lib/python3.10/site-packages',
'/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages',
'/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>>> os.environ
environ({'HOME': '/home/alm', 'LOGNAME': 'alm', 'PAGER': 'less', 'DISPLAY':
':0', 'USER': 'alm', 'TERM': 'xterm-256color', 'PATH':
'/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/bin', 'GUIX_PYTHONPATH':
'/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages',
'GUIX_ENVIRONMENT': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile'})
pgpM2JLV3bskc.pgp
Description: OpenPGP digital signature
bug#64309: Python dlopen()s musl libc
> The nvr package in ~/.local seems to be used instead of a Guix > package. That locally installed nvr package expects to use the > host's libc, but since the python interpreter being used has a > fixed RPATH and system search path it won't find it. I've just checked and temporarily removing .local/lib/python3.10 makes Guix's nvr work, and takes the .local site-packages off sys.path. pgpz6JI82iCYQ.pgp Description: OpenPGP digital signature
bug#64309: Python dlopen()s musl libc
Hi Athena,
Athena Martin writes:
> $ guix shell python-neovim-remote python -- python3
sys.path
> ['',
> '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip',
> '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10',
> '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload',
> '/home/alm/.local/lib/python3.10/site-packages',
> '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages',
>
> '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>
> (It's not from the environment:)
>
> $ guix shell python-neovim-remote python --pure -- python3
sys.path
> ['',
> '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip',
> '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10',
> '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload',
> '/home/alm/.local/lib/python3.10/site-packages',
> '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages',
>
> '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
os.environ
> environ({'HOME': '/home/alm', 'LOGNAME': 'alm', 'PAGER': 'less', 'DISPLAY':
> ':0', 'USER': 'alm', 'TERM': 'xterm-256color', 'PATH':
> '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/bin', 'GUIX_PYTHONPATH':
> '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages',
> 'GUIX_ENVIRONMENT': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile'})
So, looks like the default Python behavior is to load the usercustomize
after the sitecustomize [1], which leads to exactly the behavior you're
experiencing. I don't know what we should do here, maybe pass `-s` to
the shebang line of Python to disable loading the usercustomize? That
would probably be a world-rebuild though. CC'ing the Python team to see
what they think.
[1] https://docs.python.org/3/library/site.html
Best,
--
Josselin Poiret
signature.asc
Description: PGP signature
bug#64309: Python dlopen()s musl libc
Hi,
there’s a similar issue #63912 and a thread on the guix-devel
mailinglist at
https://lists.gnu.org/archive/html/guix-devel/2023-06/msg00066.html
> So, looks like the default Python behavior is to load the usercustomize
> after the sitecustomize [1], which leads to exactly the behavior you're
> experiencing. I don't know what we should do here, maybe pass `-s` to
> the shebang line of Python to disable loading the usercustomize? That
> would probably be a world-rebuild though. CC'ing the Python team to see
> what they think.
I think the problem is bigger than usercustomize. Any custom PYTHONPATH
also slips through and causes this issue, as well as any custom
GUIX_PYTHONPATH, because the executable wrapper appends it (think nested
`guix shell` invokations with different versions of a library for
an example where this could go wrong).
Guix-managed Python packages (libraries nor applications) should
generally not pick up dependencies from random paths – only those
from their package description, so we can keep Guix’ promise of being
self-contained.
I have experimented with customizing Python’s importing mechanism
through a custom MetaPathFinder. It works by adding a __guix_pythonpath__
variable to every Python package’s __init__.py file and modifying the
module loader’s search path accordingly if such a variable exists. It
would provide exactly that guarantee, but it’s just a PoC at this
point – see attached file.
Apart from that I don’t see a good short-term solution right now. It’s
just how Python works.
Cheers,
Lars
# Credits to
#
https://tenthousandmeters.com/blog/python-behind-the-scenes-11-how-the-python-import-system-works/
# for the very good explanation of how Python’s import statement works.
import sys, os
from importlib.abc import MetaPathFinder
from importlib.machinery import SourceFileLoader, ModuleSpec, PathFinder
class GuixPythonFinder (MetaPathFinder):
def find_spec (self, fullname, path, target=None):
# Short-circuit for non-top-level imports, which already have a path.
if path:
return None
attrname = '__guix_pythonpath__'
searchPath = None
# Search for our caller.
frame = sys._getframe ()
while frame:
# If he has a search path, use it. This is mainly for executable
# scripts with `__name__ == '__main__'`.
searchPath = frame.f_globals.get (attrname, None)
if not searchPath:
# Otherwise check the top-level package for search paths
# declared in __init__.py
package = frame.f_globals.get ('__package__')
if package:
module = sys.modules.get (package)
if module:
searchPath = getattr (module, attrname, None)
if searchPath is not None:
break
frame = frame.f_back
# If we have a caller…
if searchPath is not None:
return PathFinder.find_spec (fullname, searchPath, target=target)
else:
# Otherwise we’re not responsible for this module.
return None
sys.meta_path.insert (0, GuixPythonFinder ())
bug#64309: Python dlopen()s musl libc
Hi Lars, Lars-Dominik Braun writes: > I think the problem is bigger than usercustomize. Any custom PYTHONPATH > also slips through and causes this issue, as well as any custom > GUIX_PYTHONPATH, because the executable wrapper appends it (think nested > `guix shell` invokations with different versions of a library for > an example where this could go wrong). > > Guix-managed Python packages (libraries nor applications) should > generally not pick up dependencies from random paths – only those > from their package description, so we can keep Guix’ promise of being > self-contained. > > I have experimented with customizing Python’s importing mechanism > through a custom MetaPathFinder. It works by adding a __guix_pythonpath__ > variable to every Python package’s __init__.py file and modifying the > module loader’s search path accordingly if such a variable exists. It > would provide exactly that guarantee, but it’s just a PoC at this > point – see attached file. Woah, looks like a neat solution. Do you think it would scale for all our Python packages without manual intervention? If so, this would definitely be the way forward. > Apart from that I don’t see a good short-term solution right now. It’s > just how Python works. I mostly agree with you, but for this rather common case of having also a usercustomize it would be nice to circumvent it. In general, I don't think we ever want a Guix-produced Python script to load the usercustomize, hence my suggestion. The other case of PYTHONPATH is also annoying but can be tamed by modifying the env variable temporarily. Best, -- Josselin Poiret signature.asc Description: PGP signature
bug#64309: Python dlopen()s musl libc
Hi Josselin, > Woah, looks like a neat solution. Do you think it would scale for all > our Python packages without manual intervention? If so, this would > definitely be the way forward. I hope so, but haven’t tried it yet. > I mostly agree with you, but for this rather common case of having also > a usercustomize it would be nice to circumvent it. In general, I don't > think we ever want a Guix-produced Python script to load the > usercustomize, hence my suggestion. The other case of PYTHONPATH is > also annoying but can be tamed by modifying the env variable > temporarily. True, PYTHONPATH can be unset more easily than moving the user site dir. I’ll have a look at #64573, which should work around this issue. Cheers, Lars
