On Fri, 29 Oct 2021 22:17:48 -0400 Sandro Tosi <[email protected]> wrote: > If you feel strongly this symlink needs to be removed, please conduct > a thorough analysis of the impact that such removal would cause and > report back on this report, else i'm inclined to just close it out.
As a concrete example take sagemath (https://www.sagemath.org/) when built from source on debian [0]. See https://trac.sagemath.org/ticket/33473 for the report in sage issue tracker. The way sagemath is built: - it uses the system python if available - it creates a venv for the system python - it builds its own version of python packages, including numpy, in this venv - it compiles "sagelib" which is the main component of sagemath using this venv Because of the symlink that debian places in /usr/include/python*/numpy, this means that sagelib ends up using numpy headers from system, but the python module from the venv. This indeed happens for sage-9.5 (current released version). It turns out that numpy 1.22 changed the numpy.ndarray structure (size changed) therefore, using headers from numpy 1.22 together with python module numpy 1.21 will result in a runtime error: sage: import sage.rings.polynomial.real_roots --------------------------------------------------------------------------- ValueError Traceback (most recent call last) ... ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject In this example system numpy is 1.22 and sage numpy is 1.21. I don't use debian anymore [1] but it should be possible to reproduce this using numpy/experimental which is already 1.22. I found the issue on void linux which had the same symlink as in debian and upgraded to numpy 1.22 (see https://github.com/void-linux/void-packages/issues/36062 for the issue which is already fixed in void linux). The same issue also affects scipy.spatial (e.g. when numpy 1.22 is in the system, numpy 1.21 is built in a venv, and then scipy is built in the same venv; I think it may also fail if numpy 1.21 is in the system and numpy 1.22 in the venv but I haven't tested this). Note that it is possible to workaround the symlink: make sure that the include arguments to the compiler (-I) for python headers and numpy headers are in the right order (i.e. numpy headers come first so that the system numpy headers do not override the venv numpy headers). But this is brittle, since the order of arguments in CFLAGS is kind of hard to control. Other distros (arch, alpine, gentoo, void since today) don't ship any symlink, and dependent packages seem to work ok. Fedora ships a symlink as /usr/include/numpy, which would avoid the trouble described here (because /usr/include is usually only implicit and not added as -I option) but it makes numpy packages for different versions of python to conflict. I think if a python package doesn't compile without the symlink, this is a bug in the package which should use numpy.get_include() to know which numpy headers to use. This is the only way to make sure the package uses the correct headers when using numpy from a venv (because numpy.get_include() should be called from setup.py in the venv so it will give the correct answer, meaning the one that matches the numpy package in the venv). > i'm not sure supporting a mixed environment of debian packages and pip > installed modules is the debian maintainer's responsibility: once > users install software outside of Debian, they usually are on their > own. I beg to disagree: the purpose of compilers, interpreters, libraries and modules is to be able to build and run code and packages that may well be written by the user, or downloaded from other sources. Even for software that is shipped by debian it is often useful to be able to build and install newer, unreleased or development versions, maybe to develop or test (which sometimes may even be to the distro own advantage). Of course you do not have to support the version of numpy shipped by sagemath, but it's nice if you can make sure that the debian numpy package doesn't get in the way. Thanks! Gonzalo [0] debian ships a binary package for sagemath, but there are still many important reasons to want to build from source, for example to develop for sagemath. [1] I was a debian user 1996-2016
