Hi Guix, I’m dealing with Python in Guix pretty often, because numpy and scipy are very popular here — and I have been bitten by propagated inputs a little too often, so I’d like to discuss an alternative.
A user has the following packages installed in the default profile: python-2.7.10 python2-numpy python2-scipy This installation is already a little stale as it was done a couple of months ago, so the versions of numpy and scipy are not the latest. Now the user chooses to install the innocuous-looking “htseq” package. Guix downloads the substitute (and more) and then proceeds to build a new generation of the profile. But, oh, what is *that*? There are hundreds of lines in which Guix warns about numpy conflicts, shrugs and picks one or the other version of the conflicting file. “How could this have happened?”, the user cries out in dispair. Well, the user did not see that “htseq” is not only an executable, but also a Python library. As a Python library it depends on numpy, and thus propagates numpy as it is customary in Guix. This is necessary because Python libraries do not have a RUNPATH feature and must be able to find named imports in some directory in the PYTHONPATH. To make packages available on a simple user-controlled PYTHONPATH, we propagate all dependent Python packages, i.e. we install them into the very same profile, so that the user only has to do this export PYTHONPATH=$HOME/.guix-profile/lib/python2.7 instead of the unmanageable export PYTHONPATH=/gnu/store/...dep1/...:/gnu/store/...dep2...:... I wonder if we could do better than this. Here are two proposals for discussion; one is probably not very contentious, the other is straight from my dream diary in which I collect all sorts of strange and unrealistic ideas. 1) print a warning when a collision is expected to happen, not when a collision has happened. Guix knows what is currently installed in a profile, so it knows that, say, “python2-numpy” is installed. It also knows that installing “htseq” is also going to install “python2-numpy” into that profile. It also knows that the propagated “python2-numpy” is different from the installed “python2-numpy”. Knowing all that, it should also know that there are going to be file conflicts, no? If that’s all true, could we make Guix print a warning about impending doom *before* it creates a new profile generation? It’s a much nicer user experience (at least to me) when I’m being told about possible conflicts (at the package level) before Guix creates a new profile and nonchalantly informs me about arbitrarily resolving conflicts (at the file level). The difference is in the amount of warnings I get (package vs individual files) and about the time I would otherwise have to waste to roll back the profile and upgrade already installed libraries or choose to install “htseq” into a new profile. 2) avoid PYTHONPATH, patch all Python files invasively! Python does not have any feature that is comparable to RUNPATH. It is only concerned with finding libraries/modules by *name* in one of the directories specified by the PYTHONPATH environment variable. But actually the PYTHONPATH variable is not the only means to affect the search path for modules. It is possible to change the search path programmatically: import sys sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...") import numpy The first two lines add an explicit store item path to the search path; the third line just imports the numpy module by name (as usual). Even without setting the PYTHONPATH to include the numpy in the profile the third line won’t fail. I wonder if we could design a phase that — very much like the “wrap-program” phase — modifies *every* Python file and injects lines like the first two in the example above, appending explicit store item paths, so that all dependent Python libraries can be found without the need to have them installed in the same profile and without the need to set PYTHONPATH. Maybe this is crazy, and maybe this causes other annoying problems, but I think it is the closest we can get to a RUNPATH-like feature in Python. What do you think? Do I need a long vacation or is this something we might realistically do in an automated fashion? ~~ Ricardo