Re: Anomalies in Python search-path (was: [PATCH] Update python-pip to 9.0.1)
Hi! l...@gnu.org (Ludovic Courtès) writes: > Hello! > > Maxim Cournoyer skribis: > >> l...@gnu.org (Ludovic Courtès) writes: > > [...] > >>> ‘guix environment’ without --pure adds its own entries to the front of >>> the search paths, which is why we observe this behavior. >>> After more testing, the Python search-path we form is always problematic for pip since the system site-packages will always appear in front of the user site location (~/.local/...). My last reporting that pip was working correctly in non-environment was wrong because I had failed to logout/login after installing the python package; after doing so, the behavior is the same as in an environment: the user site location (~/.local/...) comes after site-packages rather than before, as would normally be the case. Apparently this is due to setting a site-packages location with the environment variable PYTHONPATH, which has precedence over the user site location. All of these precedence rules ared defined by the module site.py which comes with Python. It can print information about the Python search paths with: $ python -m site Normally the user site-packages location should come before any other site-packages location so that user installed Python packages can take precedence over packages visible at the system level (or in our profile in the the case Guix). >> >> Is this specific to 'guix environment' with --pure vs without --pure? > > Without ‘--pure’. Conversely, ‘--pure’ clears the environment > variables altogether: > > > https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-environment.html > >> Also, would this behavior be observed for any 3rd >> party package management system, say, the one for ruby, or >> language? If so maybe we could document this behavior somewhere. > > What would you add to the document of ‘guix environment’? It seems > quite clear to me at first sight. > I've re-read the documentation about it and yes, it is quite clear about what it does. I'll revisit any problems with environments later, after the root problem (which lies in the way we manage the Python search paths in our Python package definition/python build system) is fixed. Thanks for you time, and for Guix! Maxim signature.asc Description: PGP signature
Anomalies in Python search-path (was: [PATCH] Update python-pip to 9.0.1)
Am 06.01.2017 um 19:41 schrieb Maxim Cournoyer:
> One thing which I discovered while testing pip on Guix was that
> depending on how you use Python on Guix the PYTHONPATH differs:
I analysed this problem and found set we have another serious problem
here. But good news: There is an easy solution for both problems.
Result of the analysis (as explained below): We must not extend
PYTHONPATH for adding the profile's site-packages directory.
Proposed solution: The standard library module "site" already contains a
place where we can hook in:
# Prefixes for site-packages; add additional prefixes like /usr/local
here
PREFIXES = [sys.prefix, sys.exec_prefix]
So we simply prepend $GUIX_ENVIRONMENT to this list:
PREFIXES = [os.path.expandvar(GUIX_ENVIRONMENT), sys.prefix,
sys.exec_prefix]
Prepending to speed up searches, since within an environment almost all
site-packages will end in GUIX_ENVIRONMENT. There is no need to check if
$GUIX_ENVIRONMENT is set, since if it is not, it will be left unchanged
and the non-existing path will be skipped out automatically
This would solve both the problem Maxim described and the problem
described below, since the Guix site-packages behave exactly like any
other global site-packages directory.
When following this proposal, we *may* even remove some "wrapping" stuff
in the python-build-system. But this has to be investigated first.
Analysis (proof see below):
The python interpreter was two command line options, which are rarely
used, though:
-E ignores environment variables, esp. PYTHONPATH
-S Disable the import of the module site and the site-dependent
manipulations of sys.path.
This means:
a) When passing -E, the Guix environment's site-packages are ignored
(together with PYTHONPATH), while they should not.
b) When passing -S, the Guix environment's site-packages should be
ignored, but is not (since it is specified in $PYTHONPATH, which is not
ignored)
Conclusion: We should must not use PYTHONPATH to specify the Guix's
environment's site-package directory.
Analysis details:
(guix)$ which python3
/gnu/store/zcnb…-profile/bin/python3
Without any options:
-> /home/USER/.local/lib/python3.5/site-packages and
/gnu/store/zcnb…-profile/lib/python3.5/site-packages
should be in sys.path
(base)$ python3 -c 'import sys ; print("\n".join(sys.path))'
/usr/lib64/python34.zip
/usr/lib64/python3.4
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload
/home/USER/.local/lib/python3.4/site-packages
/usr/lib64/python3.4/site-packages
/usr/lib/python3.4/site-packages
(guix)$ python3 -c 'import sys ; print("\n".join(sys.path))'
/gnu/store/zcnb…-profile/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
/home/USER/.local/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages
-E Ignore environment variables like PYTHONPATH and PYTHONHOME
that modify the behavior of the interpreter.
-> /home/USER/.local/lib/python3.5/site-packages and
/gnu/store/zcnb…-profile/lib/python3.5/site-packages
should be in sys.path
(base)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'
/usr/lib64/python34.zip
/usr/lib64/python3.4
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload
/home/USER/.local/lib/python3.4/site-packages
/usr/lib64/python3.4/site-packages
/usr/lib/python3.4/site-packages
(guix)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'
/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
/home/USER/.local/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages
-S = Disable the import of the module site and the site-dependent
manipulations of sys.path that it entails.
-> /home/USER/.local/lib/python3.5/site-packages and
/gnu/store/zcnb…-profile/lib/python3.5/site-packages
sould *not* be in sys.path
(base)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'
/usr/lib64/python34.zip
/usr/lib64/python3.4/
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload
(guix)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'
/gnu/store/zcnb…-profile/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5/
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goe...@crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
Re: [PATCH] Update python-pip to 9.0.1
Hello! Maxim Cournoyer skribis: > l...@gnu.org (Ludovic Courtès) writes: [...] >> ‘guix environment’ without --pure adds its own entries to the front of >> the search paths, which is why we observe this behavior. >> > > Is this specific to 'guix environment' with --pure vs without --pure? Without ‘--pure’. Conversely, ‘--pure’ clears the environment variables altogether: https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-environment.html > Also, would this behavior be observed for any 3rd > party package management system, say, the one for ruby, or > language? If so maybe we could document this behavior somewhere. What would you add to the document of ‘guix environment’? It seems quite clear to me at first sight. >> In most cases this is what we want (we want to create an environment >> that prevails over the user’s profile), but I can see why this is >> problematic here. >> >> I would argue that using ‘pip’ in ‘guix environment’ is a bit convoluted >> anyway, but then again I’m no Python expert. :-) >> > > Yes, it is a bit convoluted. But with environments being so convenient, > I can think that people other than me would attempt this ;) Yes I can imagine. :-) Thanks, Ludo’.
Re: [PATCH] Update python-pip to 9.0.1
Hi Ludovic! Apologies for my delayed answer. l...@gnu.org (Ludovic Courtès) writes: >> From bfd91a59acdf3105505d7dd8483bb9cb97137cbf Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Mon, 28 Nov 2016 16:30:07 -0800 >> Subject: [PATCH] gnu: python-pip: Update to 9.0.1 >> >> * gnu/packages/python.scm (python-pip, python2-pip): Update to 9.0.1. > > I applied and expounded the commit log a bit. Thanks! > > One thing which I discovered while testing pip on Guix was that > depending on how you use Python on Guix the PYTHONPATH differs: > >> # PYTHONPATH in an environment >> $ guix environment python-wrapper >> [env]$ python3 -c 'import sys; print(sys.path)' >> ['', >> '/gnu/store/ar7k7ds90ikxv40a6lif6jv2g39l7mls-profile/lib/python3.5/site-packages', >> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python35.zip', >> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5', >> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/plat-linux', >> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/lib-dynload', >> '/home/maxim/.local/lib/python3.5/site-packages', >> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/site-packages'] > > The above command creates an environment containing the dependencies of > python-wrapper. > > Instead, you probably wanted to run this: > > $ ./pre-inst-env guix environment --ad-hoc python-wrapper -- python -c > 'import sys; print(sys.path)' Right! Thanks for pointing that out. > substitute: updating list of substitutes from > 'https://mirror.hydra.gnu.org'... 100.0% > The following derivations will be built: >/gnu/store/bxr0cw01qhgd9r741a6nnm5zwx3y12rr-profile.drv >/gnu/store/l91wf51q5ravikki293baljfx0bacqm8-info-dir.drv >/gnu/store/a4mz2l85sc11w6llh56zxlcc1bmfmzsr-ca-certificate-bundle.drv >/gnu/store/3z9hsnrwvqw70h0ssxqw0vy5y2dvzcj4-fonts-dir.drv > ['', > '/gnu/store/w63slncr9gyzjrs09n8ls81xac9fs804-profile/lib/python3.5/site-packages', > '/home/ludo/.guix-profile/lib/python3.5/site-packages', > '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python35.zip', > '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5', > '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/plat-linux', > > '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/lib-dynload', > > '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/site-packages'] > > … although it doesn’t make much of a difference in this particular case > since python-wrapper has python as its dependency. > >> Conclusion: When using python in a "guix environment", the *system* >> site-packages directory appears _before_ the *user* site-packages >> directory in the PYTHONPATH, which is wrong and causes pip to not work >> as intended. [2] > > ‘guix environment’ without --pure adds its own entries to the front of > the search paths, which is why we observe this behavior. > Is this specific to 'guix environment' with --pure vs without --pure? I'll test it out! Also, would this behavior be observed for any 3rd party package management system, say, the one for ruby, or language? If so maybe we could document this behavior somewhere. > In most cases this is what we want (we want to create an environment > that prevails over the user’s profile), but I can see why this is > problematic here. > > I would argue that using ‘pip’ in ‘guix environment’ is a bit convoluted > anyway, but then again I’m no Python expert. :-) > Yes, it is a bit convoluted. But with environments being so convenient, I can think that people other than me would attempt this ;) Thanks for the reply! Maxim signature.asc Description: PGP signature
Re: [PATCH] Update python-pip to 9.0.1
Hello! Maxim Cournoyer skribis: > This is my 2nd and hopefully correct attempt at updating python-pip to > 9.0.1. Test inputs were removed as the tarball available from pypi is > stripped from any test. > > It also incorporates suggestions that were previously made [1]; for > example we no longer need to use setuptools as an input since our Python > package definitions were recently updated to always be built with > pip/setuptools (the ensurepip flag). Sounds good. > From bfd91a59acdf3105505d7dd8483bb9cb97137cbf Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Mon, 28 Nov 2016 16:30:07 -0800 > Subject: [PATCH] gnu: python-pip: Update to 9.0.1 > > * gnu/packages/python.scm (python-pip, python2-pip): Update to 9.0.1. I applied and expounded the commit log a bit. One thing which I discovered while testing pip on Guix was that depending on how you use Python on Guix the PYTHONPATH differs: > # PYTHONPATH in an environment > $ guix environment python-wrapper > [env]$ python3 -c 'import sys; print(sys.path)' > ['', > '/gnu/store/ar7k7ds90ikxv40a6lif6jv2g39l7mls-profile/lib/python3.5/site-packages', > '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python35.zip', > '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5', > '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/plat-linux', > '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/lib-dynload', > '/home/maxim/.local/lib/python3.5/site-packages', > '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/site-packages'] The above command creates an environment containing the dependencies of python-wrapper. Instead, you probably wanted to run this: --8<---cut here---start->8--- $ ./pre-inst-env guix environment --ad-hoc python-wrapper -- python -c 'import sys; print(sys.path)' substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0% The following derivations will be built: /gnu/store/bxr0cw01qhgd9r741a6nnm5zwx3y12rr-profile.drv /gnu/store/l91wf51q5ravikki293baljfx0bacqm8-info-dir.drv /gnu/store/a4mz2l85sc11w6llh56zxlcc1bmfmzsr-ca-certificate-bundle.drv /gnu/store/3z9hsnrwvqw70h0ssxqw0vy5y2dvzcj4-fonts-dir.drv ['', '/gnu/store/w63slncr9gyzjrs09n8ls81xac9fs804-profile/lib/python3.5/site-packages', '/home/ludo/.guix-profile/lib/python3.5/site-packages', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python35.zip', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/plat-linux', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/lib-dynload', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/site-packages'] --8<---cut here---end--->8--- … although it doesn’t make much of a difference in this particular case since python-wrapper has python as its dependency. > Conclusion: When using python in a "guix environment", the *system* > site-packages directory appears _before_ the *user* site-packages > directory in the PYTHONPATH, which is wrong and causes pip to not work > as intended. [2] ‘guix environment’ without --pure adds its own entries to the front of the search paths, which is why we observe this behavior. In most cases this is what we want (we want to create an environment that prevails over the user’s profile), but I can see why this is problematic here. I would argue that using ‘pip’ in ‘guix environment’ is a bit convoluted anyway, but then again I’m no Python expert. :-) Thoughts? Thanks! Ludo’.
[PATCH] Update python-pip to 9.0.1
Hello Guix!
This is my 2nd and hopefully correct attempt at updating python-pip to
9.0.1. Test inputs were removed as the tarball available from pypi is
stripped from any test.
It also incorporates suggestions that were previously made [1]; for
example we no longer need to use setuptools as an input since our Python
package definitions were recently updated to always be built with
pip/setuptools (the ensurepip flag).
>From bfd91a59acdf3105505d7dd8483bb9cb97137cbf Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer
Date: Mon, 28 Nov 2016 16:30:07 -0800
Subject: [PATCH] gnu: python-pip: Update to 9.0.1
* gnu/packages/python.scm (python-pip, python2-pip): Update to 9.0.1.
---
gnu/packages/python.scm | 17 ++---
1 file changed, 6 insertions(+), 11 deletions(-)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 5faebae3d9..028cda6292 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -6773,26 +6773,21 @@ library.")
(define-public python-pip
(package
(name "python-pip")
-(version "8.0.2")
+(version "9.0.1")
(source
(origin
(method url-fetch)
(uri (pypi-uri "pip" version))
(sha256
(base32
- "08cm8d4228fj0qnrysy3qv1a6022zr3dcs25amd14lgxil6vvx26"
+ "03clr9c1dih5n9c00c592zzvf6r1ffimywkaq9agcqdllzhl7wh9"
(build-system python-build-system)
-(native-inputs
- `(;; Tests
-("python-virtualenv" ,python-virtualenv)
-("python-mock" ,python-mock)
-("python-pytest" ,python-pytest)
-("python-scripttest" ,python-scripttest)))
+(arguments
+ '(#:tests? #f)) ; there are no tests in the pypi archive.
(home-page "https://pip.pypa.io/";)
-(synopsis
- "Package manager for Python software")
+(synopsis "Package manager for Python software")
(description
- "Pip is a package manager for Python software, that finds packages on the
+ "Pip is a package manager for Python software, that finds packages on the
Python Package Index (PyPI).")
(license license:expat)))
--
2.11.0
One thing which I discovered while testing pip on Guix was that
depending on how you use Python on Guix the PYTHONPATH differs:
# PYTHONPATH in an environment
$ guix environment python-wrapper
[env]$ python3 -c 'import sys; print(sys.path)'
['',
'/gnu/store/ar7k7ds90ikxv40a6lif6jv2g39l7mls-profile/lib/python3.5/site-packages',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python35.zip',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/plat-linux',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/lib-dynload',
'/home/maxim/.local/lib/python3.5/site-packages',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/site-packages']
[env]$ pip install --user --upgrade setuptools
Collecting setuptools
Using cached setuptools-32.3.1-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-20.10.1
# PYTHONPATH when installed in your user profile
$ guix package -i python-wrapper
$ python3 -c 'import sys; print(sys.path)'
['',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python35.zip',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/plat-linux',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/lib-dynload',
'/home/maxim/.local/lib/python3.5/site-packages',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/site-packages']
$ pip3 uninstall setuptools
...
$ pip3 --user --upgrade setuptools
Collecting setuptools
Using cached setuptools-32.3.1-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-32.3.1
Conclusion: When using python in a "guix environment", the *system*
site-packages directory appears _before_ the *user* site-packages
directory in the PYTHONPATH, which is wrong and causes pip to not work
as intended. [2]
[1] https://lists.gnu.org/archive/html/guix-devel/2016-11/msg01242.html
[2] https://pip.pypa.io/en/stable/user_guide/#user-installs
Thanks,
Maxim
