Re: Making sudo pip Safe
On 12/07/2016 06:21 PM, Stephen John Smoogen wrote:
On 7 December 2016 at 10:27, Tomas Orsava wrote:
On 12/07/2016 01:56 PM, Neal Gompa wrote:
On Wed, Dec 7, 2016 at 7:53 AM, Michal Cyprian
wrote:
- system-python (i.e. what all programs installed via DNF will use) is
limited to site-packages under /usr, so DNF-installed software is unaffected
by anything installed with pip
system-python is not intended for this use-case. It was designed to
support DNF itself and other related system software. It's also
intentionally set up to not have everything that regular python3 has.
I would really hesitate to allow this to be used for more than that,
because then we're back to square one, again...
I think there's been a slight miscommunication: system-python will be used
instead of python3 only when building/installing packages inside spec files.
The built packages themselves depend on will be run by python3.
There has been some sort of miscommunication because from my layman
take on things, what Neal was what system-python was only meant to be
used for .. for exactly the reason he said.
System-python is a rather complex and evolving topic, so let my try to
explain it a bit deeper (as I understand it):
Up to this proposal, the system-python binary
(/usr/libexec/system-python) has been simply a direct copy of the
python3 executable (/usr/bin/python3{,.5}) [0]. The usefulness of
system-python thus does not come from the executable you use, but from
the `system-python` package that contains it. The difference between the
`python3` and `system-python` packages is that the former requires the
full Python 3 standard library (package `python3-libs`), whereas the
`system-python` package requires only a smaller subset thereof (package
`system-python-libs`). Thus saving space when Python is only needed for
system tools.
This proposal constitutes a second evolutionary step of system-python:
We now want to slightly modify how the two executables behave. Python 3
will continue as it always has—you will be able to import modules
whether you installed them through dnf or pip/setuptools. Contrary to
that, system-python will become more limited: It will only recognize
Python modules installed through dnf, and as a result, your system tolls
(dnf et al.) will become impervious to problems stemming from
non-functional/problematic modules installed using sudo pip, which up to
now could simply overwrite crucial modules installed by dnf.
To implement this modification, the locations of dnf-installed and sudo
pip–installed modules need to be separated. The default location where
Python decides to install new modules is contained within the
executable, and thus Python 3's install location will be the one where
sudo pip–installed modules belong, while system-python will install
modules to a path where dnf-installed modules ought to go. Therefore if
you run sudo pip, Python 3 executable will be invoked and pip will
install modules to the appropriate place. And in spec files of Python
RPM packages, the %{__python3} macro will be modified so that
system-python is invoked instead, and the modules will be installed to
the appropriate location for dnf-installed modules.
A good question to ask at this point is: Is it ok to use system-python
in spec files like this when it requires only a limited subset of the
Python 3 standard library? The confusing part is that while
system-python requires only a limited stdlib, it will use the full
standard library if it is installed. And when it comes to Python RPM
packages, it will always be installed because: 1. At build time all
Python 3 packages have to use `BuildRequires: python3-devel` which
brings in a full stdlib complement, and 2. at run time the resulting RPM
packages will depend on the `python(abi) = 3.X`, which is only satisfied
by the `python3` package and it's full stdlib.
I hope this may shine a bit of light on the issue.
[0]
http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec?id=ddb16c68d95603cc11233c520544adf92c3741fe#n1061
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Re: Making sudo pip Safe
On 12/07/2016 05:53 AM, Michal Cyprian wrote:
> Hello,
>
> there is a long-standing problem that `sudo pip install` cannot be safely
> used in Fedora. Many users don't know about this and break python packages on
> theirs systems. Packages installed using this command can conflict and
> overwrite Python rpm packages.
> This is a major problem and we have seen several systems broken by people
> using "sudo pip". Unfortunately, telling people to not use it will not work:
> "sudo pip" appears in documentation of too many projects online.
> We plan to solve this in Fedora 26. A more precise description of the problem
> follows.
>
> Current behavior of python packages installation tools in Fedora:
> 1) sudo dnf install python3-foo # root installs foo from rpm to
> /usr/lib/pythonX.Y/site-packages
> 2) sudo pip3 install foo # root installs foo from PyPI to
> /usr/lib/pythonX.Y/site-packages
> 3) sudo pip3 install -t /usr/local/lib/pythonX.Y/site-packages foo # root
> install package to selected location
> 4) pip3 install --user foo # user install foo from PyPI to
> ~/.local/lib/python3.5/site-packages
>
> Using the --user option with pip would be the ideal solution. However, it is
> reportedly broken in some versions of Ubuntu, so it is hard to convince
> software authors to recommend it.
>
> Packages installed using `sudo pip` (2) under /usr can be overwritten or
> removed by dnf. `dnf install python3-foo` fails if foo was `sudo pip
> installed` before etc. This problem has been reported many times. [6]
> Another issue is that packages installed under usr/local (3) cannot be
> imported unless PYTHONPATH or sys.path is set explicitly.
>
> The default install location of pip/distutils-installed packages depends on
> the value of the
> sys.prefix variable [4].
> There were several discussions on Bugzilla [1] and the pypa-dev mailing list
> [2].
> Interesting solutions were conceived at the CPython level:
> - addition of sys.local_prefix [2]
> - simplification of CPython startup sequence [4]
> Unfortunately none of them were realized and both solutions require many
> changes of Python and Python Standard Library.
>
> We discussed all the possible solutions with colleagues from the Python maint
> team.
> It would be great to fix this upstream, but it won't happen in a reasonable
> time frame.
> We realized that System Python [3], which was announced in Fedora 24, can
> help us reach the goal. We came up with the following solution:
> - sys.prefix of the python3 executable will be set to /usr/local
> - sys.prefix of system-python will be set to /usr
> - all rpm python packages will be installed using system-python (value of
> rpm macro %{__python3} will be changed to point to system-python)
> - The path /usr/lib/.../site_packages will be included in sys.path of
> /usr/bin/python3
>
> Note that no packages will be affected by other work going on around
> system-python, namely that packages can declare that they are only using a
> smaller subset of the Python stdlib. That option is still opt-in.
>
> Behavior of install methods after mentioned changes:
> 1) sudo dnf install foo # root installs foo from rpm to
> /usr/lib/pythonX.Y/site-packages
> 2) sudo pip3 install foo # root installs foo from PyPI to
> /usr/local/lib/pythonX.Y/site-packages
> - pip never touches /usr/lib/python3.5/site-packages
> - /usr/bin/python3 can import modules from
> - Python Standard Library
> - site-packages under /usr/local (priority)
> - site-packages under /usr
> - system-python (i.e. what all programs installed via DNF will use) is
> limited to site-packages under /usr, so DNF-installed software is unaffected
> by anything installed with pip
>
> Michal Cyprian
I applaud this effort. My concern comes from the question of whether or not
one should be able to extend the available library for system tools. If I'm
using the dnf installed jupyter notebook and I want to use a newer version of
some other package or something that isn't packaged and I install it via "sudo
pip install foo" I can't import it without changing PYTHONPATH, which brings
us back to the problems with method #3 above. Or can we get around this by
not changing %__python3 but changing %py3_build to be:
CFLAGS="%{optflags}" %{__system_python} %{py_setup} %{?py_setup_args} build
--executable="%{__python3} %{py3_shbang_opts}" %{?*}
And still having the installed shebangs reference /usr/bin/python3.X?
We had a discussion a while back about adding -E or -I to the py3_shbang_opts,
but this was felt to be too disruptive. I think having everything installed
with system_python shbangs would be fairly similar. Or am I missing something?
Debian deals with this by having dist-packages
(https://wiki.debian.org/Python). Is this not worth adopting?
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane
