Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 28 April 2017 at 02:01, Charalampos Stratakis  wrote:
> Fedora's equivalent to dirtbike is rewheel [0][1] and it would be great if 
> these projects could be unified somehow, however that's a separate 
> conversation for another time.
>
> I'd be happy to take a closer look at dirtbike and maybe initiate some 
> discussions when I get some free time.

My recollection from the last time I looked at this was that where
rewheel assumes that the Python level PEP 376 package metadata will be
available, dirtbike instead queried the operating system level
metadata.

So while they're technically solving the same problem (injecting a
system level Python package or bindings into a virtual environment),
they took radically different paths to get there.

The biggest benefit I see to the dirtbike approach is that it has the
potential to work with Python bindings that aren't built with the
standard Python packaging tools, and hence don't have any PEP 376
metadata.

Conversely, the biggest benefit I see to the rewheel approach is that
it lets system packages more clearly indicate whether or not they want
to support rewheel by including or omitting the PEP 376 metadata.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Barry Warsaw]

On Apr 27, 2017, at 12:01 PM, Charalampos Stratakis wrote:

>Fedora's equivalent to dirtbike is rewheel [0][1] and it would be great if
>these projects could be unified somehow, however that's a separate
>conversation for another time.
>
>I'd be happy to take a closer look at dirtbike and maybe initiate some
>discussions when I get some free time.

+1

-Barry


pgph6l0nHuHNe.pgp
Description: OpenPGP digital signature
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Charalampos Stratakis]

Fedora's equivalent to dirtbike is rewheel [0][1] and it would be great if 
these projects could be unified somehow, however that's a separate conversation 
for another time.

I'd be happy to take a closer look at dirtbike and maybe initiate some 
discussions when I get some free time.

[0] 
http://pkgs.fedoraproject.org/cgit/rpms/python3.git/plain/00189-add-rewheel-module.patch
[1] https://github.com/fedora-python/rewheel (kinda outdated)

Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat

- Original Message -
From: "Barry Warsaw" 
To: python-devel@lists.fedoraproject.org
Sent: Thursday, April 27, 2017 5:27:55 PM
Subject: Re: What is your opinion on "sudo pip" fix for Fedora 27?

On Apr 27, 2017, at 04:52 PM, Nick Coghlan wrote:

>From my perspective, the main goal of the change is to make it
>possible to fully recover from "sudo pip install " by doing
>"sudo pip uninstall ", and the approach Debian took is
>sufficient to achieve that in almost all cases. So +1 from me for
>replicating something similar to Debian's approach in Fedora rather
>than blazing new trails by fiddling with `sys.prefix` directly.

All of Debian's deltas from upstream pip are here:

https://anonscm.debian.org/cgit/python-modules/packages/python-pip.git/tree/debian/patches

We also have a somewhat more elaborate machinery to unbundle packages using a
tool called dirtbike to turn installed (at build time) packages into
wheels.  That's a separate issue I'd be happy to talk more about.

In any case, to the extend that we can upstream the more generally useful
bits, or at least share behavior so users will have a similar experience on
both ecosystems, I'd be happy to help represent Debian.  Probably not much
will change there until after the upcoming release though, since the archive
is currently frozen.

Cheers,
-Barry

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Barry Warsaw]

On Apr 27, 2017, at 04:52 PM, Nick Coghlan wrote:

>From my perspective, the main goal of the change is to make it
>possible to fully recover from "sudo pip install " by doing
>"sudo pip uninstall ", and the approach Debian took is
>sufficient to achieve that in almost all cases. So +1 from me for
>replicating something similar to Debian's approach in Fedora rather
>than blazing new trails by fiddling with `sys.prefix` directly.

All of Debian's deltas from upstream pip are here:

https://anonscm.debian.org/cgit/python-modules/packages/python-pip.git/tree/debian/patches

We also have a somewhat more elaborate machinery to unbundle packages using a
tool called dirtbike to turn installed (at build time) packages into
wheels.  That's a separate issue I'd be happy to talk more about.

In any case, to the extend that we can upstream the more generally useful
bits, or at least share behavior so users will have a similar experience on
both ecosystems, I'd be happy to help represent Debian.  Probably not much
will change there until after the upcoming release though, since the archive
is currently frozen.

Cheers,
-Barry


pgpwOOQCUGDZq.pgp
Description: OpenPGP digital signature
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Barry Warsaw]

On Apr 27, 2017, at 06:21 AM, Donald Stufft wrote:

>For whatever it’s worth, at some point in time I plan on attempting to
>formalize the Debian solution (or something akin to it) within a PEP and try
>to get baked in support for it in Python.
>
>I also feel required to point out that ``sudo pip uninstall`` isn’t always
>enough, because while the two systems install to different locations, pip
>currently will uninstall the files in /usr when installing to
>/usr/local. They’ve patched their copy of pip to refuse to uninstall files
>from /usr, but upstream pip will still do it. Again, something I hope to get
>sorted at some point (and of course, an upgrade totally restores the files in
>/usr).

Also note that `pip install` defaults to --user in Debian, and that alleviates
a lot of the "well that didn't work, let me add sudo" urges.  Upstream, this
bug tracks that, although it's long standing without much recent progress:

https://github.com/pypa/pip/issues/1668

IMHO, that's the right direction to go, and I think Donald is philosophically
aligned (apologies if I'm misreading that, DS!).  There are of course plenty
of knock-on issues to work out, which is probably why that issue is still
open.

Defaulting to --user won't solve all the problems but I think it's very much a
step in the right direction.

Even with this, over in Debuntu-land we sometimes still see bug reports from
people who were confused by importing an unexpected version of a package, and
not remembering (or knowing) that some third party version outside of the
archive had been installed, and /usr/local/lib does come before /usr/lib in
sys.path.  This lead to https://bugs.python.org/issue29546 which will be in
3.7 and should at least help in debugging such situations.

Cheers,
-Barry


pgpRQPCy_NxVn.pgp
Description: OpenPGP digital signature
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 27 April 2017 at 23:04, Daniel P. Berrange  wrote:
> On Thu, Apr 27, 2017 at 04:32:09PM +1000, Nick Coghlan wrote:
>> Their approach means that any harm caused by "sudo pip install X" can
>> subsequently be fully reversed by doing "sudo pip uninstall X".
>>
>> At this moment, this is NOT true for Fedora and derivatives. Instead,
>> the remediation step here is "sudo pip uninstall X && sudo dnf
>> reinstall " where you have to:
>>
>> 1. Figure out what "" needs to be
>> 2. Hope that whatever you broke didn't affect your ability to run
>> "sudo dnf reinstall"
>
> Yep, recovering the system python install to a pristine state after
> 'devstack' has done its stuff is very painful right now :-(
>
> devstack was originally written for Ubuntu systems, so I guess they
> don't suffer as much due to the Debian specific change you describe
> above.
>
> Which location takes priority in Debian if the same module is installed
> in both places ? The DPKG location, or the Pip location ? Presumably it
> would have to be the pip version that takes priority if you want it to
> be usable for updating the (likely older) distro provided versions.

Aye, /usr/local/lib has priority:

$ docker run --rm ncoghlan/debian-python python3 -m site
sys.path = [
'/',
'/usr/lib/python3.4',
'/usr/lib/python3.4/plat-x86_64-linux-gnu',
'/usr/lib/python3.4/lib-dynload',
'/usr/local/lib/python3.4/dist-packages',
'/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.4/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

User level package installs take priority over system level ones
(regardless of distro), so the current container build use case for
root level pip installations can typically be met equally well by
running as a non-root user inside the container and doing "pip install
--user ...". Unfortunately, as with the venv option, there are a *lot*
of container build scripts out there that don't do that :(

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Donald Stufft]

> On Apr 27, 2017, at 2:32 AM, Nick Coghlan  wrote:
> 
> Debian and derivatives already mitigate the potential harm for these
> cases by requiring the "--install-layout=deb" option to be passed to
> get distutils to install into the system directories rather than doing
> it by default: https://wiki.debian.org/Python#Deviations_from_upstream 
> 
> 
> Their approach means that any harm caused by "sudo pip install X" can
> subsequently be fully reversed by doing "sudo pip uninstall X".


For whatever it’s worth, at some point in time I plan on attempting to 
formalize the Debian solution (or something akin to it) within a PEP and try to 
get baked in support for it in Python.

I also feel required to point out that ``sudo pip uninstall`` isn’t always 
enough, because while the two systems install to different locations, pip 
currently will uninstall the files in /usr when installing to /usr/local. 
They’ve patched their copy of pip to refuse to uninstall files from /usr, but 
upstream pip will still do it. Again, something I hope to get sorted at some 
point (and of course, an upgrade totally restores the files in /usr).

—
Donald Stufft



___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 26 April 2017 at 19:19, Michal Cyprian  wrote:
> The other possibility is to limit the pip install location change
> to distutils and pip [2]. This is the "safer" option, but does
> not cover all corner cases. For example, Python software built
> locally using cmake or similar tools will be installed into
> /usr/lib and can conflict with system tools. Debian chose to
> implement similar solution.
>
> I would like to ask which solution would work for your applications
> better, and what is in your opinion the right way to go.

From my perspective, the main goal of the change is to make it
possible to fully recover from "sudo pip install " by doing
"sudo pip uninstall ", and the approach Debian took is
sufficient to achieve that in almost all cases. So +1 from me for
replicating something similar to Debian's approach in Fedora rather
than blazing new trails by fiddling with `sys.prefix` directly.

For the latter concern with CMake/autotools/Scons/etc, anyone doing
local builds of C/C++ applications already needs to be careful not to
be interfere with system packages, so I'm comfortable leaving that in
the category where running C/C++ build system install commands outside
a package build process without due care and attention is a generally
bad idea (e.g. a "sudo make altinstall" of the Python 3.5 maintenance
branch will clobber the system Python binaries in F25, and a "sudo
make install" from the main Python development branch will mess with
the default symlinks if you didn't set a suitable `/opt/` prefix).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 27 April 2017 at 11:47, Nico Kadel-Garcia  wrote:
> On Wed, Apr 26, 2017 at 7:13 AM, Charalampos Stratakis
>> At the present time, running sudo pip3 in Fedora is not safe.
>> Pip shares its installation directory with dnf, can remove
>> dnf-managed files and generally break the Python 3 interpreter.
>
> This is true of every version of pip, not merely pip3. It was also
> true of CPAN, and of many gradle, maven, and ant working environments
> that semi-randomly collate the very latest versions of indeterminate
> components and spray them on top of your system workspace with their
> own quite distinct ideas about packaging and versioning.
>
> There is no solution to this problem, except to use tools like
> "pyvenv" to set aside secluded workspaces for Python modules and their
> dependencies assembly. So, for most developers, they *should not* use
> "sudo pip". They should remain in user space, or possibly in shared
> workspaces, to avoid overwriting system components.

Nothing is changing in terms of *recommended* practices. This change
proposal is entirely about harm mitigation for the cases where users
*do* run "sudo pip ...", either because that's their instinctive
reaction to a permissions error, or because some misguided
installation instructions for a 3rd party package advised them to do
it.

Debian and derivatives already mitigate the potential harm for these
cases by requiring the "--install-layout=deb" option to be passed to
get distutils to install into the system directories rather than doing
it by default: https://wiki.debian.org/Python#Deviations_from_upstream

Their approach means that any harm caused by "sudo pip install X" can
subsequently be fully reversed by doing "sudo pip uninstall X".

At this moment, this is NOT true for Fedora and derivatives. Instead,
the remediation step here is "sudo pip uninstall X && sudo dnf
reinstall " where you have to:

1. Figure out what "" needs to be
2. Hope that whatever you broke didn't affect your ability to run
"sudo dnf reinstall"

>> Our first attempt to make sudo pip safe on Fedora [0] was
>> based on using a different binary (/usr/libexec/system-python)
>> for RPM packaging purposes and changing the behavior
>> of /usr/bin/python3 (and pip that uses its settings) to install
>> under /usr/local by default. Switching all the Python 3 packages
>
> Oh, Oh my. The extent to which this unworkable is... fascinating.

Nico, the contemptuous tone you have adopted in this post is entirely
inappropriate. Please give other list participants the courtesy of
assuming expertise at least equivalent to your own, rather than
engaging in attempted displays of assumed superiority.

> You
> will inevitably install incompatible future versions of critical
> packages which are incompatible with the Fedora system packages. And
> as soon as you install one that breaks RPM, if /usr/local/ if ahead of
> /usr in your search path for modules, you are *screwed*.

This isn't true, as long as the fix is as simple as using pip to
uninstall whatever problematic package you just installed.

At the moment, that isn't the case on Fedora and derivatives, since
"sudo pip install ..." can actually *corrupt* a system package
installation, rather than merely shadowing it at import time the way
it does on Debian et al.

> And if
> /usr/local is first, well, an RPM update may introduce a component
> that is incompatible with the pip installed modules at arbitrary
> times.

The goal of this change isn't to allow co-existence of arbitrary
upstream and downstream versions as supported combinations - it's to
make sure that forced installation of an incompatible upstream version
is an easily reversible error, rather than a potentially
system-breaking mistake.

>> We decided to try a different approach. The main idea is to detect
>>  an ongoing RPM build and modify the behavior of the Python 3
>> executable only when in normal user environment so that RPM builds
>> won't be affected at all.
>
> See https://xkcd.com/1172/ . You're inventing a workflow that isn't
> supported by *anyone*.

The problem is that the default behaviour needs to change (to mitigate
the potential harm caused by a relatively common end user error), but
unlike Debian package builds (which know they need to set
"--install-layout=deb"), most RPM builds for Python packages currently
rely on that default behaviour as part of the build process. Changing
the default indiscriminately would thus be a breaking change that not
only affects Fedora RPM builds, but also COPR and other third party
RPM builds.

So we need a solution that retains the existing behaviour for RPM
builds, but changes the behaviour to be less potentially destructive
when run interactively at a shell prompt, or as part of a
configuration management script.

>> The adjustment of the behavior can be done on different levels.
>> The first option is to set the sys.prefix variable to /usr/local
>> when the interpreter is initialized. This will