Re: [Distutils] Partial installs for in-process distribution upgrades (was: Installing packages using pip)

2015-12-10 Thread Paul Moore
On 9 December 2015 at 21:35, Robert Collins  wrote:
> On 10 December 2015 at 08:39, Erik Bray  wrote:
>> Apologies for resending this--my original message got buried in
>> unrelated commentary about how *nix filesystems work.  Resending with
>> new subject line...
>
> I think its going to break in nasty ways for users on Windows, on a
> regular basis.

Agreed. I think that, regardless of debates on how well (or otherwise)
particular operating systems handle "in place upgrades", it's a bad
idea in principle to try to upgrade a running system, unless that
system was designed from the ground up for such behaviour (e.g.,
Erlang).

Even if it can be made to work, documenting the implications and
restrictions would be a nightmare.

I think that it's better to take a simple approach, and say that you
should never upgrade a Python package that's in use, and it's
recommended not to import a package that was installed since your
Python session started (this latter probably does work in theory, but
"unsupported" is an easier position to take than trying to make sure
we didn't miss anything).

If other languages in Python's niche *do* properly support in-place
upgrading, then we can reconsider, but in that case we'd have prior
art to look at to see how they did it.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Partial installs for in-process distribution upgrades (was: Installing packages using pip)

2015-12-09 Thread Robert Collins
On 10 December 2015 at 08:39, Erik Bray  wrote:
> Apologies for resending this--my original message got buried in
> unrelated commentary about how *nix filesystems work.  Resending with
> new subject line...

I think its going to break in nasty ways for users on Windows, on a
regular basis.

For instance: start two interpreters at once with a partially
installed thing present and watch them race.

For instance: put a .dll extension in a partially installed thing and
start a second interpreter without closing the one that triggered the
install. Watch the move error because the .dll is locked.

(And this is ignoring the questions around module namespaces in
sys.modules and the poorly-supported-in-practice semantics of
reload()).

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


[Distutils] Partial installs for in-process distribution upgrades (was: Installing packages using pip)

2015-12-09 Thread Erik Bray
Apologies for resending this--my original message got buried in
unrelated commentary about how *nix filesystems work.  Resending with
new subject line...

On Fri, Nov 13, 2015 at 3:09 PM, Nathaniel Smith  wrote:
> On Nov 13, 2015 12:00 PM, "Alexander Walters" 
> wrote:
>>
>> import pip
>> pip.install(PACKAGESPEC)
>>
>> something like that?
>
> This would be extremely handy if it could be made to work reliably... But
> I'm skeptical about whether it can be made to work reliably. Consider all
> the fun things that could happen once you start upgrading packages while
> python is running, and might e.g. have half of an upgraded package already
> loaded into memory. It's like the reloading problem but even more so.

Sorry to resurrect an old thread, but I have an idea about how to do
this somewhat safely, at least insofar as the running interpreter is
concerned.  It's still a terrible idea.  Not such a terrible idea in
principle, but as a practical matter in the context of Python it's
probably a bad idea because it uses yet-another-.pth-hack.

Consider a "partial install", wherein pip installs all files into a
non-imported subdirectory of the target site-packages, along with a
.pth file.  This distribution is then considered "partially installed"
in that the files are their (whether extracted from a wheel, or
installed via distutils and the appropriate --root option or similar).
For example, consider running

>>> pip.install('requests')

It would be up to the pip.install() command to determine whether or
not the requests distribution was already installed.  If it's not
install it would proceed as normal.  For now I'm assuming the user
would still have to manually run `import requests` after this.
Auto-import would be nice, but is a separate issue.

Now, if requests were already installed and imported we don't want to
clobber the existing requests running in the interpreter.  pip would
install install into the relevant site-packages:

<...>/site-packages/requests-2.8.1.part/
requests/
requests-2.8.1.dist-info/
<...>/site-packages/requests-2.8.1.part.pth

The .part/ directory contains the results of the partial installation
(for example the contents of the wheel, for wheel installs).  The
.part.pth file is trickier, but could be something like this:

$ cat requests-2.8.1.part.pth
import inspect, shutil, sys, os, atexit;p =
inspect.currentframe().f_locals['sitedir'];part = os.path.join(p,
'requests-2.8.1.part');files = os.path.isdir(part) and
os.listdir(part);files and list(map(lambda s, d, f:
(sys.modules['shutil'].rmtree(os.path.join(d, f),
sys.modules['shutil'].move(os.path.join(s, f), os.path.join(d, f))),
[part] * len(files), [p] * len(files), files));os.rmdir(part);pth =
part + '.pth';os.path.isfile(pth) and atexit.register(os.unlink,
os.path.abspath(pth))

This rifles through the contents of requests.2.8.1.part, deletes any
existing directories in the parent site-packages of the same name,
completes the install by moving the contents of the .part/ directory
into the correct location and then deletes the .part/ directory.  The
.part.pth later deletes itself.

By the time the user restarts the interpreter and runs `import
requests` this will be completed.  Obvious it would have to be
communicated to the user that to upgrade an existing package they will
have to restart the interpreter which is less than ideal, but relates
to a deeper limitation of Python that they should get used to anyways.
At least this would enable in-process installs/upgrades.

There are of course all kinds of problems with this solution too.  It
should perhaps only work in a virtualenv and/or .local site-packages
(or at least somewhere that the user will have write permissions on
the next interpreter run), and probably other error handling too.  The
above .pth file could also be simplified by invoking a function in pip
to complete any partial installs.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig