Re: [Distutils] Installing packages using pip
On Mon, Nov 16, 2015 at 6:25 PM, Marius Gedminas wrote: > What you can do Linux that you cannot do on Windows is delete a shared > library file while it's mapped into a process's address space. Then > Linux lets you create a new file with the same name, while the old file > stays around, nameless, until it's no longer used, at which point the > disk space gets garbage-collected. (If we can call reference counting > "garbage collection".) > > The result is as you said: existing processes keep running the old code > until you restart them. There are tools (based on lsof, AFAIU) that > check for this situation and remind you to restart daemons. > Not sure what exactly was going on but whenever I did that on linux I got the most peculiar segfaults and failures. It is certainly not a safe thing to do, even if linux lets you do it. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing packages using pip
On Fri, Nov 13, 2015 at 3:09 PM, Nathaniel Smith wrote:
> On Nov 13, 2015 12:00 PM, "Alexander Walters"
> wrote:
>>
>> import pip
>> pip.install(PACKAGESPEC)
>>
>> something like that?
>
> This would be extremely handy if it could be made to work reliably... But
> I'm skeptical about whether it can be made to work reliably. Consider all
> the fun things that could happen once you start upgrading packages while
> python is running, and might e.g. have half of an upgraded package already
> loaded into memory. It's like the reloading problem but even more so.
Sorry to resurrect an old thread, but I have an idea about how to do
this somewhat safely, at least insofar as the running interpreter is
concerned. It's still a terrible idea. Not such a terrible idea in
principle, but as a practical matter in the context of Python it's
probably a bad idea because it uses yet-another-.pth-hack.
Consider a "partial install", wherein pip installs all files into a
non-imported subdirectory of the target site-packages, along with a
.pth file. This distribution is then considered "partially installed"
in that the files are their (whether extracted from a wheel, or
installed via distutils and the appropriate --root option or similar).
For example, consider running
>>> pip.install('requests')
It would be up to the pip.install() command to determine whether or
not the requests distribution was already installed. If it's not
install it would proceed as normal. For now I'm assuming the user
would still have to manually run `import requests` after this.
Auto-import would be nice, but is a separate issue.
Now, if requests were already installed and imported we don't want to
clobber the existing requests running in the interpreter. pip would
install install into the relevant site-packages:
<...>/site-packages/requests-2.8.1.part/
requests/
requests-2.8.1.dist-info/
<...>/site-packages/requests-2.8.1.part.pth
The .part/ directory contains the results of the partial installation
(for example the contents of the wheel, for wheel installs). The
.part.pth file is trickier, but could be something like this:
$ cat requests-2.8.1.part.pth
import inspect, shutil, sys, os, atexit;p =
inspect.currentframe().f_locals['sitedir'];part = os.path.join(p,
'requests-2.8.1.part');files = os.path.isdir(part) and
os.listdir(part);files and list(map(lambda s, d, f:
(sys.modules['shutil'].rmtree(os.path.join(d, f),
sys.modules['shutil'].move(os.path.join(s, f), os.path.join(d, f))),
[part] * len(files), [p] * len(files), files));os.rmdir(part);pth =
part + '.pth';os.path.isfile(pth) and atexit.register(os.unlink,
os.path.abspath(pth))
This rifles through the contents of requests.2.8.1.part, deletes any
existing directories in the parent site-packages of the same name,
completes the install by moving the contents of the .part/ directory
into the correct location and then deletes the .part/ directory. The
.part.pth later deletes itself.
By the time the user restarts the interpreter and runs `import
requests` this will be completed. Obvious it would have to be
communicated to the user that to upgrade an existing package they will
have to restart the interpreter which is less than ideal, but relates
to a deeper limitation of Python that they should get used to anyways.
At least this would enable in-process installs/upgrades.
There are of course all kinds of problems with this solution too. It
should perhaps only work in a virtualenv and/or .local site-packages
(or at least somewhere that the user will have write permissions on
the next interpreter run), and probably other error handling too. The
above .pth file could also be simplified by invoking a function in pip
to complete any partial installs.
Erik
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setup.py install using pip
On Mon, Dec 7, 2015 at 2:40 PM, Paul Moore wrote: > On 7 December 2015 at 18:58, Erik Bray wrote: >> I wasn't able to produce this problem. Even with --no-binary >> specified pip installs (by default) with >> --single-version-externally-managed. My prototype implicitly disables >> the --pip flag if --single-version-externally-managed was specified >> (true to the purpose of that flag). > > Ah - that was the bit I was missing, the > --single-version-externally-managed flag can be used to trigger > ignoring --pip. > >> What *is* a problem is if --pip is in setup.cfg, and one invokes `pip >> install --egg .`. I wasn't quite able to make that go into an >> infinite loop, but it did invoke pip.main recursively, and stuff broke >> on the second invocation for reasons not clear to me. > > Yeah, but honestly I don't think pip install --egg is that important a > use case. I may be wrong (there's lots of ways people use pip that I > know nothing of :-)) but as a starting point it might be OK just to > say that at the same time as the --pip flag was introduced, "pip > install --egg" was deprecated (and we explicitly document that pip > install --egg is known to interact badly with setup.py --pip). I'd be fine with that too. IIRC pip install --egg was introduced in part to work around problems with namespace packages. This doesn't completely eliminate the need for that workaround, but it does reduce it. In either case, if the --pip feature is implemented in setuptools it would still be good to at least have some workaround for installing with pip install --egg, at least temporarily. My goal here was to not break any existing functionality when --pip is specified (either via command line or setup.cfg). > Anyway, thanks for the explanation, I see what you're intending now. Great! ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setup.py install using pip
On 7 December 2015 at 18:58, Erik Bray wrote: > I wasn't able to produce this problem. Even with --no-binary > specified pip installs (by default) with > --single-version-externally-managed. My prototype implicitly disables > the --pip flag if --single-version-externally-managed was specified > (true to the purpose of that flag). Ah - that was the bit I was missing, the --single-version-externally-managed flag can be used to trigger ignoring --pip. > What *is* a problem is if --pip is in setup.cfg, and one invokes `pip > install --egg .`. I wasn't quite able to make that go into an > infinite loop, but it did invoke pip.main recursively, and stuff broke > on the second invocation for reasons not clear to me. Yeah, but honestly I don't think pip install --egg is that important a use case. I may be wrong (there's lots of ways people use pip that I know nothing of :-)) but as a starting point it might be OK just to say that at the same time as the --pip flag was introduced, "pip install --egg" was deprecated (and we explicitly document that pip install --egg is known to interact badly with setup.py --pip). Anyway, thanks for the explanation, I see what you're intending now. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setup.py install using pip
On Mon, Dec 7, 2015 at 11:45 AM, Paul Moore wrote: > On 7 December 2015 at 16:21, Erik Bray wrote: >> Exactly--as both a library developer / maintainer and system >> integrator I would find such a flag very useful (especially since I >> can just set it in a config file and forget it). It would be right >> for me. But wouldn't break anything for anyone else. >> >> Ironically, the default behavior of `setup.py install`, on projects >> that use setuptools, is to install an egg directory which is >> *definitely* not for everybody, especially not anymore. That's why >> --single-version-externally-managed exists. A --pip flag would be >> very much like --single-version-externally-managed (sort of a >> specialized extension of it) that also includes "do everything pip >> does" which includes installing dependencies and copying >> .egg-info/.dist-info to the appropriate location, which is what I want >> to replace all instances of `setup.py install` with. That includes >> users running `setup.py install`, who have a hard enough time as it is >> keeping up with Python build/installation "best practices" as it is. > > One thing that bothers me about this proposal. If someone does "pip > install --no-binary" for your package, and you have the "--pip" flag > in your setup.cfg, pip will use "setup.py install" to do the install. > Which, if I understand this proposal correctly, will attempt to "fall > back" to pip because "--pip" is in setup.cfg. Which results in an > infinite loop of pip and setup.py invoking each other. I wasn't able to produce this problem. Even with --no-binary specified pip installs (by default) with --single-version-externally-managed. My prototype implicitly disables the --pip flag if --single-version-externally-managed was specified (true to the purpose of that flag). What *is* a problem is if --pip is in setup.cfg, and one invokes `pip install --egg .`. I wasn't quite able to make that go into an infinite loop, but it did invoke pip.main recursively, and stuff broke on the second invocation for reasons not clear to me. This is easily worked around, however, by detecting, from the install command, if we're already using pip. As a quick hack I added to finalize_options: if 'pip' in sys.modules: self.pip = False This did the trick. pip ran fine and installed the package as an egg using the standard setup.py install. I don't think a more robust solution would be hard. pip could set an environment variable or even a variable in the pip module itself to indicate that pip is already being invoked to install this package. There may even be something like that already in pip that I'm not aware of. > I'm not sure how pip could detect a situation like this, so there's a > risk of some *very* obscure corner cases, which I'm sure people will > end up hitting. As mentioned above I don't think it should be pip's job to detect this situation. But if the setuptools install command can detect that we're already in pip then the job is done. > As a user level command line flag, "setup.py install --pip" isn't much > better than "pip install ." > As a project config, we get the issue noted above, and the user has to > edit the project code to fix it. > As a per-user or global config, we get the issue above, but we could > reasonably say it's the user's mistake and the user has the means to > fix it. But it's still not a great UX. I don't think so either. But it's also not great UX that we've told users for decades that the way to install a Python package is to run `python setup.py install`, but now the default behavior of that (for packages using setuptools) which is to install a .egg, is old and bad. I get confusion about this from users *frequently*. It's only worse that egg installs and flat installs are incompatible with each other with respect to namespace packages. If it would help, `setup.py install --pip` could also display a warning that users should run `pip install .` instead. To cut down on noise I might only do this if the --pip option came from setup.cfg, rather than when it's explicitly asked for via the command line. This would serve to inform users who don't know any better. > It's quite possible I'm missing something here (at a minimum, I'm > making huge assumptions about how the feature would be implemented) > but I think the behaviour needs to be thought through in a bit more > detail. Completely agree! I know there are corner cases I haven't thought of. I'm also undecided on whether --pip should invoke pip.main() within the same process, or run pip from a subprocess. The former seems sufficient but I don't know all the cases. Thanks, Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setup.py install using pip
On 7 December 2015 at 16:21, Erik Bray wrote: > Exactly--as both a library developer / maintainer and system > integrator I would find such a flag very useful (especially since I > can just set it in a config file and forget it). It would be right > for me. But wouldn't break anything for anyone else. > > Ironically, the default behavior of `setup.py install`, on projects > that use setuptools, is to install an egg directory which is > *definitely* not for everybody, especially not anymore. That's why > --single-version-externally-managed exists. A --pip flag would be > very much like --single-version-externally-managed (sort of a > specialized extension of it) that also includes "do everything pip > does" which includes installing dependencies and copying > .egg-info/.dist-info to the appropriate location, which is what I want > to replace all instances of `setup.py install` with. That includes > users running `setup.py install`, who have a hard enough time as it is > keeping up with Python build/installation "best practices" as it is. One thing that bothers me about this proposal. If someone does "pip install --no-binary" for your package, and you have the "--pip" flag in your setup.cfg, pip will use "setup.py install" to do the install. Which, if I understand this proposal correctly, will attempt to "fall back" to pip because "--pip" is in setup.cfg. Which results in an infinite loop of pip and setup.py invoking each other. I'm not sure how pip could detect a situation like this, so there's a risk of some *very* obscure corner cases, which I'm sure people will end up hitting. As a user level command line flag, "setup.py install --pip" isn't much better than "pip install ." As a project config, we get the issue noted above, and the user has to edit the project code to fix it. As a per-user or global config, we get the issue above, but we could reasonably say it's the user's mistake and the user has the means to fix it. But it's still not a great UX. It's quite possible I'm missing something here (at a minimum, I'm making huge assumptions about how the feature would be implemented) but I think the behaviour needs to be thought through in a bit more detail. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setup.py install using pip
On Mon, Dec 7, 2015 at 4:34 AM, Nick Coghlan wrote: > On 7 December 2015 at 17:07, Ronny Pfannschmidt > wrote: >> That's a straw man, this has enough inconsistency potential to break many >> edge cases in ugly ways, >> So global setup is out. > > No, global set up *isn't* out - the inevitable edge cases won't matter > to an application integrator if none of the components they're using > hit them, and installation related problems have the virtue of being > relatively straightforward to pick up in a continuous integration > system. > > Using such a switch wouldn't be the right fit for everyone, but that's > not the same as it being entirely useless. Exactly--as both a library developer / maintainer and system integrator I would find such a flag very useful (especially since I can just set it in a config file and forget it). It would be right for me. But wouldn't break anything for anyone else. Ironically, the default behavior of `setup.py install`, on projects that use setuptools, is to install an egg directory which is *definitely* not for everybody, especially not anymore. That's why --single-version-externally-managed exists. A --pip flag would be very much like --single-version-externally-managed (sort of a specialized extension of it) that also includes "do everything pip does" which includes installing dependencies and copying .egg-info/.dist-info to the appropriate location, which is what I want to replace all instances of `setup.py install` with. That includes users running `setup.py install`, who have a hard enough time as it is keeping up with Python build/installation "best practices" as it is. Anyways, it has been frequently requested that setuptools change the default behavior of the "install" command, so I wouldn't discount it as a valid use case. So far it hasn't been changed out of backward-compat concerns, so making it loosely opt-in represents a possible middle ground. >> Projects themselves can really just switch to pip commands, same goes for >> packagers and other tool makers >> >> Explicit is better than implicit, and in this case it also won't cost >> additional maintenance on setuptools. >> Please keep in mind, that setuptools is completely on volunteer time, and >> the time given to it is scarce. > > Sure, that's why any decision on the desirability of this feature > would be up to Jason as the setuptools lead. However, there's a > trade-off to consider here, which is that offering this kind of global > installer switch may help to lower the priority of some other > easy_install enhancement requests. Plus I've contributed to setuptools many times in the past (used to have commit access on distribute too). I'm offering to implement and maintain this feature if it's decided desirable. I think it *is* desirable by definition--I desire it, and I suspect others would as well. I'd be more interested in technical reasons why it's a bad idea but I haven't found any yet. > That's a risk assessment trade-off on future bug reports against > attempted pip support vs future RFEs against easy_install itself, as > well as a priority assessment against other open changes proposed for > setuptools. Those assessments may well come down on the side of "not > worth the hassle", but the scope of the proposed change still falls a > long way short of being a "maintenance horror". I think the only maintenance horror right now is easy_install :) Thanks, Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setup.py install using pip
On 7 December 2015 at 17:07, Ronny Pfannschmidt wrote: > That's a straw man, this has enough inconsistency potential to break many > edge cases in ugly ways, > So global setup is out. No, global set up *isn't* out - the inevitable edge cases won't matter to an application integrator if none of the components they're using hit them, and installation related problems have the virtue of being relatively straightforward to pick up in a continuous integration system. Using such a switch wouldn't be the right fit for everyone, but that's not the same as it being entirely useless. > Projects themselves can really just switch to pip commands, same goes for > packagers and other tool makers > > Explicit is better than implicit, and in this case it also won't cost > additional maintenance on setuptools. > Please keep in mind, that setuptools is completely on volunteer time, and > the time given to it is scarce. Sure, that's why any decision on the desirability of this feature would be up to Jason as the setuptools lead. However, there's a trade-off to consider here, which is that offering this kind of global installer switch may help to lower the priority of some other easy_install enhancement requests. That's a risk assessment trade-off on future bug reports against attempted pip support vs future RFEs against easy_install itself, as well as a priority assessment against other open changes proposed for setuptools. Those assessments may well come down on the side of "not worth the hassle", but the scope of the proposed change still falls a long way short of being a "maintenance horror". Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
