[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?
On Wed, Feb 24, 2021 at 12:42 PM Paul Moore wrote: > On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier wrote: > > There is probably a clever way to reuse common packages (probably via > clever symlinking) and reduce the footprint of these installations. > > Ultimately the problem is that a general tool can't deal with > conflicts (except by raising an error). If application A depends on > lib==1.0 and application B depends on lib==2.0, you simply can't have > a (consistent) environment that supports both A and B. But that's the > rare case - 99% of the time, there are no conflicts. One env per app > is a safe, but heavy handed, approach. Managing environments manually > isn't exactly *hard*, but it's annoying manual work that pipx does an > excellent job of automating, so it's a disk space vs admin time > trade-off. > There are three ways to approach the question: 1) Fully isolated envs. The safest option but uses the most space. 2) Try to minimise the number of dependencies installed by interpreting the requirements specification in the looser way possible. This is both algorithmically hard (see https://hal.archives-ouvertes.fr/hal-00149566/document for instance, or the more recent https://hal.archives-ouvertes.fr/hal-03005932/document ) and risky, as you've noted. 3) But the best way IMHO is to compute dependencies for each virtualenv independently from the others, but still share the packages, using some indirection mechanisms (hard links, symlinks or some Python-specific constructs) when the versions match exactly. The 3rd solution is probably the best of the 3, but the sharing mechanism still needs to be specified (and, if needed, implemented) properly. I've tried Christian's suggestions of using rdfind on my pipx installation, and it claims to reduce the footprint by 30% (nice, but less than I expected. This would however scale better with the number of installed packages). I'm not sure this would be practical in reality, OTOH, because I think there is a serious risk of breakage each time I would upgrade one of the packages (via 'pipx upgrade-all' for instance). So IMHO the best way to implement solution 3 would be by using some variant of the approach popularized by Nix (repository of immutable packages + links to each virtualenv). S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6GV3EPYU2SW2QKAKLXZCG3JLNCVSQZWG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?
On Wed, Feb 24, 2021 at 1:47 PM Stéfane Fermigier wrote: > > So IMHO the best way to implement solution 3 would be by using some > variant of the approach popularized by Nix (repository of immutable > packages + links to each virtualenv). > Another benefit of this kind of approach, besides sparing disk space, could be similar improvements in terms of installation time (and even bigger improvements when reinstalling a package, which happens all the time when developing). S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UO2AZHKZUNEQ34SGNKCR5RTD4GYI2SHF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?
On Wed, Feb 24, 2021, at 09:08, Paul Moore wrote: > I don't use Linux much, and I'm definitely not familiar with Linux > distribution tools, but from what I can gather Linux distributions > have made the choices: > > 1. Write key operating system utilities in Python. > 2. Share the Python interpreter and libraries. > 3. Expose that Python interpreter as the *user's* default Python. I think 1 *partially* mischaracterizes the problem, because any "system python" would logically be used by *every application written in python [or that embeds python] distributed by the OS's package management*, not just by "key operating system utilities". To suggest otherwise implies that they should not distribute any python applications at all. That also makes asking all of their package maintainers to change their #! line to point at a different interpreter [or to pass an option, as I had suggested in another post] a more significant ask than the "just change a few core utilities" that some people seem to be assuming it would be. It also means that making a "system python" does not remove the need to distribute the largish subset of python *libraries* that they distribute, because even when these libraries aren't used by key OS utilities, they are still used by packaged applications. [this, in turn, means we don't want the user's default python environment to stand entirely separate from the system python, or we'll start getting complaints along the lines of "I apt-get installed numpy, why can't I import it in my python interpreter?" - particularly from users who don't really care if it runs a couple versions behind] ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/F7YNRI7LWU2VLS6AU6PQZUHHA3M5WFQJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?
On Wed, 24 Feb 2021 at 13:12, Antoine Pitrou wrote: > > On Wed, 24 Feb 2021 13:47:40 +0100 > Stéfane Fermigier wrote: > > The 3rd solution is probably the best of the 3, but the sharing mechanism > > still needs to be specified (and, if needed, implemented) properly. > > I wouldn't want to repeat myself too often, but conda and conda-based > distributions already have sharing through hardlinks (or, on Windows, > whatever is available) baked-in, assuming you install your software > from conda packages. > > That also applies to non-Python packages, and to python itself (which > is just a package like any other). I'm not sure conda solves the problem of *application* distribution, though, so I think it's addressing a different problem. Specifically, I don't think conda addresses the use case pipx is designed for. Although to be fair, this conversation has drifted *way* off the original topic. Going back to that, my view is that Python does not have a good solution to the "write your application in Python, and then distribute it" scenario. Shipping just the app to be run on an independently installed runtime results in the conflicting dependencies issue. Shipping the app with bundled dependencies is clumsy, mostly because no-one has developed tools to make it easier. It also misses opportunities for sharing libraries (reduced maintenance, less disk usage...). Shipping the app with a bundled interpreter and libraries is safest, but hard to do and even more expensive than the "bundled libraries" approach. I'd love to see better tools for this, but the community preferred approach seems to be "ship your app as a PyPI package with a console entry point" and that's the approach pipx supports. I don't use Linux much, and I'm definitely not familiar with Linux distribution tools, but from what I can gather Linux distributions have made the choices: 1. Write key operating system utilities in Python. 2. Share the Python interpreter and libraries. 3. Expose that Python interpreter as the *user's* default Python. IMO, the mistake is (3) - because the user wants to install Python packages, and not all packages are bundled by the distribution (or if they are, they aren't updated quickly enough for the user), users want to be able to install packages using Python tools. That risks introducing unexpected library versions and/or conflicts, which breaks the OS utilities, which expect their requirements to be respected (that's what the OS packaging tools do). Hindsight is way too easy here, but if distros had a "system Python" package that OS tools depend on, and which is reserved for *only* OS tools, and a "user Python" package that users could write their code against, we'd probably have had far fewer issues (and much less FUD about the "using sudo pip breaks your OS" advice). But it's likely way too late to argue for such a sweeping change. *Shrug* I'm not the person to ask here. My view is that I avoid using Python on Linux, because it's *way* too hard. I find it so much easier to work on Windows, where I can install Python easily for myself, and I don't have to fight with system package managers, or distribution-patched tools that don't work the way I expect. And honestly, on Windows, there's no "neglect of the system environment" to worry about - if you want to install Python, and use pip to install packages into that environment for shared use, it works fine. People (including me) use virtual environments for *convenience* on Windows, not because it's a requirement. Paul ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/J7U6525AZAW4P5ZYH5WLK5IDF6TCH73O/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?
On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier wrote:
> There is probably a clever way to reuse common packages (probably via clever
> symlinking) and reduce the footprint of these installations.
Ultimately the problem is that a general tool can't deal with
conflicts (except by raising an error). If application A depends on
lib==1.0 and application B depends on lib==2.0, you simply can't have
a (consistent) environment that supports both A and B. But that's the
rare case - 99% of the time, there are no conflicts. One env per app
is a safe, but heavy handed, approach. Managing environments manually
isn't exactly *hard*, but it's annoying manual work that pipx does an
excellent job of automating, so it's a disk space vs admin time
trade-off.
As far as I know, no-one has tried to work on the more complex option
of sharing things (pipx shares the copies of pip, setuptools and wheel
that are needed to support pipx itself, but doesn't extend that to
application dependencies). It would be a reasonable request for pipx
to look at, or for a new tool, but I suspect the cost of implementing
it simply outweighs the benefit ("disk space is cheap" :-))
Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/4EAGDI66MG4WPNCHCOVHT4VHBWQUNRNP/
Code of Conduct: http://python.org/psf/codeofconduct/
