[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Steven D'Aprano]

On Wed, Feb 24, 2021 at 07:59:55PM -0800, Christopher Barker wrote:

> I used to do that — for years. But it really did cause problems.
> 
> The trick is that you have, say, your 57 apps all working. Then you need to
> update a package for one. As soon as you update, you have to go test your
> 57 apps, and if one of them is broken, you have to figure out how to deal
> with it.
> 
> Now you have 52 apps running in the main environment, and 5 running in
> their own... and you are on your way to an even harder to manage system.

I don't get it. How is it "even harder" to manage?

The five apps you have isolated are, well, isolated. And for the rest, 
you've gone down from 57 apps to 52, so there's less complexity and 
fewer dependencies, so it should be easier, not harder, to manage.

Now there is clearly one sense that it is harder to manage: updates. If 
one of the common dependencies needs updating, then you have to update 
it six times: once each for the five isolated apps, and once for the 
non-isolated apps. So that makes it harder to manage; but I guess that's 
not what you meant.

So if you extrapolate to the point that all 57 apps are isolated, what 
you save in potential-or-actual conflicts you lose in updatings. Whether 
that makes it worthwhile, I think, depends on how often you expect to be 
updating versus how often you expect to be introducing new conflicts.

Clearly there are cases where you have, say, a rapidly changing app with 
lots of dependencies that are consistently conflicting with other apps; 
or you have a legacy app that needs a frozen, stable environment 
(possibly even including the OS!). In both of these extreme cases 
isolating the app makes great sense.

But I'm not convinced that isolation makes sense every time I start to 
write a 300 line script, let alone a 30 line one.

I guess that's what annoys me about verts -- it isn't that I don't see 
their usefulness. But it seems to me, rightly or wrongly, that a 
power feature which is mostly of benefit to quite big and/or complex 
development tasks is being pushed as the One True Way that everyone must 
use, always.


-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WERN55HGV6N2CUWYB7V4SSJL5YF2HYRU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Wes Turner]

Is there a tool that (1) detects import name collisions; and (2) attempts
to read package metadata and package file checksums (maybe from the ZIP
'manifest')?

In order to:

- troubleshoot module shadowing issues
  - $PATH
  - sys.path
- `python -m site`
  - incomplete and overlapping uninstallations:

pip install a
pip install a_modified # pip uninstall a?
pip install pdbpp
pip uninstall a_modified
ls -altr "${site-packages[*]}"
strace -e trace=file python -c 'import pdb'



When shouldn't site customizations be added to the site module?
https://docs.python.org/3/library/site.html

When should customizations be built into the build instead of a runtime
conditional?


On Sat, Feb 27, 2021, 23:12 Nick Coghlan  wrote:

> On Wed, 24 Feb 2021 at 10:49, Random832  wrote:
> >
> > I was reading a discussion thread <
> https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about
> various issues with the Debian packaged version of Python, and the
> following statement stood out for me as shocking:
> >
> > Christian Heimes wrote:
> > > Core dev and PyPA has spent a lot of effort in promoting venv because
> we don't want users to break their operating system with sudo pip install.
> >
> > I don't think sudo pip install should break the operating system. And I
> think if it does, that problem should be solved rather than merely advising
> users against using it. And why is it, anyway, that distributions whose
> package managers can't coexist with pip-installed packages don't ever seem
> to get the same amount of flak for "damaging python's brand" as Debian is
> getting from some of the people in the discussion thread? Why is it that
> this community is resigned to recommending a workaround when distributions
> decide the site-packages directory belongs to their package manager rather
> than pip, instead of bringing the same amount of fiery condemnation of that
> practice as we apparently have for *checks notes* splitting parts of the
> stdlib into optional packages? Why demand that pip be present if we're not
> going to demand that it works properly?
>
> The reason venv is promoted as heavily as it is is because it's the
> only advice that can be given that is consistently correct regardless
> of the operating system the user is running locally, whereas safely
> using a system-wide Python installation varies a lot depending on
> whether you're on Windows, Mac OS X, or Linux (let alone some other
> platform outside the big 3 desktop clients).
>
> conda is also popular for the same reason: while the instructions for
> installing conda in the first place are OS-dependent, once it is up
> and running you can use consistent platform independent conda commands
> rather than having to caveat all your documentation with
> platform-specific instructions.
>
> Apple moved all of their dynamic language interpreter implementations
> to inaccessible-by-default locations so Mac OS X users would stop
> using them to run their own code.
>
> Alongside that, we *have* worked with the Linux distro vendors to help
> make "sudo pip install" safe (e.g [1]), but that only helps if a user
> is running a new enough version of a distro that has participated in
> that work.
>
> However, while the option of running "platform native" environments
> will never go away, and work will continue to make it less error
> prone, the level of knowledge of your specific OS's idiosyncrasies
> that it requires is almost certainly going to remain too high for it
> to ever again become the default recommendation that it used to be.
>
> Cheers,
> Nick.
>
> [1] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe (Note:
> this change mitigated some aspects of the problem in a way similar to
> what Debian does, but still doesn't solve it completely, as custom
> Python builds may still make arbitrary changes)
>
> P.S. "But what about user site-packages?" you ask. Until relatively
> recently, Debian didn't put the user's local bin directory on the
> system path by default, so commands provided by user level package
> installs didn't work without the user adjusting their PATH. The
> CPython Windows installer also doesn't adjust PATH by default (for
> good reasons). And unlike a venv, "python -m" doesn't let you ensure
> that the code executed is the version installed in user site-packages
> - it could be coming from a directory earlier in sys.path.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list -- python-...@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-...@python.org/message/RDLEH6DUF57UB6U4HNL2QRVAJY4KDSSJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send 

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Paul Moore]

On Thu, 25 Feb 2021 at 19:22, Mike Miller  wrote:
> Mr. Random had an interesting point to start this thread, that over-reliance 
> on
> venvs may have slowed fixes and improvements on the standard tools and
> distributions.  I suspect there is some truth to the assertion.

Arguably, your claim that using your main Python interpreter for
everything "almost never" causes you problems would imply that there's
no real *need* for fixes and improvements to handle that situation, so
work on supporting virtual environments helps people who prefer them,
and harms no-one.

I suspect the truth is somewhere between the two.

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IHZOOMX3WNGJL74GQKOENQ6KGKWHU3V6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Mike Miller]

On 2021-02-24 19:59, Christopher Barker wrote:

I used to do that — for years. But it really did cause problems.

The trick is that you have, say, your 57 apps all working. Then you need to 
update a package for one. As soon as you update, you have to go test your 57 
apps, and if one of them is broken, you have to figure out how to deal with it.



Every few years I revert whatever package upgrade that caused the issue.  Which 
brings the house back in order.  Not a substantial problem.



Now you have 52 apps running in the main environment, and 5 running in their 
own... and you are on your way to an even harder to manage system.



Almost twenty years of daily python use and this situation has never happened 
here.  Sure, if one wants to spend time and gigs of storage to guard against 
exceptional situations that's their decision.


My post was simply to push back on the idea that this is required for the 
average developer.  It isn't, as mentioned I have but a single venv for a big 
work app.  I find it less of a burden to simply fix issues as they come up, 
which is almost never.  Approximately ten minutes per year, sometimes zero.


Mr. Random had an interesting point to start this thread, that over-reliance on 
venvs may have slowed fixes and improvements on the standard tools and 
distributions.  I suspect there is some truth to the assertion.


-Mike
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LPR2VTEQR3D3SHD3LIAIRO7MGLEPT2SJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christopher Barker]

>  > I have currently 57 apps installed via pipx on my laptop, and the 57
>  > environments take almost 1 GB already.


That is a lot! But give conda a try: conda uses hard links, so no wasted
space when packages are the same.

I never understood the fear around version conflicts.


I don’t know that it’s fear. But some sod use a Lot of packages, and
version conflicts do get ugly.

Rather than installing every package this way, why not wait until a
> conflict
> actually occurs?


I used to do that — for years. But it really did cause problems.

The trick is that you have, say, your 57 apps all working. Then you need to
update a package for one. As soon as you update, you have to go test your
57 apps, and if one of them is broken, you have to figure out how to deal
with it.

Now you have 52 apps running in the main environment, and 5 running in
their own... and you are on your way to an even harder to manage system.

The nice thing about environments is that once something is working, you’re
not going to mess it up when working on something else. The stuff not being
actively maintained can just hum along.

-CHB

>
> Personally, I rarely have such conflicts, maybe every few years or so.
> When it
> happens, I fix them by uninstalling the offender and putting the more
> difficult
> one into the venv or pipx.  Right now I only have one, a giant app from
> work
> that uses pipenv, and it's fine.
>
>
> Now what about sudo and all that?  Well, I used it in the old days because
> that's what the instructions said.  But, to be honest, it never made any
> sense.


I agree — keep it all on user land.

-CHB
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7SHGFTYRRLMRYUQGZ3W6AAF2KTS3RUM7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Mike Miller]

On 2021-02-24 02:52, Stéfane Fermigier wrote:
> I have currently 57 apps installed via pipx on my laptop, and the 57
> environments take almost 1 GB already.


I never understood the fear around version conflicts.  Perhaps it has to do with 
the decline of sys-admin skills over the years?  So, the strategy above feels 
like swatting a fly with a sledgehammer to me.  Same as with a venv for every 
app. :-)


Rather than installing every package this way, why not wait until a conflict 
actually occurs?


Personally, I rarely have such conflicts, maybe every few years or so.  When it 
happens, I fix them by uninstalling the offender and putting the more difficult 
one into the venv or pipx.  Right now I only have one, a giant app from work 
that uses pipenv, and it's fine.



Now what about sudo and all that?  Well, I used it in the old days because 
that's what the instructions said.  But, to be honest, it never made any sense. 
I haven't shared a computer in decades, and when we did we used NFS for common 
tools, so it never saved any disk space.


Pip (and easy_install?) dragged their feet for years to properly support user 
installs (should have been default) but once they did I didn't look back.  I 
dump almost all packages to user, which gets cleaned up every other year when 
the distro moves to the next version.  The strategy has been working well for a 
long time.


So, --user works at the low end, and containers for the high end.  Honestly, I 
don't have much of a use case any longer for venvs.


-Mike
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J35NFRMAVAMQKWOLILMSL5AT3VGG6J5F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Random832]

I accidentally removed Python-Ideas from the recipient list while posting this 
reply.

On Wed, Feb 24, 2021, at 09:16, Random832 wrote:
> On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
> > Separate directories don't prevent clashes and system breakage. But they
> > provide an easy way to *recover* from a broken system.
> 
> I think it could be turned into a way to prevent them by A) having 
> site-packages always take precedence over dist-packages [i believe this 
> is already the case] in normal usage and B) providing an option to the 
> interpreter, used by system scripts, to exclude site-packages entirely 
> from the path.
> 
> Basically, site-packages would effectively be layered on top of "Lib + 
> dist-packages" in a similar way to how a venv is layered on top of the 
> main python installation - the inverse of the suggestion someone else 
> in the thread made for the system python to be a venv. This wouldn't 
> *exactly* be a venv because it wouldn't imply the other things that 
> entering a venv does such as "python" [and script names such as pip] 
> being an alias for the correct version of python, but it would provide 
> the same kind of one-way isolation, whereby the "system environment" 
> can influence the "normal environment" and not vice-versa, in the same 
> way that packages installed in the main installation affect a venv 
> [unless system-site-packages is disabled] but the venv obviously has no 
> effect on the main installation.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IQA6RHY24WGVVGRJT67CFABQVA5MMO3H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Paul Moore]

On Wed, 24 Feb 2021 at 13:12, Antoine Pitrou  wrote:
>
> On Wed, 24 Feb 2021 13:47:40 +0100
> Stéfane Fermigier  wrote:
> > The 3rd solution is probably the best of the 3, but the sharing mechanism
> > still needs to be specified (and, if needed, implemented) properly.
>
> I wouldn't want to repeat myself too often, but conda and conda-based
> distributions already have sharing through hardlinks (or, on Windows,
> whatever is available) baked-in, assuming you install your software
> from conda packages.
>
> That also applies to non-Python packages, and to python itself (which
> is just a package like any other).

I'm not sure conda solves the problem of *application* distribution,
though, so I think it's addressing a different problem. Specifically,
I don't think conda addresses the use case pipx is designed for.

Although to be fair, this conversation has drifted *way* off the
original topic. Going back to that, my view is that Python does not
have a good solution to the "write your application in Python, and
then distribute it" scenario. Shipping just the app to be run on an
independently installed runtime results in the conflicting
dependencies issue. Shipping the app with bundled dependencies is
clumsy, mostly because no-one has developed tools to make it easier.
It also misses opportunities for sharing libraries (reduced
maintenance, less disk usage...). Shipping the app with a bundled
interpreter and libraries is safest, but hard to do and even more
expensive than the "bundled libraries" approach.

I'd love to see better tools for this, but the community preferred
approach seems to be "ship your app as a PyPI package with a console
entry point" and that's the approach pipx supports.

I don't use Linux much, and I'm definitely not familiar with Linux
distribution tools, but from what I can gather Linux distributions
have made the choices:

1. Write key operating system utilities in Python.
2. Share the Python interpreter and libraries.
3. Expose that Python interpreter as the *user's* default Python.

IMO, the mistake is (3) - because the user wants to install Python
packages, and not all packages are bundled by the distribution (or if
they are, they aren't updated quickly enough for the user), users want
to be able to install packages using Python tools. That risks
introducing unexpected library versions and/or conflicts, which breaks
the OS utilities, which expect their requirements to be respected
(that's what the OS packaging tools do).

Hindsight is way too easy here, but if distros had a "system Python"
package that OS tools depend on, and which is reserved for *only* OS
tools, and a "user Python" package that users could write their code
against, we'd probably have had far fewer issues (and much less FUD
about the "using sudo pip breaks your OS" advice). But it's likely way
too late to argue for such a sweeping change.

*Shrug* I'm not the person to ask here. My view is that I avoid using
Python on Linux, because it's *way* too hard. I find it so much easier
to work on Windows, where I can install Python easily for myself, and
I don't have to fight with system package managers, or
distribution-patched tools that don't work the way I expect. And
honestly, on Windows, there's no "neglect of the system environment"
to worry about - if you want to install Python, and use pip to install
packages into that environment for shared use, it works fine. People
(including me) use virtual environments for *convenience* on Windows,
not because it's a requirement.

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J7U6525AZAW4P5ZYH5WLK5IDF6TCH73O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Antoine Pitrou]

On Wed, 24 Feb 2021 13:47:40 +0100
Stéfane Fermigier  wrote:
> On Wed, Feb 24, 2021 at 12:42 PM Paul Moore 
>  wrote:
> 
> > On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier 
> >  wrote:  
> > > There is probably a clever way to reuse common packages (probably via  
> > clever symlinking) and reduce the footprint of these installations.
> >
> > Ultimately the problem is that a general tool can't deal with
> > conflicts (except by raising an error). If application A depends on
> > lib==1.0 and application B depends on lib==2.0, you simply can't have
> > a (consistent) environment that supports both A and B. But that's the
> > rare case - 99% of the time, there are no conflicts. One env per app
> > is a safe, but heavy handed, approach. Managing environments manually
> > isn't exactly *hard*, but it's annoying manual work that pipx does an
> > excellent job of automating, so it's a disk space vs admin time
> > trade-off.
> >  
> 
> There are three ways to approach the question:
> 
> 1) Fully isolated envs. The safest option but uses the most space.
> 
> 2) Try to minimise the number of dependencies installed by interpreting the
> requirements specification in the looser way possible. This is both
> algorithmically hard (see
> https://hal.archives-ouvertes.fr/hal-00149566/document for instance, or the
> more recent https://hal.archives-ouvertes.fr/hal-03005932/document ) and
> risky, as you've noted.
> 
> 3) But the best way IMHO is to compute dependencies for each virtualenv
> independently from the others, but still share the packages, using some
> indirection mechanisms (hard links, symlinks or some Python-specific
> constructs) when the versions match exactly.
> 
> The 3rd solution is probably the best of the 3, but the sharing mechanism
> still needs to be specified (and, if needed, implemented) properly.

I wouldn't want to repeat myself too often, but conda and conda-based
distributions already have sharing through hardlinks (or, on Windows,
whatever is available) baked-in, assuming you install your software
from conda packages.

That also applies to non-Python packages, and to python itself (which
is just a package like any other).

Regards

Antoine.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SIO47UNLRARUCICNOVO75P3BKFE43QK2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 1:47 PM Stéfane Fermigier  wrote:

>
> So IMHO the best way to implement solution 3 would be by using some
> variant of the approach popularized by Nix (repository of immutable
> packages + links to each virtualenv).
>

Another benefit of this kind of approach, besides sparing disk space, could
be similar improvements in terms of installation time (and even bigger
improvements when reinstalling a package, which happens all the time when
developing).

  S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UO2AZHKZUNEQ34SGNKCR5RTD4GYI2SHF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 12:42 PM Paul Moore  wrote:

> On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier  wrote:
> > There is probably a clever way to reuse common packages (probably via
> clever symlinking) and reduce the footprint of these installations.
>
> Ultimately the problem is that a general tool can't deal with
> conflicts (except by raising an error). If application A depends on
> lib==1.0 and application B depends on lib==2.0, you simply can't have
> a (consistent) environment that supports both A and B. But that's the
> rare case - 99% of the time, there are no conflicts. One env per app
> is a safe, but heavy handed, approach. Managing environments manually
> isn't exactly *hard*, but it's annoying manual work that pipx does an
> excellent job of automating, so it's a disk space vs admin time
> trade-off.
>

There are three ways to approach the question:

1) Fully isolated envs. The safest option but uses the most space.

2) Try to minimise the number of dependencies installed by interpreting the
requirements specification in the looser way possible. This is both
algorithmically hard (see
https://hal.archives-ouvertes.fr/hal-00149566/document for instance, or the
more recent https://hal.archives-ouvertes.fr/hal-03005932/document ) and
risky, as you've noted.

3) But the best way IMHO is to compute dependencies for each virtualenv
independently from the others, but still share the packages, using some
indirection mechanisms (hard links, symlinks or some Python-specific
constructs) when the versions match exactly.

The 3rd solution is probably the best of the 3, but the sharing mechanism
still needs to be specified (and, if needed, implemented) properly.

I've tried Christian's suggestions of using rdfind on my pipx installation,
and it claims to reduce the footprint by 30% (nice, but less than I
expected. This would however scale better with the number of installed
packages).

I'm not sure this would be practical in reality, OTOH, because I think
there is a serious risk of breakage each time I would upgrade one of the
packages (via 'pipx upgrade-all' for instance).

So IMHO the best way to implement solution 3 would be by using some variant
of the approach popularized by Nix (repository of immutable packages +
links to each virtualenv).

  S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6GV3EPYU2SW2QKAKLXZCG3JLNCVSQZWG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Paul Moore]

On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier  wrote:
> There is probably a clever way to reuse common packages (probably via clever 
> symlinking) and reduce the footprint of these installations.

Ultimately the problem is that a general tool can't deal with
conflicts (except by raising an error). If application A depends on
lib==1.0 and application B depends on lib==2.0, you simply can't have
a (consistent) environment that supports both A and B. But that's the
rare case - 99% of the time, there are no conflicts. One env per app
is a safe, but heavy handed, approach. Managing environments manually
isn't exactly *hard*, but it's annoying manual work that pipx does an
excellent job of automating, so it's a disk space vs admin time
trade-off.

As far as I know, no-one has tried to work on the more complex option
of sharing things (pipx shares the copies of pip, setuptools and wheel
that are needed to support pipx itself, but doesn't extend that to
application dependencies). It would be a reasonable request for pipx
to look at, or for a new tool, but I suspect the cost of implementing
it simply outweighs the benefit ("disk space is cheap" :-))

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4EAGDI66MG4WPNCHCOVHT4VHBWQUNRNP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 11.52, Stéfane Fermigier wrote:
> I love pipx and I'm glad it exists at this point because it make 
> 
> The main issue is that each virtualenv takes space, lots of space.
> 
> I have currently 57 apps installed via pipx on my laptop, and the 57
> environments take almost 1 GB already.
> 
>  ~  cd .local/pipx/venvs/
>  ~/.l/p/venvs  ls
> abilian-tools/  concentration/  gitlabber/      pygount/        sphinx/
> ansible/        cookiecutter/   httpie/         pyinfra/        tentakel/
> assertize/      cruft/          isort/          pylint/         tlv/
> autoflake/      cython/         jupyterlab/     pyre-check/     towncrier/
> black/          dephell/        lektor/         pytype/         tox/
> borgbackup/     docformatter/   md2pdf/         pyupgrade/      twine/
> borgmatic/      flake8/         medikit/        radon/          virtualenv/
> bpytop/         flit/           mypy/           re-ver/         virtualfish/
> check-manifest/ flynt/          nox/            sailboat/       vulture/
> clone-github/   gh-clone/       pdoc3/          salvo/
> cloneall/       ghtop/          pdocs/          shed/
> com2ann/        gitchangelog/   pybetter/       sixer/
>  ~/.l/p/venvs  du -sh .
> 990M.
>  ~/.l/p/venvs  ls | wc
>       57      57     475
> 
> There is probably a clever way to reuse common packages (probably via
> clever symlinking) and reduce the footprint of these installations. 

There are tools like https://rdfind.pauldreik.se/rdfind.1.html that
create hard links to deduplicate files. Some files systems have
deduplicated baked in, too.

Christian
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CO6GV2CRDKBJMLE7DZVVQ4AMIPSKPMCJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 11:43 AM Henk-Jaap Wagenaar <
wagenaarhenkj...@gmail.com> wrote:

>
>>
> I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been
> quite happy with that, and it basically does what Jonathan does manually:
> clone the github repo and build python from scratch.
>

Also using pyenv myself, on platforms where either, or both, new and old
version of Python are not available (so on Ubuntu I'm sticking with Anthony
Sottile's deadsnakes.)

But I'm not using it to manage virtual envs. For my own development
projects, I'm using VirtualFish since I've switched to Fish as my main
shell, but I was happy with VirtualenvWrapper when I was still using bash
or zsh. I can't live without VirtualFish's autoactivation plugin (which
activate a given virtualenv each time you enter your project). There was
something similar for Zsh though I don't remember which. Sometimes I forget
that I'm not in the right folder, or that I didn't create the virtualenv,
or that I removed it in the case of an old project, and I end up polluting
my default Python install after a 'pip install ...'. No big deal but always
a bit unpleasant.

And as I wrote before, I'm now using pipx to install the apps that I use.

This works not so bad (except for some quirks now and then) but it took
quite some time to select the right (for me) tools, to set up the proper
config files on my computer, and it can be annoying when working on a
server which has not been set up the same way. It also takes some time to
explain to collaborators, especially interns or juniors.

So obviously there is room for improvement, especially in terms of
simplicity for beginners.

  S.

___
> Python-Dev mailing list -- python-...@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-...@python.org/message/OZRHP3Q6EZNCPCTEEXLZGOCTGY6B7YFW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/X7GSLJIICBL5TJVUG32XGBGVLPLIHKIW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Stéfane Fermigier]

I love pipx and I'm glad it exists at this point because it make

The main issue is that each virtualenv takes space, lots of space.

I have currently 57 apps installed via pipx on my laptop, and the 57
environments take almost 1 GB already.

 ~  cd .local/pipx/venvs/
 ~/.l/p/venvs  ls
abilian-tools/  concentration/  gitlabber/  pygount/sphinx/
ansible/cookiecutter/   httpie/ pyinfra/tentakel/
assertize/  cruft/  isort/  pylint/ tlv/
autoflake/  cython/ jupyterlab/ pyre-check/ towncrier/
black/  dephell/lektor/ pytype/ tox/
borgbackup/ docformatter/   md2pdf/ pyupgrade/  twine/
borgmatic/  flake8/ medikit/radon/  virtualenv/
bpytop/ flit/   mypy/   re-ver/ virtualfish/
check-manifest/ flynt/  nox/sailboat/   vulture/
clone-github/   gh-clone/   pdoc3/  salvo/
cloneall/   ghtop/  pdocs/  shed/
com2ann/gitchangelog/   pybetter/   sixer/
 ~/.l/p/venvs  du -sh .
990M .
 ~/.l/p/venvs  ls | wc
  57  57 475

There is probably a clever way to reuse common packages (probably via
clever symlinking) and reduce the footprint of these installations.

Still, I'm glad that pipx exists as it is now, and that it has been
packaged on Ubuntu 20.04 and later (and probably other distros as well).

Having pipx (or something similar) installed by the distro, and the distro
focussed on packaging only the packages that are needed for its own sake,
means that we could go past the controversies between the Python community
and the Debian (or other distros) packagers community, which are based on
different goals and assumption, such as this one:
https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e

  S.

On Wed, Feb 24, 2021 at 2:28 AM Paul Bryan  wrote:

> I think it's a classic case of dependency hell.
>
> OS packagers are rebundling Python packages as OS packages and expressing
> their own OS-package dependency graphs. Then, you sudo pip install
> something that has a conflicting dependency, it bypasses OS packaging, and
> *boom*.
>
> I find tools like pipx  go a long
> way to solve this, as they install a Python package and all of its
> dependencies in its own venv. This is great for Python apps, and (kinda)
> treats them like apps on platforms like Android, where all app dependencies
> are bundled and isolated from others.
>
> I think it would great if OS vendors did something similar to pipx for
> Python-based apps: bundle the app and all of its dependencies into its own
> venv.
>
>
> On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
>
> I was reading a discussion thread <
> https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about
> various issues with the Debian packaged version of Python, and the
> following statement stood out for me as shocking:
>
> Christian Heimes wrote:
>
> Core dev and PyPA has spent a lot of effort in promoting venv because we
> don't want users to break their operating system with sudo pip install.
>
>
> I don't think sudo pip install should break the operating system. And I
> think if it does, that problem should be solved rather than merely advising
> users against using it. And why is it, anyway, that distributions whose
> package managers can't coexist with pip-installed packages don't ever seem
> to get the same amount of flak for "damaging python's brand" as Debian is
> getting from some of the people in the discussion thread? Why is it that
> this community is resigned to recommending a workaround when distributions
> decide the site-packages directory belongs to their package manager rather
> than pip, instead of bringing the same amount of fiery condemnation of that
> practice as we apparently have for *checks notes* splitting parts of the
> stdlib into optional packages? Why demand that pip be present if we're not
> going to demand that it works properly?
>
> I think that installing packages into the actual python installation, both
> via distribution packaging tools and pip [and using both simultaneously -
> the Debian model of separated dist-packages and site-packages folders seems
> like a reasonable solution to this problem] can and should be a supported
> paradigm, and that virtual environments [or more extreme measures such as
> shipping an entire python installation as part of an application's
> deployment] should ideally be reserved for the rare corner cases where that
> doesn't work for some reason.
>
> How is it that virtual environments have become so indispensable, that
> no-one considers installing libraries centrally to be a viable model
> anymore? Are library maintainers making breaking changes too frequently,
> reasoning that if someone needs the old version they can just venv it? Is
> there some other cause?
> _

[Python-ideas] Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Henk-Jaap Wagenaar]

On Wed, 24 Feb 2021 at 10:18, Antoine Pitrou  wrote:

> On Tue, 23 Feb 2021 20:29:52 -0500
> Jonathan Goble  wrote:
> >
> > I can't speak for distributors or maintainers [1], but I can speak for
> > myself as a user. I run Debian testing (currently bullseye as that is
> > preparing for release) as my daily OS on my personal laptop, used for
> > personal matters and school assignments (I'm a university computer
> > science student in my senior year).
> >
> > I don't use the system Python for anything of my own, whether it's a
> > school assignment or a personal project, precisely because I don't
> > want to risk screwing something up. Rather, I maintain a clone/fork of
> > the official CPython GitHub repo, and periodically build from source
> > and `make altinstall` into `~/.local/`.
>
> For the record, instead of building Python by hand, you could use a
> distribution such as Anaconda or the community-maintained conda-forge.
>
> Regards
>
> Antoine.
>
>
I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been
quite happy with that, and it basically does what Jonathan does manually:
clone the github repo and build python from scratch.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OZRHP3Q6EZNCPCTEEXLZGOCTGY6B7YFW/
Code of Conduct: http://python.org/psf/codeofconduct/