[OpenStack-Infra] Should we use pip install --user ... in CI?

[Sorin Sbarnea]

As I have a few pip tickets that I am closely monitoring two new interesting 
commends came at:
https://github.com/pypa/pip/issues/1668#issuecomment-600819050


In the past I used --user to install deps w/o messing the system package during 
the last year I went more and more often into very weird issues related to use 
of --user.

Basically I ended with the conclusion that --user approach is not much better 
than messing the system and only a real virtualenv is reliable.


I know that we already had talks about migrating zuul-jobs to install tools 
inside virtualenvs but this did not happen yet.


I was told that the idea was to install each tool into each own virtulaenv in 
order to better isolate it from conflicts with others but I have some concerns 
regarding :

a) making very hard or even impossible to use multiple tools in the same 
script, as they would exist in different envs.
b) extra footprint on disk and install time.

Do we really expect to have conflicts between lets say `tox` or `bindep`? I 
really doubt.

This is why I would like to propose a default shared virtualenv to be used but 
allowing user to override the venv name.

If we would have a "default" virtualenv at a specific location, it would be 
very easy to activate it at the start of each script.


There are lots of places in tripleo* where pip --user is used and before trying 
to propose any changes there, I would like to know where are we going towards 
with zuul-jobs as I would like to avoid divergence in behaviours.


Thanks
Sorin Sbarnea


___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Should we use pip install --user ... in CI?

[Jeremy Stanley]

On 2020-03-19 09:26:23 + (+), Sorin Sbarnea wrote:
[...]
> I know that we already had talks about migrating zuul-jobs to
> install tools inside virtualenvs but this did not happen yet.

It actually has, tox is already available /usr/tox-env/bin/tox on
our systems, we just haven't yet merged the change to stop
installing a copy of tox into the system context (but that's
coming). There's also a /usr/bindep-env/bin/bindep, a
/usr/os-testr-env/ostestr and a /usr/glean/bin/glean (that last one
could stand to become consistent with the others, I suppose).

> I was told that the idea was to install each tool into each own
> virtulaenv in order to better isolate it from conflicts with
> others but I have some concerns regarding :
> 
> a) making very hard or even impossible to use multiple tools in
> the same script, as they would exist in different envs.

We're talking about installing Python-based utilities into dedicated
venvs. What you note is a concern for Python libraries. When is the
last time you "imported" tox in a Python script? How about bindep?
Remember that tox in one venv can call bindep from another venv just
fine because those are being treated as command-line tools not
Python modules (it's how I run them together on my own workstation
even). Things which need to import each other should of course be
installed into the same venv, that's just common sense.

> b) extra footprint on disk and install time.
[...]

We're not planning on pre-installing more than a handful. Our node
images are around 9GB in size, most of which is pre-cached data (Git
repositories), and the venvs I listed account for 24KB of excess
copies of files. That's something like a quarter of one percent of
the image size you're concerned about optimizing.

> There are lots of places in tripleo* where pip --user is used and
> before trying to propose any changes there, I would like to know
> where are we going towards with zuul-jobs as I would like to avoid
> divergence in behaviours.

If TripleO is installing things at job runtime, that seems like a
different case than whatever we bake into our node images, and so
doing it a different way is probably fine?
-- 
Jeremy Stanley


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra