[Cloud] Re: Kubernetes-based jobs engine -- how to use Python virtual environments?

I second that. I used to not worry too much about doing it right, but eventually got bitten by my sloppyness. The problem seems to be that the paths which get embedded in your venv executables will be different, which leads to hard-to-debug problems. > On Apr 6, 2022, at 10:12 AM,

[Cloud] Re: Kubernetes-based jobs engine -- how to use Python virtual environments?

In general, if you are using a Python virtual environment on k8s, you should only modify it from a shell in a matching k8s container. The easiest way to do this is with `webservice --backend=kubernetes python3.9 shell` (or whatever version you are using). If the python version you're using matches

[Cloud] Re: Kubernetes-based jobs engine -- how to use Python virtual environments?

You can use the one used for k8s on the bastion too, the thing is that the k8s environment expects it to be in a specific path, that is `$HOME/pyenv/`. Now, that might not work if the python version in the bastion is too different from the one the k8s job has, so you might have to have two

[Cloud] Re: Kubernetes-based jobs engine -- how to use Python virtual environments?

Hello everyone, Thanks for the suggestion. Does this mean that I have to maintain 2 virtual environments (one for bastion and the other one for k8s pod my code will actually run in)? Martin po 4. 4. 2022 v 10:06 odesílatel Arturo Borrero Gonzalez < aborr...@wikimedia.org> napsal: > > > On

[Cloud] Re: Kubernetes-based jobs engine -- how to use Python virtual environments?

On 4/2/22 14:53, Martin Urbanec wrote: Hello, I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily. I do that by calling jupyter nbconvert --to html

[Cloud] Re: Kubernetes-based jobs engine -- how to use Python virtual environments?

Hi Martin, I usually write a bash script to set up the venv and run it under toolforge-jobs. I've put a skeleton version at P24028 . You may be able to use the webservice shell since the images are similar. Regards, JJ On Sun, Apr 3, 2022 at 8:14 PM