Hello Brian, thank you for your answer.
Actually, you are not allowed to install things in your home on
computecanada, this is why you need to install everything in a
virtualenv with pip install. Also, you have to install each virtualenv
in $SLURM_TMDIR which is the local drive of the node, because
everything else is slow, so I think I cannot share homes.
Actually I succeeded at installing different virtualenvs on different
nodes using a script for each worker that creates a local virtualenv,
installs ray on it, and connects to the ray server running in the
virtualenv of the head node (I mean the primary node, yes). I just
call these scripts with srun. However, for some reason, the workers
seem to connect fine to the server but are detected as dead after a
while: https://groups.google.com/forum/#!topic/ray-dev/INB_zVS5PWY
Yann
Brian Andrus <toomuc...@gmail.com> a écrit :
I suspect when you say "head node" you mean the primary node from
the nodes your were allocated.
Normally, when you use pip as a user, it installs in your home
directory. Are you certain all your nodes share the same homes?
If they are merely synched, that would not be the same. Not actually
sharing homes could be the cause.
Brian Andrus
On 11/17/2019 11:24 AM, Yann Bouteiller wrote:
Hello,
I am trying to do this on computecanada, which is managed by slurm:
https://ray.readthedocs.io/en/latest/deploying-on-slurm.html
However, on computecanada, you cannot install things on nodes
before the job has started, and you can only install things in a
python virtualenv once the job has started.
I can do:
```
module load python/3.7.4
source venv/bin/activate
pip install ray
```
in the bash script before calling everything else, but apparently
this will only create-activate the virtualenv and install ray on
the head node, but not on the remote nodes, so calling
```
srun --nodes=1 --ntasks=1 -w $node1 ray start --block --head
--redis-port=6379 --redis-password=$redis_password & # Starting the
head
```
will succeed, but later calling
```
for (( i=1; i<=$worker_num; i++ ))
do
node2=${nodes_array[$i]}
srun --export=ALL --nodes=1 --ntasks=1 -w $node2 ray start
--block --address=$ip_head --redis-pass$
sleep 5
done
```
will produce the following error:
```
slurmstepd: error: execve(): ray: No such file or directory
srun: error: cdr768: task 0: Exited with exit code 2
srun: Terminating job step 31218604.3
[2]+ Exit 2 srun --export=ALL --nodes=1
--ntasks=1 -w $node2 ray start --block --address=$ip_head
--redis-password=$redis_password
```
How can I tackle this issue, please? I am a beginner with slurm so
I am not sure what is the problem here. Here is my whole sbatch
script:
```
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --cpus-per-task=5
#SBATCH --mem-per-cpu=1000M
#SBATCH --nodes=3
#SBATCH --tasks-per-node 1
worker_num=2 # Must be one less that the total number of nodes
nodes=$(scontrol show hostnames $SLURM_JOB_NODELIST) # Getting the
node names
nodes_array=( $nodes )
module load python/3.7.4
source venv/bin/activate
pip install ray
node1=${nodes_array[0]}
ip_prefix=$(srun --nodes=1 --ntasks=1 -w $node1 hostname
--ip-address) # Making address
suffix=':6379'
ip_head=$ip_prefix$suffix
redis_password=$(uuidgen)
export ip_head # Exporting for latter access by trainer.py
srun --nodes=1 --ntasks=1 -w $node1 ray start --block --head
--redis-port=6379 --redis-password=$redis_password & # Starting the
head
sleep 5
for (( i=1; i<=$worker_num; i++ ))
do
node2=${nodes_array[$i]}
srun --export=ALL --nodes=1 --ntasks=1 -w $node2 ray start
--block --address=$ip_head --redis-password=$redis_password & #
Starting the workers
sleep 5
done
python -u trainer.py $redis_password 15 # Pass the total number of
allocated CPUs
```
---
Regards,
Yann
