Re: [slurm-users] Nodelist dependent environment setup ?

[Bas van der Vlies]

we have solve this with a symbolic link to the optimized libraries on 
the node, eg:

 * /sw/arch ---> sara/eb/AVX2
 * /sw/arch -->  sara/eb/AVX512

and make use of '/sara/sw' in our module files.

regards


On 04/04/2020 19:36, Sajid Ali wrote:

Answering my own question so that it may serve as a reference:

This can be done by setting up a task prolog as shown below :
```
#!/bin/sh
#
# Sample TaskProlog script that will print a batch job's
# job ID and node list to the job's stdout.
# Also sets the modulepath based on nodelist
#

if [ X"$SLURM_STEP_ID" = "X" -a X"$SLURM_PROCID" = "X"0 ]
then
         echo "print =="
         echo "print SLURM_JOB_ID = $SLURM_JOB_ID"
         echo "print SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST"
         echo "print =="
fi
short_list=${SLURM_JOB_NODELIST##worker}
if [ $short_list == "01" ]
then
         echo "export MODULEPATH=/home/path1"
fi
if [ $short_list == "02" ]
then
         echo "export MODULEPATH=/home/path2"
fi
```

Note:  This was tested on my laptop using a set of docker containers 
using this configuration : https://github.com/SciDAS/slurm-in-docker .




--
Sajid Ali | PhD Candidate
Applied Physics
Northwestern University
s-sajid-ali.github.io 


--
--
Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098 XG
Amsterdam
| T +31 (0) 20 800 1300 | bas.vandervl...@surfsara.nl | www.surfsara.nl |

Re: [slurm-users] Nodelist dependent environment setup ?

[Sajid Ali]

Answering my own question so that it may serve as a reference:

This can be done by setting up a task prolog as shown below :
```
#!/bin/sh
#
# Sample TaskProlog script that will print a batch job's
# job ID and node list to the job's stdout.
# Also sets the modulepath based on nodelist
#

if [ X"$SLURM_STEP_ID" = "X" -a X"$SLURM_PROCID" = "X"0 ]
then
echo "print =="
echo "print SLURM_JOB_ID = $SLURM_JOB_ID"
echo "print SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST"
echo "print =="
fi
short_list=${SLURM_JOB_NODELIST##worker}
if [ $short_list == "01" ]
then
echo "export MODULEPATH=/home/path1"
fi
if [ $short_list == "02" ]
then
echo "export MODULEPATH=/home/path2"
fi
```

Note:  This was tested on my laptop using a set of docker containers using
this configuration : https://github.com/SciDAS/slurm-in-docker .



-- 
Sajid Ali | PhD Candidate
Applied Physics
Northwestern University
s-sajid-ali.github.io

Re: [slurm-users] Nodelist dependent environment setup ?

[Sajid Ali]

Hi Ole,

Thanks a lot for sharing the resource!

Our biggest concern is the case where a user asks for 2 nodes and one of
those is a Cascade Lake node and the other one is a Haswell node. At this
point, the environment have modules that work on both, hence my preference
for reading the slurm nodelist to set the appropriate modulepath.

PS: I'm trying to do the same thing with spack!


-- 
Sajid Ali | PhD Candidate
Applied Physics
Northwestern University
s-sajid-ali.github.io

[slurm-users] Nodelist dependent environment setup ?

[Sajid Ali]

Hi Slurm-developers/users,

If I install multiple versions of a software library, each optimized for a
different partition based on CPU architecture, how would I automatically
load the version of software based on the nodes allocated to the job ?

Ideally I'd want to store the modules for each cpu arch at a different
location and set the `MODULEPATH` at job startup to the lowest CPU-arch.
(since our cluster only runs Intel CPU's, the software for lower arch would
run on a higher arch).

Could someone point out how Slurm initializes the job environment at
startup ? Based on this I'm hoping that it will be a relatively simple task
to add a small script to determine nodelist and prepend the moduepath env
var.

Alternatively, if someone could point out how they do this at their sites
it would be useful as well.

Thanks in advance for the advice!

--
Sajid Ali | PhD Candidate
Applied Physics
Northwestern University
s-sajid-ali.github.io