Hi everyone, I'm trying to figure out a way to set environment variables into the environment that a Slurm job runs in, depending on the characteristics of the job.
Here's the background: our new cluster has Omni-Path interconnect, which uses hardware contexts that are associated with each MPI process or rank on the node. We allow node sharing and in some cases when there are multiple MPI jobs on the same node (don't ask…) one job apparently uses up too many contexts and the other job crashes. So I'd like to set the PSM2_SHAREDCONTEXTS_MAX environment variable to an appropriate value for each job based on the number of cores or contexts available on the node and the number of cores requested by the job. Presumably the job_submit script would be the logical place to do this but I can't figure out how to set environment variables for the job in it. Any suggestions if this is the right track? Other ideas? Thanks, Pete Ruprecht CU-Boulder Research Computing
