Relu, Thank you. Looks like the fix is indeed the missing file /etc/slurm/cgroup_allowed_devices_file.conf
-SS- -----Original Message----- From: slurm-users <slurm-users-boun...@lists.schedmd.com> On Behalf Of Christopher Samuel Sent: Thursday, October 8, 2020 6:10 PM To: slurm-users@lists.schedmd.com Subject: Re: [slurm-users] CUDA environment variable not being set EXTERNAL SENDER Hi Sajesh, On 10/8/20 11:57 am, Sajesh Singh wrote: > debug: common_gres_set_env: unable to set env vars, no device files > configured I suspect the clue is here - what does your gres.conf look like? Does it list the devices in /dev for the GPUs? All the best, Chris -- Chris Samuel : https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=01%7C01%7Cssingh%40amnh.org%7C1bf5374fd6454b3fcd5a08d86bd6f427%7Cbe0003e8c6b9496883aeb34586974b76%7C0&sdata=INvZvw%2FiTrdf52patYRF9TtrQ0vuXRSivrxC8MJYLM4%3D&reserved=0 : Berkeley, CA, USA