Re: [slurm-users] addressing NVIDIA MIG + non MIG devices in Slurm

2022-01-31 Thread Bas van der Vlies
haring for a specific gres. *From:* slurm-users on behalf of Matthias Leopold *Sent:* Thursday, January 27, 2022 16:27 *To:* Slurm User Community List *Subject:* [slurm-users] addressing NVIDIA MIG + non MIG devices in Slu

Re: [slurm-users] addressing NVIDIA MIG + non MIG devices in Slurm - solved

2022-01-31 Thread Matthias Leopold
I looked at option > 2.2.3 using partial "AutoDetect=nvml" again and saw that the reason for failure was indeed the sanity check, but it was my fault because I set an invalid "Links" value for the "hardcoded" GPUs. So this variant of gres.conf setup works and gives me everything I want, sorry

Re: [slurm-users] addressing NVIDIA MIG + non MIG devices in Slurm - within one node

2022-01-27 Thread Matthias Leopold
I should have mentioned more prominently that my problem is about addressing NVIDIA MIG + non MIG devices in Slurm WITHIN ONE NODE (although this is obvious from reading my report). That also gives me the last resort solution of having one node with all MIG devices and one node with no MIG

Re: [slurm-users] addressing NVIDIA MIG + non MIG devices in Slurm

2022-01-27 Thread EPF (Esben Peter Friis)
ag implied. This flag is not combatible with one_sharing for a specific gres. From: slurm-users on behalf of Matthias Leopold Sent: Thursday, January 27, 2022 16:27 To: Slurm User Community List Subject: [slurm-users] addressing NVIDIA MIG + non MIG devices in

[slurm-users] addressing NVIDIA MIG + non MIG devices in Slurm

2022-01-27 Thread Matthias Leopold
Hi, we have 2 DGX A100 systems which we would like to use with Slurm. We want to use the MIG feature for _some_ of the GPUs. As I somehow suspected I couldn't find a working setup for this in Slurm yet. I'll describe the configuration variants I tried after creating the MIG instances, it