What operating system are you running? Modern versions of systemd automatically put login sessions into their own cgroup which are themselves in a "user" group. When slurm is running parallel to this, it makes its own slurm cgroup.
It should be possible to have something at boot modify the systemd user cgroup to remove the gpus. Alternately, there may be options in pam-slurm-adopt to automatically move login sessions to a default cgroup. (I can't remember without looking how easy this would be.)
