date:20210123

Re: [slurm-users] [External] Re: Cluster nodes on multiple cluster networks

2021-01-23 Thread Florian Zillner

Chiming in on Michael's suggestion.

You can specify the same hostname in the slurm.conf but for the on-premise 
nodes you either set the DNS or the /etc/hosts entry to the local (=private) IP 
address.
For the cloud nodes you set DNS or the hosts entry to the publicly reachable IP.

example /etc/hosts on-prem node:
10.10.10.10   myslurmcontroller

example /etc/hosts cloud node:
50.50.50.50   myslurmcontroller

example slurm.conf for both locations
ControlMachine=myslurmcontroller

From: slurm-users  on behalf of Sajesh 
Singh 
Sent: Friday, 22 January 2021 22:17
To: Slurm User Community List 
Subject: [External] Re: [slurm-users] Cluster nodes on multiple cluster networks


Thank you for the recommendation. Will try that out. Unfortunately the on-prem 
nodes cannot reach the head node via the public IP



-Sajesh-



From: slurm-users  On Behalf Of Michael 
Gutteridge
Sent: Friday, January 22, 2021 3:18 PM
To: Slurm User Community List 
Subject: Re: [slurm-users] Cluster nodes on multiple cluster networks



EXTERNAL SENDER



I don't believe the IP address is required- if you can configure a DNS/hosts 
entry differently for cloud nodes you can set:



   SlurmCtldhost = controllername



Then have "controllername" resolve to the private IP for the controller for the 
on-prem cluster, the public IP for the nodes in the cloud.  Theoretically 
anyway- I haven't run a config like that and I'm not sure how the controller 
will react to such a configuration (i.e. getting slurm traffic on both 
interfaces).



If the on-prem nodes can reach the public IP address of the controller it may 
be simpler to use only the public IP for the controller, but I don't know how 
your routing is set up.



HTH



 - Michael







On Fri, Jan 22, 2021 at 11:26 AM Sajesh Singh 
mailto:ssi...@amnh.org>> wrote:

How would I deal with the address of the head node defined in the slurm.conf as 
I have it defined as



SlurmctldHost=private-hostname(private.ip.addr)



The private.ip.addr address is not reachable from the cloud nodes



-Sajesh-



From: slurm-users 
mailto:slurm-users-boun...@lists.schedmd.com>>
 On Behalf Of Brian Andrus
Sent: Friday, January 22, 2021 1:45 PM
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] Cluster nodes on multiple cluster networks



EXTERNAL SENDER



You would need to have a direct connect/vpn so the cloud nodes can connect to 
your head node.

Brian Andrus

On 1/22/2021 10:37 AM, Sajesh Singh wrote:

We are looking at rolling out cloud bursting to our on-prem Slurm cluster and I 
am wondering how to deal with the slurm.conf variable SlurmctldHost. It is 
currently configured with the private cluster network address that the on-prem 
nodes use to contact it. The nodes in the cloud would contact the head node via 
its public IP address. How can I configure Slurm so that both IPs are 
recognized as the head node?





-Sajesh-

[slurm-users] Exclude Slurm packages from the EPEL yum repository

2021-01-23 Thread Ole Holm Nielsen

We use the EPEL yum repository on our CentOS 7 nodes.  Today EPEL 
surprisingly delivers Slurm 20.11.2 RPMs, and the daily yum updates 
(luckily) fail with some errors:


--> Running transaction check
---> Package slurm.x86_64 0:20.02.6-1.el7 will be updated
--> Processing Dependency: slurm(x86-64) = 20.02.6-1.el7 for package: 
slurm-libpmi-20.02.6-1.el7.x86_64
--> Processing Dependency: libslurmfull.so()(64bit) for package: 
slurm-libpmi-20.02.6-1.el7.x86_64

---> Package slurm.x86_64 0:20.11.2-2.el7 will be an update
--> Processing Dependency: pmix for package: slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libfreeipmi.so.17()(64bit) for package: 
slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libipmimonitoring.so.6()(64bit) for package: 
slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libslurmfull-20.11.2.so()(64bit) for package: 
slurm-20.11.2-2.el7.x86_64

---> Package slurm-contribs.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-contribs.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-devel.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-devel.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-perlapi.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-perlapi.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-slurmdbd.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-slurmdbd.x86_64 0:20.11.2-2.el7 will be an update
--> Running transaction check
---> Package freeipmi.x86_64 0:1.5.7-3.el7 will be installed
---> Package pmix.x86_64 0:1.1.3-1.el7 will be installed
---> Package slurm.x86_64 0:20.02.6-1.el7 will be updated
--> Processing Dependency: slurm(x86-64) = 20.02.6-1.el7 for package: 
slurm-libpmi-20.02.6-1.el7.x86_64
--> Processing Dependency: libslurmfull.so()(64bit) for package: 
slurm-libpmi-20.02.6-1.el7.x86_64

---> Package slurm-libs.x86_64 0:20.11.2-2.el7 will be installed
--> Finished Dependency Resolution
Error: Package: slurm-libpmi-20.02.6-1.el7.x86_64 
(@/slurm-libpmi-20.02.6-1.el7.x86_64)

   Requires: libslurmfull.so()(64bit)
   Removing: slurm-20.02.6-1.el7.x86_64 
(@/slurm-20.02.6-1.el7.x86_64)

   libslurmfull.so()(64bit)
   Updated By: slurm-20.11.2-2.el7.x86_64 (epel)
   Not found
Error: Package: slurm-libpmi-20.02.6-1.el7.x86_64 
(@/slurm-libpmi-20.02.6-1.el7.x86_64)

   Requires: slurm(x86-64) = 20.02.6-1.el7
   Removing: slurm-20.02.6-1.el7.x86_64 
(@/slurm-20.02.6-1.el7.x86_64)

   slurm(x86-64) = 20.02.6-1.el7
   Updated By: slurm-20.11.2-2.el7.x86_64 (epel)
   slurm(x86-64) = 20.11.2-2.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest


We still run Slurm 20.02 and don't want EPEL to introduce any Slurm 
updates!!  Slurm must be upgraded with some care, see for example

https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

Therefore we must disable EPEL's slurm RPMs permanently.  The fix is to 
add to the file /etc/yum.repos.d/epel.repo an "exclude=slurm*" line like 
the last line in:


[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
exclude=slurm*

/Ole

Re: [slurm-users] Job not running with Resource Reason even though resources appear to be available

2021-01-23 Thread Paul Raines


Yes, I meant job 38692.  Sorry.

I am still having the problem.  I suspect it has something to do with
the GPU configuration as this does not happen on my non-GPU node partitions.
Also, if I submit non-GPU jobs to the rtx8000 partition here, they
use up all the cores on the nodes just fine.

The upshot is on my 10 GPU nodes, I never see more than 6 GPUs in use
and jobs just asking for 1 or 2 GPUs are just made to wait in the qeuue.

Here is an example.  The state of the nodes in rtx8000 queue before
I queue jobs:

rtx-04
   CfgTRES=cpu=32,mem=1546000M,billing=99,gres/gpu=10
   AllocTRES=cpu=15,mem=120G,gres/gpu=5
rtx-05
   CfgTRES=cpu=32,mem=1546000M,billing=99,gres/gpu=10
   AllocTRES=cpu=15,mem=328G,gres/gpu=5
rtx-06
   CfgTRES=cpu=32,mem=1546000M,billing=99,gres/gpu=10
   AllocTRES=cpu=15,mem=224G,gres/gpu=5
rtx-07
   CfgTRES=cpu=32,mem=1546000M,billing=99,gres/gpu=10
   AllocTRES=cpu=16,mem=232G,gres/gpu=6
rtx-08
   CfgTRES=cpu=32,mem=1546000M,billing=81,gres/gpu=4

I then submit 10 jobs.  Then the queue for rtx8000 is:

NODELISTJOBID PARTITION  ST TIME_LIMIT  TRES_ALLOC   TRES_PER
rtx-04  40365 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-04  38676 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-04  38673 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-04  38670 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-04  38409 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-05  40214 rtx8000R  6-10:00:00  cpu=3,mem=128G,node= gpu:1
rtx-05  38677 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-05  38674 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-05  37450 rtx8000R  6-10:00:00  cpu=3,mem=128G,node= gpu:1
rtx-05  37278 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-06  40366 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-06  40364 rtx8000R  6-10:00:00  cpu=3,mem=128G,node= gpu:1
rtx-06  38648 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-06  38646 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-06  37267 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-07  40760 rtx8000R  50:00   cpu=4,mem=32G,node=1 gpu:2
rtx-07  38675 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-07  38672 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-07  38671 rtx8000R  7-00:00:00  cpu=3,mem=24G,node=1 gpu:1
rtx-07  37451 rtx8000R  6-10:00:00  cpu=3,mem=128G,node= gpu:1
rtx-08  40785 rtx8000R  50:00   cpu=4,mem=32G,node=1 gpu:2
rtx-08  40786 rtx8000R  50:00   cpu=4,mem=32G,node=1 gpu:2
(Priorit40794 rtx8000PD 50:00   cpu=4,mem=32G,node=1 gpu:2
(Priorit40793 rtx8000PD 50:00   cpu=4,mem=32G,node=1 gpu:2
(Priorit40792 rtx8000PD 50:00   cpu=4,mem=32G,node=1 gpu:2
(Priorit40791 rtx8000PD 50:00   cpu=4,mem=32G,node=1 gpu:2
(Priorit40790 rtx8000PD 50:00   cpu=4,mem=32G,node=1 gpu:2
(Priorit40789 rtx8000PD 50:00   cpu=4,mem=32G,node=1 gpu:2
(Priorit40788 rtx8000PD 50:00   cpu=4,mem=32G,node=1 gpu:2
(Resourc40787 rtx8000PD 50:00   cpu=4,mem=32G,node=1 gpu:2

[root@mlsc-head ~]# scontrol show job=40787
JobId=40787 JobName=sjob_5
   UserId=raines(5829) GroupId=raines(5829) MCS_label=N/A
   Priority=19836243 Nice=0 Account=sysadm QOS=normal
   JobState=PENDING Reason=Resources Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=00:50:00 TimeMin=N/A
   SubmitTime=2021-01-23T12:37:51 EligibleTime=2021-01-23T12:37:51
   AccrueTime=2021-01-23T12:37:51
   StartTime=2021-01-23T13:08:52 EndTime=2021-01-23T13:58:52 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2021-01-23T12:38:36
   Partition=rtx8000 AllocNode:Sid=mlsc-head:1268664
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=rtx-07
   NumNodes=1-2 NumCPUs=4 NumTasks=1 CPUs/Task=4 ReqB:S:C:T=0:0:*:*
   TRES=cpu=4,mem=32G,node=1,billing=11,gres/gpu=2
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=*
   MinCPUsNode=4 MinMemoryNode=32G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/autofs/cluster/batch/raines/sjob_5
   WorkDir=/autofs/cluster/batch/raines
   StdErr=/autofs/cluster/batch/raines/sjob_5.err40787
   StdIn=/dev/null
   StdOut=/autofs/cluster/batch/raines/sjob_5.out40787
   Power=
   TresPerJob=gpu:2
   MailUser=(null) MailType=NONE


[root@mlsc-head ~]# scontrol show node=rtx-04
NodeName=rtx-04 Arch=x86_64 CoresPerSocket=16
   CPUAlloc=15 CPUTot=32 CPULoad=18.21
   AvailableFeatures=intel,cascade,rtx8000
   ActiveFeatures=intel,cascade,rtx8000
   Gres=gpu:quadro_rtx_8000:10(S:0)
   NodeAddr=rtx-04 NodeHostName=rtx-04 Version=20.02.3
   OS=Linux 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020
   RealMemory=1546000 AllocMem=122880

Re: [slurm-users] Job not running with Resource Reason even though resources appear to be available

2021-01-23 Thread Chris Samuel

On Saturday, 23 January 2021 9:54:11 AM PST Paul Raines wrote:

> Now rtx-08 which has only 4 GPUs seems to always get all 4 uses.
> But the others seem to always only get half used (except rtx-07
> which somehow gets 6 used so another wierd thing).
> 
> Again if I submit non-GPU jobs, they end up allocating all hte
> cores/cpus on the nodes just fine.

What does your gres.conf look like for these nodes?

One thing I've seen in the past is where the core specifications for the GPUs 
are out of step with the hardware and so Slurm thinks they're on the wrong 
socket.  Then when all the cores in that socket are used up Slurm won't put 
more GPU jobs on the node without the jobs explicitly asking to not do 
locality.

One thing I've noticed is that in prior to Slurm 20.02 the documentation for 
gres.conf used to say:

# If your cores contain multiple threads only the first thread
# (processing unit) of each core needs to be listed.

but that language is gone from 20.02 and later and the change isn't mentioned 
in the release notes for 20.02 so I'm not sure what happened there, the only 
clue is this commit:

https://github.com/SchedMD/slurm/commit/
7461b6ba95bb8ae70b36425f2c7e4961ac35799e#diff-
cac030b65a8fc86123176971a94062fafb262cb2b11b3e90d6cc69e353e3bb89

which says "xcpuinfo_abs_to_mac() expects a core list, not a CPU list."

Best of luck!
Chris
-- 
  Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

Re: [slurm-users] Exclude Slurm packages from the EPEL yum repository

2021-01-23 Thread Philip Kovacs

 I can assure you it was easier for you to filter slurm from your repos than it 
was for me to make them available to both epel7 and epel8.
No good deed goes unpunished I guess.On Saturday, January 23, 2021, 
07:03:08 AM EST, Ole Holm Nielsen  wrote:  
 
 We use the EPEL yum repository on our CentOS 7 nodes.  Today EPEL 
surprisingly delivers Slurm 20.11.2 RPMs, and the daily yum updates 
(luckily) fail with some errors:

--> Running transaction check
---> Package slurm.x86_64 0:20.02.6-1.el7 will be updated
--> Processing Dependency: slurm(x86-64) = 20.02.6-1.el7 for package: 
slurm-libpmi-20.02.6-1.el7.x86_64
--> Processing Dependency: libslurmfull.so()(64bit) for package: 
slurm-libpmi-20.02.6-1.el7.x86_64
---> Package slurm.x86_64 0:20.11.2-2.el7 will be an update
--> Processing Dependency: pmix for package: slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libfreeipmi.so.17()(64bit) for package: 
slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libipmimonitoring.so.6()(64bit) for package: 
slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libslurmfull-20.11.2.so()(64bit) for package: 
slurm-20.11.2-2.el7.x86_64
---> Package slurm-contribs.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-contribs.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-devel.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-devel.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-perlapi.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-perlapi.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-slurmdbd.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-slurmdbd.x86_64 0:20.11.2-2.el7 will be an update
--> Running transaction check
---> Package freeipmi.x86_64 0:1.5.7-3.el7 will be installed
---> Package pmix.x86_64 0:1.1.3-1.el7 will be installed
---> Package slurm.x86_64 0:20.02.6-1.el7 will be updated
--> Processing Dependency: slurm(x86-64) = 20.02.6-1.el7 for package: 
slurm-libpmi-20.02.6-1.el7.x86_64
--> Processing Dependency: libslurmfull.so()(64bit) for package: 
slurm-libpmi-20.02.6-1.el7.x86_64
---> Package slurm-libs.x86_64 0:20.11.2-2.el7 will be installed
--> Finished Dependency Resolution
Error: Package: slurm-libpmi-20.02.6-1.el7.x86_64 
(@/slurm-libpmi-20.02.6-1.el7.x86_64)
            Requires: libslurmfull.so()(64bit)
            Removing: slurm-20.02.6-1.el7.x86_64 
(@/slurm-20.02.6-1.el7.x86_64)
                libslurmfull.so()(64bit)
            Updated By: slurm-20.11.2-2.el7.x86_64 (epel)
                Not found
Error: Package: slurm-libpmi-20.02.6-1.el7.x86_64 
(@/slurm-libpmi-20.02.6-1.el7.x86_64)
            Requires: slurm(x86-64) = 20.02.6-1.el7
            Removing: slurm-20.02.6-1.el7.x86_64 
(@/slurm-20.02.6-1.el7.x86_64)
                slurm(x86-64) = 20.02.6-1.el7
            Updated By: slurm-20.11.2-2.el7.x86_64 (epel)
                slurm(x86-64) = 20.11.2-2.el7
  You could try using --skip-broken to work around the problem
  You could try running: rpm -Va --nofiles --nodigest


We still run Slurm 20.02 and don't want EPEL to introduce any Slurm 
updates!!  Slurm must be upgraded with some care, see for example
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

Therefore we must disable EPEL's slurm RPMs permanently.  The fix is to 
add to the file /etc/yum.repos.d/epel.repo an "exclude=slurm*" line like 
the last line in:

[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
exclude=slurm*

/Ole

Re: [slurm-users] [External] Re: Cluster nodes on multiple cluster networks

[slurm-users] Exclude Slurm packages from the EPEL yum repository

Re: [slurm-users] Job not running with Resource Reason even though resources appear to be available

Re: [slurm-users] Job not running with Resource Reason even though resources appear to be available

Re: [slurm-users] Exclude Slurm packages from the EPEL yum repository

5 matches

Site Navigation

Mail list logo

Footer information