[slurm-dev] Re: Backup controller not responding to requests
Aha! That was just far too large for me. Set it down to 60 seconds and things seem happier (along with the users). Thanks! Brian -Original Message- From: TO_Webmaster [mailto:luftha...@gmail.com] Sent: Tuesday, January 31, 2017 12:26 AM To: slurm-dev Subject: [slurm-dev] Re: Backup controller not responding to requests What is the output of scontrol show config | grep SlurmctldTimeout ? 2017-01-31 6:57 GMT+01:00 Andrus, Brian Contractor : > Yes, if I do scontrol takeover, it successfully goes to the backup. > > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > > > > -Original Message- > From: TO_Webmaster [mailto:luftha...@gmail.com] > Sent: Monday, January 30, 2017 11:02 AM > To: slurm-dev > Subject: [slurm-dev] Re: Backup controller not responding to requests > > > Does it work if you use "scontrol takeover" to shut down the primary > controller and switch immediately to the backup controller? > > 2017-01-30 19:41 GMT+01:00 Andrus, Brian Contractor : >> Paddy, >> >> I will enable those and try. The backup controller does have access to the >> directory and it is the same version as the master. >> >> Not seeing much more in the logs.. >> The backup controller ends with a ping of the master and then just >> sits. I restart the master and the backup starts saying "Invalid RPC". When >> the master comes back up, it says it is ignoring the RPC: REQUEST_CONTROL >> So, for some reason, it seems the backup will not promote itself... >> >> -- >> [2017-01-30T10:30:21.321] debug3: Success. >> [2017-01-30T10:30:21.322] trigger pulled for SLURMCTLD event 16384 >> successful [2017-01-30T10:30:27.323] debug3: pinging slurmctld at >> 10.1.1.127 [2017-01-30T10:31:55.814] error: Invalid RPC received 2009 >> while in standby mode [2017-01-30T10:32:04.839] debug3: Ignoring RPC: >> REQUEST_CONTROL [2017-01-30T10:32:06.133] error: Invalid RPC received >> 2009 while in standby mode [2017-01-30T10:32:07.338] debug3: pinging >> slurmctld at 10.1.1.127 [2017-01-30T10:32:07.339] debug2: >> slurm_connect failed: Connection refused [2017-01-30T10:32:07.339] >> debug2: Error connecting slurm stream socket at 10.1.1.127:6817: >> Connection refused [2017-01-30T10:32:07.339] error: >> _ping_controller/slurm_send_node_msg error: Connection refused >> [2017-01-30T10:33:47.351] debug3: pinging slurmctld at 10.1.1.127 >> [2017-01-30T10:35:27.366] debug3: pinging slurmctld at 10.1.1.127 >> [2017-01-30T10:35:33.758] debug3: Ignoring RPC: REQUEST_CONTROL >> - >> >> >> Brian Andrus >> ITACS/Research Computing >> Naval Postgraduate School >> Monterey, California >> voice: 831-656-6238 >> >> >> >> -Original Message- >> From: Paddy Doyle [mailto:pa...@tchpc.tcd.ie] >> Sent: Monday, January 30, 2017 9:48 AM >> To: slurm-dev >> Subject: [slurm-dev] Re: Backup controller not responding to requests >> >> >> Hi Brian, >> >> You could turn up the SlurmctldDebug and SlurmdDebug values in slurm.conf to >> get it to be more verbose. >> >> As a wild guess, perhaps your backup control doesn't have access to the >> StateSaveLocation directory? >> >> Or another possibility could be it's running a different version of slurm. >> >> Paddy >> >> On Mon, Jan 30, 2017 at 08:21:59AM -0800, Andrus, Brian Contractor wrote: >> >>> All, >>> >>> I have configured a backup slurmctld system and it appears to work at >>> first, but not in practice. >>> In particular, when I start it, it says it is running in background mode: >>> [2017-01-25T14:23:37.648] slurmctld version 16.05.6 started on >>> cluster hamming [2017-01-25T14:23:37.650] slurmctld running in >>> background mode >>> >>> But if I stop the primary daemon, it does not take over. I keep getting >>> Invalid RPC errors (random snippets): >>> [2017-01-25T15:50:37.664] error: Invalid RPC received 2007 while in >>> standby mode [2017-01-25T15:53:50.495] error: Invalid RPC received >>> 5018 while in standby mode [2017-01-25T15:59:36.847] error: Invalid >>> RPC received 2007 while in standby mode [2017-01-25T15:59:37.499] >>> error: Invalid RPC received 2007 while in standby mode >>> [2017-01-25T15:59:38.923] error: Invalid RPC received 2007 while in >>> standby mode [2017-01-25T15:59:38.985] error: Invalid RPC received >>> 2007 while in standby mode
[slurm-dev] Re: Backup controller not responding to requests
Yes, if I do scontrol takeover, it successfully goes to the backup. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: TO_Webmaster [mailto:luftha...@gmail.com] Sent: Monday, January 30, 2017 11:02 AM To: slurm-dev Subject: [slurm-dev] Re: Backup controller not responding to requests Does it work if you use "scontrol takeover" to shut down the primary controller and switch immediately to the backup controller? 2017-01-30 19:41 GMT+01:00 Andrus, Brian Contractor : > Paddy, > > I will enable those and try. The backup controller does have access to the > directory and it is the same version as the master. > > Not seeing much more in the logs.. > The backup controller ends with a ping of the master and then just > sits. I restart the master and the backup starts saying "Invalid RPC". When > the master comes back up, it says it is ignoring the RPC: REQUEST_CONTROL So, > for some reason, it seems the backup will not promote itself... > > -- > [2017-01-30T10:30:21.321] debug3: Success. > [2017-01-30T10:30:21.322] trigger pulled for SLURMCTLD event 16384 > successful [2017-01-30T10:30:27.323] debug3: pinging slurmctld at > 10.1.1.127 [2017-01-30T10:31:55.814] error: Invalid RPC received 2009 > while in standby mode [2017-01-30T10:32:04.839] debug3: Ignoring RPC: > REQUEST_CONTROL [2017-01-30T10:32:06.133] error: Invalid RPC received > 2009 while in standby mode [2017-01-30T10:32:07.338] debug3: pinging > slurmctld at 10.1.1.127 [2017-01-30T10:32:07.339] debug2: > slurm_connect failed: Connection refused [2017-01-30T10:32:07.339] > debug2: Error connecting slurm stream socket at 10.1.1.127:6817: > Connection refused [2017-01-30T10:32:07.339] error: > _ping_controller/slurm_send_node_msg error: Connection refused > [2017-01-30T10:33:47.351] debug3: pinging slurmctld at 10.1.1.127 > [2017-01-30T10:35:27.366] debug3: pinging slurmctld at 10.1.1.127 > [2017-01-30T10:35:33.758] debug3: Ignoring RPC: REQUEST_CONTROL > - > > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > > > > -Original Message- > From: Paddy Doyle [mailto:pa...@tchpc.tcd.ie] > Sent: Monday, January 30, 2017 9:48 AM > To: slurm-dev > Subject: [slurm-dev] Re: Backup controller not responding to requests > > > Hi Brian, > > You could turn up the SlurmctldDebug and SlurmdDebug values in slurm.conf to > get it to be more verbose. > > As a wild guess, perhaps your backup control doesn't have access to the > StateSaveLocation directory? > > Or another possibility could be it's running a different version of slurm. > > Paddy > > On Mon, Jan 30, 2017 at 08:21:59AM -0800, Andrus, Brian Contractor wrote: > >> All, >> >> I have configured a backup slurmctld system and it appears to work at first, >> but not in practice. >> In particular, when I start it, it says it is running in background mode: >> [2017-01-25T14:23:37.648] slurmctld version 16.05.6 started on >> cluster hamming [2017-01-25T14:23:37.650] slurmctld running in >> background mode >> >> But if I stop the primary daemon, it does not take over. I keep getting >> Invalid RPC errors (random snippets): >> [2017-01-25T15:50:37.664] error: Invalid RPC received 2007 while in >> standby mode [2017-01-25T15:53:50.495] error: Invalid RPC received >> 5018 while in standby mode [2017-01-25T15:59:36.847] error: Invalid >> RPC received 2007 while in standby mode [2017-01-25T15:59:37.499] >> error: Invalid RPC received 2007 while in standby mode >> [2017-01-25T15:59:38.923] error: Invalid RPC received 2007 while in >> standby mode [2017-01-25T15:59:38.985] error: Invalid RPC received >> 2007 while in standby mode [2017-01-25T15:59:39.246] error: Invalid >> RPC received 2007 while in standby mode [2017-01-25T15:59:39.293] >> error: Invalid RPC received 2009 while in standby mode >> [2017-01-25T15:59:39.522] error: Invalid RPC received 5018 while in >> standby mode [2017-01-25T15:59:43.839] error: Invalid RPC received >> 2009 while in standby mode [2017-01-25T15:59:43.930] error: Invalid >> RPC received 2009 while in standby mode [2017-01-25T16:19:47.215] >> error: Invalid RPC received 6012 while in standby mode >> [2017-01-25T16:19:48.238] error: Invalid RPC received 6012 while in >> standby mode >> >> And on any client running 'sinfo' for instance, it merely hangs. >> The interfaces for both slurmctld controllers are in the 'trusted' firewall >> group and ther
[slurm-dev] Re: Backup controller not responding to requests
Paddy, I will enable those and try. The backup controller does have access to the directory and it is the same version as the master. Not seeing much more in the logs.. The backup controller ends with a ping of the master and then just sits. I restart the master and the backup starts saying "Invalid RPC". When the master comes back up, it says it is ignoring the RPC: REQUEST_CONTROL So, for some reason, it seems the backup will not promote itself... -- [2017-01-30T10:30:21.321] debug3: Success. [2017-01-30T10:30:21.322] trigger pulled for SLURMCTLD event 16384 successful [2017-01-30T10:30:27.323] debug3: pinging slurmctld at 10.1.1.127 [2017-01-30T10:31:55.814] error: Invalid RPC received 2009 while in standby mode [2017-01-30T10:32:04.839] debug3: Ignoring RPC: REQUEST_CONTROL [2017-01-30T10:32:06.133] error: Invalid RPC received 2009 while in standby mode [2017-01-30T10:32:07.338] debug3: pinging slurmctld at 10.1.1.127 [2017-01-30T10:32:07.339] debug2: slurm_connect failed: Connection refused [2017-01-30T10:32:07.339] debug2: Error connecting slurm stream socket at 10.1.1.127:6817: Connection refused [2017-01-30T10:32:07.339] error: _ping_controller/slurm_send_node_msg error: Connection refused [2017-01-30T10:33:47.351] debug3: pinging slurmctld at 10.1.1.127 [2017-01-30T10:35:27.366] debug3: pinging slurmctld at 10.1.1.127 [2017-01-30T10:35:33.758] debug3: Ignoring RPC: REQUEST_CONTROL - Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: Paddy Doyle [mailto:pa...@tchpc.tcd.ie] Sent: Monday, January 30, 2017 9:48 AM To: slurm-dev Subject: [slurm-dev] Re: Backup controller not responding to requests Hi Brian, You could turn up the SlurmctldDebug and SlurmdDebug values in slurm.conf to get it to be more verbose. As a wild guess, perhaps your backup control doesn't have access to the StateSaveLocation directory? Or another possibility could be it's running a different version of slurm. Paddy On Mon, Jan 30, 2017 at 08:21:59AM -0800, Andrus, Brian Contractor wrote: > All, > > I have configured a backup slurmctld system and it appears to work at first, > but not in practice. > In particular, when I start it, it says it is running in background mode: > [2017-01-25T14:23:37.648] slurmctld version 16.05.6 started on cluster > hamming [2017-01-25T14:23:37.650] slurmctld running in background mode > > But if I stop the primary daemon, it does not take over. I keep getting > Invalid RPC errors (random snippets): > [2017-01-25T15:50:37.664] error: Invalid RPC received 2007 while in > standby mode [2017-01-25T15:53:50.495] error: Invalid RPC received > 5018 while in standby mode [2017-01-25T15:59:36.847] error: Invalid > RPC received 2007 while in standby mode [2017-01-25T15:59:37.499] > error: Invalid RPC received 2007 while in standby mode > [2017-01-25T15:59:38.923] error: Invalid RPC received 2007 while in > standby mode [2017-01-25T15:59:38.985] error: Invalid RPC received > 2007 while in standby mode [2017-01-25T15:59:39.246] error: Invalid > RPC received 2007 while in standby mode [2017-01-25T15:59:39.293] > error: Invalid RPC received 2009 while in standby mode > [2017-01-25T15:59:39.522] error: Invalid RPC received 5018 while in > standby mode [2017-01-25T15:59:43.839] error: Invalid RPC received > 2009 while in standby mode [2017-01-25T15:59:43.930] error: Invalid > RPC received 2009 while in standby mode [2017-01-25T16:19:47.215] > error: Invalid RPC received 6012 while in standby mode > [2017-01-25T16:19:48.238] error: Invalid RPC received 6012 while in > standby mode > > And on any client running 'sinfo' for instance, it merely hangs. > The interfaces for both slurmctld controllers are in the 'trusted' firewall > group and there is no filtering between them. > Is there something I am missing to make the backup controller 'kick in' and > start responding to requests? > > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > -- Paddy Doyle Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. Phone: +353-1-896-3725 http://www.tchpc.tcd.ie/
[slurm-dev] Backup controller not responding to requests
All, I have configured a backup slurmctld system and it appears to work at first, but not in practice. In particular, when I start it, it says it is running in background mode: [2017-01-25T14:23:37.648] slurmctld version 16.05.6 started on cluster hamming [2017-01-25T14:23:37.650] slurmctld running in background mode But if I stop the primary daemon, it does not take over. I keep getting Invalid RPC errors (random snippets): [2017-01-25T15:50:37.664] error: Invalid RPC received 2007 while in standby mode [2017-01-25T15:53:50.495] error: Invalid RPC received 5018 while in standby mode [2017-01-25T15:59:36.847] error: Invalid RPC received 2007 while in standby mode [2017-01-25T15:59:37.499] error: Invalid RPC received 2007 while in standby mode [2017-01-25T15:59:38.923] error: Invalid RPC received 2007 while in standby mode [2017-01-25T15:59:38.985] error: Invalid RPC received 2007 while in standby mode [2017-01-25T15:59:39.246] error: Invalid RPC received 2007 while in standby mode [2017-01-25T15:59:39.293] error: Invalid RPC received 2009 while in standby mode [2017-01-25T15:59:39.522] error: Invalid RPC received 5018 while in standby mode [2017-01-25T15:59:43.839] error: Invalid RPC received 2009 while in standby mode [2017-01-25T15:59:43.930] error: Invalid RPC received 2009 while in standby mode [2017-01-25T16:19:47.215] error: Invalid RPC received 6012 while in standby mode [2017-01-25T16:19:48.238] error: Invalid RPC received 6012 while in standby mode And on any client running 'sinfo' for instance, it merely hangs. The interfaces for both slurmctld controllers are in the 'trusted' firewall group and there is no filtering between them. Is there something I am missing to make the backup controller 'kick in' and start responding to requests? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238
[slurm-dev] RE: Restrict access for a user group to certain nodes
The way we did that was to put the nodes in their own partition which is only accessible by that group. PartitionName=beardq Nodes=compute-8-[1,5,9,13,17] AllowGroups=beards DefaultTime=01:00:00 MaxTime=INFINITE State=UP So here is a partition "beardq" which is only available to folks in the group "beards" (yah, all they guys in that group have beards) Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: Felix Willenborg [mailto:felix.willenb...@uni-oldenburg.de] Sent: Thursday, December 01, 2016 6:53 AM To: slurm-dev Subject: [slurm-dev] Restrict access for a user group to certain nodes Dear everybody, I'd like to restrict submissions from a certain user group or allow only one certain user group to submit jobs to certain nodes. Does Slurm offer groups which can handle such an occassion? It'd be prefered if there is a linux user group support, because this would save time setting up a new user group environment. The intention is that only administrators can submit jobs to those certain nodes to perform some tests, which might be disturbed by users submitting their jobs to those nodes. Various Search Engines didn't offer answers to my question, which is why I'm writing you here. Looking forward to some answers! Best, Felix Willenborg
[slurm-dev] Re: squeue returns "invalid user" for a user that has jobs running
I take that back. It was indeed the issue. User name is clwalton1... Doh! Thanks for pointing me in the right direction. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: Andrus, Brian Contractor [mailto:bdand...@nps.edu] Sent: Monday, November 28, 2016 9:53 AM To: slurm-dev Subject: [slurm-dev] Re: squeue returns "invalid user" for a user that has jobs running Paddy, Nope, it is exactly 8 characters: clwalton Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: Paddy Doyle [mailto:pa...@tchpc.tcd.ie] Sent: Monday, November 28, 2016 2:35 AM To: slurm-dev Subject: [slurm-dev] Re: squeue returns "invalid user" for a user that has jobs running Hi Brian, Is there actual username longer than 8 characters? The default squeue format includes "%.8u" for the username. Paddy On Fri, Nov 25, 2016 at 08:26:36PM -0800, Andrus, Brian Contractor wrote: > All, > > Don't quite get this: > > # squeue|head > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > 751071_17703 primary PARAMEIG clwalton CG 3-00:00:19 1 > compute-3-87 > 751071_[36752-6220 primary PARAMEIG clwalton PD 0:00 1 > (Resources) > 787831 primary runwrfam hale PD 0:00 1 > (Reservation) > 751071_36750 primary PARAMEIG clwalton R 0:02 1 > compute-3-33 > 751071_36751 primary PARAMEIG clwalton R 0:02 1 > compute-3-37 > 751071_36749 primary PARAMEIG clwalton R 0:06 1 > compute-8-17 > 787294 primary wrfrunmp hale R3:34:03 1 > compute-7-1 > # squeue -u clwalton > squeue: error: Invalid user: clwalton > > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > # squeue -u hale > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > 787831 primary runwrfam hale PD 0:00 1 > (Reservation) > 787294 primary wrfrunmp hale R3:34:03 1 > compute-7-1 > > > Clearly user clwalton is a valid user and has jobs running, but if I try to > specify him, squeue isn't happy. It is fine with other users... > What would cause this? > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > -- Paddy Doyle Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. Phone: +353-1-896-3725 http://www.tchpc.tcd.ie/
[slurm-dev] Re: squeue returns "invalid user" for a user that has jobs running
Carlos, Yes. The cluster uses LDAP for users. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: Carlos Fenoy [mailto:mini...@gmail.com] Sent: Monday, November 28, 2016 9:58 AM To: slurm-dev Subject: [slurm-dev] Re: squeue returns "invalid user" for a user that has jobs running Hi, Is the user defined in all the compute nodes? Does it has the same UID in all the hosts? Regards, Carlos On Mon, Nov 28, 2016 at 6:54 PM, Andrus, Brian Contractor mailto:bdand...@nps.edu>> wrote: Paddy, Nope, it is exactly 8 characters: clwalton Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: Paddy Doyle [mailto:pa...@tchpc.tcd.ie<mailto:pa...@tchpc.tcd.ie>] Sent: Monday, November 28, 2016 2:35 AM To: slurm-dev mailto:slurm-dev@schedmd.com>> Subject: [slurm-dev] Re: squeue returns "invalid user" for a user that has jobs running Hi Brian, Is there actual username longer than 8 characters? The default squeue format includes "%.8u" for the username. Paddy On Fri, Nov 25, 2016 at 08:26:36PM -0800, Andrus, Brian Contractor wrote: > All, > > Don't quite get this: > > # squeue|head > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > 751071_17703 primary PARAMEIG clwalton CG 3-00:00:19 1 > compute-3-87 > 751071_[36752-6220 primary PARAMEIG clwalton PD 0:00 1 > (Resources) > 787831 primary runwrfam hale PD 0:00 1 > (Reservation) > 751071_36750 primary PARAMEIG clwalton R 0:02 1 > compute-3-33 > 751071_36751 primary PARAMEIG clwalton R 0:02 1 > compute-3-37 > 751071_36749 primary PARAMEIG clwalton R 0:06 1 > compute-8-17 > 787294 primary wrfrunmp hale R3:34:03 1 > compute-7-1 > # squeue -u clwalton > squeue: error: Invalid user: clwalton > > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > # squeue -u hale > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > 787831 primary runwrfam hale PD 0:00 1 > (Reservation) > 787294 primary wrfrunmp hale R3:34:03 1 > compute-7-1 > > > Clearly user clwalton is a valid user and has jobs running, but if I try to > specify him, squeue isn't happy. It is fine with other users... > What would cause this? > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > -- Paddy Doyle Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. Phone: +353-1-896-3725 http://www.tchpc.tcd.ie/ -- -- Carles Fenoy
[slurm-dev] Re: squeue returns "invalid user" for a user that has jobs running
Paddy, Nope, it is exactly 8 characters: clwalton Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: Paddy Doyle [mailto:pa...@tchpc.tcd.ie] Sent: Monday, November 28, 2016 2:35 AM To: slurm-dev Subject: [slurm-dev] Re: squeue returns "invalid user" for a user that has jobs running Hi Brian, Is there actual username longer than 8 characters? The default squeue format includes "%.8u" for the username. Paddy On Fri, Nov 25, 2016 at 08:26:36PM -0800, Andrus, Brian Contractor wrote: > All, > > Don't quite get this: > > # squeue|head > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > 751071_17703 primary PARAMEIG clwalton CG 3-00:00:19 1 > compute-3-87 > 751071_[36752-6220 primary PARAMEIG clwalton PD 0:00 1 > (Resources) > 787831 primary runwrfam hale PD 0:00 1 > (Reservation) > 751071_36750 primary PARAMEIG clwalton R 0:02 1 > compute-3-33 > 751071_36751 primary PARAMEIG clwalton R 0:02 1 > compute-3-37 > 751071_36749 primary PARAMEIG clwalton R 0:06 1 > compute-8-17 > 787294 primary wrfrunmp hale R3:34:03 1 > compute-7-1 > # squeue -u clwalton > squeue: error: Invalid user: clwalton > > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > # squeue -u hale > JOBID PARTITION NAME USER ST TIME NODES > NODELIST(REASON) > 787831 primary runwrfam hale PD 0:00 1 > (Reservation) > 787294 primary wrfrunmp hale R3:34:03 1 > compute-7-1 > > > Clearly user clwalton is a valid user and has jobs running, but if I try to > specify him, squeue isn't happy. It is fine with other users... > What would cause this? > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > -- Paddy Doyle Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. Phone: +353-1-896-3725 http://www.tchpc.tcd.ie/
[slurm-dev] Re: PIDfile on CentOS7 and compute nodes
Ole, I am building the RPMs on CentOS7. I merely do: rpmbuild -tb slurm-16.05.6.tar.bz2 I do see the resulting rpm has both the init file and the unit files: # rpm -qlp ../RPMS/x86_64/slurm-16.05.6-1.el7.centos.x86_64.rpm|egrep "init.d|service$" /etc/init.d/slurm /usr/lib/systemd/system/slurmctld.service /usr/lib/systemd/system/slurmd.service Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: Ole Holm Nielsen [mailto:ole.h.niel...@fysik.dtu.dk] Sent: Sunday, November 27, 2016 11:12 PM To: slurm-dev Subject: [slurm-dev] Re: PIDfile on CentOS7 and compute nodes Hi Brian, Did you build and install the Slurm RPMs on CentOS 7, or is it a manual install? Which Slurm and CentOS versions do you run? We run Slurm 16.05 on CentOS 7, see instructions in our Wiki https://wiki.fysik.dtu.dk/niflheim/SLURM /Ole On 11/25/2016 05:04 PM, Andrus, Brian Contractor wrote: > All, > > I have been having an issue where if I try to run the slurm daemon > under systemd, it hangs for some time and then errors out with: > > > > systemd[1]: Starting LSB: slurm daemon management... > > systemd[1]: PID file /var/run/slurmctld.pid not readable (yet?) after start. > > systemd[1]: slurm.service: control process exited, code=exited > status=203 > > systemd[1]: Failed to start LSB: slurm daemon management. > > systemd[1]: Unit slurm.service entered failed state. > > systemd[1]: slurm.service failed. > > > > > > Now it does actually start and is running when I do a ‘ps’. > > So I DID figure out a work-around, which, for now, I will code for > changing the scripts. > > If I remove the lines from the /etc/init.d/slurm file: > > > > # processname: /usr/sbin/slurmctld > > # pidfile: /var/run/slurmctld.pid > > > > Then systemd is happy running just slurm. > > > > Not sure what the appropriate fix is for this, but that is a > work-around that seems effective.
[slurm-dev] squeue returns "invalid user" for a user that has jobs running
All, Don't quite get this: # squeue|head JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 751071_17703 primary PARAMEIG clwalton CG 3-00:00:19 1 compute-3-87 751071_[36752-6220 primary PARAMEIG clwalton PD 0:00 1 (Resources) 787831 primary runwrfam hale PD 0:00 1 (Reservation) 751071_36750 primary PARAMEIG clwalton R 0:02 1 compute-3-33 751071_36751 primary PARAMEIG clwalton R 0:02 1 compute-3-37 751071_36749 primary PARAMEIG clwalton R 0:06 1 compute-8-17 787294 primary wrfrunmp hale R3:34:03 1 compute-7-1 # squeue -u clwalton squeue: error: Invalid user: clwalton JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) # squeue -u hale JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 787831 primary runwrfam hale PD 0:00 1 (Reservation) 787294 primary wrfrunmp hale R3:34:03 1 compute-7-1 Clearly user clwalton is a valid user and has jobs running, but if I try to specify him, squeue isn't happy. It is fine with other users... What would cause this? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238
[slurm-dev] PIDfile on CentOS7 and compute nodes
All, I have been having an issue where if I try to run the slurm daemon under systemd, it hangs for some time and then errors out with: systemd[1]: Starting LSB: slurm daemon management... systemd[1]: PID file /var/run/slurmctld.pid not readable (yet?) after start. systemd[1]: slurm.service: control process exited, code=exited status=203 systemd[1]: Failed to start LSB: slurm daemon management. systemd[1]: Unit slurm.service entered failed state. systemd[1]: slurm.service failed. Now it does actually start and is running when I do a 'ps'. So I DID figure out a work-around, which, for now, I will code for changing the scripts. If I remove the lines from the /etc/init.d/slurm file: # processname: /usr/sbin/slurmctld # pidfile: /var/run/slurmctld.pid Then systemd is happy running just slurm. Not sure what the appropriate fix is for this, but that is a work-around that seems effective. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238
[slurm-dev] Re: Fully utilizing nodes
Marius, Yep! I had commented out my DefMemPerNode so it was defaulting to all the memory on a node. Thanks for helping me see that! Brian From: Marius Bjørnstad [mailto:p.m.bjorns...@medisin.uio.no] Sent: Tuesday, August 16, 2016 3:23 AM To: slurm-dev Subject: [slurm-dev] Re: Fully utilizing nodes Hi Brian, Looks like your default memory allocation for jobs is 258307 MB, which is just how much memory you have on the node. Try to request less memory with --mem. Best wishes, Marius 16. aug. 2016 kl. 01.44 skrev Andrus, Brian Contractor mailto:bdand...@nps.edu>>: NodeName=compute-2-1 Arch=x86_64 CoresPerSocket=8 OS=Linux RealMemory=258307 AllocMem=0 FreeMem=254172 Sockets=8 Boards=1 When I check the running job it shows: JobId=133887 JobName=bash MinCPUsNode=1 MinMemoryNode=258307M MinTmpDiskNode=0 It sits and waits until I end the first job, at which point it immediately starts. Any idea what am I missing here that is stopping my second job from running on the same node at the same time?
[slurm-dev] Re: Fully utilizing nodes
Ok, I am still having trouble here and am not sure where to look. Slurm is configured with: SelectType = select/cons_res SelectTypeParameters= CR_CORE_MEMORY,CR_ONE_TASK_PER_CORE I have a node which has 64 cores: NodeName=compute-2-1 Arch=x86_64 CoresPerSocket=8 CPUAlloc=0 CPUErr=0 CPUTot=64 CPULoad=0.06 AvailableFeatures=amd ActiveFeatures=amd Gres=(null) NodeAddr=compute-2-1 NodeHostName=compute-2-1 Version=16.05 OS=Linux RealMemory=258307 AllocMem=0 FreeMem=254172 Sockets=8 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=3903 Weight=1 Owner=N/A MCS_label=N/A BootTime=2016-08-05T09:48:40 SlurmdStartTime=2016-08-13T14:21:49 CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s I am merely trying to start two interactive bash sessions on the same node. I use srun --ntasks=1 --share --nodelist=compute-2-1 --pty bash and I get a command prompt right away. In another session, I also do: srun --nodelist=compute-2-1 --ntasks=1 --share --pty bash which responds with: srun: job 133888 queued and waiting for resources When I check the running job it shows: JobId=133887 JobName=bash UserId=bdandrus(30274) GroupId=hamming(20) MCS_label=N/A Priority=2 Nice=0 Account=admin QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:25 TimeLimit=01:00:00 TimeMin=N/A SubmitTime=2016-08-15T16:38:58 EligibleTime=2016-08-15T16:38:58 StartTime=2016-08-15T16:38:58 EndTime=2016-08-15T17:38:58 Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 Partition=primary AllocNode:Sid=submit-0:25370 ReqNodeList=compute-2-1 ExcNodeList=(null) NodeList=compute-2-1 BatchHost=compute-2-1 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=258307M,node=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=* MinCPUsNode=1 MinMemoryNode=258307M MinTmpDiskNode=0 Features=(null) Gres=(null) Reservation=(null) OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash WorkDir=/work/bdandrus/MPI/OPENMPI Power= It sits and waits until I end the first job, at which point it immediately starts. Any idea what am I missing here that is stopping my second job from running on the same node at the same time? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238
[slurm-dev] Re: Fully utilizing nodes
Lyn, I am confused then. In the man page for slurm.conf: Shared The Shared configuration parameter has been replaced by the OverSubscribe parameter described above. I have exactly the settings you list for SelectType and SelectTypeParameters as well. I had already tried the shared=no setting for the partition, but it seemed to be ignored, which is why I looked into the oversubscribe option. This is running slurm 16.05 Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: Lyn Gerner [mailto:schedulerqu...@gmail.com] Sent: Tuesday, August 09, 2016 2:29 PM To: slurm-dev Subject: [slurm-dev] Re: Fully utilizing nodes Hi Brian, You'll need Shared=No in the partition definition (Oversubscribe not required). That will cap your allocations to one user task per core. To allocate >1 job per node, you can use something like these two values: SelectType = select/cons_res SelectTypeParameters= CR_CORE_MEMORY (See the other CR_* options; CR_LLN is what you *don't* want.) WIth the above, your users should be able to specify and obtain --exclusive as desired. Best, Lyn On Tue, Aug 9, 2016 at 11:06 AM, Andrus, Brian Contractor mailto:bdand...@nps.edu>> wrote: All, I am trying to figure out the bits required to allow users to use part of a node and not block others from using remaining resources. It looks like the “OverSubscribe” option is what I need, but that doesn’t seem to quite be all of it. I would like users to be able to request --exclusive if needed. However, I would like when users don’t then slurm prefers to pack jobs onto as few nodes as possible when they start. I suspect it may be a combination of the settings in slurm.conf as well as how users are requesting nodes. Currently, I have a user running an array that only needs one core. His script uses: #SBATCH --time=00:10:00 #SBATCH --ntasks=1 But slurm is allocating one node per task and not putting multiple tasks on a node. Additionally, it appears nobody else is allowed to use that node until his job completes. Could someone point me to the proper settings to have set for both users and slurm to accomplish this? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238
[slurm-dev] Fully utilizing nodes
All, I am trying to figure out the bits required to allow users to use part of a node and not block others from using remaining resources. It looks like the "OverSubscribe" option is what I need, but that doesn't seem to quite be all of it. I would like users to be able to request --exclusive if needed. However, I would like when users don't then slurm prefers to pack jobs onto as few nodes as possible when they start. I suspect it may be a combination of the settings in slurm.conf as well as how users are requesting nodes. Currently, I have a user running an array that only needs one core. His script uses: #SBATCH --time=00:10:00 #SBATCH --ntasks=1 But slurm is allocating one node per task and not putting multiple tasks on a node. Additionally, it appears nobody else is allowed to use that node until his job completes. Could someone point me to the proper settings to have set for both users and slurm to accomplish this? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238
[slurm-dev] List resources used/available
All, I am trying to find a way to see what resources are used/remaining on a per node basis. In particular memory and sockets/cpus/cores/threads Not seeing anything in the sinfo or scontrol man pages that show specifically that.. Any insight is appreciated. Brian Andrus
[slurm-dev] Re: distribution for array jobs
Ryan, The work I am doing is indeed homogenous. I am basically reading in a remote file and generating a checksum for comparison. Each array job is working through it’s own list of files. (eg: list1.txt, list2.txt, etc) Mpi is used to do a parallel run against the list. Each child process is grabbing a file and returning the checksum to the master. In this case, I want to spread the array jobs out because too many connections from a single box is causing a bottleneck. When the jobs land on separate nodes, the totals being processed is much greater than if they are packed onto a single node. Kinda new to using slurm (been using torque) and had issues trying to use srun. It would launch process 0 for every core or cpu requested, rather than what mpirun does (-np = number of cores requested). I think you may be right to a degree, but I have not been able to figure how to get srun to do something like run one instance of the program but allocate it 10 cores (eg: mpirun -np 10) Brian Andrus From: Ryan Cox [mailto:ryan_...@byu.edu] Sent: Thursday, January 28, 2016 9:45 AM To: slurm-dev Subject: [slurm-dev] Re: distribution for array jobs Unless I'm misunderstanding something, it sounds like you should be using jobs (sbatch) and job steps (srun) instead of job arrays (sbatch -a). The way I think of it, srun is like subletting a property that you're renting. Job arrays are for launching homogeneous work that differ just in an index number, though there are some other creative things you can do with them. Ryan On 01/27/2016 06:47 PM, Andrus, Brian Contractor wrote: I ended up just doing ‘scancel’ on all the jobs and resubmitting them. I seem to be making progress. Now I am having trouble figuring out the –distribution option. I want to have it such that each node runs 1 of each array job, but shares the remaining resources for other jobs. Here is what is in my script: #SBATCH --nodes=1 #SBATCH --sockets-per-node=1 #SBATCH --cores-per-socket=5 #SBATCH --threads-per-core=2 #SBATCH --distribution=cyclic:block,NoPack So I am getting 10 threads on a box. This is to run an mpi program. I do a sbatch: sbatch --array=1-100%2 slurm_array.sh I would expect my job to have 1 running on node1 and 1 on node2, but both start on node1. From: John Desantis [mailto:desan...@mail.usf.edu] Sent: Wednesday, January 27, 2016 7:37 AM To: slurm-dev <mailto:slurm-dev@schedmd.com> Subject: [slurm-dev] Re: Update job and partition for shared jobs Brian, I've never run into that message with SLURM yet. Have you tried releasing the jobs with scontrol, e.g. "scontrol release ID" where "ID" is the job number? We do not automatically requeue jobs due to a bug (fixed!) which caused the controller to crash because of an empty task_id_bitmap. John DeSantis 2016-01-26 20:05 GMT-05:00 Andrus, Brian Contractor mailto:bdand...@nps.edu>>: John, Thanks. That seemed to help; a job started on a node that had a job on it once the job that had been on it (‘using’ all the memory) completed. But now all my jobs won’t start and have a status of ‘JobHoldMaxRequeue’ From the docs, it seems that is because MAX_BATCH_REQUEUE is too low, but I don’t see where to change that. Even worse, I cannot seem to scancel any of those jobs just to clean things up and test stuff. Anyone know how to get rid of jobs with a status of ‘JobHoldMaxRequeue’? Brian Andrus From: John Desantis [mailto:desan...@mail.usf.edu<mailto:desan...@mail.usf.edu>] Sent: Tuesday, January 26, 2016 12:37 PM To: slurm-dev mailto:slurm-dev@schedmd.com>> Subject: [slurm-dev] Re: Update job and partition for shared jobs Brian, Try setting a default memory per CPU in the partition definition. Later versions of SLURM (>= 14.11.6?) require this value to be set, otherwise all memory per node is scheduled. HTH, John DeSantis 2016-01-26 15:20 GMT-05:00 Andrus, Brian Contractor mailto:bdand...@nps.edu>>: All, I am in the process of transitioning from Torque to Slurm. So far it is doing very well, especially handling arrays. Now I have one array job that is running across several nodes, but only using some of the node resources. I would like to have slurm start sharing the nodes so some of the array jobs will start where there are unused resources. I ran a scontrol update to force sharing and see the partition did change: #scontrol show partitions PartitionName=debug AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=YES QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=compute[45-49] Priority=1 RootOnly=NO ReqResv=NO Shared=FORCE:4 PreemptMode=OFF State=UP TotalCPUs=280 TotalNodes=5 SelectTypeParameters=N/A DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED But it is not starting job 416_37 on any node as I woul
[slurm-dev] distribution for array jobs
I ended up just doing ‘scancel’ on all the jobs and resubmitting them. I seem to be making progress. Now I am having trouble figuring out the –distribution option. I want to have it such that each node runs 1 of each array job, but shares the remaining resources for other jobs. Here is what is in my script: #SBATCH --nodes=1 #SBATCH --sockets-per-node=1 #SBATCH --cores-per-socket=5 #SBATCH --threads-per-core=2 #SBATCH --distribution=cyclic:block,NoPack So I am getting 10 threads on a box. This is to run an mpi program. I do a sbatch: sbatch --array=1-100%2 slurm_array.sh I would expect my job to have 1 running on node1 and 1 on node2, but both start on node1. From: John Desantis [mailto:desan...@mail.usf.edu] Sent: Wednesday, January 27, 2016 7:37 AM To: slurm-dev Subject: [slurm-dev] Re: Update job and partition for shared jobs Brian, I've never run into that message with SLURM yet. Have you tried releasing the jobs with scontrol, e.g. "scontrol release ID" where "ID" is the job number? We do not automatically requeue jobs due to a bug (fixed!) which caused the controller to crash because of an empty task_id_bitmap. John DeSantis 2016-01-26 20:05 GMT-05:00 Andrus, Brian Contractor mailto:bdand...@nps.edu>>: John, Thanks. That seemed to help; a job started on a node that had a job on it once the job that had been on it (‘using’ all the memory) completed. But now all my jobs won’t start and have a status of ‘JobHoldMaxRequeue’ From the docs, it seems that is because MAX_BATCH_REQUEUE is too low, but I don’t see where to change that. Even worse, I cannot seem to scancel any of those jobs just to clean things up and test stuff. Anyone know how to get rid of jobs with a status of ‘JobHoldMaxRequeue’? Brian Andrus From: John Desantis [mailto:desan...@mail.usf.edu<mailto:desan...@mail.usf.edu>] Sent: Tuesday, January 26, 2016 12:37 PM To: slurm-dev mailto:slurm-dev@schedmd.com>> Subject: [slurm-dev] Re: Update job and partition for shared jobs Brian, Try setting a default memory per CPU in the partition definition. Later versions of SLURM (>= 14.11.6?) require this value to be set, otherwise all memory per node is scheduled. HTH, John DeSantis 2016-01-26 15:20 GMT-05:00 Andrus, Brian Contractor mailto:bdand...@nps.edu>>: All, I am in the process of transitioning from Torque to Slurm. So far it is doing very well, especially handling arrays. Now I have one array job that is running across several nodes, but only using some of the node resources. I would like to have slurm start sharing the nodes so some of the array jobs will start where there are unused resources. I ran a scontrol update to force sharing and see the partition did change: #scontrol show partitions PartitionName=debug AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=YES QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=compute[45-49] Priority=1 RootOnly=NO ReqResv=NO Shared=FORCE:4 PreemptMode=OFF State=UP TotalCPUs=280 TotalNodes=5 SelectTypeParameters=N/A DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED But it is not starting job 416_37 on any node as I would expect. #squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 416_[37-1013%6] debug slurm_ar user1 PD 0:00 1 (Resources) 416_36 debug slurm_ar user1 R 35:46 1 compute49 416_35 debug slurm_ar user1 R1:47:25 1 compute46 416_33 debug slurm_ar user1 R7:30:50 1 compute45 416_32 debug slurm_ar user1 R7:38:39 1 compute47 416_31 debug slurm_ar user1 R8:53:26 1 compute48 In my config, I have: SelectType = select/cons_res SelectTypeParameters= CR_CORE_MEMORY What am I missing to get more than one job to run on a node? Thanks in advance, Brian Andrus
[slurm-dev] Re: Update job and partition for shared jobs
John, Thanks. That seemed to help; a job started on a node that had a job on it once the job that had been on it (‘using’ all the memory) completed. But now all my jobs won’t start and have a status of ‘JobHoldMaxRequeue’ From the docs, it seems that is because MAX_BATCH_REQUEUE is too low, but I don’t see where to change that. Even worse, I cannot seem to scancel any of those jobs just to clean things up and test stuff. Anyone know how to get rid of jobs with a status of ‘JobHoldMaxRequeue’? Brian Andrus From: John Desantis [mailto:desan...@mail.usf.edu] Sent: Tuesday, January 26, 2016 12:37 PM To: slurm-dev Subject: [slurm-dev] Re: Update job and partition for shared jobs Brian, Try setting a default memory per CPU in the partition definition. Later versions of SLURM (>= 14.11.6?) require this value to be set, otherwise all memory per node is scheduled. HTH, John DeSantis 2016-01-26 15:20 GMT-05:00 Andrus, Brian Contractor mailto:bdand...@nps.edu>>: All, I am in the process of transitioning from Torque to Slurm. So far it is doing very well, especially handling arrays. Now I have one array job that is running across several nodes, but only using some of the node resources. I would like to have slurm start sharing the nodes so some of the array jobs will start where there are unused resources. I ran a scontrol update to force sharing and see the partition did change: #scontrol show partitions PartitionName=debug AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=YES QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=compute[45-49] Priority=1 RootOnly=NO ReqResv=NO Shared=FORCE:4 PreemptMode=OFF State=UP TotalCPUs=280 TotalNodes=5 SelectTypeParameters=N/A DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED But it is not starting job 416_37 on any node as I would expect. #squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 416_[37-1013%6] debug slurm_ar user1 PD 0:00 1 (Resources) 416_36 debug slurm_ar user1 R 35:46 1 compute49 416_35 debug slurm_ar user1 R1:47:25 1 compute46 416_33 debug slurm_ar user1 R7:30:50 1 compute45 416_32 debug slurm_ar user1 R7:38:39 1 compute47 416_31 debug slurm_ar user1 R8:53:26 1 compute48 In my config, I have: SelectType = select/cons_res SelectTypeParameters= CR_CORE_MEMORY What am I missing to get more than one job to run on a node? Thanks in advance, Brian Andrus
[slurm-dev] Update job and partition for shared jobs
All, I am in the process of transitioning from Torque to Slurm. So far it is doing very well, especially handling arrays. Now I have one array job that is running across several nodes, but only using some of the node resources. I would like to have slurm start sharing the nodes so some of the array jobs will start where there are unused resources. I ran a scontrol update to force sharing and see the partition did change: #scontrol show partitions PartitionName=debug AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=YES QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=compute[45-49] Priority=1 RootOnly=NO ReqResv=NO Shared=FORCE:4 PreemptMode=OFF State=UP TotalCPUs=280 TotalNodes=5 SelectTypeParameters=N/A DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED But it is not starting job 416_37 on any node as I would expect. #squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 416_[37-1013%6] debug slurm_ar user1 PD 0:00 1 (Resources) 416_36 debug slurm_ar user1 R 35:46 1 compute49 416_35 debug slurm_ar user1 R1:47:25 1 compute46 416_33 debug slurm_ar user1 R7:30:50 1 compute45 416_32 debug slurm_ar user1 R7:38:39 1 compute47 416_31 debug slurm_ar user1 R8:53:26 1 compute48 In my config, I have: SelectType = select/cons_res SelectTypeParameters= CR_CORE_MEMORY What am I missing to get more than one job to run on a node? Thanks in advance, Brian Andrus
[slurm-dev] Re: Adjust an array job's maximum simultaneous running tasks
That’s it thanks! I looked right past it since they changed terminology from “simultaneously” (in the sbatch man page) To “execute at the same time” (in the scontrol man page) ☺ Brian From: Ryan Clough [mailto:ryan.clo...@dsic.com] Sent: Wednesday, January 20, 2016 10:49 PM To: slurm-dev Subject: [slurm-dev] Re: Adjust an array job's maximum simultaneous running tasks Might I suggest this(gleaned from scontrol manpage): scontrol update ArrayTaskThrottle= JobId= ___ ¯\_(ツ)_/¯ Ryan Clough Information Systems Decision Sciences International Corporation<http://www.decisionsciencescorp.com/> On Wed, Jan 20, 2016 at 6:49 PM, Andrus, Brian Contractor mailto:bdand...@nps.edu>> wrote: All, Is there a way to change the maximum simultaneous running tasks of an array job that is currently running? For example I have sbatch --array=1-100%2 and I want to change it to effectively be: sbatch --array=1-100%5 to cause slurm to start running 5 at a time right away. And be able to do the inverse to effectively throttle or slow down the rate of a particular array job. Brian Andrus This email and its contents are confidential. If you are not the intended recipient, please do not disclose or use the information within this email or its attachments. If you have received this email in error, please report the error to the sender by return email and delete this communication from your records.
[slurm-dev] Adjust an array job's maximum simultaneous running tasks
All, Is there a way to change the maximum simultaneous running tasks of an array job that is currently running? For example I have sbatch --array=1-100%2 and I want to change it to effectively be: sbatch --array=1-100%5 to cause slurm to start running 5 at a time right away. And be able to do the inverse to effectively throttle or slow down the rate of a particular array job. Brian Andrus
[slurm-dev] Re: NodeName and PartitionName format in slurm.conf
That's what I have done for now. I'm just a little OCD about how the conf file looks and don't care for 8 lines worth of wraparound. Managing is done at the pxeboot/kickstart level and yum. I can dynamically install the bits necessary for the various hardware differences (eg: GPUs, MIC cards, Infiniband, etc). Brian Andrus -Original Message- From: Benjamin Redling [mailto:benjamin.ra...@uni-jena.de] Sent: Wednesday, January 20, 2016 2:00 AM To: slurm-dev Subject: [slurm-dev] Re: NodeName and PartitionName format in slurm.conf Am 19.01.2016 um 20:37 schrieb Andrus, Brian Contractor: > I am testing our slurm to replace our torque/moab setup here. > > The issue I have is to try and put all our node names in the NodeName > and PartitionName entries. > In our cluster, we name our nodes compute-- That seems to > be problem enough with the abilities to use ranges in slurm, but it is > compounded with the fact that the folks put the nodes in keeping 1u of > space in between. > So I have compute-1-[1,3,5,7,9,11...41] Why not simply use a comma separated list _generated_ from your inventory / DNS / /etc/hosts / etc. .? When you have outliers (2U, 4U -- do they have more resources too!?) it would make sense to group/partition by resources anyway. What are you using to manage inventory? Most configuration management and provisioning tools I know provide you with the necessary tools -- have a look at puppetlabs facter (or alternatives). http://slurm.schedmd.com/slurm.conf.html Multiple node names may be comma separated (e.g. "alpha,beta,gamma") and/or a simple node range expression may optionally be used to specify numeric ranges of nodes to avoid building a configuration file with large numbers of entries. The node range expression can contain one pair of square brackets with a sequence of comma separated numbers and/or ranges of numbers separated by a "-" (e.g. "linux[0-64,128]", or "lx[15,18,32-33]"). Note that the numeric ranges can include one or more leading zeros to indicate the numeric portion has a fixed number of digits (e.g. "linux[-1023]"). Up to two numeric ranges can be included in the expression (e.g. "rack[0-63]_blade[0-41]"). If one or more numeric expressions are included, one of them must be at the end of the name (e.g. "unit[0-31]rack" is invalid), but arbitrary names can always be used in a comma separated list. Complicating that logic wouldn't make much sense to me. Mapping host names to partitions shouldn't be too hard to script. In the worst case you copy the full/per-rack/per-resources host list to partitions and manually cherry-pick afterwards. Regards, Benjamin -- FSU Jena | JULIELab.de/Staff/Benjamin+Redling.html vox: +49 3641 9 44323 | fax: +49 3641 9 44321
[slurm-dev] NodeName and PartitionName format in slurm.conf
All, I am testing our slurm to replace our torque/moab setup here. The issue I have is to try and put all our node names in the NodeName and PartitionName entries. In our cluster, we name our nodes compute-- That seems to be problem enough with the abilities to use ranges in slurm, but it is compounded with the fact that the folks put the nodes in keeping 1u of space in between. So I have compute-1-[1,3,5,7,9,11...41] Add to that, some nodes are 2u or 4u and some are dual nodes, so something like that would be compute-2-5a and compute-2-5b Slurm does not seem happy if you put anything outside a range bracket (eg: NodeName=compute-3-[31-37]a) It complains: scontrol: error: Unable to create NodeAddr list from compute-3-[31-37]a So is there a clean way to have such weird naming in NodeName and PartitionName? If not, a suggestion: If it could do something like pdsh uses, that would be nice; comma separated with brackets that are expanded. And/Or even be able to line wrap would be good NodeName=compute-1-[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39], compute-2-[3,5,7,9,11,13,15,17,19,23,25,27,29], compute-3-[31-37]a, compute-3-[31-37]b, compute-4-[1,5,9,13,17,26,31,35,42] Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238
