Re: [slurm-users] Slurm - UnkillableStepProgram
Hi Mike, On 22/3/21 7:12 pm, Yap, Mike wrote: # I presume UnkillableStepTimeout is set in slurm.conf. and it act as a timer to trigger UnkillableStepProgram That is correct. # UnkillableStepProgram can be use to send email or reboot compute node – question is how do we configure it ? Also - or to automate collecting debug info (which is what we do) and then we manually intervene to reboot the node once we've determined there's no more useful info to collect. It's just configured in your slurm.conf. UnkillableStepProgram=/path/to/the/unkillable/step/script.sh Of course this script has to be present on every compute node. All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
[slurm-users] Slurm - UnkillableStepProgram
Hi All Have been reading on the archive hoping to implement unkillablesteptimeout and unkillablesteprogram to the slurm But I'm kind of confuse with it application 1. I presume UnkillableStepTimeout is set in slurm.conf. and it act as a timer to trigger UnkillableStepProgram 2. UnkillableStepProgram can be use to send email or reboot compute node - question is how do we configure it ? scontrol show config | grep -i kill KillOnBadExit = 1 KillWait= 30 sec UnkillableStepProgram = (null) UnkillableStepTimeout = 300 sec Please advise Thanks Mike
[slurm-users] Slurm prolog export variable
Hi All Can anyone assist the following We're using Bright Cluster 9.1 with CentOS 7.9 running with slurm 2.02.6 We have a script running on prolog exporting the SCRATCH as variable for user running job Addition command on the script to create a user folder accordingly When submitting the job, the folder is created without any error on the compute node (which indicate the prolog is being executed without an issue) but $SCRATCH seems to be undone once the prolog script is done We also tried to create an additional script to only export SCRATCH but no luck We have been using the same script for an older setup and it work fine. Any suggestion if there is any workaround to define $SCRACTH beside getting user to define it in their submit script ? Prolog script #!/bin/sh export SCRATCH=/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID mkdir --parents $SCRATCH Thanks Mike
Re: [slurm-users] [External] srun at front-end nodes with --enable_configless fails with "Can't find an address, check slurm.conf"
Also check the settings on your nodeaddr in slurm.conf On 3/22/21, 2:48 PM, "slurm-users on behalf of Michael Robbert" wrote: I haven't tried configless setup yet, but the problem you're hitting looks like it could be a DNS issue. Can you do a dns lookup of n26 from the login node? The way that non-interactive batch jobs are started may not require that, but I believe that it is required for interactive jobs. Mike Robbert Cyberinfrastructure Specialist, Cyberinfrastructure and Advanced Research Computing Information and Technology Solutions (ITS) 303-273-3786 | mrobb...@mines.edu Our values: Trust | Integrity | Respect | Responsibility On 3/22/21, 11:24, "slurm-users on behalf of Josef Dvoracek" wrote: Hi @list; I was able to configure "configless" slurm cluster with quite minimalistic slurm.conf everywhere, of-course excepting slurmctld server. All nodes are running slurmd, including front-end/login nodes to pull the config. Submitting jobs using sbatch scripts works fine, but interactive jobs using srun are failing with $ srun --verbose -w n26 --pty /bin/bash ... srun: error: fwd_tree_thread: can't find address for host n26, check slurm.conf srun: error: Task launch for 200137.0 failed on node n26: Can't find an address, check slurm.conf srun: error: Application launch failed: Can't find an address, check slurm.conf ... Does it mean that on submit hosts one has to manually specify all relevant NodeNames? I thought that running slurmd there will pull configuration from slurmserver. (I can see the file is actually sucessfully pulled into /run/slurm/conf/slurm.conf ). So far I found two workarounds: workaround1: specify nodenames at login/front-end nodes in slurm.conf: NodeName=n[(...)n26(...)] Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN then, srun works as expected. workaround2: directing environment variable SLURM_CONF to the slurm.conf pulled by slurmd: export SLURM_CONF=/run/slurm/conf/slurm.conf then again, srun works as expected. Is this expected behavior? I actually expected that srun at configless login/front-end node with running slurmd recognizes the pulled configuration, but apparently, that's not the case. cheers josef setup at front-end and compute nodes: [root@FRONTEND ~]# slurmd --version slurm 20.02.5 [root@FRONTEND ~]# [root@FRONTEND ~]# cat /etc/sysconfig/slurmd SLURMD_OPTIONS="--conf-server slurmserver2.DOMAIN" [root@FRONTEND ~]# [root@FRONTEND ~]# cat /etc/slurm/slurm.conf ClusterName=CLUSTERNAME ControlMachine=slurmserver2.DOMAIN AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=slurmserver2.DOMAIN AccountingStoragePort=7031 SlurmctldParameters=enable_configless [root@FRONTEND ~]# ClusterName=X ControlMachine=slurmserver2.DOMAIN AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=slurmserver2.DOMAIN AccountingStoragePort=7031 SlurmctldParameters=enable_configless -- Josef Dvoracek Institute of Physics | Czech Academy of Sciences cell: +420 608 563 558 | https://telegram.me/jose_d | FZU phone nr. : 2669
Re: [slurm-users] [External] srun at front-end nodes with --enable_configless fails with "Can't find an address, check slurm.conf"
I haven't tried configless setup yet, but the problem you're hitting looks like it could be a DNS issue. Can you do a dns lookup of n26 from the login node? The way that non-interactive batch jobs are started may not require that, but I believe that it is required for interactive jobs. Mike Robbert Cyberinfrastructure Specialist, Cyberinfrastructure and Advanced Research Computing Information and Technology Solutions (ITS) 303-273-3786 | mrobb...@mines.edu Our values: Trust | Integrity | Respect | Responsibility On 3/22/21, 11:24, "slurm-users on behalf of Josef Dvoracek" wrote: Hi @list; I was able to configure "configless" slurm cluster with quite minimalistic slurm.conf everywhere, of-course excepting slurmctld server. All nodes are running slurmd, including front-end/login nodes to pull the config. Submitting jobs using sbatch scripts works fine, but interactive jobs using srun are failing with $ srun --verbose -w n26 --pty /bin/bash ... srun: error: fwd_tree_thread: can't find address for host n26, check slurm.conf srun: error: Task launch for 200137.0 failed on node n26: Can't find an address, check slurm.conf srun: error: Application launch failed: Can't find an address, check slurm.conf ... Does it mean that on submit hosts one has to manually specify all relevant NodeNames? I thought that running slurmd there will pull configuration from slurmserver. (I can see the file is actually sucessfully pulled into /run/slurm/conf/slurm.conf ). So far I found two workarounds: workaround1: specify nodenames at login/front-end nodes in slurm.conf: NodeName=n[(...)n26(...)] Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN then, srun works as expected. workaround2: directing environment variable SLURM_CONF to the slurm.conf pulled by slurmd: export SLURM_CONF=/run/slurm/conf/slurm.conf then again, srun works as expected. Is this expected behavior? I actually expected that srun at configless login/front-end node with running slurmd recognizes the pulled configuration, but apparently, that's not the case. cheers josef setup at front-end and compute nodes: [root@FRONTEND ~]# slurmd --version slurm 20.02.5 [root@FRONTEND ~]# [root@FRONTEND ~]# cat /etc/sysconfig/slurmd SLURMD_OPTIONS="--conf-server slurmserver2.DOMAIN" [root@FRONTEND ~]# [root@FRONTEND ~]# cat /etc/slurm/slurm.conf ClusterName=CLUSTERNAME ControlMachine=slurmserver2.DOMAIN AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=slurmserver2.DOMAIN AccountingStoragePort=7031 SlurmctldParameters=enable_configless [root@FRONTEND ~]# ClusterName=X ControlMachine=slurmserver2.DOMAIN AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=slurmserver2.DOMAIN AccountingStoragePort=7031 SlurmctldParameters=enable_configless -- Josef Dvoracek Institute of Physics | Czech Academy of Sciences cell: +420 608 563 558 | https://telegram.me/jose_d | FZU phone nr. : 2669 smime.p7s Description: S/MIME cryptographic signature
[slurm-users] srun at front-end nodes with --enable_configless fails with "Can't find an address, check slurm.conf"
Hi @list; I was able to configure "configless" slurm cluster with quite minimalistic slurm.conf everywhere, of-course excepting slurmctld server. All nodes are running slurmd, including front-end/login nodes to pull the config. Submitting jobs using sbatch scripts works fine, but interactive jobs using srun are failing with $ srun --verbose -w n26 --pty /bin/bash ... srun: error: fwd_tree_thread: can't find address for host n26, check slurm.conf srun: error: Task launch for 200137.0 failed on node n26: Can't find an address, check slurm.conf srun: error: Application launch failed: Can't find an address, check slurm.conf ... Does it mean that on submit hosts one has to manually specify all relevant NodeNames? I thought that running slurmd there will pull configuration from slurmserver. (I can see the file is actually sucessfully pulled into /run/slurm/conf/slurm.conf ). So far I found two workarounds: workaround1: specify nodenames at login/front-end nodes in slurm.conf: NodeName=n[(...)n26(...)] Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN then, srun works as expected. workaround2: directing environment variable SLURM_CONF to the slurm.conf pulled by slurmd: export SLURM_CONF=/run/slurm/conf/slurm.conf then again, srun works as expected. Is this expected behavior? I actually expected that srun at configless login/front-end node with running slurmd recognizes the pulled configuration, but apparently, that's not the case. cheers josef setup at front-end and compute nodes: [root@FRONTEND ~]# slurmd --version slurm 20.02.5 [root@FRONTEND ~]# [root@FRONTEND ~]# cat /etc/sysconfig/slurmd SLURMD_OPTIONS="--conf-server slurmserver2.DOMAIN" [root@FRONTEND ~]# [root@FRONTEND ~]# cat /etc/slurm/slurm.conf ClusterName=CLUSTERNAME ControlMachine=slurmserver2.DOMAIN AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=slurmserver2.DOMAIN AccountingStoragePort=7031 SlurmctldParameters=enable_configless [root@FRONTEND ~]# ClusterName=X ControlMachine=slurmserver2.DOMAIN AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=slurmserver2.DOMAIN AccountingStoragePort=7031 SlurmctldParameters=enable_configless -- Josef Dvoracek Institute of Physics | Czech Academy of Sciences cell: +420 608 563 558 | https://telegram.me/jose_d | FZU phone nr. : 2669 smime.p7s Description: S/MIME Cryptographic Signature
Re: [slurm-users] Set Fairshare by Hand
I asked something similar a few months ago and wasn't able to find anything to suit my needs. https://groups.google.com/g/slurm-users/c/ude1M5w_4IU/m/R2GziD9JAQAJ Good luck! Luke -Original Message- From: slurm-users On Behalf Of Paul Edmon Sent: Monday, March 22, 2021 6:31 AM To: slurm-users@lists.schedmd.com Subject: Re: [slurm-users] Set Fairshare by Hand External email: Use caution opening links or attachments No, there is no way to my knowledge to do this. You can zero out some one's fairshare (by removing and readding them) or a groups fairshare but you can't set it to an arbitrary value. You can always adjust their RawShares for a somewhat similar effect but that will have all the normal consequences of changing their RawShares. -Paul Edmon- On 3/22/2021 5:12 AM, Michael Müller wrote: > Dear Slurm users and admins, > > can we set the faireshare values manually, i.e., they are not > (re)calculated be Slurm? > > With kind regards > Michael >
Re: [slurm-users] Slurm version 20.11.5 is now available
I create the link at job runtime. For my case, I use the prologue to validate the link since I know ahead of time what is needed based on job templates that are used. In the more general case, I would have it be part of the batch script the user runs. Just ensure they clean up afterwards (I use the epilogue for this). So long as the destination directory is writable (chmod 1777) this would work well. Yet another option is to tell them to use an environment variable rather than a path and you could have those available. Eg: FAST_SCRATCH=/mnt/ssd BIG_SCRATCH=/mnt/sdb1 RAM_SCRATCH=/dev/shm Then have them use $SCRATCH after something like SCRATCH=$FAST_SCRATCH Just set SCRATCH to the one you want to use. Brian Andrus On 3/21/2021 11:32 PM, Loris Bennett wrote: Brian Andrus writes: The method I use for jobs is to make /scratch a symlink to where ever it may be best suited. Then all users just use /scratch eg: /scratch -> /dev/shm for a ramdisk or /scrach->/mnt/ssd for local ssd, etc When do you create the symlink and how do you decide what link to create? Doesn't it depend on the job, e.g. whether scratch needs to be, say, particularly large or particularly fast? Brian Andrus On 3/19/2021 6:25 AM, Paul Edmon wrote: I was about to ask this as well as we use /scratch as our tmp space not /tmp. I haven't kicked the tires on this to know how it works but after I take a look at it I will probably file a feature request to make the name of the tmp dir flexible. -Paul Edmon- On 3/19/2021 7:19 AM, Tina Friedrich wrote: That's excellent; I've been using the 'auto_tmpdir' plugin for this; having that functionality within SLURM will be good. Have a question though - we have a need to also create a per-job /scratch/ (on a shared fast file system) in much the same way. I don't see a way that the currentl tmpfs plugin can be used to do that, as it would seem that it's hard-coded to mount things into /tmp/ (i.e. where to mount a file system can not be changed). Or am I misreading this? Tina On 16/03/2021 22:26, Tim Wickberg wrote: One errant backspace snuck into that announcement: the job_container.conf man page (with an 'r') serves as the initial documentation for this new job_container/tmpfs plugin. The link to the HTML version of the man page has been corrected in the text below: On 3/16/21 4:16 PM, Tim Wickberg wrote: We are pleased to announce the availability of Slurm version 20.11.5. This includes a number of moderate severity bug fixes, alongside a new job_container/tmpfs plugin developed by NERSC that can be used to create per-job filesystem namespaces. Initial documentation for this plugin is available at: https://slurm.schedmd.com/job_container.conf.html Slurm can be downloaded from https://www.schedmd.com/downloads.php . - Tim
Re: [slurm-users] Set Fairshare by Hand
No, there is no way to my knowledge to do this. You can zero out some one's fairshare (by removing and readding them) or a groups fairshare but you can't set it to an arbitrary value. You can always adjust their RawShares for a somewhat similar effect but that will have all the normal consequences of changing their RawShares. -Paul Edmon- On 3/22/2021 5:12 AM, Michael Müller wrote: Dear Slurm users and admins, can we set the faireshare values manually, i.e., they are not (re)calculated be Slurm? With kind regards Michael
[slurm-users] Set Fairshare by Hand
Dear Slurm users and admins, can we set the faireshare values manually, i.e., they are not (re)calculated be Slurm? With kind regards Michael -- Michael Müller Application Developer Department of System and Service Design (SDE) Center of Information Services and High Performance Computing (ZIH) Dresden University of Technology 01069 Dresden phone: (0351)463-35261 www: www.tu-dresden.de/zih smime.p7s Description: S/MIME Cryptographic Signature
Re: [slurm-users] Slurm version 20.11.5 is now available
Brian Andrus writes: > The method I use for jobs is to make /scratch a symlink to where ever it may > be > best suited. Then all users just use /scratch > > eg: /scratch -> /dev/shm for a ramdisk or /scrach->/mnt/ssd for local ssd, etc When do you create the symlink and how do you decide what link to create? Doesn't it depend on the job, e.g. whether scratch needs to be, say, particularly large or particularly fast? > Brian Andrus > > On 3/19/2021 6:25 AM, Paul Edmon wrote: >> I was about to ask this as well as we use /scratch as our tmp space not >> /tmp. >> I haven't kicked the tires on this to know how it works but after I take a >> look at it I will probably file a feature request to make the name of the tmp >> dir flexible. >> >> -Paul Edmon- >> >> On 3/19/2021 7:19 AM, Tina Friedrich wrote: >>> That's excellent; I've been using the 'auto_tmpdir' plugin for this; having >>> that functionality within SLURM will be good. >>> >>> Have a question though - we have a need to also create a per-job /scratch/ >>> (on a shared fast file system) in much the same way. >>> >>> I don't see a way that the currentl tmpfs plugin can be used to do that, as >>> it would seem that it's hard-coded to mount things into /tmp/ (i.e. where to >>> mount a file system can not be changed). Or am I misreading this? >>> >>> Tina >>> >>> On 16/03/2021 22:26, Tim Wickberg wrote: One errant backspace snuck into that announcement: the job_container.conf man page (with an 'r') serves as the initial documentation for this new job_container/tmpfs plugin. The link to the HTML version of the man page has been corrected in the text below: On 3/16/21 4:16 PM, Tim Wickberg wrote: > We are pleased to announce the availability of Slurm version 20.11.5. > > This includes a number of moderate severity bug fixes, alongside a new > job_container/tmpfs plugin developed by NERSC that can be used to create > per-job filesystem namespaces. > > Initial documentation for this plugin is available at: > https://slurm.schedmd.com/job_container.conf.html > Slurm can be downloaded from https://www.schedmd.com/downloads.php . > > - Tim > >>> >>
[slurm-users] About memory limits with srun
Hi all, I sent this mail from a bad email address this week-end. I apologize if it is published duplicate (but not found in the archive yet). May be this is a basic question but I'm stuck with it. I'm quite new in managing a small cluster with slurm instead of a local batch scheduler. On the nodes I've set memory limits in slurm.conf. DefMemPerCPU=2048 MaxMemPerCPU=4096 Requesting 1.2GB of RAM works: srun --ntasks-per-node=1 --mem-per-cpu=1500M -p tenibre-gpu --pty bash -i and my testcase can allocate until 1.5GB: ./a.out allocation de 1000Mo.Ok allocation de 1419Mo.Ok allocation de 1524Mo.Ok Killed Now I would like to use more memory than MaxMemPerCPU: srun --ntasks-per-node=1 --mem-per-cpu=12G -p tenibre-gpu --pty bash -i So, if I understand the documentation, as mem-per-cpu > MaxMemPerCPU this is a limitation applied to the task and it agregates cpu and memory. The squeue command show 3 cpu agregated on the node to reach the 3*MaxMemPerCPU memory requested so all seams correct. JOBID PARTITION NAME USER ST TIME START_TIME TIME_LIMIT CPUS NODELIST(REASON) 497 tenibre-gpu bash begou R 1:23 2021-03-20T14:42:47 12:00:00 3 tenibre-gpu-0 But my task is unable to exceed the MaxMemPerCPU value ? ./a.out allocation de 1000Mo.Ok allocation de 4145Mo.Ok allocation de 4250Mo.Ok Killed So, I'm wrong somewhere but ? Running the testcase in a ssh sessions (ssh as root then su as a basic user) allows using more memory so it is related to my bad slurm setup/use Patrick
