This is essentially what I do as well. To save me modifying the actual init script I add the ulimit entries to /etc/sysconfig/slurm:
# The below line raises the memlock limit so that when slurmd is launched user tasks will have unlimited memlock ulimit -l unlimited # Increase number of open files ulimit -n 8192 # Memlocks the slurmd process's memory so that when a node starts swapping, the slurmd will continue to respond SLURMD_OPTIONS="-M" -Aaron On Mon, Oct 24, 2011 at 2:39 PM, Williams, Kevin E. (Federal SIP) < kevin.e.willi...@hp.com> wrote: > Just an observation. I have found that to ensure larger or unlimited > limits for the slurm processes, I needed to add appropriate limits sentences > to the /etc/init.d/slurm file prior to the actual start stanza. Seems the > daemons are started prior to the instantiation of the > /etc/security/limits.conf action. > > Hope this helps. > > -----Original Message----- > From: owner-slurm-...@lists.llnl.gov [mailto: > owner-slurm-...@lists.llnl.gov] On Behalf Of Lipari, Don > Sent: Monday, October 24, 2011 2:21 PM > To: slurm-dev@lists.llnl.gov > Subject: RE: [slurm-dev] Max Locked Memory Problem (sbatch not able to > change it) > > Also the limits in effect when the slurmd was started on lbs02 might be the > cause of what you're seeing. > > > > -----Original Message----- > > From: owner-slurm-...@lists.llnl.gov [mailto:owner-slurm- > > d...@lists.llnl.gov] On Behalf Of Mark A. Grondona > > Sent: Monday, October 24, 2011 9:39 AM > > To: Matteo Guglielmi; SLURM > > Subject: Re: [slurm-dev] Max Locked Memory Problem (sbatch not able to > > change it) > > > > > > Do you have PAM enabled in SLURM? > > > > https://computing.llnl.gov/linux/slurm/faq.html#pam > > > > On Mon, 24 Oct 2011 07:26:58 -0700, Matteo Guglielmi > > <matteo.guglie...@epfl.ch> wrote: > > > On all nodes we have: > > > > > > $ cat /etc/security/limits.conf > > > * soft memlock unlimited > > > * hard memlock unlimited > > > > > > > > > ############# jobfile ############### > > > #!/bin/bash > > > #SBATCH -o job-%N-%j.out > > > #SBATCH -J testlimits > > > #SBATCH --nodes=1 > > > > > > whoami > > > > > > echo > > > > > > hostname > > > > > > echo > > > > > > ulimit -a > > > > > > echo > > > > > > ulimit -l unlimited > > > > > > echo > > > > > > ulimit -a > > > > > > ulimit -s unlimited > > > > > > echo > > > > > > ulimit -a > > > ############################# > > > > > > $ sbatch job.sh > > > > > > Submitted batch job 1723 > > > > > > $ cat job-lbs02-1723.out > > > > > > software > > > > > > lbs02 > > > > > > core file size (blocks, -c) 0 > > > data seg size (kbytes, -d) 131072 > > > scheduling priority (-e) 0 > > > file size (blocks, -f) unlimited > > > pending signals (-i) 204800 > > > max locked memory (kbytes, -l) 32 <<<<<==== > > > max memory size (kbytes, -m) unlimited > > > open files (-n) 1024 > > > pipe size (512 bytes, -p) 8 > > > POSIX message queues (bytes, -q) 819200 > > > real-time priority (-r) 0 > > > stack size (kbytes, -s) 8192 <<<<===== > > > cpu time (seconds, -t) unlimited > > > max user processes (-u) 71680 > > > virtual memory (kbytes, -v) unlimited > > > file locks (-x) unlimited > > > > > > /var/run/slurm/slurmd/job01723/slurm_script: line 18: ulimit: max > > locked memory: cannot modify limit: Operation not permitted > > > > > > core file size (blocks, -c) 0 > > > data seg size (kbytes, -d) 131072 > > > scheduling priority (-e) 0 > > > file size (blocks, -f) unlimited > > > pending signals (-i) 204800 > > > max locked memory (kbytes, -l) 32 <<<<<==== > > > max memory size (kbytes, -m) unlimited > > > open files (-n) 1024 > > > pipe size (512 bytes, -p) 8 > > > POSIX message queues (bytes, -q) 819200 > > > real-time priority (-r) 0 > > > stack size (kbytes, -s) 8192 <<<<<==== > > > cpu time (seconds, -t) unlimited > > > max user processes (-u) 71680 > > > virtual memory (kbytes, -v) unlimited > > > file locks (-x) unlimited > > > > > > core file size (blocks, -c) 0 > > > data seg size (kbytes, -d) 131072 > > > scheduling priority (-e) 0 > > > file size (blocks, -f) unlimited > > > pending signals (-i) 204800 > > > max locked memory (kbytes, -l) 32 <<<<<==== > > > max memory size (kbytes, -m) unlimited > > > open files (-n) 1024 > > > pipe size (512 bytes, -p) 8 > > > POSIX message queues (bytes, -q) 819200 > > > real-time priority (-r) 0 > > > stack size (kbytes, -s) unlimited <<<<<==== > > > cpu time (seconds, -t) unlimited > > > max user processes (-u) 71680 > > > virtual memory (kbytes, -v) unlimited > > > file locks (-x) unlimited > > > > > > > > > But if I login directly on node "lbs02" via ssh both > > > "ulimit -l unlimited" and "ulimit -s unlimited" commands > > > give me no problem at all. > > > > > > [software@lbs02:~]$ ulimit -l unlimited > > > > > > [software@lbs02:~]$ ulimit -s unlimited > > > > > > [software@lbs02:~]$ ulimit -a > > > core file size (blocks, -c) 0 > > > data seg size (kbytes, -d) unlimited > > > scheduling priority (-e) 0 > > > file size (blocks, -f) unlimited > > > pending signals (-i) 204800 > > > max locked memory (kbytes, -l) unlimited <<<<<==== > > > max memory size (kbytes, -m) unlimited > > > open files (-n) 1024 > > > pipe size (512 bytes, -p) 8 > > > POSIX message queues (bytes, -q) 819200 > > > real-time priority (-r) 0 > > > stack size (kbytes, -s) unlimited <<<<<==== > > > cpu time (seconds, -t) unlimited > > > max user processes (-u) 204800 > > > virtual memory (kbytes, -v) unlimited > > > file locks (-x) unlimited > > > > > > > > > Thanks, > > > > > > --matt > > > > > > -- Aaron Knister Systems Administrator Division of Information Technology University of Maryland, Baltimore County aar...@umbc.edu
