Just an observation. I have found that to ensure larger or unlimited limits for the slurm processes, I needed to add appropriate limits sentences to the /etc/init.d/slurm file prior to the actual start stanza. Seems the daemons are started prior to the instantiation of the /etc/security/limits.conf action.
Hope this helps. -----Original Message----- From: owner-slurm-...@lists.llnl.gov [mailto:owner-slurm-...@lists.llnl.gov] On Behalf Of Lipari, Don Sent: Monday, October 24, 2011 2:21 PM To: slurm-dev@lists.llnl.gov Subject: RE: [slurm-dev] Max Locked Memory Problem (sbatch not able to change it) Also the limits in effect when the slurmd was started on lbs02 might be the cause of what you're seeing. > -----Original Message----- > From: owner-slurm-...@lists.llnl.gov [mailto:owner-slurm- > d...@lists.llnl.gov] On Behalf Of Mark A. Grondona > Sent: Monday, October 24, 2011 9:39 AM > To: Matteo Guglielmi; SLURM > Subject: Re: [slurm-dev] Max Locked Memory Problem (sbatch not able to > change it) > > > Do you have PAM enabled in SLURM? > > https://computing.llnl.gov/linux/slurm/faq.html#pam > > On Mon, 24 Oct 2011 07:26:58 -0700, Matteo Guglielmi > <matteo.guglie...@epfl.ch> wrote: > > On all nodes we have: > > > > $ cat /etc/security/limits.conf > > * soft memlock unlimited > > * hard memlock unlimited > > > > > > ############# jobfile ############### > > #!/bin/bash > > #SBATCH -o job-%N-%j.out > > #SBATCH -J testlimits > > #SBATCH --nodes=1 > > > > whoami > > > > echo > > > > hostname > > > > echo > > > > ulimit -a > > > > echo > > > > ulimit -l unlimited > > > > echo > > > > ulimit -a > > > > ulimit -s unlimited > > > > echo > > > > ulimit -a > > ############################# > > > > $ sbatch job.sh > > > > Submitted batch job 1723 > > > > $ cat job-lbs02-1723.out > > > > software > > > > lbs02 > > > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) 131072 > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 204800 > > max locked memory (kbytes, -l) 32 <<<<<==== > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) 8192 <<<<===== > > cpu time (seconds, -t) unlimited > > max user processes (-u) 71680 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > > > > /var/run/slurm/slurmd/job01723/slurm_script: line 18: ulimit: max > locked memory: cannot modify limit: Operation not permitted > > > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) 131072 > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 204800 > > max locked memory (kbytes, -l) 32 <<<<<==== > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) 8192 <<<<<==== > > cpu time (seconds, -t) unlimited > > max user processes (-u) 71680 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > > > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) 131072 > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 204800 > > max locked memory (kbytes, -l) 32 <<<<<==== > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) unlimited <<<<<==== > > cpu time (seconds, -t) unlimited > > max user processes (-u) 71680 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > > > > > > But if I login directly on node "lbs02" via ssh both > > "ulimit -l unlimited" and "ulimit -s unlimited" commands > > give me no problem at all. > > > > [software@lbs02:~]$ ulimit -l unlimited > > > > [software@lbs02:~]$ ulimit -s unlimited > > > > [software@lbs02:~]$ ulimit -a > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) unlimited > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 204800 > > max locked memory (kbytes, -l) unlimited <<<<<==== > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) unlimited <<<<<==== > > cpu time (seconds, -t) unlimited > > max user processes (-u) 204800 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > > > > > > Thanks, > > > > --matt > >
