Stuart Barkley wrote:
We have applied parts of Mark's patch set to our 8.1.8 system. The consolidated patches we use are attached. I don't believe we did anything significant except update them for 8.1.8 sources.
I've given the h_vmem patch some limited testing and things look good. My test job was:alexis@te-ahux01$ cat job2.sh #$ -clear #$ -w e #$ -notify
#$ -S /bin/bash
#$ -j y
#$ -cwd
#$ -m beas
#$ -M [email protected]
#$ -N testjob
#$ -l h_vmem=500M <==============
#$ -l h_rt=1000
echo "start of job"
# mem is a compiled C program that loops calling malloc()
# (and then writes to the allocated memory to ensure
# it has been allocated). In this case every 1s it
# requests 4MB and prints "+" to indicate progress.
# We run it under a 300s timeout.
timeout 300 ./mem 1.0 4m +
RC=$?
echo
# Examine mem's exit code.
if [ $RC = 124 ]; then
echo "timeout"
elif [ $RC -ge 128 ]; then
echo "killed with signal $(kill -l $((RC-128)))"
else
echo "exited with status $RC"
fi
echo "end of job"
alexis@te-ahux01$ qsub job2.sh
Your job 79 ("testjob") has been submitted
alexis@te-ahux01$
While it was running, I found this under <my-memory-cgroup-mountpoint>
on the exec node:
te-ahux02# cat 79.1/memory.memsw.limit_in_bytes
524288000
te-ahux02#
and after a couple of minutes (once mem's memory usage reached
the 500MB threshold) I got this in dmesg:
te-ahux02# dmesg
[ 8202.462817] mem invoked oom-killer: gfp_mask=0xd0, order=0,
oom_adj=0, oom_score_adj=0
[ 8202.462822] mem cpuset=6096 mems_allowed=0
[ 8202.462825] Pid: 6260, comm: mem Tainted: G W E X
3.0.101-0.47.52-default #1
[ 8202.462826] Call Trace:
[ 8202.462838] [<ffffffff81004b95>] dump_trace+0x75/0x300
[ 8202.462843] [<ffffffff81461d43>] dump_stack+0x69/0x6f
[ 8202.462847] [<ffffffff810fdfae>] dump_header+0x8e/0x110
[ 8202.462851] [<ffffffff810fe366>] oom_kill_process+0xa6/0x350
[ 8202.462855] [<ffffffff8115450c>]
mem_cgroup_out_of_memory+0x19c/0x2e0
[ 8202.462858] [<ffffffff811547b8>] mem_cgroup_handle_oom+0x168/0x370
[ 8202.462861] [<ffffffff81154ade>] mem_cgroup_do_charge+0x11e/0x150
[ 8202.462864] [<ffffffff81154ef8>] __mem_cgroup_try_charge+0x138/0x3c0
[ 8202.462867] [<ffffffff81156e64>] mem_cgroup_charge_common+0x74/0xd0
[ 8202.462871] [<ffffffff8111eabb>] do_anonymous_page+0x15b/0x300
[ 8202.462876] [<ffffffff8146873d>] do_page_fault+0x1fd/0x4c0
[ 8202.462879] [<ffffffff81465345>] page_fault+0x25/0x30
[ 8202.462932] [<0000000000400b2e>] 0x400b2d
[ 8202.462936] Task in /sge/79.1 killed as a result of limit of
/sge/79.1
[ 8202.462939] memory: usage 512000kB, limit 512000kB, failcnt 128
[ 8202.462940] memory+swap: usage 512000kB, limit 512000kB, failcnt 0
[ 8202.462941] Memory cgroup stats for /sge/79.1: cache:12KB
rss:511988KB mapped_file:0KB swap:0KB pgpgin:0KB pgpgout:0KB inactive_anon:0KB
active_anon:511988KB inactive_file:12KB active_file:0KB unevictable:0KB
[ 8202.462961] [ pid ] uid tgid total_vm rss cpu oom_adj
oom_score_adj name
[ 8202.462985] [ 6096] 0 6096 4211 416 3 0
0 sge_shepherd
[ 8202.462987] [ 6103] 29920 6103 2949 404 0 0
0 bash
[ 8202.462989] [ 6259] 29920 6259 1233 76 1 0
0 timeout
[ 8202.462991] [ 6260] 29920 6260 129092 127920 1 0
0 mem
[ 8202.462994] Memory cgroup out of memory: Kill process 6260 (mem)
score 1001 or sacrifice child
[ 8202.462996] Killed process 6260 (mem) total-vm:516368kB,
anon-rss:511236kB, file-rss:444kB
te-ahux02#
Job output was:
alexis@te-ahux01$ cat testjob.o79
start of job
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
killed with signal KILL
end of job
alexis@te-ahux01$ fgrep + testjob.o79 | wc -c
125
alexis@te-ahux01$
i.e. mem made 124 successful malloc(4MB)'s before it got killed. The job itself continued, but that's just because this is a test, so I wasn't going to insert 'set -e' or handle the error more gracefully.
I think there might be some interactions with cgroup based cpuset/core binding support which will eventually need addressing. We haven't looked at that yet.
I wonder if I ran into this. I saw a chicken-and-egg situation in sge-8.1.8/source/libs/uti2/sge_cgroup.c:init_cgroups(). This function ultimately sets global initialized=true but before it does that it calls copy_from_parent(), which calls cgroup_dir(), which aborts if not yet'initialized' :-) But obviously, this isn't what normally happens. I killed sge_execd and, when the init.d script failed to start it again, then I ran it manually, which worked. I've not seen the problem again, but if I do then I'll dig deeper.
Dave Love wrote:
People should probably move to SLURM for better resource management.
Noted, but currently not an option. Thanks very much for your help and advice! Alexis
<<attachment: alexis_huxley.vcf>>
_______________________________________________ SGE-discuss mailing list [email protected] https://arc.liv.ac.uk/mailman/listinfo/sge-discuss
