it's still not constraining memory... a memhog job continues to memhog:
boris@rod:~/scripts$ sacct --starttime=2023-05-01 --format=jobid,user,start,elapsed,reqmem,maxrss,maxvmsize,nodelist,state,exit -j 199 JobID User Start Elapsed ReqMem MaxRSS MaxVMSize NodeList State ExitCode ------------ --------- ------------------- ---------- ---------- ---------- ---------- --------------- ---------- -------- 199 boris 2023-06-23T02:42:30 00:01:21 1M milhouse COMPLETED 0:0 199.batch 2023-06-23T02:42:30 00:01:21 104857988K 104858064K milhouse COMPLETED 0:0 One thing I noticed is that the machines I'm working on do not have libcgroup and libcgroup-dev installed - but slurm does have its own cgroup implementation? the slurmd processes do utilize /usr/lib/slurm/*cgroup.so objects. I will try to recompile slurm with those cgrouplib packages present. On Thu, Jun 22, 2023 at 6:04 PM Ozeryan, Vladimir < vladimir.ozer...@jhuapl.edu> wrote: > No worries, > > No, we don’t have any OS level settings, only “allowed_devices.conf” which > just has /dev/random, /dev/tty and stuff like that. > > > > But I think this could be the culprit, check out man page for cgroup.conf > AllowedRAMSpace=100 > > > > I would just leave these four: > > CgroupAutomount=yes > ConstrainCores=yes > ConstrainDevices=yes > ConstrainRAMSpace=yes > > > > Vlad. > > > > *From:* slurm-users <slurm-users-boun...@lists.schedmd.com> *On Behalf Of > *Boris Yazlovitsky > *Sent:* Thursday, June 22, 2023 5:40 PM > *To:* Slurm User Community List <slurm-users@lists.schedmd.com> > *Subject:* Re: [slurm-users] [EXT] --mem is not limiting the job's memory > > > > *APL external email warning: *Verify sender > slurm-users-boun...@lists.schedmd.com before clicking links or attachments > > > > thank you Vlad - looks like we have the same yes's > > Do you remember if you had to make any settings on the OS level or in the > kernel to make it work? > > > > -b > > > > On Thu, Jun 22, 2023 at 5:31 PM Ozeryan, Vladimir < > vladimir.ozer...@jhuapl.edu> wrote: > > Hello, > > > > We have the following configured and it seems to be working ok. > > > > CgroupAutomount=yes > ConstrainCores=yes > ConstrainDevices=yes > ConstrainRAMSpace=yes > > Vlad. > > > > *From:* slurm-users <slurm-users-boun...@lists.schedmd.com> *On Behalf Of > *Boris Yazlovitsky > *Sent:* Thursday, June 22, 2023 4:50 PM > *To:* Slurm User Community List <slurm-users@lists.schedmd.com> > *Subject:* Re: [slurm-users] [EXT] --mem is not limiting the job's memory > > > > *APL external email warning: *Verify sender > slurm-users-boun...@lists.schedmd.com before clicking links or attachments > > > > Hello Vladimir, thank you for your response. > > > > this is the cgroups.conf file: > > CgroupAutomount=yes > ConstrainCores=yes > ConstrainDevices=yes > ConstrainRAMSpace=yes > ConstrainSwapSpace=yes > MaxRAMPercent=90 > AllowedSwapSpace=0 > AllowedRAMSpace=100 > MemorySwappiness=0 > MaxSwapPercent=0 > > > > /etc/default/grub: > > GRUB_DEFAULT=0 > GRUB_TIMEOUT_STYLE=hidden > GRUB_TIMEOUT=0 > GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` > GRUB_CMDLINE_LINUX_DEFAULT="" > GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 cgroup_enable=memory > swapaccount=1" > > > > what other cgroup settings need to be set? > > > > && thank you! > > -b > > > > On Thu, Jun 22, 2023 at 4:02 PM Ozeryan, Vladimir < > vladimir.ozer...@jhuapl.edu> wrote: > > --mem=5G. Should allocate 5G of memory per node. > > Are your cgroups configured? > > > > *From:* slurm-users <slurm-users-boun...@lists.schedmd.com> *On Behalf Of > *Boris Yazlovitsky > *Sent:* Thursday, June 22, 2023 3:28 PM > *To:* slurm-users@lists.schedmd.com > *Subject:* [EXT] [slurm-users] --mem is not limiting the job's memory > > > > *APL external email warning: *Verify sender > slurm-users-boun...@lists.schedmd.com before clicking links or attachments > > > > Running slurm 22.03.02 on Ubunutu 22.04 server. > > Jobs submitted with --mem=5g are able to allocate an unlimited amount of > memory. > > > > how to limit on the job submission level how much memory it can grab? > > > > thanks, and best regards! > Boris > > > >