Re: cgroups: cannot write on file memory.oom_control
Hi Rami, I have looked at the example code in your slides and wrote something similar to handle the OoM condition and it seems to work fine. Thank you for that. Since it would be even better for me to avoid playing with OoM, I explored another option for monitoring memory usage: keeping the OoM killer enabled and using the memory threshold feature (see below) to get notified when a group's memory usage exceeds the limit. First, my C code for monitoring the memory usage looked like this (running as a separate process): // Assuming that initial memory usage is below the threshold while(1) { if (read(efd, counter, sizeof(uint64_t)) != sizeof(uint64_t)) die(read eventfd); printf(Event received: memory usage ABOVE limit\n); if (read(efd, counter, sizeof(uint64_t)) != sizeof(uint64_t)) die(read eventfd); printf(Event received: memory usage BELOW limit\n); } Then I thought: what happens if mem usage crosses the threshold more than once between 2 consecutive calls to read() ? (a scenario that cannot be formally excluded). I realised that in this situation the counter value returned by read() would contain the number of occurrences crossing events. So for example if, between 2 calls to read(), the memory usage goes beyond then below the threshold, the value of counter after the 2nd read will be 2. I ended up using this counter behaviour to determine if the current usage is below or beyond the limit: counter = 0; above_mem_limit = 0; while(1) { if (read(efd, counter, sizeof(uint64_t)) != sizeof(uint64_t)) die(read eventfd); if(counter%2) above_mem_limit = !above_mem_limit; printf(Event received! (counter=%d), above_mem_limit=%d\n, counter, above_mem_limit); } My question: I have never seen this event counter behaviour documented anywhere. The cgroups docs and forums do not say anything about the 64bit value returned by the read() call. Do you think one can safely rely on this to count the number of crossing events ? Thanks and Regards, Hedi From https://www.kernel.org/doc/Documentation/cgroups/memory.txt: 9. Memory thresholds Memory cgroup implements memory thresholds using the cgroups notification API (see cgroups.txt). It allows to register multiple memory and memsw thresholds and gets notifications when it crosses. To register a threshold, an application must: - create an eventfd using eventfd(2); - open memory.usage_in_bytes or memory.memsw.usage_in_bytes; - write string like event_fd fd of memory.usage_in_bytes threshold to cgroup.event_control. Application will be notified through eventfd when memory usage crosses threshold in any direction. It's applicable for root and non-root cgroup. Best Regard, Hedi From: Hedi Boufaied/NCE/AMADEUS To: Rami Rosen roszenr...@gmail.com@AMAEXTMAIL_MUC, Cc: kernelnewbies kernelnewbies@kernelnewbies.org Date: 22/04/2014 13:12 Subject:Re: cgroups: cannot write on file memory.oom_control Hi Rami, Thank you for the advice. Yes, I will try using the event handler. Best Regards, Hedi From: Rami Rosen roszenr...@gmail.com To: Hedi Boufaied hedi.boufa...@amadeus.com, Cc: kernelnewbies kernelnewbies@kernelnewbies.org Date: 18/04/2014 22:38 Subject:Re: cgroups: cannot write on file memory.oom_control Sent by:kernelnewbies-boun...@kernelnewbies.org Hi Hedi, Well, you can use disabling oom in conjunction with event handler. Thus, you can free memory by changing memory setting of other cgroups when an oom occurs in a specific cgroup, and avoid having an oops in that cgroup. You can see an example of using event handler in conjunction with disabling the oom killer in Namespaces and cgroups in linux, in slides 101/102, see: http://ramirose.wix.com/ramirosen A shameless plug - I wrote this presentation of 121 slides about namespaces and cgroups. Regards, Rami Rosen בתאריך 17 באפר 2014 11:24, Hedi Boufaied hedi.boufa...@amadeus.com כתב: Hi Rami, Thanks a lot for your sequence. I followed it and it does work. I now see my process paused in sleep state instead of getting killed when the cgroup reaches OoO. This being said: * I do not really understand why the use_hierarchy option needs to be disabled in order to allow disabling of the oom killer... * I'm not sure that disabling the OoO killer is safe/reliable enough: in some of the trials I made, I saw other processes (xterms) not part of the OoO cgroup getting completely stuck and I had to reboot my machine... Best Regards, Hedi From:Rami Rosen roszenr...@gmail.com To:Hedi Boufaied hedi.boufa...@amadeus.com, Cc:kernelnewbies kernelnewbies@kernelnewbies.org Date:15/04/2014 15:39 Subject:Re: cgroups: cannot write on file memory.oom_control Hi, Hedi, This sequence works for me: echo 0 /sys/fs/cgroup/memory/memory.use_hierarchy mkdir /sys/fs/cgroup/memory/0 Then, the following two commands changes
Re: cgroups: cannot write on file memory.oom_control
Hi Rami, Thank you for the advice. Yes, I will try using the event handler. Best Regards, Hedi From: Rami Rosen roszenr...@gmail.com To: Hedi Boufaied hedi.boufa...@amadeus.com, Cc: kernelnewbies kernelnewbies@kernelnewbies.org Date: 18/04/2014 22:38 Subject:Re: cgroups: cannot write on file memory.oom_control Sent by:kernelnewbies-boun...@kernelnewbies.org Hi Hedi, Well, you can use disabling oom in conjunction with event handler. Thus, you can free memory by changing memory setting of other cgroups when an oom occurs in a specific cgroup, and avoid having an oops in that cgroup. You can see an example of using event handler in conjunction with disabling the oom killer in Namespaces and cgroups in linux, in slides 101/102, see: http://ramirose.wix.com/ramirosen A shameless plug - I wrote this presentation of 121 slides about namespaces and cgroups. Regards, Rami Rosen בתאריך 17 באפר 2014 11:24, Hedi Boufaied hedi.boufa...@amadeus.com כתב: Hi Rami, Thanks a lot for your sequence. I followed it and it does work. I now see my process paused in sleep state instead of getting killed when the cgroup reaches OoO. This being said: * I do not really understand why the use_hierarchy option needs to be disabled in order to allow disabling of the oom killer... * I'm not sure that disabling the OoO killer is safe/reliable enough: in some of the trials I made, I saw other processes (xterms) not part of the OoO cgroup getting completely stuck and I had to reboot my machine... Best Regards, Hedi From:Rami Rosen roszenr...@gmail.com To:Hedi Boufaied hedi.boufa...@amadeus.com, Cc:kernelnewbies kernelnewbies@kernelnewbies.org Date:15/04/2014 15:39 Subject:Re: cgroups: cannot write on file memory.oom_control Hi, Hedi, This sequence works for me: echo 0 /sys/fs/cgroup/memory/memory.use_hierarchy mkdir /sys/fs/cgroup/memory/0 Then, the following two commands changes the value of oom_kill_disable: echo 1/sys/fs/cgroup/memory/0/memory.oom_control echo 0/sys/fs/cgroup/memory/0/memory.oom_control For more info, please look for the text about enabling/disabling use_hierarchy in section 6, http://lxr.free-electrons.com/source/Documentation/cgroups/memory.txt. Regards, Rami Rosen http://ramirose.wix.com/ramirosen On Mon, Apr 14, 2014 at 12:22 PM, Hedi Boufaied hedi.boufa...@amadeus.com wrote: Hi everyone, I am working with cgroups on OpenSuse (over VirtualBox) and I cannot disable the OOM killer by writing to the file memory.oom_control: echo 1 memory.oom_control -bash: echo: write error: Invalid argument I can change the memory limit and several other settings by writing to the appropriate files (like memory.limit_in_bytes) but I could never write to file memory.oom_control although I am root. I saw someone posted a similar issue a few days ago but there was no reply... I have copied below the sequence of command I am using. Any idea what could be the issue ? Thanks in advance for your help! Hedi /sys/fs su - root /sys/fs cd cgroup /sys/fs/cgroup cgcreate -g memory:/mygroup /sys/fs/cgroup cd memory/mygroup /sys/fs/cgroup/memory/mygroup echo 32M memory.limit_in_bytes /sys/fs/cgroup/memory/mygroup cat memory.limit_in_bytes 33554432 /sys/fs/cgroup/memory/mygroup echo 1 memory.oom_control -bash: echo: write error: Invalid argument /sys/fs/cgroup/memory/mygroup cat memory.oom_control oom_kill_disable 0 under_oom 0 --- ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: cgroups: cannot write on file memory.oom_control
Hi Rami, Thanks a lot for your sequence. I followed it and it does work. I now see my process paused in sleep state instead of getting killed when the cgroup reaches OoO. This being said: * I do not really understand why the use_hierarchy option needs to be disabled in order to allow disabling of the oom killer... * I'm not sure that disabling the OoO killer is safe/reliable enough: in some of the trials I made, I saw other processes (xterms) not part of the OoO cgroup getting completely stuck and I had to reboot my machine... Best Regards, Hedi From: Rami Rosen roszenr...@gmail.com To: Hedi Boufaied hedi.boufa...@amadeus.com, Cc: kernelnewbies kernelnewbies@kernelnewbies.org Date: 15/04/2014 15:39 Subject:Re: cgroups: cannot write on file memory.oom_control Hi, Hedi, This sequence works for me: echo 0 /sys/fs/cgroup/memory/memory.use_hierarchy mkdir /sys/fs/cgroup/memory/0 Then, the following two commands changes the value of oom_kill_disable: echo 1/sys/fs/cgroup/memory/0/memory.oom_control echo 0/sys/fs/cgroup/memory/0/memory.oom_control For more info, please look for the text about enabling/disabling use_hierarchy in section 6, http://lxr.free-electrons.com/source/Documentation/cgroups/memory.txt. Regards, Rami Rosen http://ramirose.wix.com/ramirosen On Mon, Apr 14, 2014 at 12:22 PM, Hedi Boufaied hedi.boufa...@amadeus.com wrote: Hi everyone, I am working with cgroups on OpenSuse (over VirtualBox) and I cannot disable the OOM killer by writing to the file memory.oom_control: echo 1 memory.oom_control -bash: echo: write error: Invalid argument I can change the memory limit and several other settings by writing to the appropriate files (like memory.limit_in_bytes) but I could never write to file memory.oom_control although I am root. I saw someone posted a similar issue a few days ago but there was no reply... I have copied below the sequence of command I am using. Any idea what could be the issue ? Thanks in advance for your help! Hedi /sys/fs su - root /sys/fs cd cgroup /sys/fs/cgroup cgcreate -g memory:/mygroup /sys/fs/cgroup cd memory/mygroup /sys/fs/cgroup/memory/mygroup echo 32M memory.limit_in_bytes /sys/fs/cgroup/memory/mygroup cat memory.limit_in_bytes 33554432 /sys/fs/cgroup/memory/mygroup echo 1 memory.oom_control -bash: echo: write error: Invalid argument /sys/fs/cgroup/memory/mygroup cat memory.oom_control oom_kill_disable 0 under_oom 0 --- ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
cgroups: cannot write on file memory.oom_control
Hi everyone, I am working with cgroups on OpenSuse (over VirtualBox) and I cannot disable the OOM killer by writing to the file memory.oom_control: echo 1 memory.oom_control -bash: echo: write error: Invalid argument I can change the memory limit and several other settings by writing to the appropriate files (like memory.limit_in_bytes) but I could never write to file memory.oom_control although I am root. I saw someone posted a similar issue a few days ago but there was no reply... I have copied below the sequence of command I am using. Any idea what could be the issue ? Thanks in advance for your help! Hedi /sys/fs su - root /sys/fs cd cgroup /sys/fs/cgroup cgcreate -g memory:/mygroup /sys/fs/cgroup cd memory/mygroup /sys/fs/cgroup/memory/mygroup echo 32M memory.limit_in_bytes /sys/fs/cgroup/memory/mygroup cat memory.limit_in_bytes 33554432 /sys/fs/cgroup/memory/mygroup echo 1 memory.oom_control -bash: echo: write error: Invalid argument /sys/fs/cgroup/memory/mygroup cat memory.oom_control oom_kill_disable 0 under_oom 0 ---___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies