Thanks for the detailed explanation of why the OOM Killer strikes. I have dome some reading about it, and am tinkering with how to stop it from killing varnishd.
What I am curious about is - how did the OOM killer get invoked at all? My python process is fairly basic, and wouldn't have consumed any memory at all. When varnish reaches it's malloc limit, I thought cached objects would start getting Nuked. My LRU nuke counters were 0 through the process. So, instead of nuking objects gracefully, I had a varnish-child-restart. This is what I am worried about. If I can get nuking by reducing the overall memory footprint by reducing the malloc limits, I will gladly do it. Do you think that might help? -T On Wed, Mar 23, 2011 at 4:33 PM, Thiago Figueiro <[email protected] > wrote: > From: Tejaswi Nadahalli > > I am running my Python origin-server on the same machine. It seems like > > the Python interpreter caused the OOM killer to kill Varnish. If that's > > the case, is there anything I can do prevent this from happening? > > > I've been meaning to write-up a blog entry regarding the OOM killer in > Linux (what a dumb idea) but in the mean time this should get you started. > > The OOM Killer is there because Linux, by default in most distros, > allocates more memory than available (swap+ram) on the assumption that > applications will never need it (this is called overcommiting). Mostly this > is true but when it's not the oom_kill is called to free-up some memory so > the kernel can keep its promise. Usually it does a shit job (as you just > noticed) and I hate it so much. > > One way to solve this is to tweak oom_kill so it doesn't kill varnish > processes. It's a bit cumbersome because you need to do that based on the > PID, which you only learn after the process has started, leaving room for > some nifty race conditions. Still, adding these to Varnish's init scripts > should do what you need - look up online for details. > > The other way is to disable memory overcommit. Add to /etc/sysctl.conf: > > # Disables memory overcommit > vm.overcommit_memory = 2 > # Tweak to fool VM (read manual for setting above) > vm.overcommit_ratio = 100 > # swap only if really needed > vm.swappiness = 10 > > and sudo /sbin/sysctl -e -p /etc/sysctl.conf > > The problem with setting overcommit_memory to 2 is that the VM will not > allocate more memory than you have available (the actual rule is a function > of RAM, swap and overcommit_ratio, hence the tweak above). > > This could be a problem for Varnish depending on the storage used. The > file storage will mmap the file, resulting in a VM size as large as the > file. If you don't have enough RAM the kernel will deny memory allocation > and varnish will fail to start. At this point you either buy more RAM or > tweak your swap size to account for greedy processes (ie.: processes that > allocate a lot of memory but never use it). > > > TL;DR: buy more memory; get rid of memory hungry scripts in your varnish > box > > > Good luck. > > > ______________________________________________________ > CONFIDENTIALITY NOTICE > This electronic mail message, including any and/or all attachments, is for > the sole use of the intended recipient(s), and may contain confidential > and/or privileged information, pertaining to business conducted under the > direction and supervision of the sending organization. All electronic mail > messages, which may have been established as expressed views and/or opinions > (stated either within the electronic mail message or any of its > attachments), are left to the sole responsibility of that of the sender, and > are not necessarily attributed to the sending organization. Unauthorized > interception, review, use, disclosure or distribution of any such > information contained within this electronic mail message and/or its > attachment(s), is (are) strictly prohibited. If you are not the intended > recipient, please contact the sender by replying to this electronic mail > message, along with the destruction all copies of the original electronic > mail message (along with any attachments). > ______________________________________________________ >
_______________________________________________ varnish-misc mailing list [email protected] http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
