For a long time we've tried to reduce VM page thrashing by running cmmflush on all our non-production servers at night. Cmmflush is a script that Rob van der Heij wrote about a decade ago that goes through the Linux cache looking for pages that it wants to return to VM. We run it in the wee hours on most dev systems trying to kick things out of the Linux buffer cache because double buffering leads to bad performance. Our memory is also heavily overcommitted in dev.
Recently, we found a serious problem that cmmflush might be causing and IBM via Suse recommended that we just stop. They say that newer versions of the kernel have background cmma flushing enabled by default, and if it's working we shouldn't need cmmflush. A Redhat bug report I found says that one shouldn't use old-style CMM (like cmmflush) and hot-pluggable memory together; we do. So: should we just retire cmmflush? How do we make sure that the newer cmma is enabled? It's supposedly the default but I don't see it in /proc/cmdline on any guests where I've looked. Are there kernel parameters in /sys we should look for? In case you're not familiar with it, cmmflush pretty much freezes the server as it runs through the buffers. We already don't do it if there's software that doesn't like it. The new problem was that a few servers with an application that had filled more than half of swap lost their primary network interfaces. We couldn't recover the network so we had to just halt and reboot the servers. The problem seemed to start right after cmmflush ran. We're not certain that cmmflush caused the problem but the correlation is striking. I don't want to sound like I'm criticizing Mr. van der Heij for this at all. cmmflush was a big help when we started using it. If we missed updates that made it unnecessary that's on us. Rob's presentation at SHARE 2013: https://share.confex.com/share/121/webprogram/Handout/Session13486/2013-13486.pdf IBM kernel docs: cmma - Reduce hypervisor paging I/O overhead - IBM Documentation<https://www.ibm.com/docs/en/linux-on-systems?topic=kp-cmma-2> Redhat bug report (membership needed): https://access.redhat.com/solutions/4621911 Ted Rodriguez-Bell z/VM and z/Linux, Mainframe and Midrange Services Wells Fargo | 333 Market St, Floor 10, San Francisco, CA 94105-2102 MAC A0109-103 | 415-516-7913 mobile, [email protected]<mailto:[email protected]> or http://www.vtext.com<http://www.vtext.com/> text paging [email protected]<mailto:[email protected]> Company policy requires: This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
