Thanks again! I've got some research ahead of me. Jon
-----Original Message----- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Rob van der Heij Sent: Friday, August 16, 2013 7:51 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: linux cache When the pages are owned by the JVM, there's nothing Linux can do about it. Since the application is not touching the stuff in the heap, VM will eventually page it out. When the application ramps up again, paging will slow you down because VM block paging can't help you much. What you could do is set your minimum heap size low enough that you encourage the JVM to clean up earlier and give extents back to Linux when it can. That way Linux sees the memory is free and could return it to z/VM again (with my cmmflush). Problem is knowing when to do that. As you point out, when the JVM shrinks the heap, that gives Linux room to grow the page cache. In some cases there are tasks that grow the page cache very fast (backup, file scanning). I have some ideas to handle that: http://zvmperf.wordpress.com/2013/01/27/taming-linux-page-cache/ Another thing I would want to try in your case is to use large pages for the JVM. Because that is pre-allocated, z/VM block paging could be an order of magnitude less painful when the application comes back after a long idle period. As long as memory is used by the JVM, cpuplugd can't do anything either. When you have the JVM return the extents when possible, cpuplugd could return the memory to z/VM. But my experience is that it keeps things too tight and it would take very long before it deflates the balloon enough to let the heap grow again. My script has a modprobe of the cmm module, that's the only thing I need to be prepared. It also tolerates a fixed setting of the balloon if you had that before. Rob On 16 August 2013 14:21, Veencamp, Jonathon D. <jdveenc...@fedins.com>wrote: > I have some development zLinux guests with 25 JVM's. Most of these aren't > used on a typical day and can idle along with a 200MB heap. But if a > development team is doing load testing, then the heap and memory footprint > for a JVM can grow to 1.5GB+. If 2-3 teams are doing testing, my Linux > memory utilization on that guest just grow by 4-6 GB. I cannot "right > size" some of these guests without fixed heaps which seems wasteful, and > relying on multiple 500MB VDISK linux swaps for Linux can have downsides. > I've had guest VDISKS paged out by VM with terrible performance penalties. > I'd hypothesize VDISK can be far more likely to get paged out than memory > by the file cache! Swappiness=0 doesn't prevent the cache from being > populated. If it's populated, VM still might not see that memory as LRU. > Unless I can help out the hypervisor by preventing some memory from being > used, then it limits our ability to overcommit memory on VM. > > So a month ago I asked the list about cpuplugd, CMM, & CMMA. Everyone > said to stay away from these. But this is exactly the problem I was trying > to solve, and why I went with the drop caches approach. I don't need > "always on" help, just the ability to suppress the linux file cache. Drop > caches seems to help. On a guest described like above, I'm going into the > day with a 1.5GB cache (still too big), but 3.5GB of free memory that is a > candidate for LRU. That seems to work for me. > > Rob - For your example script to work, does CMM need to be fully > configured? > > And with "fairness in opinions" in mind, how would you recommend I > approach my situation Mauro? > > Jon > > > -----Original Message----- > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Rob > van der Heij > Sent: Friday, August 16, 2013 5:22 AM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: linux cache > > The thing that happened to your servers is that drop_caches frees up page > frames in Linux that have not been referenced for a long time. Linux starts > to use them for something else, and requires z/VM to page it in. To make > room for that, VM will page out other parts of the guests. > > You're still missing the point. We don't drop the caches because Linux > needs memory but because we want to determine what memory to give back to > VM. Yes, you also lose the good things from cache, but that's a trade-off > to make with performance instrumentation. As I started my first response in > this thread, if you don't plan to give the pages back with CMM or CMMA, it > is not a good idea to use drop_caches and swappiness might be a better way > to help workloads that get in trouble. > > Rob > > > On 15 August 2013 19:16, Mauro Souza <thoriu...@gmail.com> wrote: > > > I have read it now. And as I think there's no CMM involved, I still > > disagree. > > I said I would never ever drop caches because I saw it almost killing a > > LPAR. I had a customer that put drop_caches on crontab, and at that time > > *every* guest started dropping caches and zVM was crazy doing a lot of > > page-ins and page-outs. Nobody understood why the system was > unresponsible. > > I think is less costly to zVM to handle dozens of page-ins and outs for a > > Linux needing 10MB of memory here and there, than to drop 2GB of caches > at > > the same time. If you drop all the cache at the same time, Linux will > have > > to make an I/O operation for every file it will read, and zVM will have > to > > read a lot of data from PAGE that will be just ignored. > > No matter if you drop all the cache at the same time, or drop a page when > > you need it, zVM will have to bring it back. Dropping all caches is like > > paying a $100k loan in one time, instead of splitting it in 1000 times of > > $100. > > > > > > Mauro > > http://mauro.limeiratem.com - registered Linux User: 294521 > > Scripture is both history, and a love letter from God. > > > > > > 2013/8/15 Rob van der Heij <rvdh...@gmail.com> > > > > > On 15 August 2013 17:34, Mauro Souza <thoriu...@gmail.com> wrote: > > > > > > > > > > You can safely ignore the cache usage on Linux, zVM will realise the > > page > > > > was not in use and drop it itself. When Linux needs memory, it will > > > reclaim > > > > cache pages automatically. > > > > > > > > So we disagree. Maybe we wouldn't if you had read the page that I > > linked > > > to. > > > > > > The point that you miss is when Linux needs memory and reclaims a page > > > frame from page cache, z/VM will first bring back the old contents that > > > Linux did not care about anymore. This is where getting a free page > will > > > slow you down. Especially since the LRU patterns interfere. The idea is > > > that you want to drop the backing page from z/VM either before it is > > paged > > > out or before z/VM will page it back in. If you have CMMA enabled, then > > > having Linux free the page (by drop_caches) will prevent z/VM from > paging > > > it out. > > > > > > Rob > > > > > > ---------------------------------------------------------------------- > > > For LINUX-390 subscribe / signoff / archive access instructions, > > > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 > or > > > visit > > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > > ---------------------------------------------------------------------- > > > For more information on Linux on System z, visit > > > http://wiki.linuxvm.org/ > > > > > > > ---------------------------------------------------------------------- > > For LINUX-390 subscribe / signoff / archive access instructions, > > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > > visit > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > ---------------------------------------------------------------------- > > For more information on Linux on System z, visit > > http://wiki.linuxvm.org/ > > > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > > ________________________________ > > The information contained in this e-mail message is intended only for the > personal and confidential use of the designated recipient(s) named above. > This message may be an attorney-client or work product communication which > is privileged and confidential. It may also contain protected health > information that is protected by federal law. If you have received this > communication in error, please notify us immediately by telephone and > destroy (shred) the original message and all attachments. Any review, > dissemination, distribution or copying of this message by any person other > than the intended recipient(s) or their authorized agents is strictly > prohibited. Thank you. > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/ ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the designated recipient(s) named above. This message may be an attorney-client or work product communication which is privileged and confidential. It may also contain protected health information that is protected by federal law. If you have received this communication in error, please notify us immediately by telephone and destroy (shred) the original message and all attachments. Any review, dissemination, distribution or copying of this message by any person other than the intended recipient(s) or their authorized agents is strictly prohibited. Thank you. ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
