Re: Memory use question
True. After the instances have settled down and stabilized, the WSS of the swapless server is appreciably larger than that of the others. And yes there is a risk in doing this that I'm not willing to take in our production environments. It would be nice to *someday* be able to simply define instances with CP storage allocation and let the software take care of how its all used, maybe through better control of file and buffer caching or something. Ray -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Martin, Terry R. (CMS/CTR) (CTR) Sent: Monday, June 28, 2010 11:24 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Memory use question Hi Well this is not something I would do in my shop. Not having swap available for Linux is just asking for trouble. One reason is that you really want to size your guests in such a way that they use just a trickle of swap this allows you to size a guest in terms of memory defined to the guest to allow z/Linux to maximize the handling of the memory and from a performance standpoint you really want it to be VDISK. One of the things that makes running z/Linux under VM so appealing is the ability to take a server running in Solaris let's say that has 20G and cut that in half or more when it is migrated to z/Linux under VM. To do this you need to have flexibility when it comes to paging and swapping for the guest. Thank You, Terry Martin Lockheed Martin - Citic z/OS and z/VM Performance Tuning and Operating Systems Support Office - 443 348-2102 Cell - 443 632-4191 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
This led me into an interesting area. I just set a couple of our test servers
to run without swap space. This could put a bigger paging load on VM at times,
but then again simplifying the Linux configuration and having VM do the heavy
lifting are both good selling points. I'm also thinking about when I eventually
hand this work over to someone else. Not having to explain v-disks and the
interplay of paging and swapping between the hypervisor and guests could avoid
confusion and mistakes being made. Of course Linux memory utilization becomes
critical, and we can watch it with something like this and adjust virtual CP
storage accordingly:
ps -eo pmem | awk '{pmem += $1}; END {print pmem = pmem%}';
Right now our biggest WebSphere server is running comfortably at ~60% memory
used, out of 2G. Is anyone else running swapless servers big or small? If swap
space is considered a safety valve to keep a server up, then I would just as
soon put it on real DASD, and avoid its use with proper memory sizing.
Ray
-Original Message-
From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On
Behalf Of Marcy Cortes
Sent: Thursday, June 24, 2010 10:34 AM
To: LINUX-390@VM.MARIST.EDU
Subject: Re: Memory use question
So with swappiness higher, Linux is making decsions to
preemptively move something from memory to vdisk. Well, your
vdisk is in VM's pageable memory too.
So moving something from one piece of VM memory to another
piece of VM memory means both parts will have to become
resident. The source page probably had already been paged
out by VM if Linux hadn't been using it recently and the
vdisk page may not have even existed or if it did since it
had been used before, it was likely paged out too.
Marcy
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
Re: Memory use question
Hi
Well this is not something I would do in my shop. Not having swap
available for Linux is just asking for trouble. One reason is that you
really want to size your guests in such a way that they use just a
trickle of swap this allows you to size a guest in terms of memory
defined to the guest to allow z/Linux to maximize the handling of the
memory and from a performance standpoint you really want it to be VDISK.
One of the things that makes running z/Linux under VM so appealing is
the ability to take a server running in Solaris let's say that has 20G
and cut that in half or more when it is migrated to z/Linux under VM. To
do this you need to have flexibility when it comes to paging and
swapping for the guest.
Thank You,
Terry Martin
Lockheed Martin - Citic
z/OS and z/VM Performance Tuning and Operating Systems Support
Office - 443 348-2102
Cell - 443 632-4191
-Original Message-
From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of
Mrohs, Ray
Sent: Monday, June 28, 2010 10:21 AM
To: LINUX-390@VM.MARIST.EDU
Subject: Re: Memory use question
This led me into an interesting area. I just set a couple of our test
servers to run without swap space. This could put a bigger paging load
on VM at times, but then again simplifying the Linux configuration and
having VM do the heavy lifting are both good selling points. I'm also
thinking about when I eventually hand this work over to someone else.
Not having to explain v-disks and the interplay of paging and swapping
between the hypervisor and guests could avoid confusion and mistakes
being made. Of course Linux memory utilization becomes critical, and we
can watch it with something like this and adjust virtual CP storage
accordingly:
ps -eo pmem | awk '{pmem += $1}; END {print pmem = pmem%}';
Right now our biggest WebSphere server is running comfortably at ~60%
memory used, out of 2G. Is anyone else running swapless servers big or
small? If swap space is considered a safety valve to keep a server up,
then I would just as soon put it on real DASD, and avoid its use with
proper memory sizing.
Ray
-Original Message-
From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On
Behalf Of Marcy Cortes
Sent: Thursday, June 24, 2010 10:34 AM
To: LINUX-390@VM.MARIST.EDU
Subject: Re: Memory use question
So with swappiness higher, Linux is making decsions to
preemptively move something from memory to vdisk. Well, your
vdisk is in VM's pageable memory too.
So moving something from one piece of VM memory to another
piece of VM memory means both parts will have to become
resident. The source page probably had already been paged
out by VM if Linux hadn't been using it recently and the
vdisk page may not have even existed or if it did since it
had been used before, it was likely paged out too.
Marcy
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
Re: Memory use question
Rob van der Heij wrote: On Wed, Jun 23, 2010 at 8:47 PM, Mrohs, Ray ray.mr...@usdoj.gov wrote: Rob, Heap size is set to 500M/1000M. I've read recommendations to make default/max the same number, but I would just be guessing at a value. It has grow close to 800M at least once, but it always falls back to around the default size and stays there most of the time. Ray The suggestion to set min and max the same goes well with the folks who have plenty of memory and just want to avoid garbage collection... whether it applies to a shared resource environment needs to be seen (uh, measured..) I'm always suspicious about recommendations without motivation or measurements. If it really were the best setting, would we not expect the defaults to be like that? Well, bearing in mind both the Sun and IBM JVMs default to memory settings that both IBM and Sun say are crap for app servers (e.g. neither using their 1.4 or later GC algorithms by default), I'm not sure I'd place too much stock in their defaults. Our experince, both on standalone Solaris boxes using the Sun JVM, and on our zLinux guests using the IBM JVM, has been that large min/max separation has produced poor results, as the JVM tends to consistently try to push memory use back down to the min setting, often invoking unnecessary GCs to do so. Using our actual applications with our stress test suites (which tend to have pretty good predictave power) we tend to see worse application behaviour (small stalls, higher average response times) and higher CPU use when giving a big spread for min/max; on an older Sun JVM/Solaris combo we also saw failure in the application to ramp up quickly enough to deal with load spikes, with big stalls as the JVM tried to alloc memory. (That last behaviour was common to connection pooling - if we didn't open enough connections to Oracle at startup time, the app could have bad performance or even fail as load ramped up due to an inability to open connections quickly enough.) -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
On Thu, Jun 24, 2010 at 9:59 AM, Rodger Donaldson rodg...@diaspora.gen.nz wrote: Well, bearing in mind both the Sun and IBM JVMs default to memory settings that both IBM and Sun say are crap for app servers (e.g. neither using their 1.4 or later GC algorithms by default), I'm not sure I'd place too much stock in their defaults. It's my understanding that the WAS properties (or such) override the setting. I guess I'll have to look into the typical settings then... One of the design choices with the JVM is that memory management data (counters, pointers, etc) are within the objects being managed. This is not entirely lucky for virtualized environment because it means that pages with allocated old objects are referenced and thus paged back in. With small enough objects, a GC scan needs the entire heap resident. I thought modern GC strategies identified generations of data in the heap that would allow old data to be skipped in most scans. Our experince, both on standalone Solaris boxes using the Sun JVM, and on our zLinux guests using the IBM JVM, has been that large min/max separation has produced poor results, as the JVM tends to consistently try to push memory use back down to the min setting, often invoking unnecessary GCs to do so. Memory has to come from somewhere. Whether a GC was unnecessary is hard to tell, especially in a virtualized environment. While it is true GC takes valuable resources, so does any (memory) management. Simply the fact that you can't identify the cost does not mean it's for free... With an application server that does not hold data (data resides in the database elsewhere) we could expect memory requirements to consist of a base level plus some amount for each active connection. I've talked to application developers who had very good understanding of the requirements per transaction. Depending on typical volume, duration and freqeuency of such active connections, you get a typical heap size. If you set the minimum a bit aobve that, malloc() / free() calls should happen only around periods with significantly more active connections. I recall you can also change the chunk size to create some hystheresis. Don't know whether this model still works now that developers start to implement cache with in-memory databases on top of GC-managed objects... Using our actual applications with our stress test suites (which tend to have pretty good predictave power) we tend to see worse application behaviour (small stalls, higher average response times) and higher CPU use when giving a big spread for min/max; on an older Sun JVM/Solaris combo we also saw failure in the application to ramp up quickly enough to deal with load spikes, with big stalls as the JVM tried to alloc memory. Don't know about yours, but stress tests often involve periods with high utilization. While you need that stake in the ground, it's not the best one to measure scalability: if you need the resource 90% of the time, you worry less about how to share it during the remaining 10%. But if you have an average utilization of 5% and a duty cycle of an hour, then it makes sense trying to share resources even if it costs you something extra (during the 5% that your application is active). (That last behaviour was common to connection pooling - if we didn't open enough connections to Oracle at startup time, the app could have bad performance or even fail as load ramped up due to an inability to open connections quickly enough.) Though it might sound nice to have a large connection pool, if that means the data is referenced very rarely (would not be surprised to see round-robin rather than push-down stack) then you may get paged on z/VM level and held back even longer. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
Rob has a good point in resource usage and with GC it definitely is the lesser of two evils. I have never had an instance where after determining the GC Collector settings performance and % utilization suffered negatively. One can also look at GC as being Java feeble attempt at memory management. Richard (Gaz) Gasiorowski SAE Solution Architect CSC 3170 Fairview Park Dr., Falls Church, VA 22042 845-889-8533|Work|845-392-7889 Cell|rgasi...@csc.com|www.csc.com This is a PRIVATE message. If you are not the intended recipient, please delete without copying and kindly advise us by e-mail of the mistake in delivery. NOTE: Regardless of content, this e-mail shall not operate to bind CSC to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose. From: Rob van der Heij rvdh...@velocitysoftware.com To: LINUX-390@vm.marist.edu Date: 06/24/2010 04:45 AM Subject: Re: Memory use question On Thu, Jun 24, 2010 at 9:59 AM, Rodger Donaldson rodg...@diaspora.gen.nz wrote: Well, bearing in mind both the Sun and IBM JVMs default to memory settings that both IBM and Sun say are crap for app servers (e.g. neither using their 1.4 or later GC algorithms by default), I'm not sure I'd place too much stock in their defaults. It's my understanding that the WAS properties (or such) override the setting. I guess I'll have to look into the typical settings then... One of the design choices with the JVM is that memory management data (counters, pointers, etc) are within the objects being managed. This is not entirely lucky for virtualized environment because it means that pages with allocated old objects are referenced and thus paged back in. With small enough objects, a GC scan needs the entire heap resident. I thought modern GC strategies identified generations of data in the heap that would allow old data to be skipped in most scans. Our experince, both on standalone Solaris boxes using the Sun JVM, and on our zLinux guests using the IBM JVM, has been that large min/max separation has produced poor results, as the JVM tends to consistently try to push memory use back down to the min setting, often invoking unnecessary GCs to do so. Memory has to come from somewhere. Whether a GC was unnecessary is hard to tell, especially in a virtualized environment. While it is true GC takes valuable resources, so does any (memory) management. Simply the fact that you can't identify the cost does not mean it's for free... With an application server that does not hold data (data resides in the database elsewhere) we could expect memory requirements to consist of a base level plus some amount for each active connection. I've talked to application developers who had very good understanding of the requirements per transaction. Depending on typical volume, duration and freqeuency of such active connections, you get a typical heap size. If you set the minimum a bit aobve that, malloc() / free() calls should happen only around periods with significantly more active connections. I recall you can also change the chunk size to create some hystheresis. Don't know whether this model still works now that developers start to implement cache with in-memory databases on top of GC-managed objects... Using our actual applications with our stress test suites (which tend to have pretty good predictave power) we tend to see worse application behaviour (small stalls, higher average response times) and higher CPU use when giving a big spread for min/max; on an older Sun JVM/Solaris combo we also saw failure in the application to ramp up quickly enough to deal with load spikes, with big stalls as the JVM tried to alloc memory. Don't know about yours, but stress tests often involve periods with high utilization. While you need that stake in the ground, it's not the best one to measure scalability: if you need the resource 90% of the time, you worry less about how to share it during the remaining 10%. But if you have an average utilization of 5% and a duty cycle of an hour, then it makes sense trying to share resources even if it costs you something extra (during the 5% that your application is active). (That last behaviour was common to connection pooling - if we didn't open enough connections to Oracle at startup time, the app could have bad performance or even fail as load ramped up due to an inability to open connections quickly enough.) Though it might sound nice to have a large connection pool, if that means the data is referenced very rarely (would not be surprised to see round-robin rather than push-down stack) then you may get paged on z/VM level and held back even longer. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu
Re: Memory use question
On Thu, Jun 24, 2010 at 1:09 PM, Richard Gasiorowski rgasi...@csc.com wrote: Rob has a good point in resource usage and with GC it definitely is the lesser of two evils. I have never had an instance where after determining the GC Collector settings performance and % utilization suffered negatively. One can also look at GC as being Java feeble attempt at memory management. I recently heard someone claim one might find classic programs spend 30% of their CPU in malloc() / free() calls and tasks that would otherwise be provided by the JVM But he's a true Java person, so I would like to see some data... OTOH I was working with a customer recently who was fighting the embedded DB2 inside his TSM 6 server. Apparently DB2 could not decide whether to keep the block or give it back to Linux. After changing a memory threshold in DB2, he got back a lot of system time in Linux... Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
-Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Marcy Cortes Sent: Wednesday, June 23, 2010 3:45 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: Memory use question Rob mentioned the vm.swappiness setting and he and I have had a lot of discussions about that one. You do want to probably set that to zero on a WAS server (and probably others). It's recommended in this WAS tuning paper too http://download.boulder.ibm.com/ibmdl/pub/software/dw/linux390 /perf/ZSW03132USEN.PDF I have trouble seeing whether using memory or v-disk in memory is more effective, though I might have read that VM needs to page more for large seldom referenced v-disk spaces. My natural inclination is to use RAM first and swap last, perhaps based on outdated notions. Anyway I set swappiness to zero. The publication also seems to recommend a heap size that is ~70% of available memory. In our case that would equal 1400M which is 400M more than our current max. setting. Ray -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
So with swappiness higher, Linux is making decsions to preemptively move something from memory to vdisk. Well, your vdisk is in VM's pageable memory too. So moving something from one piece of VM memory to another piece of VM memory means both parts will have to become resident. The source page probably had already been paged out by VM if Linux hadn't been using it recently and the vdisk page may not have even existed or if it did since it had been used before, it was likely paged out too. Marcy marcy.d.cor...@wellsfargo.com This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Mrohs, Ray Sent: Thursday, June 24, 2010 5:53 AM To: LINUX-390@vm.marist.edu Subject: Re: [LINUX-390] Memory use question -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Marcy Cortes Sent: Wednesday, June 23, 2010 3:45 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: Memory use question Rob mentioned the vm.swappiness setting and he and I have had a lot of discussions about that one. You do want to probably set that to zero on a WAS server (and probably others). It's recommended in this WAS tuning paper too http://download.boulder.ibm.com/ibmdl/pub/software/dw/linux390 /perf/ZSW03132USEN.PDF I have trouble seeing whether using memory or v-disk in memory is more effective, though I might have read that VM needs to page more for large seldom referenced v-disk spaces. My natural inclination is to use RAM first and swap last, perhaps based on outdated notions. Anyway I set swappiness to zero. The publication also seems to recommend a heap size that is ~70% of available memory. In our case that would equal 1400M which is 400M more than our current max. setting. Ray -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
Also, I don't think I'd set the heap higher. If you don't need it, that's just wasted memory. And garbage collection will run longer when it does run. Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Mrohs, Ray Sent: Thursday, June 24, 2010 5:53 AM To: LINUX-390@vm.marist.edu Subject: Re: [LINUX-390] Memory use question -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Marcy Cortes Sent: Wednesday, June 23, 2010 3:45 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: Memory use question Rob mentioned the vm.swappiness setting and he and I have had a lot of discussions about that one. You do want to probably set that to zero on a WAS server (and probably others). It's recommended in this WAS tuning paper too http://download.boulder.ibm.com/ibmdl/pub/software/dw/linux390 /perf/ZSW03132USEN.PDF I have trouble seeing whether using memory or v-disk in memory is more effective, though I might have read that VM needs to page more for large seldom referenced v-disk spaces. My natural inclination is to use RAM first and swap last, perhaps based on outdated notions. Anyway I set swappiness to zero. The publication also seems to recommend a heap size that is ~70% of available memory. In our case that would equal 1400M which is 400M more than our current max. setting. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
I think you might be talking about generational garbage collection (-Xgcpolicy:gencon ). It's not the default in WAS 6 (not sure about 7). Our biggest app sees much better throughput and less CPU usage using it. Marcy marcy.d.cor...@wellsfargo.com This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Rob van der Heij Sent: Thursday, June 24, 2010 1:44 AM To: LINUX-390@vm.marist.edu Subject: Re: [LINUX-390] Memory use question On Thu, Jun 24, 2010 at 9:59 AM, Rodger Donaldson rodg...@diaspora.gen.nz wrote: Well, bearing in mind both the Sun and IBM JVMs default to memory settings that both IBM and Sun say are crap for app servers (e.g. neither using their 1.4 or later GC algorithms by default), I'm not sure I'd place too much stock in their defaults. It's my understanding that the WAS properties (or such) override the setting. I guess I'll have to look into the typical settings then... One of the design choices with the JVM is that memory management data (counters, pointers, etc) are within the objects being managed. This is not entirely lucky for virtualized environment because it means that pages with allocated old objects are referenced and thus paged back in. With small enough objects, a GC scan needs the entire heap resident. I thought modern GC strategies identified generations of data in the heap that would allow old data to be skipped in most scans. Our experince, both on standalone Solaris boxes using the Sun JVM, and on our zLinux guests using the IBM JVM, has been that large min/max separation has produced poor results, as the JVM tends to consistently try to push memory use back down to the min setting, often invoking unnecessary GCs to do so. Memory has to come from somewhere. Whether a GC was unnecessary is hard to tell, especially in a virtualized environment. While it is true GC takes valuable resources, so does any (memory) management. Simply the fact that you can't identify the cost does not mean it's for free... With an application server that does not hold data (data resides in the database elsewhere) we could expect memory requirements to consist of a base level plus some amount for each active connection. I've talked to application developers who had very good understanding of the requirements per transaction. Depending on typical volume, duration and freqeuency of such active connections, you get a typical heap size. If you set the minimum a bit aobve that, malloc() / free() calls should happen only around periods with significantly more active connections. I recall you can also change the chunk size to create some hystheresis. Don't know whether this model still works now that developers start to implement cache with in-memory databases on top of GC-managed objects... Using our actual applications with our stress test suites (which tend to have pretty good predictave power) we tend to see worse application behaviour (small stalls, higher average response times) and higher CPU use when giving a big spread for min/max; on an older Sun JVM/Solaris combo we also saw failure in the application to ramp up quickly enough to deal with load spikes, with big stalls as the JVM tried to alloc memory. Don't know about yours, but stress tests often involve periods with high utilization. While you need that stake in the ground, it's not the best one to measure scalability: if you need the resource 90% of the time, you worry less about how to share it during the remaining 10%. But if you have an average utilization of 5% and a duty cycle of an hour, then it makes sense trying to share resources even if it costs you something extra (during the 5% that your application is active). (That last behaviour was common to connection pooling - if we didn't open enough connections to Oracle at startup time, the app could have bad performance or even fail as load ramped up due to an inability to open connections quickly enough.) Though it might sound nice to have a large connection pool, if that means the data is referenced very rarely (would not be surprised to see round-robin rather than push-down stack) then you may get paged on z/VM level and held back even longer. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Memory use question
On Wed, Jun 23, 2010 at 4:46 PM, Mrohs, Ray ray.mr...@usdoj.gov wrote: The cache number looks interesting because it remains large while swap space is being used up. This particular instance has been up for 30+ days, so maybe there is incremental swap space saturation over time? Swappiness setting makes Linux swap out process data to retain data in page cache. It's likely to happen when the JVM heap is too large and continues to grow. The old stuff never gets referenced and is eventually swapped out, and then later z/VM will page out that VDISK again. Still find, until the point where you do a garbage collect and re-use the heap pages. That requires Linux to swap it in, which means z/VM must page it in.. etc. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
It's called a native memory leak. Lots of things can cause it. Thread pools, asynci i/o are two areas where this can happen. See http://www-01.ibm.com/support/docview.wss?uid=swg21368248 Also http://www-01.ibm.com/support/docview.wss?uid=swg21373312 You may have to open a PMR to get IBM to assist. Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Mrohs, Ray Sent: Wednesday, June 23, 2010 7:46 AM To: LINUX-390@vm.marist.edu Subject: [LINUX-390] Memory use question Hi, I am experimentally minimizing the footprint of a SLES10 WebSphere 7 instance and seeing the following. Swap is to v-disk. top: Tasks: 120 total, 3 running, 117 sleeping, 0 stopped, 0 zombie Cpu(s): 5.3%us, 0.3%sy, 0.0%ni, 94.0%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 2050776k total, 2041236k used, 9540k free, 120320k buffers Swap: 849964k total, 819676k used,30288k free, 431684k cached rmf: Total memory size 2002MB Swap space size 830MB Total memory used 1507MB % Swap space used 96.4% Used for buffer 117MB Swap-in rate 1.233/s Used for shared 0MB Swap-out rate 0/s Used for cache 422MB Page-in rate 4.933/s Total free memory7MB Page-out rate 35.888/s The cache number looks interesting because it remains large while swap space is being used up. This particular instance has been up for 30+ days, so maybe there is incremental swap space saturation over time? Ray Mrohs U.S. Department of Justice 202-307-6896 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
Also: so maybe there is incremental swap space saturation over time? Use your perf tool and plot out the last 30 days of swap size to see if it looks like a leak. Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
Marcy, thanks for the pointers. I verified that the thread pool default/max numbers are the same, and the async I/O box remains unchecked. The swap space used stays at 0 for a day or two, but slowly climbs. There are also dips but it never returns to 0 until reboot. The WebSphere java proc grows from 800M to 1200M RES in that time. I suppose it's also possible we have some WebSphere user code that's not behaving as expected. Ray -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Marcy Cortes Sent: Wednesday, June 23, 2010 11:12 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Memory use question It's called a native memory leak. Lots of things can cause it. Thread pools, asynci i/o are two areas where this can happen. See http://www-01.ibm.com/support/docview.wss?uid=swg21368248 Also http://www-01.ibm.com/support/docview.wss?uid=swg21373312 You may have to open a PMR to get IBM to assist. Marcy -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
Rob, Heap size is set to 500M/1000M. I've read recommendations to make default/max the same number, but I would just be guessing at a value. It has grow close to 800M at least once, but it always falls back to around the default size and stays there most of the time. Ray -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Rob van der Heij Sent: Wednesday, June 23, 2010 11:06 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Memory use question On Wed, Jun 23, 2010 at 4:46 PM, Mrohs, Ray ray.mr...@usdoj.gov wrote: The cache number looks interesting because it remains large while swap space is being used up. This particular instance has been up for 30+ days, so maybe there is incremental swap space saturation over time? Swappiness setting makes Linux swap out process data to retain data in page cache. It's likely to happen when the JVM heap is too large and continues to grow. The old stuff never gets referenced and is eventually swapped out, and then later z/VM will page out that VDISK again. Still find, until the point where you do a garbage collect and re-use the heap pages. That requires Linux to swap it in, which means z/VM must page it in.. etc. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
Rob mentioned the vm.swappiness setting and he and I have had a lot of discussions about that one. You do want to probably set that to zero on a WAS server (and probably others). It's recommended in this WAS tuning paper too http://download.boulder.ibm.com/ibmdl/pub/software/dw/linux390/perf/ZSW03132USEN.PDF Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Mrohs, Ray Sent: Wednesday, June 23, 2010 11:40 AM To: LINUX-390@vm.marist.edu Subject: Re: [LINUX-390] Memory use question Marcy, thanks for the pointers. I verified that the thread pool default/max numbers are the same, and the async I/O box remains unchecked. The swap space used stays at 0 for a day or two, but slowly climbs. There are also dips but it never returns to 0 until reboot. The WebSphere java proc grows from 800M to 1200M RES in that time. I suppose it's also possible we have some WebSphere user code that's not behaving as expected. Ray -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Marcy Cortes Sent: Wednesday, June 23, 2010 11:12 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Memory use question It's called a native memory leak. Lots of things can cause it. Thread pools, asynci i/o are two areas where this can happen. See http://www-01.ibm.com/support/docview.wss?uid=swg21368248 Also http://www-01.ibm.com/support/docview.wss?uid=swg21373312 You may have to open a PMR to get IBM to assist. Marcy -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Memory use question
On Wed, Jun 23, 2010 at 8:47 PM, Mrohs, Ray ray.mr...@usdoj.gov wrote: Rob, Heap size is set to 500M/1000M. I've read recommendations to make default/max the same number, but I would just be guessing at a value. It has grow close to 800M at least once, but it always falls back to around the default size and stays there most of the time. Ray The suggestion to set min and max the same goes well with the folks who have plenty of memory and just want to avoid garbage collection... whether it applies to a shared resource environment needs to be seen (uh, measured..) When the minimum heap size is smaller than the maximum, JVM moves objects in the heap such that entire chunks of memory can be returned to Linux. This makes sense when peak usage is termporary because you avoid the JVM to sit on all the memory when it does not have to. But when the memory requirement is fairly constant, it would be a waste to give the chunk back and then ask for it again. I'm always suspicious about recommendations without motivation or measurements. If it really were the best setting, would we not expect the defaults to be like that? Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
