Re: What about these VMSTAT 5 numbers?
Also set the Java Heap as low as possible relative what the appl needs. A too big heap just cause problems with cache and longer time for gc. Cordialement / Vriendelijke Groeten / Best Regards / Med Vänliga Hälsningar Tore Agblad Volvo Information Technology Infrastructure Mainframe Design Development SE-405 08, Gothenburg Sweden E-mail: tore.agb...@volvo.com http://www.volvo.com/volvoit/global/en-gb/ From: Linux on 390 Port [linux-...@vm.marist.edu] On Behalf Of Rob van der Heij [rvdh...@velocitysoftware.com] Sent: Wednesday, February 17, 2010 16:19 To: LINUX-390@VM.MARIST.EDU Subject: Re: What about these VMSTAT 5 numbers? On Wed, Feb 17, 2010 at 2:49 PM, Mrohs, Ray ray.mr...@usdoj.gov wrote: Hi, We are running SLES10 and WebSphere on 1 IFL and 2G storage. These bursts of activity seem to happen every few hours. Is occasional swapping in the thousands too high if its going to VDISK? What can cause the high 'wa' values? Our info is limited since this is a test partition with no VM performance monitoring (yet). Thanks for any insight! The wa column is % of time waiting for I/O. Your vmstat data shows swapping, so my guess is that you're swapping to real disk and the process is in I/O wait for the swap-in. So the virtual machine is too small for this workload. Since this spike happens frequently, it could be the JVM Garbage Collection going through the entire JVM heap and force all pages to be swapped in. Your page cache is rather large for a WAS workload, might be a good thing to lower the swappiness. Increasing the virtual machine size or adding VDISK for swap would help, but can't say whether that fits without seeing z/VM data. Did I already mention Performance Monitor? Oh, you did ;-) -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: What about these VMSTAT 5 numbers?
The Java heap size in WAS is 512M/1024M. Sizing is based on some ROT from the app vendor so I'm not sure how much leeway there is in making adjustments. Thanks for all the info pointers so far. Ray Mrohs -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Agblad Tore Sent: Thursday, February 18, 2010 7:25 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: What about these VMSTAT 5 numbers? Also set the Java Heap as low as possible relative what the appl needs. A too big heap just cause problems with cache and longer time for gc. Cordialement / Vriendelijke Groeten / Best Regards / Med Vänliga Hälsningar Tore Agblad Volvo Information Technology Infrastructure Mainframe Design Development SE-405 08, Gothenburg Sweden E-mail: tore.agb...@volvo.com http://www.volvo.com/volvoit/global/en-gb/ From: Linux on 390 Port [linux-...@vm.marist.edu] On Behalf Of Rob van der Heij [rvdh...@velocitysoftware.com] Sent: Wednesday, February 17, 2010 16:19 To: LINUX-390@VM.MARIST.EDU Subject: Re: What about these VMSTAT 5 numbers? On Wed, Feb 17, 2010 at 2:49 PM, Mrohs, Ray ray.mr...@usdoj.gov wrote: Hi, We are running SLES10 and WebSphere on 1 IFL and 2G storage. These bursts of activity seem to happen every few hours. Is occasional swapping in the thousands too high if its going to VDISK? What can cause the high 'wa' values? Our info is limited since this is a test partition with no VM performance monitoring (yet). Thanks for any insight! The wa column is % of time waiting for I/O. Your vmstat data shows swapping, so my guess is that you're swapping to real disk and the process is in I/O wait for the swap-in. So the virtual machine is too small for this workload. Since this spike happens frequently, it could be the JVM Garbage Collection going through the entire JVM heap and force all pages to be swapped in. Your page cache is rather large for a WAS workload, might be a good thing to lower the swappiness. Increasing the virtual machine size or adding VDISK for swap would help, but can't say whether that fits without seeing z/VM data. Did I already mention Performance Monitor? Oh, you did ;-) -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: What about these VMSTAT 5 numbers?
You can monitor it and see how much the appl need. Try 512 to begin with. Cordialement / Vriendelijke Groeten / Best Regards / Med Vänliga Hälsningar Tore Agblad Volvo Information Technology Infrastructure Mainframe Design Development SE-405 08, Gothenburg Sweden E-mail: tore.agb...@volvo.com http://www.volvo.com/volvoit/global/en-gb/ From: Linux on 390 Port [linux-...@vm.marist.edu] On Behalf Of Mrohs, Ray [ray.mr...@usdoj.gov] Sent: Thursday, February 18, 2010 14:31 To: LINUX-390@VM.MARIST.EDU Subject: Re: What about these VMSTAT 5 numbers? The Java heap size in WAS is 512M/1024M. Sizing is based on some ROT from the app vendor so I'm not sure how much leeway there is in making adjustments. Thanks for all the info pointers so far. Ray Mrohs -Original Message- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Agblad Tore Sent: Thursday, February 18, 2010 7:25 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: What about these VMSTAT 5 numbers? Also set the Java Heap as low as possible relative what the appl needs. A too big heap just cause problems with cache and longer time for gc. Cordialement / Vriendelijke Groeten / Best Regards / Med Vänliga Hälsningar Tore Agblad Volvo Information Technology Infrastructure Mainframe Design Development SE-405 08, Gothenburg Sweden E-mail: tore.agb...@volvo.com http://www.volvo.com/volvoit/global/en-gb/ From: Linux on 390 Port [linux-...@vm.marist.edu] On Behalf Of Rob van der Heij [rvdh...@velocitysoftware.com] Sent: Wednesday, February 17, 2010 16:19 To: LINUX-390@VM.MARIST.EDU Subject: Re: What about these VMSTAT 5 numbers? On Wed, Feb 17, 2010 at 2:49 PM, Mrohs, Ray ray.mr...@usdoj.gov wrote: Hi, We are running SLES10 and WebSphere on 1 IFL and 2G storage. These bursts of activity seem to happen every few hours. Is occasional swapping in the thousands too high if its going to VDISK? What can cause the high 'wa' values? Our info is limited since this is a test partition with no VM performance monitoring (yet). Thanks for any insight! The wa column is % of time waiting for I/O. Your vmstat data shows swapping, so my guess is that you're swapping to real disk and the process is in I/O wait for the swap-in. So the virtual machine is too small for this workload. Since this spike happens frequently, it could be the JVM Garbage Collection going through the entire JVM heap and force all pages to be swapped in. Your page cache is rather large for a WAS workload, might be a good thing to lower the swappiness. Increasing the virtual machine size or adding VDISK for swap would help, but can't say whether that fits without seeing z/VM data. Did I already mention Performance Monitor? Oh, you did ;-) -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: What about these VMSTAT 5 numbers?
On Wed, Feb 17, 2010 at 2:49 PM, Mrohs, Ray ray.mr...@usdoj.gov wrote: Hi, We are running SLES10 and WebSphere on 1 IFL and 2G storage. These bursts of activity seem to happen every few hours. Is occasional swapping in the thousands too high if its going to VDISK? What can cause the high 'wa' values? Our info is limited since this is a test partition with no VM performance monitoring (yet). Thanks for any insight! The wa column is % of time waiting for I/O. Your vmstat data shows swapping, so my guess is that you're swapping to real disk and the process is in I/O wait for the swap-in. So the virtual machine is too small for this workload. Since this spike happens frequently, it could be the JVM Garbage Collection going through the entire JVM heap and force all pages to be swapped in. Your page cache is rather large for a WAS workload, might be a good thing to lower the swappiness. Increasing the virtual machine size or adding VDISK for swap would help, but can't say whether that fits without seeing z/VM data. Did I already mention Performance Monitor? Oh, you did ;-) -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: What about these VMSTAT 5 numbers?
On 2/17/2010 at 08:49 AM, Mrohs, Ray ray.mr...@usdoj.gov wrote: Hi, We are running SLES10 and WebSphere on 1 IFL and 2G storage. These bursts of activity seem to happen every few hours. Is occasional swapping in the thousands too high if its going to VDISK? No. The numbers you show are actually quite low. What can cause the high 'wa' values? Our info is limited since this is a test partition with no VM performance monitoring (yet). As Rob mentioned, %wa is the percentage of time the system was waiting for I/O to complete, and nothing else was running at that moment. When overall CPU use is very low, as in your case, just about any I/O will result in very high %wa numbers. I only see 4 instances of more than trivial CPU use in your vmstat output. The pattern of swap out versus swap in is rather unusual, but not necessarily anything to worry over. As Rob also pointed out, you have a very large amount of storage out on paging space (700-850MB), and very little free/buffer/cache. Since your paging *rates* aren't terribly high, this may just be a lot of unused code getting paged out. The main concern I would have is if the usage pattern changes and a lot more of that would need to be in and not out you could run into page thrashing. Things could get very ugly very fast in that case. Since it is just a test machine, you might not want to spend much time doing anything about it, but keep an eye on it as time goes on. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: What about these VMSTAT 5 numbers?
VMSTAT - in the limited metrics it provides it manages to mix stats based on pages, blocks and kilobytes. All without telling anyone. Swap is only used for dirty anonymous storage. Given that (nearly) all that swap is one way, I'd be guessing that a lot of it is contained in the swap cache - your swap usage is only slowly growing. As an adjunct to what Mark added to the %wa discussion - there is no (direct) correlation between %wa and tasks actually waiting for the I/O to complete. Probable maybe, but not necessary. Linux has awful (performance) metrics - perf counters may help in the future. Be a while before they are available in Enterprise kernels, and only the Böblingen folks will be able to tell you if s390x will (ever) be supported. Or, if/when they are, if they'll be available as a z/VM guest, or only in LPAR mode. Shane ... On Wed, 2010-02-17 at 08:49 -0500, Mrohs, Ray wrote: Hi, We are running SLES10 and WebSphere on 1 IFL and 2G storage. These bursts of activity seem to happen every few hours. Is occasional swapping in the thousands too high if its going to VDISK? What can cause the high 'wa' values? Our info is limited since this is a test partition with no VM performance monitoring (yet). -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
