Swap partition "filling up" on RHEL4 - NOT
some things in this thread. 1) as Rob pointed out, you are slowing down your linux server on purpose. A swap device to disk maybe can do 200 swaps per second, so ok, add 3, and you can do 600. With vdisk, it could be 40,000 per second - so linux doesn't notice it is swapping. 2) you do not have to back up vdisk with storage. You have to back it up with paging devices. So if you converted those dog slow swap devices to vm paging, problem solved. nothing lost, no extra storage required, just linux doesn't get arbitrarily slowed when it happens to need more storage. and vm very nicely balances the paging load across all it's paging devices. For the time period linux needs to swap to vdisk it has all this extra fast memory, then that extra memory gets paged out. 3) linux measurement of anything based on time is skewed. linux might measure the swap device 100% busy - where from the vm side, it shows 30%. depends on other server's CPu requirements. Anything measuring time inside linux and busy time is wrong. you need the vm perspective which is different. I'd say your data is misleading and incomplete, too many times does such data lead to misleading conclusions. >Date: Wed, 30 Aug 2006 11:25:53 -0400 >Reply-To: Linux on 390 Port >From: "Hall, Ken (GTI)" <[EMAIL PROTECTED]> > >I tracked down a copy of the actual report from the performance >guy, and it turns out we've been (partially) barking up the >wrong tree. Sorry for the confusion. =20 > >The real story is that the large swap partition DEVICE was 100% >BUSY during part of the test. The original summary report said >100% FULL, so we've all been going around trying to explain >that. What I now suspect was actually happening was that the >memory load increased to the point that the instance started >paging heavily and saturated the path to one of the swap >devices. > >This makes a lot more sense, but the problem is still there, >just a little different. > >We definitely have a memory constraint, so we're going to >increase the memory allocation for the instance, but would it >also help to use multiple small-ish swap partitions as a >safety net for peak periods? (Please don't start up the debate >on whether to use disk-based swap devices at all. I've been >through that, and the alternatives won't fly here right now.) > >Right now, the busy (larger) swap device has a priority of -2, >and the other (smaller) one has a priority of -1 (defaults). >The load problem appeared on the larger one, implying that it >was getting hit harder (far harder than the smaller partition >during peak periods). I found a howto that indicates that if >you set multiple partitions to the same priority, they get >used "round robin", instead in in "spillover" mode, but this >might not help either. =20 > >If we set several partitions to equal priority, does the kernel >do any kind of load balancing between partitions? It wouldn't >do much good if it keeps trying to use a heavily loaded swap >device. > >Thanks for all of the suggestions, and apologies for starting >down the wrong road earlier. "If you can't measure it, I'm Just NOT interested!"(tm) // Barton Robinson - CBW Internet: [EMAIL PROTECTED] Velocity Software, IncMailing Address: 196-D Castro Street P.O. Box 390640 Mountain View, CA 94041 Mountain View, CA 94039-0640 VM Performance Hotline: 650-964-8867 Fax: 650-964-9012 Web Page: WWW.VELOCITY-SOFTWARE.COM // -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4 - NOT
Rob van der Heij commented: > On 8/30/06, John Campbell <[EMAIL PROTECTED]> wrote: > > > The hell of it is that I *don't* have a zSeries-- or even an > > instance within z/VM-- where I can run Linux, so I can only do > > some speculation. > > Yes, and that is where it makes a difference. On discrete servers it > may make sense to tune your configuration for maximum single server > performance. Any resources that you don't use are wasted anyway, and > there's nothing wrong to increase CPU usage from 10% to 20% if that > buys you 10% more throughput. Your tuning is not complicated by other > things happening on the same machine (although a shared disk system > like a SAN is going to impact your tuning if the bottleneck is other > than on your own doorstep). > > With Linux on z/VM the tuning objective often is to achieve lowest > cost per transaction (as long as you meet the required response time). > It is very rare that you need to size your virtual machine to consume > all available resources of the entire machine (like CPU or I/O). In > most cases the z/VM system will also run other virtual machines at the > same time and normally your objective is to have all of them make some > progress rather than let one virtual machine run away with the system. > Oversized systems then require extra monitoring to prevent them from > taking all that you gave them. Well, I commented to Ken that this is because you're not merely tuning for "best throughput" but you have to balance it with "being a good neighbor", so there are two targets you have to consider. Now if only I had an opportunity to amass *real* experience. I was once pretty good at working w/ standalone Unix systems and even did some performance analysis on mainframes... but not of zSeries, but on Sperry 1100 systems. It's been a long time since I wore _that_ hat. (laughs) I have *way* too much "useless knowledge" stowed away in my head. Well, at least I'm pretty good w/ pSeries boxes... John R. Campbell, Speaker to Machines (GNUrd) (813) 356-5322 (t/l 697) Adsumo ergo raptus sum MacOS X: Because making Unix user-friendly was easier than debugging Windows. Red Hat Certified Engineer (#803004680310286) -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Fw: [LINUX-390] Swap partition "filling up" on RHEL4 - NOT
On 8/30/06, John Campbell <[EMAIL PROTECTED]> wrote: The hell of it is that I *don't* have a zSeries-- or even an instance within z/VM-- where I can run Linux, so I can only do some speculation. Yes, and that is where it makes a difference. On discrete servers it may make sense to tune your configuration for maximum single server performance. Any resources that you don't use are wasted anyway, and there's nothing wrong to increase CPU usage from 10% to 20% if that buys you 10% more throughput. Your tuning is not complicated by other things happening on the same machine (although a shared disk system like a SAN is going to impact your tuning if the bottleneck is other than on your own doorstep). With Linux on z/VM the tuning objective often is to achieve lowest cost per transaction (as long as you meet the required response time). It is very rare that you need to size your virtual machine to consume all available resources of the entire machine (like CPU or I/O). In most cases the z/VM system will also run other virtual machines at the same time and normally your objective is to have all of them make some progress rather than let one virtual machine run away with the system. Oversized systems then require extra monitoring to prevent them from taking all that you gave them. Even on a mainframe you will not be happy with a Linux server constantly swapping to disk. If you're swapping so much that you would enjoy multiple paths to your swap devices, there's probably something wrong in your setup. The only good thing of swapping to real disk is that you slow down the server enough to prevent it from taking up a lot of resources. Swapping to VDISK however is very fast and is a good trade-off to fool Linux memory management. Things like CMM let us fool Linux even better, and if you can steer that with the system wide performance metrics from z/VM it can be very effective. Rob -- Rob van der Heij Velocity Software, Inc http://velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Fw: [LINUX-390] Swap partition "filling up" on RHEL4 - NOT
Ken Hall said: > I tracked down a copy of the actual report from the performance guy, > and it turns out we've been (partially) barking up the wrong tree. > Sorry for the confusion. > > The real story is that the large swap partition DEVICE was 100% BUSY > during part of the test. The original summary report said 100% > FULL, so we've all been going around trying to explain that. What I > now suspect was actually happening was that the memory load > increased to the point that the instance started paging heavily and > saturated the path to one of the swap devices. > > This makes a lot more sense, but the problem is still there, just a > little different. > > We definitely have a memory constraint, so we're going to increase > the memory allocation for the instance, but would it also help to > use multiple small-ish swap partitions as a safety net for peak > periods? (Please don't start up the debate on whether to use disk- > based swap devices at all. I've been through that, and the > alternatives won't fly here right now.) > > Right now, the busy (larger) swap device has a priority of -2, and > the other (smaller) one has a priority of -1 (defaults). The load > problem appeared on the larger one, implying that it was getting hit > harder (far harder than the smaller partition during peak periods). > I found a howto that indicates that if you set multiple partitions > to the same priority, they get used "round robin", instead in in > "spillover" mode, but this might not help either. > > If we set several partitions to equal priority, does the kernel do > any kind of load balancing between partitions? It wouldn't do much > good if it keeps trying to use a heavily loaded swap device. > > Thanks for all of the suggestions, and apologies for starting down > the wrong road earlier. Actually, I'd suspect a mix of round-robin and spillover might be a good approach, depending upon I/O paths. Using a SAN may be... ahem... unhelpful, since it has few paths to the DASD, and it's these paths that get busy, too, not just the devices. Additionally, has anyone done any research on multiple small paging spaces at the same priority level (round-robin) which are accessed by different I/O paths so that there's an effect like striping across the paging spaces (it's not _real_ striping, of course, but it's like setting the inter-policy on AIX's LVM to "maximum") in order to minimize which path is busiest? The idea of multiple small chunks on different drives centers on the idea of locality of reference, an attempt to minimize HDA motion. While SAN devices pretty much render this debate moot by making such factors much harder to tune for (since the whole disk farm is being carved up into logical volumes that look like physical volumes for the client systems) I'm wondering how much this is impacting the ability to use paging spaces and distributing the I/O load across multiple I/O paths. So that's why you may want multiple spaces of the same size on as many drives as you can arrange for. That's easy to say and to do on, say, a PC. On an xSeries server, this might be more difficult to arrange since one of the defaults I've run across in my travails is that the drives are all arranged in a large RAID5 array. Here's another one: can the swap devices be allocated on drives that AREN'T in a RAID5 array? After all, writing to such an array is expensive in terms of time... and placing paging space within such an array, especially if it's active, is gonna come to haunt you given the slow writes to the device (you're never far from BUSY on a RAID5 array). (sighs) The hell of it is that I *don't* have a zSeries-- or even an instance within z/VM-- where I can run Linux, so I can only do some speculation. So maybe I shoulda worn a toga for this post given that I've just played the part of a stand-up philosopher. John R. Campbell, Speaker to Machines (GNUrd) (813) 356-5322 (t/l 697) Adsumo ergo raptus sum MacOS X: Because making Unix user-friendly was easier than debugging Windows. Red Hat Certified Engineer (#803004680310286) -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4 - NOT
This is the road I've been down before re disk-based swapping, and in principle, I agree, BUT... I can't justify that at the moment, even if it's just because 1) We have no experience with it, and 2) We're at a point in the process where we can't make major configuration changes. I also believe virtual disk requires zVM memory backing, and we're not sure we can afford that without a lot of analysis. > -Original Message- > From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of > Rich Smrcina > Sent: Wednesday, August 30, 2006 11:37 AM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 - NOT > > > If you're running under VM, make the swap devices a virtual > disk. They > can handle a significantly higher load than the real disk. > > Hall, Ken (GTI) wrote: > > I tracked down a copy of the actual report from the > performance guy, and it turns out we've been (partially) > barking up the wrong tree. Sorry for the confusion. > > > > The real story is that the large swap partition DEVICE was > 100% BUSY during part of the test. The original summary > report said 100% FULL, so we've all been going around trying > to explain that. What I now suspect was actually happening > was that the memory load increased to the point that the > instance started paging heavily and saturated the path to one > of the swap devices. > > > > This makes a lot more sense, but the problem is still > there, just a little different. > > > > We definitely have a memory constraint, so we're going to > increase the memory allocation for the instance, but would it > also help to use multiple small-ish swap partitions as a > safety net for peak periods? (Please don't start up the > debate on whether to use disk-based swap devices at all. > I've been through that, and the alternatives won't fly here > right now.) > > > > Right now, the busy (larger) swap device has a priority of > -2, and the other (smaller) one has a priority of -1 > (defaults). The load problem appeared on the larger one, > implying that it was getting hit harder (far harder than the > smaller partition during peak periods). I found a howto that > indicates that if you set multiple partitions to the same > priority, they get used "round robin", instead in in > "spillover" mode, but this might not help either. > > > > If we set several partitions to equal priority, does the > kernel do any kind of load balancing between partitions? It > wouldn't do much good if it keeps trying to use a heavily > loaded swap device. > > > > Thanks for all of the suggestions, and apologies for > starting down the wrong road earlier. > > > > > > If you are not an intended recipient of this e-mail, please > notify the sender, delete it and do not read, act upon, > print, disclose, copy, retain or redistribute it. Click here > for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ > > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4 - NOT
On 8/30/06, Rich Smrcina <[EMAIL PROTECTED]> wrote: If you're running under VM, make the swap devices a virtual disk. They can handle a significantly higher load than the real disk. Sure. And that's where the different priorities make sense to avoid infinite growth of the footprint of that VDISK. If you have multiple real disks for swap you probably want to have them same priority so that traffic is evenly spread. But if he's really (?) driving the swap device at 100% busy, replacing it with VDISK will at best hide the problem. Most likely his real memory is just a bit too small for these two JVMs. Rob -- Rob van der Heij Velocity Software, Inc http://velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4 - NOT
If you're running under VM, make the swap devices a virtual disk. They can handle a significantly higher load than the real disk. Hall, Ken (GTI) wrote: I tracked down a copy of the actual report from the performance guy, and it turns out we've been (partially) barking up the wrong tree. Sorry for the confusion. The real story is that the large swap partition DEVICE was 100% BUSY during part of the test. The original summary report said 100% FULL, so we've all been going around trying to explain that. What I now suspect was actually happening was that the memory load increased to the point that the instance started paging heavily and saturated the path to one of the swap devices. This makes a lot more sense, but the problem is still there, just a little different. We definitely have a memory constraint, so we're going to increase the memory allocation for the instance, but would it also help to use multiple small-ish swap partitions as a safety net for peak periods? (Please don't start up the debate on whether to use disk-based swap devices at all. I've been through that, and the alternatives won't fly here right now.) Right now, the busy (larger) swap device has a priority of -2, and the other (smaller) one has a priority of -1 (defaults). The load problem appeared on the larger one, implying that it was getting hit harder (far harder than the smaller partition during peak periods). I found a howto that indicates that if you set multiple partitions to the same priority, they get used "round robin", instead in in "spillover" mode, but this might not help either. If we set several partitions to equal priority, does the kernel do any kind of load balancing between partitions? It wouldn't do much good if it keeps trying to use a heavily loaded swap device. Thanks for all of the suggestions, and apologies for starting down the wrong road earlier. If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Swap partition "filling up" on RHEL4 - NOT
I tracked down a copy of the actual report from the performance guy, and it turns out we've been (partially) barking up the wrong tree. Sorry for the confusion. The real story is that the large swap partition DEVICE was 100% BUSY during part of the test. The original summary report said 100% FULL, so we've all been going around trying to explain that. What I now suspect was actually happening was that the memory load increased to the point that the instance started paging heavily and saturated the path to one of the swap devices. This makes a lot more sense, but the problem is still there, just a little different. We definitely have a memory constraint, so we're going to increase the memory allocation for the instance, but would it also help to use multiple small-ish swap partitions as a safety net for peak periods? (Please don't start up the debate on whether to use disk-based swap devices at all. I've been through that, and the alternatives won't fly here right now.) Right now, the busy (larger) swap device has a priority of -2, and the other (smaller) one has a priority of -1 (defaults). The load problem appeared on the larger one, implying that it was getting hit harder (far harder than the smaller partition during peak periods). I found a howto that indicates that if you set multiple partitions to the same priority, they get used "round robin", instead in in "spillover" mode, but this might not help either. If we set several partitions to equal priority, does the kernel do any kind of load balancing between partitions? It wouldn't do much good if it keeps trying to use a heavily loaded swap device. Thanks for all of the suggestions, and apologies for starting down the wrong road earlier. If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
On 8/30/06, Hall, Ken (GTI) <[EMAIL PROTECTED]> wrote: Yes, it does, and we haven't, although the performance folks use it extensively on x86 Linux, and have great confidence in it. Not unlikely that application exhausted the JVM heap and someone (or something) jumped to conclusions seeing one swap device full (especially since in other areas people have swap devices with the same priority). With some bad luck you probably could have the two JVMs fill up your real memory plus the first swap device. Rob -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
Maybe if this occurs again you should terminate Teamquest while memory usage is high and then check if the memory usage starts going back down. At least you can maybe see (1) if usage still goes up the memory problem or leak is due to the application that's running or (2) if usage goes down maybe it's Teamquest. -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Hall, Ken (GTI) Sent: Wednesday, August 30, 2006 9:52 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Swap partition "filling up" on RHEL4 Yes, it does, and we haven't, although the performance folks use it extensively on x86 Linux, and have great confidence in it. > -Original Message- > From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of > Rich Smrcina > Sent: Wednesday, August 30, 2006 9:43 AM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 > > > Does 'Teamquest' run on the websphere machine? If so, don't discount > the possibility that it caused the problem. > > Hall, Ken (GTI) wrote: > > You've put your finger on the crux of the matter there. > We're not sure. > > > > We believe it was Teamquest that reported the error, but > the first mention of this issue came from someone who's on vacation > right now. > > > > I would have been inclined to discount the message except > for the hanging threads and IBM's report that this was caused by a > memory shortage. > > > >> -Original Message- > >> From: Linux on 390 Port [mailto:[EMAIL PROTECTED] > Behalf Of > >> Carsten Otte > >> Sent: Wednesday, August 30, 2006 10:55 AM > >> To: LINUX-390@VM.MARIST.EDU > >> Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 > >> > >> > >> Hall, Ken (GTI) wrote: > >>> Here's how it looks NOW. Can't speak for the time of the failure. > >> You stated, one swap device is full and another is not (at > the time of > >> failure). Where do you get that data from? > >> > >> cheers, > >> Carsten > >> -- > >> Carsten Otte has stopped smoking: Ich habe in 3 Monate, 5 > Tage und 22 > >> Stunden schon 470,07 Euro gespart anstatt 1.958,63 Zigaretten zu > >> kaufen. > >> > > -- > Rich Smrcina > VM Assist, Inc. > Phone: 414-491-6001 > Ans Service: 360-715-2467 > rich.smrcina at vmassist.com > > Catch the WAVV! http://www.wavv.org > WAVV 2007 - Green Bay, WI - May 18-22, 2007 > > -- > For LINUX-390 subscribe / signoff / archive access instructions, send > email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or > visit http://www.marist.edu/htbin/wlvindex?LINUX-390 > If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
If you can drive the test remotely, like one of those x86 machines, it will certainly eliminate his possibility. Hall, Ken (GTI) wrote: Yes, it does, and we haven't, although the performance folks use it extensively on x86 Linux, and have great confidence in it. -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of Rich Smrcina Sent: Wednesday, August 30, 2006 9:43 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 Does 'Teamquest' run on the websphere machine? If so, don't discount the possibility that it caused the problem. Hall, Ken (GTI) wrote: You've put your finger on the crux of the matter there. We're not sure. We believe it was Teamquest that reported the error, but the first mention of this issue came from someone who's on vacation right now. I would have been inclined to discount the message except for the hanging threads and IBM's report that this was caused by a memory shortage. -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of Carsten Otte Sent: Wednesday, August 30, 2006 10:55 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 Hall, Ken (GTI) wrote: Here's how it looks NOW. Can't speak for the time of the failure. You stated, one swap device is full and another is not (at the time of failure). Where do you get that data from? cheers, Carsten -- Carsten Otte has stopped smoking: Ich habe in 3 Monate, 5 Tage und 22 Stunden schon 470,07 Euro gespart anstatt 1.958,63 Zigaretten zu kaufen. -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
Yes, it does, and we haven't, although the performance folks use it extensively on x86 Linux, and have great confidence in it. > -Original Message- > From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of > Rich Smrcina > Sent: Wednesday, August 30, 2006 9:43 AM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 > > > Does 'Teamquest' run on the websphere machine? If so, don't discount > the possibility that it caused the problem. > > Hall, Ken (GTI) wrote: > > You've put your finger on the crux of the matter there. > We're not sure. > > > > We believe it was Teamquest that reported the error, but > the first mention of this issue came from someone who's on > vacation right now. > > > > I would have been inclined to discount the message except > for the hanging threads and IBM's report that this was caused > by a memory shortage. > > > >> -Original Message- > >> From: Linux on 390 Port [mailto:[EMAIL PROTECTED] > Behalf Of > >> Carsten Otte > >> Sent: Wednesday, August 30, 2006 10:55 AM > >> To: LINUX-390@VM.MARIST.EDU > >> Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 > >> > >> > >> Hall, Ken (GTI) wrote: > >>> Here's how it looks NOW. Can't speak for the time of the failure. > >> You stated, one swap device is full and another is not (at > the time of > >> failure). Where do you get that data from? > >> > >> cheers, > >> Carsten > >> -- > >> Carsten Otte has stopped smoking: Ich habe in 3 Monate, 5 > Tage und 22 > >> Stunden schon 470,07 Euro gespart anstatt 1.958,63 Zigaretten > >> zu kaufen. > >> > > -- > Rich Smrcina > VM Assist, Inc. > Phone: 414-491-6001 > Ans Service: 360-715-2467 > rich.smrcina at vmassist.com > > Catch the WAVV! http://www.wavv.org > WAVV 2007 - Green Bay, WI - May 18-22, 2007 > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [EMAIL PROTECTED] with the message: INFO > LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
Does 'Teamquest' run on the websphere machine? If so, don't discount the possibility that it caused the problem. Hall, Ken (GTI) wrote: You've put your finger on the crux of the matter there. We're not sure. We believe it was Teamquest that reported the error, but the first mention of this issue came from someone who's on vacation right now. I would have been inclined to discount the message except for the hanging threads and IBM's report that this was caused by a memory shortage. -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of Carsten Otte Sent: Wednesday, August 30, 2006 10:55 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 Hall, Ken (GTI) wrote: Here's how it looks NOW. Can't speak for the time of the failure. You stated, one swap device is full and another is not (at the time of failure). Where do you get that data from? cheers, Carsten -- Carsten Otte has stopped smoking: Ich habe in 3 Monate, 5 Tage und 22 Stunden schon 470,07 Euro gespart anstatt 1.958,63 Zigaretten zu kaufen. -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
You've put your finger on the crux of the matter there. We're not sure. We believe it was Teamquest that reported the error, but the first mention of this issue came from someone who's on vacation right now. I would have been inclined to discount the message except for the hanging threads and IBM's report that this was caused by a memory shortage. > -Original Message- > From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of > Carsten Otte > Sent: Wednesday, August 30, 2006 10:55 AM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 > > > Hall, Ken (GTI) wrote: > > Here's how it looks NOW. Can't speak for the time of the failure. > You stated, one swap device is full and another is not (at the time of > failure). Where do you get that data from? > > cheers, > Carsten > -- > Carsten Otte has stopped smoking: Ich habe in 3 Monate, 5 Tage und 22 > Stunden schon 470,07 Euro gespart anstatt 1.958,63 Zigaretten > zu kaufen. > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [EMAIL PROTECTED] with the message: INFO > LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
Hall, Ken (GTI) wrote: Here's how it looks NOW. Can't speak for the time of the failure. You stated, one swap device is full and another is not (at the time of failure). Where do you get that data from? cheers, Carsten -- Carsten Otte has stopped smoking: Ich habe in 3 Monate, 5 Tage und 22 Stunden schon 470,07 Euro gespart anstatt 1.958,63 Zigaretten zu kaufen. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
This is consistent with what I've observed, but it's nice to have confirmation. Here's the current swap configuration: FilenameTypeSizeUsedPriority /dev/dasdd1 partition 516136 40360 -1 /dev/dasdf1 partition 1032460 11732 -2 At the request of another, I supplied /proc/meminfo in a separate post. Note that the above reflects the current (idle) state of the instance. We have another load test scheduled this week, so we hope to get some more info then. > -Original Message- > From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of > Alan Cox > Sent: Wednesday, August 30, 2006 9:06 AM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 > > > Ar Mer, 2006-08-30 am 08:21 -0400, ysgrifennodd Hall, Ken (GTI): > > We've been going around trying to figure out: 1) why we suddenly > > should have run out of virtual memory (if we did, because we never > > even got close to that before), and 2) why it was reported in this > > way. Does anyone know if there's anything in the swap space > > management mechanism that would cause Linux to fail malloc > if a single > > swap partition fills up? Is there some kind of process affinity for > > paging to swap spaces? > > #1 no idea - maybe you hit a worst case or a memory leak in it ? Java > and other garbage collectors are also a common source of large brief > spikes in memory usage. > > #2 Linux uses the swap according to the priority of the swap files and > swap partitions. When one fills it will move onto the next. Memory > allocations are failed according to the overcommit settings. > > The default behaviour is "fail allocations that will obviously fail". > This allows overcommit of resources (so you can get process > kills due to > out of memory) but tries to avoid obvious cases. > > A seperate mode is available which assumes all allocation requests > should be allowed until the box goes totally out of memory. > This is only > for specialist scientific applications > > The third mode prevents overcommit and will fail allocations > when it is > theoretically possible for an overcommit situation to occur, this is > used for high reliability environments where having to deploy > extra swap > "just in case" is preferable to, say, having your air traffic control > system self destruct. > > Alan > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [EMAIL PROTECTED] with the message: INFO > LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
Here's how it looks NOW. Can't speak for the time of the failure. MemTotal: 1345376 kB MemFree: 6968 kB Buffers: 73984 kB Cached: 134016 kB SwapCached: 10784 kB Active:1110932 kB Inactive: 105864 kB HighTotal: 0 kB HighFree:0 kB LowTotal: 1345376 kB LowFree: 6968 kB SwapTotal: 1548596 kB SwapFree: 1496504 kB Dirty: 364 kB Writeback: 0 kB Mapped:1024648 kB Slab:91356 kB Committed_AS: 4402024 kB PageTables: 5536 kB VmallocTotal: 4293582848 kB VmallocUsed: 3540 kB VmallocChunk: 4293577724 kB There are two WAS JVM instances running (for two separate apps). To meet service targets, they've been increasing the memory allocation for each of these. I believe they're currently set to 512m, and the "real" storage allocation was increased to 1.3 gb. to allow for this. Various people have suggested increasing "real" memory still further, but we're approaching the point where it might affect other instances under VM, so we need justification and we haven't seen anything concrete. > -Original Message- > From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of > Carsten Otte > Sent: Wednesday, August 30, 2006 10:38 AM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: [LINUX-390] Swap partition "filling up" on RHEL4 > > > Carsten Otte wrote: > > Malloc fails when there is no more free memory except the > emergency pool > > and no swap slots are free. Note that anonymous pages in > memory may well > > occupy both a swap slot and a pysical page in swap cache at > the same time. > May we see /proc/meminfo content of the system? > > Carsten > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [EMAIL PROTECTED] with the message: INFO > LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
Ar Mer, 2006-08-30 am 08:21 -0400, ysgrifennodd Hall, Ken (GTI): > We've been going around trying to figure out: 1) why we suddenly > should have run out of virtual memory (if we did, because we never > even got close to that before), and 2) why it was reported in this > way. Does anyone know if there's anything in the swap space > management mechanism that would cause Linux to fail malloc if a single > swap partition fills up? Is there some kind of process affinity for > paging to swap spaces? #1 no idea - maybe you hit a worst case or a memory leak in it ? Java and other garbage collectors are also a common source of large brief spikes in memory usage. #2 Linux uses the swap according to the priority of the swap files and swap partitions. When one fills it will move onto the next. Memory allocations are failed according to the overcommit settings. The default behaviour is "fail allocations that will obviously fail". This allows overcommit of resources (so you can get process kills due to out of memory) but tries to avoid obvious cases. A seperate mode is available which assumes all allocation requests should be allowed until the box goes totally out of memory. This is only for specialist scientific applications The third mode prevents overcommit and will fail allocations when it is theoretically possible for an overcommit situation to occur, this is used for high reliability environments where having to deploy extra swap "just in case" is preferable to, say, having your air traffic control system self destruct. Alan -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
Carsten Otte wrote: Malloc fails when there is no more free memory except the emergency pool and no swap slots are free. Note that anonymous pages in memory may well occupy both a swap slot and a pysical page in swap cache at the same time. May we see /proc/meminfo content of the system? Carsten -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Swap partition "filling up" on RHEL4
Hall, Ken (GTI) wrote: 1) why we suddenly should have run out of virtual memory (if we did, because we never even got close to that before), and Most probably because your application has sudden demand for anonymous memory. Your report indicates the processes are hanging in "malloc". Maybe an infinite loop containing a memory leak? 2) why it was reported in this way. Does anyone know if there's anything in the swap space management mechanism that would cause Linux to fail malloc if a single swap partition fills up? Malloc fails when there is no more free memory except the emergency pool and no swap slots are free. Note that anonymous pages in memory may well occupy both a swap slot and a pysical page in swap cache at the same time. Is there some kind of process affinity for paging to swap spaces? No. If malloc fails, all swap slots are in use. cheers, Carsten -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Swap partition "filling up" on RHEL4
We've been doing some load testing with WAS on RHEL4 under VM, and the other day something weird came up. The machine is defined with 1.3 gb. storage under zVM 5.2, and about 1.5 gb. swap space in two partitions: One about 1 gb. (dasdf), and the other about .5 gb. (dasdd) We use a product called "Teamquest" to monitor the system during the test, and as we were running, it suddently reported "dasdf" was 100% full. WAS started to choke, and analysis of a crashdump appeared to show threads hung consistent with malloc failures. We've been going around trying to figure out: 1) why we suddenly should have run out of virtual memory (if we did, because we never even got close to that before), and 2) why it was reported in this way. Does anyone know if there's anything in the swap space management mechanism that would cause Linux to fail malloc if a single swap partition fills up? Is there some kind of process affinity for paging to swap spaces? If you are not an intended recipient of this e-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute it. Click here for important additional terms relating to this e-mail. http://www.ml.com/email_terms/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
