Re: zVM CPU allocation
On Tue, May 18, 2010 at 6:29 PM, Mark Post mp...@novell.com wrote: Looking at column one, you had a maximum of one process waiting to run. It could very well have been that the system was voluntarily giving up its time slice. The sudden jump in context switches and interrupts, combined with that one process being in uninterruptable sleep (column two) when the steal time goes high suggests that the process was waiting on something. I think cause and response are the other way around. The steal percentage is when Linux had work to do but did not get the cycles. If Linux does not need the cycles, it's idle. When Linux waits for CPU 85% of the time, it only had 15% of the time to do something. So you would expect context switches and interrupts to also drop a lot. The high steal ratio means that Linux did not get the cycles it wanted to have. The z/VM monitor data reveals why that was happening. Without that, we're just guessing. There's many possible reasons for it. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/
Re: zVM CPU allocation
Check the steal column. This is an anomaly, but the numbers on this server are high and often, which is why I am questioning. We do see performance drops at this point, so something is not right. 0 0860 14652 937188 88826000 0 0 108 24 0 0 100 0 0 0 0860 14652 937188 88826000 0 0 109 23 0 1 98 0 1 1 0860 14232 937188 88826000 0 0 107 33 1 1 43 0 55 1 0860 13984 937196 888252002048 133 223 0 1 9 5 85 0 1860 13860 937200 88824800 848 511 1283 1 1 26 2 70 0 0860 13860 937200 8882480016 0 713 1283 1 2 26 7 64 0 0860 13860 937200 88824800 0 0 510 1118 1 1 71 0 27 -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Rob van der Heij Sent: Saturday, November 14, 2009 11:13 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: zVM CPU allocation On Fri, Nov 13, 2009 at 6:05 PM, Michael MacIsaac mike...@us.ibm.com wrote: As Rob and Alan have less blatantly stated, Linux CPU numbers are bogus in a virtual environment. However, with the addition of steal percentage (%st in top), the amount of CPU that is being stolen by the hipervisor, I believe many would agree that they are less bogus. I like your less bogus qualification. I'm trying to be more PC and call it different And if we're into word games; I don't like steal in this context. It suggests something you had was taken away. But in this case, you did not have it and it could not be taken from you :-) Most people *do* agree that you need both Linux and z/VM data to make sense of it or understand whether there is a problem. When someone claims to have wisdom in only a single metric, you normally don't have to try very hard to show him wrong. It is very easy to explain why the old Linux numbers were wrong (and by how much) when you had the z/VM data already. We use the VM monitor data to correct the Linux data. It is true that with the virtual CPU time accounting in Linux (that what produces the steal time) are not affected by that virtualization effect anymore. The numbers are still a bit off, but in normal situation the difference can be ignored. Unfortunately I often deal with abnormal situations where people have performance problems. In my Understanding CPU Usage presentation I show a case where z/VM claims the guest uses 30% of a CPU, Linux says it uses 6% of a CPU, and when you look for detailed per-process usage it adds up to 3% of a CPU (with the new improved numbers). I bring a stuffed penguin for someone in the audience who thinks Linux numbers are correct. And each time it goes back home with me ;-) This was indeed caused by a kernel bug. I think we identified 3 problems with the new CPU accounting in Linux because we match both numbers and want them to be correct. Eventually those bugs get fixed in your systems too. Whether the numbers are more correct or more often correct is not really the issue. I believe it is more important what the value of those numbers is for those who see it and whether they let you solve the performance problem. The reason the Linux admin looks at CPU usage is not because he is worried to wear out the CPU, but because he believes the rest is still available for him to use. In a virtualized environment it does not work like that. There's still a load of tools out there that don't show steal as part of the metrics, but just have user, nice, system. In that case it is better to see 99% to show the system is out of CPU than to see 25% and have no clue. But back to the original problem. If the 3 IFL's run at 10-15% we do not expect the old metrics in Linux to show 99%. There must be something else in the system causing this, and the monitor data would reveal the cause. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ - Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: zVM CPU allocation
On 5/18/2010 at 11:45 AM, Dean, David (I/S) david_d...@bcbst.com wrote: Check the steal column. This is an anomaly, but the numbers on this server are high and often, which is why I am questioning. We do see performance drops at this point, so something is not right. 0 0860 14652 937188 88826000 0 0 108 24 0 0 100 0 0 0 0860 14652 937188 88826000 0 0 109 23 0 1 98 0 1 1 0860 14232 937188 88826000 0 0 107 33 1 1 43 0 55 1 0860 13984 937196 888252002048 133 223 0 1 9 5 85 0 1860 13860 937200 88824800 848 511 1283 1 1 26 2 70 0 0860 13860 937200 8882480016 0 713 1283 1 2 26 7 64 0 0860 13860 937200 88824800 0 0 510 1118 1 1 71 0 27 Looking at column one, you had a maximum of one process waiting to run. It could very well have been that the system was voluntarily giving up its time slice. The sudden jump in context switches and interrupts, combined with that one process being in uninterruptable sleep (column two) when the steal time goes high suggests that the process was waiting on something. Mark Post
Re: zVM CPU allocation
On Thu, Nov 12, 2009 at 7:17 PM, Dean, David (I/S) david_d...@bcbst.com wrote: We have a z10 running zVM 5.4 in a dedicated LPAR with 3 dedicated IFL’s. We have approximately 30 zLinux servers. Using IBM PerfKit, I list all of the individual USER / zLinux CPU usages, each of which generally run in the 1 to 5 % range. These servers do of course peak higher but on average they are pretty low. We then take a look at the 3 IFL’s and see usage of maybe 5% on each. Now, let’s say we have a USER / server or two or three go berserk and peak CPU at 99 % for an extended period of time. We then look at the IFL CPU usages and all three have climbed to maybe 10 to 15% each. Your numbers don't add up. If you have 30 Linux guests at 1-5% each, then you use 30-150% of an engine. With a 3-way z/VM you would have 10-50% utilization of each IFL - not 5% (sidebar: the cautious reader will notice a problem when the idle Linux guests use 5% each, you use half of your capacity just to keep the server warm - you need cheap agents to make this scale) If you would have 3 Linux guests with strong demand for CPU resources, you will indeed see all IFLs go to the max. If that max is very much below 100% then you're probably held back because you share the IFLs with other LPARs. There are some configuration mistakes to make. But you say you have dedicated IFLs for the LPAR, so we must rule that out. You can't have 3 Linux guests use 99% each and still have each of the 3 IFLs utilized for only 15%. That does not add up. So you're probably looking at the wrong things. What makes you determine these 3 Linux guests are not held back by something else? You're not looking at the Linux internal vmstat or top metrics, are you? And I hope you're not using the CP INDICATE command either (since that does a funny average). How did the CPU get allocated? Is it always spread evenly across the IFL’s, is that a setting? Why, if there were 3 USER / servers running at 99%, was more CPU not allocated from the 3 IFL’s? Why did the IFL’s decide to allocate X amount and go no further. You don't really allocate CPU. Basically, VM will give the available CPU resources to those who want it. When there is more demand than available resources, some users will have to wait before they get their share. The business may want to decide who should suffer less in that case, which is why we have SHARE settings and some other scheduler commands. In the USER DIRECTORY I allocate storage but not CPU. As Marty mentions, you can set the SHARE value for the guests via commands or directory statements. A word of caution though: this stuff is not intuitive and you should not expect to get it right by reading the chapter of the book during your coffee break. You can do this work full time and still learn from new workloads. I have seen several customers fiddle with the settings based on intuition, casual reading, and bad examples. And they get it wrong. Always. You end up giving CPU resources to the wrong servers and buying more hardware to overcome wrong settings. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/
Re: zVM CPU allocation
On Friday, 11/13/2009 at 05:13 EST, Rob van der Heij rvdh...@velocitysoftware.com wrote: You end up giving CPU resources to the wrong servers and buying more hardware to overcome wrong settings. HEY! You say that like it's a bad thing! If people want to buy more hardware to solve their problems, what right do we have to try to stop them? Throw hardware at it until it goes away has been on The Top 5 List of IT Strategies since the 1950s. They say that people are more expensive than machines these days (h. ahh. big deal. Have you seen how cheap the h/w is?) so why waste your valuable time? Poppiespoppiespoppiessleeep.. Though now that you outed them, They (and you know of whom I speak), will be forced to come up with Plan B for World Domination. (Get it? Domin-ation? wink). At least with Plan A you always knew what They were trying to do. Now, not so much. I didn't have always keep an eye on HRH, because I could predict his movements. Now I will have to give up afternoon naps and stand watch. Drats. -- Chuckie
Re: zVM CPU allocation
On Thu, 12 Nov 2009 13:01:52 -0600, Alan Ackerman alan.acker...@bankofamerica.com wrote: Where are you getting the 99% number? Which version of Linux are you running? Older versions of Linux were fooled by the CPU being taken away and reported high values of CPU utilization in 'top' and elsewhere. If all the numbers are from PerfKit, they don't make sense. You cannot have 3 users running at 99% and have the 3 IFLs running at 10-15% each. Are you sure you are looking at the same time interval? If you are looki ng at both Linxu and VM tools, they may not match up. Alan Ackerman Alan (dot) Ackerman (at) Bank of America (dot) com On Thu, 12 Nov 2009 13:17:26 -0500, Dean, David (I/S) david_d...@bcbst.com wrote: OK, this is for all you guys that understand the magic tunnel in which CPU processing (usage) flows from an IFL to a zLinux server. We have a z10 running zVM 5.4 in a dedicated LPAR with 3 dedicated IFL's. We have approximately 30 zLinux servers. Using IBM PerfKit, I list all of the individual USER / zLinux CPU usages, each of which generally run in the 1 to 5 % range. These servers do of course peak higher but on average they are pretty low. We then take a look at the 3 IFL's and see usage of maybe 5% on each. Now, let's say we have a USER / server or two or three go berserk and peak CPU at 99 % for an extended period of time. We then look at the IFL CPU usages and all three have climbed to maybe 10 to 15% each. How did the CPU get allocated? Is it always spread evenly across the IFL's, is that a setting? Why, if there were 3 USER / servers running a t 99%, was more CPU not allocated from the 3 IFL's? Why did the IFL's decide to allocate X amount and go no further. In the USER DIRECTORY I allocate storage but not CPU. David M. Dean Information Systems BlueCross BlueShield Tennnessee - Please see the following link for the BlueCross BlueShield of Tennessee E- mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm = ===
Re: zVM CPU allocation
Regarding the Share command... Apologize for the haphazard responses, I don't seem to be getting my zvm mail... -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of David Dean Sent: Friday, November 13, 2009 9:26 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: zVM CPU allocation Don't know, good point, I did not know that. - Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: zVM CPU allocation
Don't know, good point, I did not know that.
Re: zVM CPU allocation
As Rob and Alan have less blatantly stated, Linux CPU numbers are bogus in a virtual environment. The CPU reporting problem has been corrected in ESALPS (now zVPS) for those that want to use linux numbers for anything useful. If you don't have a mechanism to correlate the linux cpu numbers to reality, then one would not want to take the virtual reality too serious David Dean wrote: On Thu, 12 Nov 2009 13:01:52 -0600, Alan Ackerman alan.acker...@bankofamerica.com wrote: Where are you getting the 99% number? Which version of Linux are you running? Older versions of Linux were fooled by the CPU being taken away and reported high values of CPU utilization in 'top' and elsewhere. If all the numbers are from PerfKit, they don't make sense. You cannot have 3 users running at 99% and have the 3 IFLs running at 10-15% each. Are you sure you are looking at the same time interval? If you are looking at both Linxu and VM tools, they may not match up. Alan Ackerman Alan (dot) Ackerman (at) Bank of America (dot) com On Thu, 12 Nov 2009 13:17:26 -0500, Dean, David (I/S) david_d...@bcbst.com wrote: OK, this is for all you guys that understand the magic tunnel in which CPU processing (usage) flows from an IFL to a zLinux server. We have a z10 running zVM 5.4 in a dedicated LPAR with 3 dedicated IFL's. We have approximately 30 zLinux servers. Using IBM PerfKit, I list all of the individual USER / zLinux CPU usages, each of which generally run in the 1 to 5 % range. These servers do of course peak higher but on average they are pretty low. We then take a look at the 3 IFL's and see usage of maybe 5% on each. Now, let's say we have a USER / server or two or three go berserk and peak CPU at 99 % for an extended period of time. We then look at the IFL CPU usages and all three have climbed to maybe 10 to 15% each. How did the CPU get allocated? Is it always spread evenly across the IFL's, is that a setting? Why, if there were 3 USER / servers running at 99%, was more CPU not allocated from the 3 IFL's? Why did the IFL's decide to allocate X amount and go no further. In the USER DIRECTORY I allocate storage but not CPU. David M. Dean Information Systems BlueCross BlueShield Tennnessee - Please see the following link for the BlueCross BlueShield of Tennessee E- mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: zVM CPU allocation
As Rob and Alan have less blatantly stated, Linux CPU numbers are bogus in a virtual environment. However, with the addition of steal percentage (%st in top), the amount of CPU that is being stolen by the hipervisor, I believe many would agree that they are less bogus. Mike MacIsaac mike...@us.ibm.com (845) 433-7061
Re: zVM CPU allocation
Thank you Now that you mention it, Rob has told me that in the past at a SHARE. I sincerely wish I could use zVPS...8( I will continue to not take reality - virtual or otherwise - seriously. Life is too important to be taken seriously - Oscar Wilde. -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Barton Robinson Sent: Friday, November 13, 2009 11:37 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: zVM CPU allocation As Rob and Alan have less blatantly stated, Linux CPU numbers are bogus in a virtual environment. The CPU reporting problem has been corrected in ESALPS (now zVPS) for those that want to use linux numbers for anything useful. If you don't have a mechanism to correlate the linux cpu numbers to reality, then one would not want to take the virtual reality too serious David Dean wrote: On Thu, 12 Nov 2009 13:01:52 -0600, Alan Ackerman alan.acker...@bankofamerica.com wrote: Where are you getting the 99% number? Which version of Linux are you running? Older versions of Linux were fooled by the CPU being taken away and reported high values of CPU utilization in 'top' and elsewhere. If all the numbers are from PerfKit, they don't make sense. You cannot have 3 users running at 99% and have the 3 IFLs running at 10-15% each. Are you sure you are looking at the same time interval? If you are looking at both Linxu and VM tools, they may not match up. Alan Ackerman Alan (dot) Ackerman (at) Bank of America (dot) com On Thu, 12 Nov 2009 13:17:26 -0500, Dean, David (I/S) david_d...@bcbst.com wrote: OK, this is for all you guys that understand the magic tunnel in which CPU processing (usage) flows from an IFL to a zLinux server. We have a z10 running zVM 5.4 in a dedicated LPAR with 3 dedicated IFL's. We have approximately 30 zLinux servers. Using IBM PerfKit, I list all of the individual USER / zLinux CPU usages, each of which generally run in the 1 to 5 % range. These servers do of course peak higher but on average they are pretty low. We then take a look at the 3 IFL's and see usage of maybe 5% on each. Now, let's say we have a USER / server or two or three go berserk and peak CPU at 99 % for an extended period of time. We then look at the IFL CPU usages and all three have climbed to maybe 10 to 15% each. How did the CPU get allocated? Is it always spread evenly across the IFL's, is that a setting? Why, if there were 3 USER / servers running at 99%, was more CPU not allocated from the 3 IFL's? Why did the IFL's decide to allocate X amount and go no further. In the USER DIRECTORY I allocate storage but not CPU. David M. Dean Information Systems BlueCross BlueShield Tennnessee - Please see the following link for the BlueCross BlueShield of Tennessee E- mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm - Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: zVM CPU allocation
less bogus? Now I feel better. Just kidding, I sincerely appreciate all the help I get from you guys. This mail list has been priceless. From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Michael MacIsaac Sent: Friday, November 13, 2009 12:05 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: zVM CPU allocation As Rob and Alan have less blatantly stated, Linux CPU numbers are bogus in a virtual environment. However, with the addition of steal percentage (%st in top), the amount of CPU that is being stolen by the hipervisor, I believe many would agree that they are less bogus. Mike MacIsaac mike...@us.ibm.com (845) 433-7061 - Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: zVM CPU allocation
“less bogus”? More meaningful? Mike MacIsaac mike...@us.ibm.com (845) 433-7061
Re: zVM CPU allocation
Less bogus is like less pregnant. If it is still bogus, it is only useful for governmental purposes. :-) Regards, Richard Schuh From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Michael MacIsaac Sent: Friday, November 13, 2009 9:42 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: zVM CPU allocation less bogus? More meaningful? Mike MacIsaac mike...@us.ibm.com (845) 433-7061
Re: zVM CPU allocation
Mike, Rob probably left you out of the loop when he showed the new bogusness of the data with Sles10/11. It's more fun showing customers... Michael MacIsaac wrote: “less bogus”? More meaningful? Mike MacIsaac mike...@us.ibm.com (845) 433-7061
zVM CPU allocation
OK, this is for all you guys that understand the magic tunnel in which CPU processing (usage) flows from an IFL to a zLinux server. We have a z10 running zVM 5.4 in a dedicated LPAR with 3 dedicated IFL's. We have approximately 30 zLinux servers. Using IBM PerfKit, I list all of the individual USER / zLinux CPU usages, each of which generally run in the 1 to 5 % range. These servers do of course peak higher but on average they are pretty low. We then take a look at the 3 IFL's and see usage of maybe 5% on each. Now, let's say we have a USER / server or two or three go berserk and peak CPU at 99 % for an extended period of time. We then look at the IFL CPU usages and all three have climbed to maybe 10 to 15% each. How did the CPU get allocated? Is it always spread evenly across the IFL's, is that a setting? Why, if there were 3 USER / servers running at 99%, was more CPU not allocated from the 3 IFL's? Why did the IFL's decide to allocate X amount and go no further. In the USER DIRECTORY I allocate storage but not CPU. David M. Dean Information Systems BlueCross BlueShield Tennnessee - Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: zVM CPU allocation
Where are you getting the 99% number? Which version of Linux are you running? Older versions of Linux were fooled by the CPU being taken away and reported high values of CPU utilization in 'top' and elsewhere. If all the numbers are from PerfKit, they don't make sense. You cannot have 3 users running at 99% and have the 3 IFLs running at 10-15% each. Are you sure you are looking at the same time interval? If you are lookin g at both Linxu and VM tools, they may not match up. Alan Ackerman Alan (dot) Ackerman (at) Bank of America (dot) com On Thu, 12 Nov 2009 13:17:26 -0500, Dean, David (I/S) david_d...@bcbst.com wrote: OK, this is for all you guys that understand the magic tunnel in which CPU processing (usage) flows from an IFL to a zLinux server. We have a z10 running zVM 5.4 in a dedicated LPAR with 3 dedicated IFL's. We have approximately 30 zLinux servers. Using IBM PerfKit, I list all of the individual USER / zLinux CPU usages, each of which generally run in the 1 to 5 % range. These servers do of course peak higher but on average they are pretty low. We then take a look at the 3 IFL's and see usage of maybe 5% on each. Now, let's say we have a USER / server or two or three go berserk and peak CPU at 99 % for an extended period of time. We then look at the IFL CPU usages and all three have climbed to maybe 10 to 15% each. How did the CPU get allocated? Is it always spread evenly across the IFL's, is that a setting? Why, if there were 3 USER / servers running at 99%, was more CPU not allocated from the 3 IFL's? Why did the IFL's decide to allocate X amount and go no further. In the USER DIRECTORY I allocate storage but not CPU. David M. Dean Information Systems BlueCross BlueShield Tennnessee - Please see the following link for the BlueCross BlueShield of Tennessee E- mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: zVM CPU allocation
David, What kind of SHARE statements do you have in the Linux servers' directory entries. That's what determines how much of the available CPU resource the users get. Marty Martin Zimelis Principal maz/Consultancy _ From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Dean, David (I/S) Sent: Thursday, November 12, 2009 1:17 PM To: IBMVM@LISTSERV.UARK.EDU Subject: zVM CPU allocation OK, this is for all you guys that understand the magic tunnel in which CPU processing (usage) flows from an IFL to a zLinux server. We have a z10 running zVM 5.4 in a dedicated LPAR with 3 dedicated IFL's. We have approximately 30 zLinux servers. Using IBM PerfKit, I list all of the individual USER / zLinux CPU usages, each of which generally run in the 1 to 5 % range. These servers do of course peak higher but on average they are pretty low. We then take a look at the 3 IFL's and see usage of maybe 5% on each. Now, let's say we have a USER / server or two or three go berserk and peak CPU at 99 % for an extended period of time. We then look at the IFL CPU usages and all three have climbed to maybe 10 to 15% each. How did the CPU get allocated? Is it always spread evenly across the IFL's, is that a setting? Why, if there were 3 USER / servers running at 99%, was more CPU not allocated from the 3 IFL's? Why did the IFL's decide to allocate X amount and go no further. In the USER DIRECTORY I allocate storage but not CPU. David M. Dean Information Systems BlueCross BlueShield Tennnessee - Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
