Re: CPU usage formula
VM does have a single performance/capacity/accounting data stream, it is provided via Velocity Software's Linux Performance Suite. It includes Linux data to the process, application and user level, as well as all the z/VM data. But you knew that? Thomas Kern wrote: It would be nice if linux could be convinced to deliver consumption data to VM on a per account/user by interval. I don't think it has to be into the 'Accounting' data stream but maybe to the 'Monitor' data stream. I still think VM need a single performance/capacity/accounting data stream (think SMF/RMF). Yeah, I know that can rub some VM people the wrong way but I have done enough accounting/capacity reports from SMF data that I like the idea of a single data stream for all of this data. Back to the linux implementation, I don't think it has to be something invasive that requires acceptance from all the x86 linux authorities, but could be an add-on. Does the SYSSTAT package require any kernel level interference? But basically take that idea, modify to write the data to the Monitor data stream, and write summary data in linux as well as the Monitor data stream. This lets you add data collection on a per linux system so if you have a mix of servers, some dedicated (only one customer to bill for ALL of its resources) and some shared (multiple customers), you don't need the extra overhead of data collection/reporting for the dedicated servers. /Tom Kern Scott Rohling wrote: Has anyone played around with using the VM accounting data, along with Linux usage data (sar data for example - capturing process usage) to come up with a way to assign usage as the VM level (i.e. host CPU hours) to individual processes? I'm thinking of a grid environment, where you would want to assign usage to accounts -- not at the server level - but at the process level. Given a set CPU hour rate at the VM level - you could (hopefully) accurately determine the real cost of individual Linux processes. Maybe cut C0 z/VM accounting records daily from Linux (using cpint) to feed the data to the VM accounting file. I'm not sure it's even possible.. but perhaps through some statistical formula (overall cost of CPU for VM guest is x - process y used 10% of it) you can get close? Any thoughts.. ? ScottR -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 begin:vcard fn:Barton Robinson n:Robinson;Barton adr;dom:;;PO 391209;Mountain View;CA;94039-1209 email;internet:bar...@velocitysoftware.com title:Sr. Architect tel;work:650-964-8867 x-mozilla-html:FALSE url:VelocitySoftware.com version:2.1 end:vcard
Re: CPU usage formula
Yes, done, ESALPS provides you all the data you want. Scott Rohling wrote: Has anyone played around with using the VM accounting data, along with Linux usage data (sar data for example - capturing process usage) to come up with a way to assign usage as the VM level (i.e. host CPU hours) to individual processes? I'm thinking of a grid environment, where you would want to assign usage to accounts -- not at the server level - but at the process level. Given a set CPU hour rate at the VM level - you could (hopefully) accurately determine the real cost of individual Linux processes. Maybe cut C0 z/VM accounting records daily from Linux (using cpint) to feed the data to the VM accounting file. I'm not sure it's even possible.. but perhaps through some statistical formula (overall cost of CPU for VM guest is x - process y used 10% of it) you can get close? Any thoughts.. ? ScottR -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 begin:vcard fn:Barton Robinson n:Robinson;Barton adr;dom:;;PO 391209;Mountain View;CA;94039-1209 email;internet:bar...@velocitysoftware.com title:Sr. Architect tel;work:650-964-8867 x-mozilla-html:FALSE url:VelocitySoftware.com version:2.1 end:vcard
Re: CPU usage formula
I knew that, I just cannot pay for it. /Tom Kern Barton Robinson wrote: VM does have a single performance/capacity/accounting data stream, it is provided via Velocity Software's Linux Performance Suite. It includes Linux data to the process, application and user level, as well as all the z/VM data. But you knew that? Thomas Kern wrote: It would be nice if linux could be convinced to deliver consumption data to VM on a per account/user by interval. I don't think it has to be into the 'Accounting' data stream but maybe to the 'Monitor' data stream. I still think VM need a single performance/capacity/accounting data stream (think SMF/RMF). Yeah, I know that can rub some VM people the wrong way but I have done enough accounting/capacity reports from SMF data that I like the idea of a single data stream for all of this data. Back to the linux implementation, I don't think it has to be something invasive that requires acceptance from all the x86 linux authorities, but could be an add-on. Does the SYSSTAT package require any kernel level interference? But basically take that idea, modify to write the data to the Monitor data stream, and write summary data in linux as well as the Monitor data stream. This lets you add data collection on a per linux system so if you have a mix of servers, some dedicated (only one customer to bill for ALL of its resources) and some shared (multiple customers), you don't need the extra overhead of data collection/reporting for the dedicated servers. /Tom Kern Scott Rohling wrote: Has anyone played around with using the VM accounting data, along with Linux usage data (sar data for example - capturing process usage) to come up with a way to assign usage as the VM level (i.e. host CPU hours) to individual processes? I'm thinking of a grid environment, where you would want to assign usage to accounts -- not at the server level - but at the process level. Given a set CPU hour rate at the VM level - you could (hopefully) accurately determine the real cost of individual Linux processes. Maybe cut C0 z/VM accounting records daily from Linux (using cpint) to feed the data to the VM accounting file. I'm not sure it's even possible.. but perhaps through some statistical formula (overall cost of CPU for VM guest is x - process y used 10% of it) you can get close? Any thoughts.. ? ScottR -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CPU usage formula
On Sun, Aug 2, 2009 at 1:26 AM, Scott Rohlingscott.rohl...@gmail.com wrote: Has anyone played around with using the VM accounting data, along with Linux usage data (sar data for example - capturing process usage) to come up with a way to assign usage as the VM level (i.e. host CPU hours) to individual processes? That does not work in general with sar data because in most situations the average life time of a process is much shorter. Also, for most shops memory usage today is the main issue, rather than CPU usage. And that's not in accounting data either. You're probably aware that we do this with monitor data. ESALPS takes both process data from Linux and z/VM monitor data to provide data that is good enough to be used for process level accounting on process, application or user level. We collect Linux data via SNMP but have our own agent to get a high (near 100% capture ratio). With the normal Linux SNMP data you often don't get more than 10% of your usage reported (that means that you may be a factor 10 off in your charges) I'm thinking of a grid environment, where you would want to assign usage to accounts -- not at the server level - but at the process level. Given a set CPU hour rate at the VM level - you could (hopefully) accurately determine the real cost of individual Linux processes. Maybe cut C0 z/VM accounting records daily from Linux (using cpint) to feed the data to the VM accounting file. There is an interface to Linux to put per-process data into the accounting data, but this is incomplete and not useful for what you want. Even when you charge people by virtual machine usage, you still want to have complete usage data on process level. Most customers don't accept a bill with just the totals on it, they want you to break it down to a level where they recognize the usage.That way they can tell that the charges were higher on tuesday because db2 was looping, or things like that. If your detail only covers 10% or so, it does not help to convince the customer that your data is correct. I'm not sure it's even possible.. but perhaps through some statistical formula (overall cost of CPU for VM guest is x - process y used 10% of it) you can get close? Have a look at this presentation: http://www.rvdheij.nl/Presentations/zLX44.pdf I will present (part of it, unfortunately) at SHARE in Denver (S9217, Mon 11:00) Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CPU usage formula
On Sun, Aug 2, 2009 at 3:04 AM, Thomas Kerntlk_sysp...@yahoo.com wrote: Back to the linux implementation, I don't think it has to be something invasive that requires acceptance from all the x86 linux authorities, One of the risks of single operating system for multiple platforms is that you end up restricting yourself to only supporting the function that all platforms have in common. Or in the case of Linux, because of the pure numbers: when it is also available and needed on x86. When many developers have their fingers in the code, it is very hard to justify complicating the code just for a small percentage of the installations. That's why CMM-2 ended up the way it did. And it turns out that the requirements in those areas are also different. If you're used to dedicated hardware and consider top your main/only source of information, then you probably don't worry about capture ratio of process usage accounting. Why else do you think we find the bugs in Linux process accounting. The same thing also happens on application level. When the vendor decides to use a single source for his application on different operating systems, the easiest approach is to dumb down the OS interface to the minimum supported anywhere. So instead of proper inter-process communication, we find the application polling. By the time the application is used on shared hardware and it causes performance issues, it is too late to change the design. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
CPU usage formula
Has anyone played around with using the VM accounting data, along with Linux usage data (sar data for example - capturing process usage) to come up with a way to assign usage as the VM level (i.e. host CPU hours) to individual processes? I'm thinking of a grid environment, where you would want to assign usage to accounts -- not at the server level - but at the process level. Given a set CPU hour rate at the VM level - you could (hopefully) accurately determine the real cost of individual Linux processes. Maybe cut C0 z/VM accounting records daily from Linux (using cpint) to feed the data to the VM accounting file. I'm not sure it's even possible.. but perhaps through some statistical formula (overall cost of CPU for VM guest is x - process y used 10% of it) you can get close? Any thoughts.. ? ScottR -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CPU usage formula
It would be nice if linux could be convinced to deliver consumption data to VM on a per account/user by interval. I don't think it has to be into the 'Accounting' data stream but maybe to the 'Monitor' data stream. I still think VM need a single performance/capacity/accounting data stream (think SMF/RMF). Yeah, I know that can rub some VM people the wrong way but I have done enough accounting/capacity reports from SMF data that I like the idea of a single data stream for all of this data. Back to the linux implementation, I don't think it has to be something invasive that requires acceptance from all the x86 linux authorities, but could be an add-on. Does the SYSSTAT package require any kernel level interference? But basically take that idea, modify to write the data to the Monitor data stream, and write summary data in linux as well as the Monitor data stream. This lets you add data collection on a per linux system so if you have a mix of servers, some dedicated (only one customer to bill for ALL of its resources) and some shared (multiple customers), you don't need the extra overhead of data collection/reporting for the dedicated servers. /Tom Kern Scott Rohling wrote: Has anyone played around with using the VM accounting data, along with Linux usage data (sar data for example - capturing process usage) to come up with a way to assign usage as the VM level (i.e. host CPU hours) to individual processes? I'm thinking of a grid environment, where you would want to assign usage to accounts -- not at the server level - but at the process level. Given a set CPU hour rate at the VM level - you could (hopefully) accurately determine the real cost of individual Linux processes. Maybe cut C0 z/VM accounting records daily from Linux (using cpint) to feed the data to the VM accounting file. I'm not sure it's even possible.. but perhaps through some statistical formula (overall cost of CPU for VM guest is x - process y used 10% of it) you can get close? Any thoughts.. ? ScottR -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
