Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)
On Thu, May 3, 2012 at 11:59 AM, Loic Dachary l...@enovance.com wrote: On 05/03/2012 09:58 AM, Robert Collins wrote: On Wed, May 2, 2012 at 10:28 AM, Loic Dachary l...@enovance.com wrote: As I wrote in a previous mail, once we manage to provide an implementation that proves useful, we will be in a position to approach the core OpenStack components. Integrating the metering agents as part of the core component, much in the same way it's currently done in nova. That will reduce the overall complexity of deploying OpenStack with metering (which must not be mandatory). However, there is very little chance that all components developed around OpenStack are convinced and there will always be a need for a metering that is external to the component. Therefore, even if metering eventually manages to become a first class concern for the core OpenStack components, the proposed architecture of the metering project ( per node agents when necessary and a collector harvesting them into a storage ) will keep being used for other components. Do you think I'm wrong ? We're at a very early stage and now is the time to question everything :-) I would avoid node agents as a primary interface: they can always be written to workaround components that don't provide metering themselves - like the nagios plugins example given by Andrew. node agents are more complex than implementing metering in each component because they require a handoff, parsing of local logs (or whatever relevant store the component uses), and they are another process to keep running; they probably devolve to polling, and polling is a good way to use a lot of resources up :(. The key thing I see is having a clear contract for how metering is done, so that anyone writing a new component can add metering easily. This is easier for the component author than having to write their component *and* write a metering node agent for that component. E.g. - define the signals, extension points, and python API for implementing metering in a component. - implement a no-op implementation of that metering API which discards all data. - implement a AMQP based implementation which shoves all the data out to some queue somewhere. Do you envision that writing to AMQP directly, or using the existing notification API (which seems to be a wrapper over the RPC layer, which is in turn a wrapper for AMQP)? - submit patches to existing components to add metering - and if a component owner rejects metering altogether, you can still implement a node agent to gather the data and push it out via the metering API - that will always be an option. Then we will have a situation where: - new components can meter easily - folk with different delivery constraints can change the network layer easily (drop in a different implementation of the Python API) [e.g. to log directly to a NoSQL store] I could not agree more. This way we have the best of both worlds : integration in components when possible and standalone agents when it can't be done for whatever reason. Cheers -- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ✉ l...@enovance.com ☎ +33 1 49 70 99 82 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)
On Fri, May 4, 2012 at 4:51 PM, Christopher MacGown ch...@pistoncloud.comwrote: Do you envision that writing to AMQP directly, or using the existing notification API (which seems to be a wrapper over the RPC layer, which is in turn a wrapper for AMQP)? The RPC layer is not merely a wrapper for AMQP, in addition to Kombu and QPid it also supports brokered 0mq. Sorry, I didn't mean to imply that it was only for AMQP. I just meant that instead of talking directly to AMQP, it would probably be better to use one of those wrapper layers that handles the other messaging systems so that deployers have as much flexibility as possible. -- Christopher MacGown, CTO Piston Cloud Computing, Inc. w: (650) 24-CLOUD m: (415) 300-0944 http://www.pistoncloud.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)
On 05/03/2012 05:25 AM, Andrew Clay Shafer wrote: Integrating the metering agents as part of the core component, much in the same way it's currently done in nova. What specifically is done? If metering is not integrated in the beginning it will likely never be. Hi, I'm refering to http://wiki.openstack.org/SystemUsageData . In another thread on this list ( Subject: Understanding SystemUsageData and its evolution ) I asked the author Dragon Monsyne about its evolution and how to get per-IP meters which does not seem to be possible at the moment. Do you know who would be most interested in discussing metering among the swift or quantum developers ? Cheers -- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ? l...@enovance.com ? +33 1 49 70 99 82 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)
On 05/03/2012 09:58 AM, Robert Collins wrote: On Wed, May 2, 2012 at 10:28 AM, Loic Dachary l...@enovance.com wrote: As I wrote in a previous mail, once we manage to provide an implementation that proves useful, we will be in a position to approach the core OpenStack components. Integrating the metering agents as part of the core component, much in the same way it's currently done in nova. That will reduce the overall complexity of deploying OpenStack with metering (which must not be mandatory). However, there is very little chance that all components developed around OpenStack are convinced and there will always be a need for a metering that is external to the component. Therefore, even if metering eventually manages to become a first class concern for the core OpenStack components, the proposed architecture of the metering project ( per node agents when necessary and a collector harvesting them into a storage ) will keep being used for other components. Do you think I'm wrong ? We're at a very early stage and now is the time to question everything :-) I would avoid node agents as a primary interface: they can always be written to workaround components that don't provide metering themselves - like the nagios plugins example given by Andrew. node agents are more complex than implementing metering in each component because they require a handoff, parsing of local logs (or whatever relevant store the component uses), and they are another process to keep running; they probably devolve to polling, and polling is a good way to use a lot of resources up :(. The key thing I see is having a clear contract for how metering is done, so that anyone writing a new component can add metering easily. This is easier for the component author than having to write their component *and* write a metering node agent for that component. E.g. - define the signals, extension points, and python API for implementing metering in a component. - implement a no-op implementation of that metering API which discards all data. - implement a AMQP based implementation which shoves all the data out to some queue somewhere. - submit patches to existing components to add metering - and if a component owner rejects metering altogether, you can still implement a node agent to gather the data and push it out via the metering API - that will always be an option. Then we will have a situation where: - new components can meter easily - folk with different delivery constraints can change the network layer easily (drop in a different implementation of the Python API) [e.g. to log directly to a NoSQL store] I could not agree more. This way we have the best of both worlds : integration in components when possible and standalone agents when it can't be done for whatever reason. Cheers -- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ✉ l...@enovance.com ☎ +33 1 49 70 99 82 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)
It would be better if all OpenStack core components agreed on unified interfaces / messages for metering that would be easy to harvest without installing agents on nodes. This is also true for many services outside of the OpenStack eco-system. However, much in the same way munin and nagios plugins are developped outside of the project for which they provide graphing and monitoring (for instance we recently published swift munin plugins in the repository where ops usually look for them : https://github.com/munin-monitoring/contrib/tree/master/plugins/swift and there is no glance plugin or up-to-date nova plugins yet ), metering agents will be developed separately, most of the time. This is Conway's Law manifest. The people developing nagios plugins are often in an operators role running software they didn't necessarily write (neither what they have deployed as a service nor the monitoring framework). People make it work, but it's rarely the best solution. Regardless of the monitoring solution, having application level metrics exposed by the application and implemented by people who understand the application has always led to a qualitatively better solution in my experience. As I wrote in a previous mail, once we manage to provide an implementation that proves useful, we will be in a position to approach the core OpenStack components. I don't follow this statement. Integrating the metering agents as part of the core component, much in the same way it's currently done in nova. What specifically is done? If metering is not integrated in the beginning it will likely never be. That will reduce the overall complexity of deploying OpenStack with metering (which must not be mandatory). I'm confused what you are using 'that' to refer to in this sentence. The integrated solution or the standalone service? We have a framework that's operation is entirely dependent on generating most of the events that need to be metered. What could be less complex than exposing that in a sensible way? Sadly, I think too many believe deploying OpenStack without monitoring must not be mandatory. I personally hope we can get to the point where fault tolerance is a first class concern for OpenStack services, and in my opinion getting there is somewhat dependent on solving this same problem in a sensible way. However, there is very little chance that all components developed around OpenStack are convinced and there will always be a need for a metering that is external to the component. If that is true, then it is a sad state of affairs. I would hope people have a more holistic understanding of what OpenStack could and should become. Therefore, even if metering eventually manages to become a first class concern for the core OpenStack components, the proposed architecture of the metering project ( per node agents when necessary and a collector harvesting them into a storage ) will keep being used for other components. I agree there is potentially interesting engineering work to be done on the transport and collection of metrics. I have an aversion to thinking the starting point for that should be defining a schema and deciding on a db. Do you think I'm wrong ? We're at a very early stage and now is the time to question everything :-) I don't think your motives are wrong. I could also be 'wrong'. I think the answer to that depends on what we think OpenStack should be in the end and what is good enough to get there. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)
On 05/01/2012 09:57 PM, Andrew Clay Shafer wrote: I'm glad to see people championing the effort to implement metering. Is there someway to refocus the enthusiasm for solving the metering problem into engineering a general solution in OpenStack? I'm just going to apologize in advance, but I don't think this project is headed in the right direction. I believe metering should be a first class concern of OpenStack and the way this project is starting is almost exactly backwards from what I think a solution to metering should look like. The last thing I want to see right now is a blessed OpenStack metering project adding more agents, coupled to a particular db and making policy decisions about what is quantifiable. I think there are really three problems that need to be solved to do metering, what data to get, getting the data and doing things with the data. From my perspective, a lot if not all of the data events should be coming out of the services themselves. There is already a service that should know when an instance gets started by what tenant. A cross cutting system for publishing those events and a service definition for collecting them seems like a reasonable place to start. To me that should look awful lot like a message queue or centralized logging. Once the first two problems are solved well, if you are so inclined to collect the data into a relational model, the schema will be obvious. If the first two problems are solved well, then I could be persuaded that a service that provides some of the aggregation functionality is a great idea and a reference implementation on a relational database isn't the worst thing in the world. Without a general solution for the first two problems, I believe the primary focus on a schema and db is premature and sub-optimal. I also believe the current approach likely results in a project that is generally unusable. Does anyone else share my perspective? It would be better if all OpenStack core components agreed on unified interfaces / messages for metering that would be easy to harvest without installing agents on nodes. This is also true for many services outside of the OpenStack eco-system. However, much in the same way munin and nagios plugins are developped outside of the project for which they provide graphing and monitoring (for instance we recently published swift munin plugins in the repository where ops usually look for them : https://github.com/munin-monitoring/contrib/tree/master/plugins/swift and there is no glance plugin or up-to-date nova plugins yet ), metering agents will be developed separately, most of the time. As I wrote in a previous mail, once we manage to provide an implementation that proves useful, we will be in a position to approach the core OpenStack components. Integrating the metering agents as part of the core component, much in the same way it's currently done in nova. That will reduce the overall complexity of deploying OpenStack with metering (which must not be mandatory). However, there is very little chance that all components developed around OpenStack are convinced and there will always be a need for a metering that is external to the component. Therefore, even if metering eventually manages to become a first class concern for the core OpenStack components, the proposed architecture of the metering project ( per node agents when necessary and a collector harvesting them into a storage ) will keep being used for other components. Do you think I'm wrong ? We're at a very early stage and now is the time to question everything :-) ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
