Re: [Openstack] Monitoring / Billing Architecture proposed

On Apr 24, 2012, at 2:46 PM, Brian Schott wrote: Yeah, but does that mean the instance is alive and billable :-)? I guess that counts! I thought they were only in response to external API/admin requests. - Brian Schott, CTO Nimbis Services, Inc.

Re: [Openstack] Monitoring / Billing Architecture proposed

On Apr 24, 2012, at 2:31 PM, Brian Schott wrote: I take it that the instance manager doesn't generate any kind of heartbeat, so whatever monitoring/archiving service we do should internally poll the status over MQ? Actually, it generates periodic 'heartbeat' events ('exists' events) for each

Re: [Openstack] Monitoring / Billing Architecture proposed

On Apr 24, 2012, at 11:00 AM, Loic Dachary wrote: On 04/24/2012 04:45 PM, Monsyne Dragon wrote: On Apr 24, 2012, at 9:03 AM, Loic Dachary wrote: On 04/24/2012 03:06 PM, Monsyne Dragon wrote: Yes, we emit bandwidth (bytes in/out) on a per VIF basis from each instance The event has the somewha

Re: [Openstack] Monitoring / Billing Architecture proposed

I think so. If it is in response or not, it's a truly heartbeat On Tue, Apr 24, 2012 at 9:46 PM, Brian Schott < brian.sch...@nimbisservices.com> wrote: > Yeah, but does that mean the instance is alive and billable :-)? I guess > that counts! I thought they were only in response to external API

Re: [Openstack] Monitoring / Billing Architecture proposed

Yeah, but does that mean the instance is alive and billable :-)? I guess that counts! I thought they were only in response to external API/admin requests. - Brian Schott, CTO Nimbis Services, Inc. brian.sch...@nimbisservices.com ph: 443-274-6064

Re: [Openstack] Monitoring / Billing Architecture proposed

This kind of messages are coming from nova exchange aprox. each 60 secs Can be this considered as a heartbeat for you? [x] Received '{"_context_roles": ["admin"], "_msg_id": " a2d13735baad4613b89c6132e0fa8302", "_context_read_deleted": "no", "_context_request_id": "req-d7ffbe78-7a9c-4d20-9ac5-3e

Re: [Openstack] Monitoring / Billing Architecture proposed

I take it that the instance manager doesn't generate any kind of heartbeat, so whatever monitoring/archiving service we do should internally poll the status over MQ? - Brian Schott, CTO Nimbis Services, Inc. brian.sch...@nimbisservices.com ph: 443

Re: [Openstack] Monitoring / Billing Architecture proposed

Hi Monsyne, I have set the notification_driver param, but no notification.* queues created. I'm using devstack stable/essex. stack@ubuntu:/$ sudo rabbitmqctl list_queues Listing queues ... volume_fanout_e0923a8bbb9f45dc9e63d382796a4c8f 0 cert.ubuntu 0 consoleauth.ubuntu 0 compute 0 comp

Re: [Openstack] Monitoring / Billing Architecture proposed

Probably an extra audit system is required. I'm searching for solutions in the IT market. Regards On Tue, Apr 24, 2012 at 6:00 PM, Loic Dachary wrote: > ** > On 04/24/2012 04:45 PM, Monsyne Dragon wrote: > > > On Apr 24, 2012, at 9:03 AM, Loic Dachary wrote: > > On 04/24/2012 03:06 PM, Monsyn

Re: [Openstack] Monitoring / Billing Architecture proposed

On 04/24/2012 04:45 PM, Monsyne Dragon wrote: > > On Apr 24, 2012, at 9:03 AM, Loic Dachary wrote: > >> On 04/24/2012 03:06 PM, Monsyne Dragon wrote: >>> Yes, we emit bandwidth (bytes in/out) on a per VIF basis from each >>> instance The event has the somewhat generic name of >>> 'compute.instan

Re: [Openstack] Monitoring / Billing Architecture proposed

On Apr 24, 2012, at 9:03 AM, Loic Dachary wrote: On 04/24/2012 03:06 PM, Monsyne Dragon wrote: Yes, we emit bandwidth (bytes in/out) on a per VIF basis from each instance The event has the somewhat generic name of 'compute.instance.exists' and is emitted on an periodic basis, currently by a

Re: [Openstack] Monitoring / Billing Architecture proposed

On 04/24/2012 03:06 PM, Monsyne Dragon wrote: > Yes, we emit bandwidth (bytes in/out) on a per VIF basis from each instance > The event has the somewhat generic name of 'compute.instance.exists' and is > emitted on an periodic basis, currently by a cronjob. > Currently, we only populate bandwi

Re: [Openstack] Monitoring / Billing Architecture proposed

Yes, we emit bandwidth (bytes in/out) on a per VIF basis from each instance The event has the somewhat generic name of 'compute.instance.exists' and is emitted on an periodic basis, currently by a cronjob. Currently, we only populate bandwidth data from XenServer, but if the hook is implement

Re: [Openstack] Monitoring / Billing Architecture proposed

I think we have support for this currently in some fashion, Dragon? -S On 04/24/2012 12:55 AM, Loic Dachary wrote: > Metering needs to account for the "volume of data sent to external network > destinations " ( i.e. n4 in http://wiki.openstack.org/EfficientMetering ) or > the disk I/O etc. Th

Re: [Openstack] Monitoring / Billing Architecture proposed

On 04/23/2012 10:45 PM, Doug Hellmann wrote: > > > On Mon, Apr 23, 2012 at 4:14 PM, Brian Schott > > wrote: > > Doug, > > Do we mirror the table structure of nova, etc. and add > created/modified columns? > > > Or do we flatten into an

Re: [Openstack] Monitoring / Billing Architecture proposed

On 04/23/2012 10:09 PM, Sandy Walsh wrote: > Flavor information is copied to the Instance table on creation so the > Flavors can change and still be tracked in the Instance. It may just > need to be sent in the notification payload. > > The current events in the system are documented here: > http:/

Re: [Openstack] Monitoring / Billing Architecture proposed

This looks like just the standard RPC traffic. You need to turn notifications on (set: notification_driver=nova.notifier.rabbit_notifier in nova's config file) and listen on the notification.* queues On Apr 23, 2012, at 2:26 PM, Luis Gervaso wrote: Joshua, I have performed a create instance o

Re: [Openstack] Monitoring / Billing Architecture proposed

StackTach is a Django-based web interface for capturing, displaying and navigating OpenStack notifications https://github.com/rackspace/stacktach -S On 04/23/2012 04:26 PM, Luis Gervaso wrote: > Joshua, > > I have performed a create instance operation and here is an example data > obtained fro

Re: [Openstack] Monitoring / Billing Architecture proposed

Great, Now we just need to "officialize" those as a first step to making a real interchange format that can be versioned, documented... and all the other goodies u would normally expect. On 4/23/12 12:26 PM, "Luis Gervaso" wrote: Joshua, I have performed a create instance operation and here

Re: [Openstack] Monitoring / Billing Architecture proposed

On Mon, Apr 23, 2012 at 4:14 PM, Brian Schott < brian.sch...@nimbisservices.com> wrote: > Doug, > > Do we mirror the table structure of nova, etc. and add created/modified > columns? > > Or do we flatten into an instance event record with everything? > I lean towards flattening the data as it is

Re: [Openstack] Monitoring / Billing Architecture proposed

Doug, Do we mirror the table structure of nova, etc. and add created/modified columns? Or do we flatten into an instance event record with everything? Bran - Brian Schott, CTO Nimbis Services, Inc. brian.sch...@nimbisservices.com ph: 443-274-6

Re: [Openstack] Monitoring / Billing Architecture proposed

Flavor information is copied to the Instance table on creation so the Flavors can change and still be tracked in the Instance. It may just need to be sent in the notification payload. The current events in the system are documented here: http://wiki.openstack.org/SystemUsageData -Sandy On 04/23

Re: [Openstack] Monitoring / Billing Architecture proposed

The following links to work from OGF in the usage accounting and tracking area might be useful. First of all, we have the UsageRecord format (UR), which has been used in a variety of distributed computing environments for tracking usage. It defines an XML-based format for exchange of usage r

Re: [Openstack] Monitoring / Billing Architecture proposed

Joshua, I have performed a create instance operation and here is an example data obtained from stable/essex rabbitmq nova catch all exchange. [*] Waiting for messages. To exit press CTRL+C [x] Received '{"_context_roles": ["admin"], "_msg_id": "a2d13735baad4613b89c6132e0fa8302", "_context_read_

Re: [Openstack] Monitoring / Billing Architecture proposed

On Mon, Apr 23, 2012 at 1:50 PM, Brian Schott < brian.sch...@nimbisservices.com> wrote: > So, we could build on this. No reason to reinvent, but we might want to > expand the number of events. I'm concerned about things like what happens > when flavors change over time. Maybe the answer is, alwa

Re: [Openstack] Monitoring / Billing Architecture proposed

I like http://wiki.openstack.org/SystemUsageData I just wonder if there should be offical formats for the data sent across the wire. Or at least high level formats that define the data, maybe in a simple python "map" style layout. With examples would be awesome. On 4/23/12 10:40 AM, "Luis Ger

Re: [Openstack] Monitoring / Billing Architecture proposed

Brian, Dough isn't dependent on Horizon. Dough has a client that can be inserted anywhere to notify the start/end of using a resource. We inserted the client in horizon just to try out the many ways our billing system could be integrated with our existing deployment. We are planning to make a N

Re: [Openstack] Monitoring / Billing Architecture proposed

On 04/23/2012 07:06 PM, Luis Gervaso wrote: > Nick, I want contribute in the discussion group. Luis, I just sent your an invite to the doodle to pick the best time for this irc meeting. Once the date will have been finalized (I'll close the poll tomorrow EOD), I'll announce the date, time and pla

Re: [Openstack] Monitoring / Billing Architecture proposed

So, we could build on this. No reason to reinvent, but we might want to expand the number of events. I'm concerned about things like what happens when flavors change over time. Maybe the answer is, always append to the flavor/instance-type table. The code I remember and the admin interface th

Re: [Openstack] Monitoring / Billing Architecture proposed

.@woorea.es>> Cc: "mailto:openstack@lists.launchpad.net>>" mailto:openstack@lists.launchpad.net>> Subject: Re: [Openstack] Monitoring / Billing Architecture proposed This already exists in trunk. The Notification system was designed specifically to feed billing and m

Re: [Openstack] Monitoring / Billing Architecture proposed

I have been looking at : http://wiki.openstack.org/SystemUsageData On Mon, Apr 23, 2012 at 7:35 PM, Brian Schott < brian.sch...@nimbisservices.com> wrote: > Is there a document somewhere on what events the services emit? > > - > Brian Schott, CTO >

Re: [Openstack] Monitoring / Billing Architecture proposed

Is there a document somewhere on what events the services emit? - Brian Schott, CTO Nimbis Services, Inc. brian.sch...@nimbisservices.com ph: 443-274-6064 fx: 443-274-6060 On Apr 23, 2012, at 12:39 PM, Monsyne Dragon wrote: > This already exis

Re: [Openstack] Monitoring / Billing Architecture proposed

On Sun, Apr 22, 2012 at 5:29 PM, Brian Schott < brian.sch...@nimbisservices.com> wrote: > Dough is a proposed billing service. There was a session at Folsom design > summit. This is a practical project for an OpenStack provider with test > code on github. > http://summit.openstack.org/sessions/v

Re: [Openstack] Monitoring / Billing Architecture proposed

Nick, I want contribute in the discussion group. I have seen yagi implementation, and i like it. I like the protocol. But I think that it's a protocol that fits very well when no sensible data is involved (content feeds). When money, resources, companies are involved i think we need to use a more

Re: [Openstack] Monitoring / Billing Architecture proposed

This already exists in trunk. The Notification system was designed specifically to feed billing and monitoring systems. Basically, we don't want Nova/Glance/etc to be in the business of trying to determine billing logic, since it is different for pretty much everyone, so we just emit notifica

Re: [Openstack] Monitoring / Billing Architecture proposed

On Sun, 2012-04-22 at 20:50 +0200, Luis Gervaso wrote: > I want to share the architecture i am developing in order to perform > the monitorig / billing OpenStack support: > > > 1. AMQP Client which listen to RabbitMQ / QPid (this should be > interchangeable) (Own Stuff or ServiceMix / Camel) > >

Re: [Openstack] Monitoring / Billing Architecture proposed

On 04/23/2012 04:39 PM, Alexey Ababilov wrote: > Hi! > > We have developed Nova Billing v2. Its documentation is currently > available at http://aababilov.github.com/nova-billing-doc.github.com/. > The documentation includes a glossary and architecture and API descriptions. > > Nova Billing v2 is

Re: [Openstack] Monitoring / Billing Architecture proposed

Hi! We have developed Nova Billing v2. Its documentation is currently available at http://aababilov.github.com/nova-billing-doc.github.com/. The documentation includes a glossary and architecture and API descriptions. Nova Billing v2 is a totally new solution. Its API and architecture were rewrit

Re: [Openstack] Monitoring / Billing Architecture proposed

On 04/23/2012 11:17 AM, Ahn, Jaesuk wrote: > Hi, > > Although metering and billing always comes together for the deployment, > for the sake of clarity, I also think metering should be a separate > project from the billing, especially in openstack. > (As you mentioned it, billing is complex and

Re: [Openstack] Monitoring / Billing Architecture proposed

Hi, Although metering and billing always comes together for the deployment, for the sake of clarity, I also think metering should be a separate project from the billing, especially in openstack. (As you mentioned it, billing is complex and has too many different requirements per provider) A

Re: [Openstack] Monitoring / Billing Architecture proposed

Hello Luis, I presented a blueprint last week [1] which proposes to clearly differentiate metering from the overall billing process. It is my understanding that billing is too complex a beast to be solved for each requirement in a satisfactory way and have therefore proposed that we should first

Re: [Openstack] Monitoring / Billing Architecture proposed

rote: >> >>> I see this is an accounting system, a billing system needs things like >>> promotional codes, vat, invoices ... >>> >>> I'm proposing the way the events should be orchestated >>> >>> Please, correct me, if i'm wron

Re: [Openstack] Monitoring / Billing Architecture proposed

t;>> - >>>>> Brian Schott, CTO >>>>> Nimbis Services, Inc. >>>>> brian.sch...@nimbisservices.com >>>>> ph: 443-274-6064 fx: 443-274-6060 >>>>> >>>>> >>>&g

Re: [Openstack] Monitoring / Billing Architecture proposed

s an accounting system, a billing system needs things like >>>> promotional codes, vat, invoices ... >>>> >>>> I'm proposing the way the events should be orchestated >>>> >>>> Please, correct me, if i'm wrong >>&g

Re: [Openstack] Monitoring / Billing Architecture proposed

should be orchestated >>> >>> Please, correct me, if i'm wrong >>> >>> Luis >>> >>> On Sun, Apr 22, 2012 at 11:16 PM, Atul Jha wrote: >>> >>>> Hi, >>>> Has anyone checked this >>>> http:

Re: [Openstack] Monitoring / Billing Architecture proposed

griddynamics.com/openstack/docs/nova-billing/quickstart.html >> >> From: openstack-bounces+atul.jha=csscorp@lists.launchpad.net >> [openstack-bounces+atul.jha=csscorp@lists.launchpad.net] on behalf of >> Endre Karlson [endre.karl...@gm

Re: [Openstack] Monitoring / Billing Architecture proposed

kstart.html >>> >>> From: >>> openstack-bounces+atul.jha=csscorp@lists.launchpad.net[openstack-bounces+atul.jha= >>> csscorp@lists.launchpad.net] on behalf of Endre Karlson [ >>> endre.karl...@gmail.com] >

Re: [Openstack] Monitoring / Billing Architecture proposed

; >> Hi, > >> Has anyone checked this > >> http://www.griddynamics.com/openstack/docs/nova-billing/quickstart.html > >> > >> From: openstack-bounces+atul.jha=csscorp....@lists.launchpad.net > >> [opensta

Re: [Openstack] Monitoring / Billing Architecture proposed

of Endre Karlson [ >> endre.karl...@gmail.com] >> Sent: Monday, April 23, 2012 2:27 AM >> To: openstack@lists.launchpad.net >> Subject: Re: [Openstack] Monitoring / Billing Architecture proposed >> >> What is Dough then compared to what you want to do ? >> >

Re: [Openstack] Monitoring / Billing Architecture proposed

penstack-bounces+atul.jha=csscorp@lists.launchpad.net] on behalf of >> Endre Karlson [endre.karl...@gmail.com] >> Sent: Monday, April 23, 2012 2:27 AM >> To: openstack@lists.launchpad.net >> Subject: Re: [Openstack] Monitoring / Billing Architecture proposed >> >

Re: [Openstack] Monitoring / Billing Architecture proposed

I agree. Keypoint: Metering always feeds an accounting system Then, 2 possible scenarios: 1) Meetering also feeds billing system (this is a sync operation) 2) Meetering does *not* feeds billing system. Then : The billing system has to pull the accounting system, then the billing system can't be

Re: [Openstack] Monitoring / Billing Architecture proposed

+atul.jha=csscorp@lists.launchpad.net > [openstack-bounces+atul.jha=csscorp@lists.launchpad.net] on behalf of > Endre Karlson [endre.karl...@gmail.com] > Sent: Monday, April 23, 2012 2:27 AM > To: openstack@lists.launchpad.net > Subject: Re: [Openstack] Monitoring / Billing Archite

Re: [Openstack] Monitoring / Billing Architecture proposed

s+atul.jha=csscorp@lists.launchpad.net] on behalf of > Endre Karlson [endre.karl...@gmail.com] > Sent: Monday, April 23, 2012 2:27 AM > To: openstack@lists.launchpad.net > Subject: Re: [Openstack] Monitoring / Billing Architecture proposed > > What is Dough then compared to what y

Re: [Openstack] Monitoring / Billing Architecture proposed

son [ > endre.karl...@gmail.com] > Sent: Monday, April 23, 2012 2:27 AM > To: openstack@lists.launchpad.net > Subject: Re: [Openstack] Monitoring / Billing Architecture proposed > > What is Dough then compared to what you want to do ? > > 2012/4/22 Endre Karlson endre.karl...@g

Re: [Openstack] Monitoring / Billing Architecture proposed

Dough is a proposed billing service. There was a session at Folsom design summit. This is a practical project for an OpenStack provider with test code on github. http://summit.openstack.org/sessions/view/37 " The billing system consists of three components. 1) API Server, which receives subs

Re: [Openstack] Monitoring / Billing Architecture proposed

On Sun, Apr 22, 2012 at 11:18 PM, Luis Gervaso wrote: > I think Dough takes care about pricing and billing. > > I have proposed an arch and technologies to fill the gap between openstack > and > whatever billing system / monitoring system. > > > On Sun, Apr 22, 2012 at 10:57 PM, Endre Karlson >

Re: [Openstack] Monitoring / Billing Architecture proposed

[endre.karl...@gmail.com] Sent: Monday, April 23, 2012 2:27 AM To: openstack@lists.launchpad.net Subject: Re: [Openstack] Monitoring / Billing Architecture proposed What is Dough then compared to what you want to do ? 2012/4/22 Endre Karlson mailto:endre.karl...@gmail.com>> What is Doug

Re: [Openstack] Monitoring / Billing Architecture proposed

my comments below On Sun, Apr 22, 2012 at 10:32 PM, Brian Schott < brian.sch...@nimbisservices.com> wrote: > Have you started a blueprint and/or Etherpad? We should do that. > > Couple of comments: > > 1. I had an idea for naming the metering service after Nipper, the famous > RCA dog :-) > http

Re: [Openstack] Monitoring / Billing Architecture proposed

What is Dough then compared to what you want to do ? 2012/4/22 Endre Karlson > What is Dough then ? > > > 2012/4/22 Brian Schott > >> I see this blueprint for metering, but none for Dough currently. >> http://wiki.openstack.org/EfficientMetering >> >> Here are the Dough slides, however: >> http

Re: [Openstack] Monitoring / Billing Architecture proposed

I see this blueprint for metering, but none for Dough currently. http://wiki.openstack.org/EfficientMetering Here are the Dough slides, however: http://www.slideshare.net/lzyeval/dough-openstack-billing-project We collectively need to talk more about the user scenarios, because I don't think you

Re: [Openstack] Monitoring / Billing Architecture proposed

Have you started a blueprint and/or Etherpad? We should do that. Couple of comments: 1. I had an idea for naming the metering service after Nipper, the famous RCA dog :-) http://en.wikipedia.org/wiki/Nipper 2. A common metering service that listens on Rabbit/ZeroMQ bus for registering events

Re: [Openstack] Monitoring / Billing Architecture proposed

On Sun, Apr 22, 2012 at 9:57 PM, Matt Joyce wrote: > Correct me if I am wrong but mongo has not been used in openstack > previously? What is the benefit here that justifies bringing in new > technology? > Mongo is document oriented, which i consider fits perfectly with the needs. It allows to

Re: [Openstack] Monitoring / Billing Architecture proposed

Correct me if I am wrong but mongo has not been used in openstack previously? What is the benefit here that justifies bringing in new technology? Also are you planning an active polling process over AMPQ or passive listening for the monitor? It seems to me that most of the main components today

Re: [Openstack] Monitoring / Billing Architecture proposed

Dough is the proposed billing platform/product (where the billing rules live), isn't it? I don't know Dough enough, so please me correct me if i'm wrong. I'm trying to define a generic/agnostic integration process, obviously where Dough can fit perfectly. I would like it become part to the refere

Re: [Openstack] Monitoring / Billing Architecture proposed

What about using the Dough project? Endre. 2012/4/22 Endre Karlson > What about using the Dough project ? > > Endre. > > 2012/4/22 Luis Gervaso > >> Hi, >> >> I want to share the architecture i am developing in order to perform the >> monitorig / billing OpenStack support: >> >> 1. AMQP Client

[Openstack] Monitoring / Billing Architecture proposed

Hi, I want to share the architecture i am developing in order to perform the monitorig / billing OpenStack support: 1. AMQP Client which listen to RabbitMQ / QPid (this should be interchangeable) (Own Stuff or ServiceMix / Camel) 2. Events should be stored on a NoSQL document oriented database (