Re: [openstack-dev] [Ceilometer]Cumulative metrics resetting
On 31/01/14 21:50, Julien Danjou wrote: On Fri, Jan 31 2014, Adrian Turjak wrote: Is it working on Havana? As I've gone through and tried all the possible state changes (reboot, suspend, pause, resize, terminate, etc), and not one triggers a poll cycle outside of the standard polling interval. So in all those cases apart from pause (which doesn't cause a reset), I'm losing cumulative data. It should. Did you enable it in the notification_drivers? It hadn't been, devstack doesn't enable it by default it seems. I have since added it to ceilometer.conf: notification_driver = ceilometer.compute.nova_notifier and done a ./unstack ./stack Still nothing. I'm not sure if there is some documentation on this I'm missing, but I've been unable to find anything useful. Nova_notifier as is only covers the terminate case (compute.instance.delete.start) anyway, but once I have it working at least for that case, I can alter it to work for the rest easily enough by adding in a list of tracked notifications. Also, side note, see comments #18 and #19 of this thread: https://bugs.launchpad.net/nova/+bug/1221987 If that is the case, is nova_notifier what I should be using at all? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer]Cumulative metrics resetting
On Thu, Jan 30 2014, Adrian Turjak wrote: example: 10min pipeline interval, a reset/shutdown happens 7 mins after the last poll. The data for those 7 mins is gone. Even terminating a VM will mean we lose the data in that last interval. If the shutdown is down properly, the nova notifier plugin that we provide in Ceilometer do a lost polling of the instance before shutting it down, so you don't lose anything. OTOH if the compute node crashes, that's true that you lose 7 mins of data. I guess you also have bigger problem. On the other hand, would it be possible to setup a notification based metric that updates cumulative metrics, or triggers a poll right before the reset/shutdown/suspension/terminate, so we have an entry right before it resets and don't lose any data? This would pretty much solve the issue, and as long as it is documented that the cumulative metrics reset, this would solve most problems. Yes, we have the nova notifier plugin doing exactly that. :) -- Julien Danjou # Free Software hacker # independent consultant # http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer]Cumulative metrics resetting
On 30/01/2014 23:31, Julien Danjou wrote: On the other hand, would it be possible to setup a notification based metric that updates cumulative metrics, or triggers a poll right before the reset/shutdown/suspension/terminate, so we have an entry right before it resets and don't lose any data? This would pretty much solve the issue, and as long as it is documented that the cumulative metrics reset, this would solve most problems. Yes, we have the nova notifier plugin doing exactly that. :) Awesome! But where in the source are they? As the only two notification plugins that seem to exist on the compute agent are for cpu and instance. Is there meant to be one that handles network.outgoing/incoming updates on those VM state changes? On Havana I know those network notifier plugins aren't there, so I assume master then and possibly going towards icehouse. I'll have a play with it again, but last time i did a clean devstack off of master, notification plugins weren't working at all for me. Has anyone else had this issue? Sorry for all the questions, just poking at ceilometer and trying to get data from it that I can be sure I can trust, and use for billing. I don't mind digging in the source, or even extending/fixing things, just need to know where to look. Cheers, -Adrian Turjak ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer]Cumulative metrics resetting
On Thu, Jan 30 2014, Adrian Turjak wrote: Awesome! But where in the source are they? As the only two notification plugins that seem to exist on the compute agent are for cpu and instance. Is there meant to be one that handles network.outgoing/incoming updates on those VM state changes? This does not rely on notifications but on polling. The notifier is in ceilometer.compute.nova_notifier. On Havana I know those network notifier plugins aren't there, so I assume master then and possibly going towards icehouse. That's not based on notification, that's based on polling. -- Julien Danjou -- Free Software hacker - independent consultant -- http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer]Cumulative metrics resetting
A question about this bug: https://bugs.launchpad.net/ceilometer/+bug/1061817 In the billing program we are currently writing we've partly accounted for resetting of values in a given period for the cumulative metrics, but since we need high accuracy especially for metrics like network.incoming/outgoing, we are likely to lose chargeable data if someone resets a VM, or a VM goes down. example: 10min pipeline interval, a reset/shutdown happens 7 mins after the last poll. The data for those 7 mins is gone. Even terminating a VM will mean we lose the data in that last interval. Fixing the bug so resets don't happen is likely a lot of work, and I have a feeling will require work in Nova, and probably won't account for the terminate case. On the other hand, would it be possible to setup a notification based metric that updates cumulative metrics, or triggers a poll right before the reset/shutdown/suspension/terminate, so we have an entry right before it resets and don't lose any data? This would pretty much solve the issue, and as long as it is documented that the cumulative metrics reset, this would solve most problems. Since the 'process_notification' function only gets passed a 'message', I don't know if there is a way to pull the needed data out from nova using only what is in the message though. Any thoughts, or suggestions as to where to start experimenting? -Adrian Turjak ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
