Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On 6/22/2015 4:55 AM, Daniel P. Berrange wrote: On Sun, Jun 21, 2015 at 11:14:00AM -0500, Matt Riedemann wrote: On 6/20/2015 3:35 PM, Daniel P. Berrange wrote: On Sat, Jun 20, 2015 at 01:50:53PM -0500, Matt Riedemann wrote: Waking up from a rare nap opportunity on a Saturday, this is what was bothering me: The proposal in the etherpad assumes that we are just getting bulk host/domain/guest VM stats from the hypervisor and sending those in a notification, but how do we go about filtering those out to only instances that were booted through Nova? In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. Having said that, when nova uses libguestfs, it will create some temp guests via libvirt, so we do have to consider that possibility. Even today with the general list domains virt driver call, we could be getting domains that weren't launched by Nova I believe. Jason pointed out the ceilometer code gets all of the non-error state instances from nova first [1] and then for each of those it does the domain lookup from libvirt, filtering out any that are in SHUTOFF state [2]. When talking about the new virt driver API for bulk stats, danpb said to use virConnectGetAllDomainStats with libvirt [3] but I'm not aware of that being able to filter out instances that weren't created by nova. I don't think we want a notification from nova about the hypervisor stats to include things that were created outside nova, like directly through virsh or vCenter. For at least libvirt, if virConnectGetAllDomainStats returns the domain metadata then we can filter those since there is nova-specific metadata in the domains created through nova [4] but I'm not sure that's true about the other virt types in nova (I think the vCenter driver tags VMs somehow as being created by OpenStack/Nova, but not sure about xen/hyper-v/ironic). The nova database hsa a list of domains that it owns, so if you query the database for a list of valid UUIDs for the host, you can use that to filter the domains that libvirt reports by comparing UUIDs. Regards, Daniel Dan, is virsh domstats using virConnectGetAllDomainStats? I have libvirt 1.2.8 on RHEL 7.1, created two m1.tiny instances through nova and got this from virsh domstats: http://paste.openstack.org/show/310874/ Is that similar to what we'd see from virConnectGetAllDomainStats? I haven't yet written any code in the libvirt driver to use virConnectGetAllDomainStats to see what that looks like. Yes, that's the kind of data you'd expect. Regards, Daniel Here is another issue I just thought of. There are limits to the size of a message you can send through RPC right? So what if you have a lot of instances running and you're pulling bulk stats on them and sending over rpc via a notification? Is there the possibility that we blow that up on message size limits? For libvirt/xen/hyper-v this is maybe not a big deal since the compute node is 1:1 with the hypervisor and I'd think in most cases you don't have enough instances running on that compute host to blow the size limit on the message payload, unless you have a big ass compute host. But what about clustered virt drivers like vcenter and ironic? That one compute node could be getting bulk stats on an entire cloud (vcenter cluster at least). Maybe we could just chunk the messages/notifications if we know the rpc message limit? -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On 6/26/2015 2:17 PM, Matt Riedemann wrote: On 6/22/2015 4:55 AM, Daniel P. Berrange wrote: On Sun, Jun 21, 2015 at 11:14:00AM -0500, Matt Riedemann wrote: On 6/20/2015 3:35 PM, Daniel P. Berrange wrote: On Sat, Jun 20, 2015 at 01:50:53PM -0500, Matt Riedemann wrote: Waking up from a rare nap opportunity on a Saturday, this is what was bothering me: The proposal in the etherpad assumes that we are just getting bulk host/domain/guest VM stats from the hypervisor and sending those in a notification, but how do we go about filtering those out to only instances that were booted through Nova? In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. Having said that, when nova uses libguestfs, it will create some temp guests via libvirt, so we do have to consider that possibility. Even today with the general list domains virt driver call, we could be getting domains that weren't launched by Nova I believe. Jason pointed out the ceilometer code gets all of the non-error state instances from nova first [1] and then for each of those it does the domain lookup from libvirt, filtering out any that are in SHUTOFF state [2]. When talking about the new virt driver API for bulk stats, danpb said to use virConnectGetAllDomainStats with libvirt [3] but I'm not aware of that being able to filter out instances that weren't created by nova. I don't think we want a notification from nova about the hypervisor stats to include things that were created outside nova, like directly through virsh or vCenter. For at least libvirt, if virConnectGetAllDomainStats returns the domain metadata then we can filter those since there is nova-specific metadata in the domains created through nova [4] but I'm not sure that's true about the other virt types in nova (I think the vCenter driver tags VMs somehow as being created by OpenStack/Nova, but not sure about xen/hyper-v/ironic). The nova database hsa a list of domains that it owns, so if you query the database for a list of valid UUIDs for the host, you can use that to filter the domains that libvirt reports by comparing UUIDs. Regards, Daniel Dan, is virsh domstats using virConnectGetAllDomainStats? I have libvirt 1.2.8 on RHEL 7.1, created two m1.tiny instances through nova and got this from virsh domstats: http://paste.openstack.org/show/310874/ Is that similar to what we'd see from virConnectGetAllDomainStats? I haven't yet written any code in the libvirt driver to use virConnectGetAllDomainStats to see what that looks like. Yes, that's the kind of data you'd expect. Regards, Daniel Here is another issue I just thought of. There are limits to the size of a message you can send through RPC right? So what if you have a lot of instances running and you're pulling bulk stats on them and sending over rpc via a notification? Is there the possibility that we blow that up on message size limits? For libvirt/xen/hyper-v this is maybe not a big deal since the compute node is 1:1 with the hypervisor and I'd think in most cases you don't have enough instances running on that compute host to blow the size limit on the message payload, unless you have a big ass compute host. But what about clustered virt drivers like vcenter and ironic? That one compute node could be getting bulk stats on an entire cloud (vcenter cluster at least). Maybe we could just chunk the messages/notifications if we know the rpc message limit? With respect to message size limit, I found a thread in the rabbitmq mailing list [1] talking about message size limits which basically says you're only bounded by resources available, but sending things too large is obviously a bad idea since you starve the system and can potentially screw up the heartbeat checking. The actual 64K size limit I was really thinking of originally was a Qpid limitation that was fixed in the long long ago by bnemec [2]. So I guess for the purpose of a bulk stats notification, we'd probably be safe to keep the messages under 64K and just chunk through the list of instances. [1] http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-March/018699.html [2] https://review.openstack.org/#/c/28711/ -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On Mon, 2015-06-22 at 18:10 +0100, Daniel P. Berrange wrote: On Mon, Jun 22, 2015 at 11:10:40AM -0500, Kevin L. Mitchell wrote: On Sat, 2015-06-20 at 21:35 +0100, Daniel P. Berrange wrote: In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. On the other hand, this is exactly how compute nodes themselves are often deployed—a random guest on the hypervisor node… In a devstack env maybe, but in production we expect the hypervisor to be exclusively used by Nova, or things Nova uses. In production, at least in some Xen deployments, the hypervisor does not run the nova-compute service; instead, we have a VM specifically set aside for running the nova-compute. This is done because the hypervisor (in older Xen deployments, at least) has an old version of Python, and because some operations are most conveniently done by a co-located compute node. -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On Mon, Jun 22, 2015 at 11:10:40AM -0500, Kevin L. Mitchell wrote: On Sat, 2015-06-20 at 21:35 +0100, Daniel P. Berrange wrote: In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. On the other hand, this is exactly how compute nodes themselves are often deployed—a random guest on the hypervisor node… In a devstack env maybe, but in production we expect the hypervisor to be exclusively used by Nova, or things Nova uses. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On Sat, 2015-06-20 at 21:35 +0100, Daniel P. Berrange wrote: In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. On the other hand, this is exactly how compute nodes themselves are often deployed—a random guest on the hypervisor node… -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On 06/22/15 20:10, Daniel P. Berrange wrote: On Mon, Jun 22, 2015 at 11:10:40AM -0500, Kevin L. Mitchell wrote: On Sat, 2015-06-20 at 21:35 +0100, Daniel P. Berrange wrote: In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. On the other hand, this is exactly how compute nodes themselves are often deployed—a random guest on the hypervisor node… In a devstack env maybe, but in production we expect the hypervisor to be exclusively used by Nova, or things Nova uses. Regards, Daniel +1 to that!! I think it would be safe to say that in any production environment I would run, Nova would control the instances exclusively. -- Best Regards, Maish Saidel-Keesing __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On Sun, Jun 21, 2015 at 11:14:00AM -0500, Matt Riedemann wrote: On 6/20/2015 3:35 PM, Daniel P. Berrange wrote: On Sat, Jun 20, 2015 at 01:50:53PM -0500, Matt Riedemann wrote: Waking up from a rare nap opportunity on a Saturday, this is what was bothering me: The proposal in the etherpad assumes that we are just getting bulk host/domain/guest VM stats from the hypervisor and sending those in a notification, but how do we go about filtering those out to only instances that were booted through Nova? In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. Having said that, when nova uses libguestfs, it will create some temp guests via libvirt, so we do have to consider that possibility. Even today with the general list domains virt driver call, we could be getting domains that weren't launched by Nova I believe. Jason pointed out the ceilometer code gets all of the non-error state instances from nova first [1] and then for each of those it does the domain lookup from libvirt, filtering out any that are in SHUTOFF state [2]. When talking about the new virt driver API for bulk stats, danpb said to use virConnectGetAllDomainStats with libvirt [3] but I'm not aware of that being able to filter out instances that weren't created by nova. I don't think we want a notification from nova about the hypervisor stats to include things that were created outside nova, like directly through virsh or vCenter. For at least libvirt, if virConnectGetAllDomainStats returns the domain metadata then we can filter those since there is nova-specific metadata in the domains created through nova [4] but I'm not sure that's true about the other virt types in nova (I think the vCenter driver tags VMs somehow as being created by OpenStack/Nova, but not sure about xen/hyper-v/ironic). The nova database hsa a list of domains that it owns, so if you query the database for a list of valid UUIDs for the host, you can use that to filter the domains that libvirt reports by comparing UUIDs. Regards, Daniel Dan, is virsh domstats using virConnectGetAllDomainStats? I have libvirt 1.2.8 on RHEL 7.1, created two m1.tiny instances through nova and got this from virsh domstats: http://paste.openstack.org/show/310874/ Is that similar to what we'd see from virConnectGetAllDomainStats? I haven't yet written any code in the libvirt driver to use virConnectGetAllDomainStats to see what that looks like. Yes, that's the kind of data you'd expect. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On 6/20/2015 3:35 PM, Daniel P. Berrange wrote: On Sat, Jun 20, 2015 at 01:50:53PM -0500, Matt Riedemann wrote: Waking up from a rare nap opportunity on a Saturday, this is what was bothering me: The proposal in the etherpad assumes that we are just getting bulk host/domain/guest VM stats from the hypervisor and sending those in a notification, but how do we go about filtering those out to only instances that were booted through Nova? In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. Having said that, when nova uses libguestfs, it will create some temp guests via libvirt, so we do have to consider that possibility. Even today with the general list domains virt driver call, we could be getting domains that weren't launched by Nova I believe. Jason pointed out the ceilometer code gets all of the non-error state instances from nova first [1] and then for each of those it does the domain lookup from libvirt, filtering out any that are in SHUTOFF state [2]. When talking about the new virt driver API for bulk stats, danpb said to use virConnectGetAllDomainStats with libvirt [3] but I'm not aware of that being able to filter out instances that weren't created by nova. I don't think we want a notification from nova about the hypervisor stats to include things that were created outside nova, like directly through virsh or vCenter. For at least libvirt, if virConnectGetAllDomainStats returns the domain metadata then we can filter those since there is nova-specific metadata in the domains created through nova [4] but I'm not sure that's true about the other virt types in nova (I think the vCenter driver tags VMs somehow as being created by OpenStack/Nova, but not sure about xen/hyper-v/ironic). The nova database hsa a list of domains that it owns, so if you query the database for a list of valid UUIDs for the host, you can use that to filter the domains that libvirt reports by comparing UUIDs. Regards, Daniel Dan, is virsh domstats using virConnectGetAllDomainStats? I have libvirt 1.2.8 on RHEL 7.1, created two m1.tiny instances through nova and got this from virsh domstats: http://paste.openstack.org/show/310874/ Is that similar to what we'd see from virConnectGetAllDomainStats? I haven't yet written any code in the libvirt driver to use virConnectGetAllDomainStats to see what that looks like. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On Sat, Jun 20, 2015 at 01:50:53PM -0500, Matt Riedemann wrote: Waking up from a rare nap opportunity on a Saturday, this is what was bothering me: The proposal in the etherpad assumes that we are just getting bulk host/domain/guest VM stats from the hypervisor and sending those in a notification, but how do we go about filtering those out to only instances that were booted through Nova? In general I would say that is an unsupported deployment scenario to have other random virt guests running on a nova compute node. Having said that, when nova uses libguestfs, it will create some temp guests via libvirt, so we do have to consider that possibility. Even today with the general list domains virt driver call, we could be getting domains that weren't launched by Nova I believe. Jason pointed out the ceilometer code gets all of the non-error state instances from nova first [1] and then for each of those it does the domain lookup from libvirt, filtering out any that are in SHUTOFF state [2]. When talking about the new virt driver API for bulk stats, danpb said to use virConnectGetAllDomainStats with libvirt [3] but I'm not aware of that being able to filter out instances that weren't created by nova. I don't think we want a notification from nova about the hypervisor stats to include things that were created outside nova, like directly through virsh or vCenter. For at least libvirt, if virConnectGetAllDomainStats returns the domain metadata then we can filter those since there is nova-specific metadata in the domains created through nova [4] but I'm not sure that's true about the other virt types in nova (I think the vCenter driver tags VMs somehow as being created by OpenStack/Nova, but not sure about xen/hyper-v/ironic). The nova database hsa a list of domains that it owns, so if you query the database for a list of valid UUIDs for the host, you can use that to filter the domains that libvirt reports by comparing UUIDs. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
On 6/17/2015 10:52 AM, Matt Riedemann wrote: Without getting into the details from the etherpad [1], a few of us in IRC today were talking about how the ceilometer compute-agent polls libvirt directly for guest VM statistics and how ceilometer should really be getting this information from nova via notifications sent from a periodic task in the nova compute manager. Nova already has the get_instance_diagnostics virt driver API which is nice in that it has structured versioned instance diagnostic information regardless of virt driver (unlike the v2 os-server-diagnostics API which is a free-form bag of goodies depending on which virt driver is used, which makes it mostly untestable and not portable). The problem is the get_instance_diagnostics virt driver API is per-instance, so it's not efficient in the case that you want bulk instance data for a given compute host. So the idea is to add a new virt driver API to get the bulk data and emit that via a structured versioned payload similar to get_instance_diagnostics but for all instances. Eventually the goal is for nova to send what ceilometer is collecting today [2] and then ceilometer can just consume that notification rather than doing the direct hypervisor polling it has today. Anyway, this is the high level idea, the details/notes are in the etherpad along with next steps. Feel free to chime in now with reasons why this is crazy and will never work and we shouldn't waste our time on it. [1] https://etherpad.openstack.org/p/nova-hypervisor-bulk-stats-notify [2] http://docs.openstack.org/admin-guide-cloud/content/section_telemetry-compute-meters.html Waking up from a rare nap opportunity on a Saturday, this is what was bothering me: The proposal in the etherpad assumes that we are just getting bulk host/domain/guest VM stats from the hypervisor and sending those in a notification, but how do we go about filtering those out to only instances that were booted through Nova? Jason pointed out the ceilometer code gets all of the non-error state instances from nova first [1] and then for each of those it does the domain lookup from libvirt, filtering out any that are in SHUTOFF state [2]. When talking about the new virt driver API for bulk stats, danpb said to use virConnectGetAllDomainStats with libvirt [3] but I'm not aware of that being able to filter out instances that weren't created by nova. I don't think we want a notification from nova about the hypervisor stats to include things that were created outside nova, like directly through virsh or vCenter. For at least libvirt, if virConnectGetAllDomainStats returns the domain metadata then we can filter those since there is nova-specific metadata in the domains created through nova [4] but I'm not sure that's true about the other virt types in nova (I think the vCenter driver tags VMs somehow as being created by OpenStack/Nova, but not sure about xen/hyper-v/ironic). I guess adding support for something like bulk guest VM stats in a nova virt driver would have a pre-requisite of being able to uniquely identify guest VMs from the hypervisor as being created by nova. Otherwise we'd just basically be moving the same thing that ceilometer is doing today into nova, as a per-instance loop and then using the os-server-diagnostics virt driver calls from the compute manager (at least saving us a compute API call from ceilometer every 10 minutes). Thoughts? [1] https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/discovery.py#L35 [2] https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/virt/libvirt/inspector.py#L111 [3] http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectGetAllDomainStats [4] http://paste.openstack.org/show/308305/ -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
not familiar/smart enough to comment on design within Nova but i'm very much in favour if this is possible. the polling option in Ceilometer was always a means to get information not readily available via notifications. that said, i don't think we can completely do away with polling. i'm aware of some cases where in addition to polling for measurements, some users leverage the polling agent as a means of performing health checks as well but the less load our polls generate the better. On 17/06/2015 11:52 AM, Matt Riedemann wrote: Without getting into the details from the etherpad [1], a few of us in IRC today were talking about how the ceilometer compute-agent polls libvirt directly for guest VM statistics and how ceilometer should really be getting this information from nova via notifications sent from a periodic task in the nova compute manager. Nova already has the get_instance_diagnostics virt driver API which is nice in that it has structured versioned instance diagnostic information regardless of virt driver (unlike the v2 os-server-diagnostics API which is a free-form bag of goodies depending on which virt driver is used, which makes it mostly untestable and not portable). The problem is the get_instance_diagnostics virt driver API is per-instance, so it's not efficient in the case that you want bulk instance data for a given compute host. So the idea is to add a new virt driver API to get the bulk data and emit that via a structured versioned payload similar to get_instance_diagnostics but for all instances. Eventually the goal is for nova to send what ceilometer is collecting today [2] and then ceilometer can just consume that notification rather than doing the direct hypervisor polling it has today. Anyway, this is the high level idea, the details/notes are in the etherpad along with next steps. Feel free to chime in now with reasons why this is crazy and will never work and we shouldn't waste our time on it. [1] https://etherpad.openstack.org/p/nova-hypervisor-bulk-stats-notify [2] http://docs.openstack.org/admin-guide-cloud/content/section_telemetry-compute-meters.html -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
Without getting into the details from the etherpad [1], a few of us in IRC today were talking about how the ceilometer compute-agent polls libvirt directly for guest VM statistics and how ceilometer should really be getting this information from nova via notifications sent from a periodic task in the nova compute manager. Nova already has the get_instance_diagnostics virt driver API which is nice in that it has structured versioned instance diagnostic information regardless of virt driver (unlike the v2 os-server-diagnostics API which is a free-form bag of goodies depending on which virt driver is used, which makes it mostly untestable and not portable). The problem is the get_instance_diagnostics virt driver API is per-instance, so it's not efficient in the case that you want bulk instance data for a given compute host. So the idea is to add a new virt driver API to get the bulk data and emit that via a structured versioned payload similar to get_instance_diagnostics but for all instances. Eventually the goal is for nova to send what ceilometer is collecting today [2] and then ceilometer can just consume that notification rather than doing the direct hypervisor polling it has today. Anyway, this is the high level idea, the details/notes are in the etherpad along with next steps. Feel free to chime in now with reasons why this is crazy and will never work and we shouldn't waste our time on it. [1] https://etherpad.openstack.org/p/nova-hypervisor-bulk-stats-notify [2] http://docs.openstack.org/admin-guide-cloud/content/section_telemetry-compute-meters.html -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
I was involved in the IRC discussion, but to reiterate here this would be a wonderful thing to get done. The current polling can be terribly inefficient, and the ceilo team is already looking into other ways to reduce the impact of it. I'm looking forward to helping with this. Cheers, Jason On Wed, Jun 17, 2015, at 10:52 AM, Matt Riedemann wrote: Without getting into the details from the etherpad [1], a few of us in IRC today were talking about how the ceilometer compute-agent polls libvirt directly for guest VM statistics and how ceilometer should really be getting this information from nova via notifications sent from a periodic task in the nova compute manager. Nova already has the get_instance_diagnostics virt driver API which is nice in that it has structured versioned instance diagnostic information regardless of virt driver (unlike the v2 os-server-diagnostics API which is a free-form bag of goodies depending on which virt driver is used, which makes it mostly untestable and not portable). The problem is the get_instance_diagnostics virt driver API is per-instance, so it's not efficient in the case that you want bulk instance data for a given compute host. So the idea is to add a new virt driver API to get the bulk data and emit that via a structured versioned payload similar to get_instance_diagnostics but for all instances. Eventually the goal is for nova to send what ceilometer is collecting today [2] and then ceilometer can just consume that notification rather than doing the direct hypervisor polling it has today. Anyway, this is the high level idea, the details/notes are in the etherpad along with next steps. Feel free to chime in now with reasons why this is crazy and will never work and we shouldn't waste our time on it. [1] https://etherpad.openstack.org/p/nova-hypervisor-bulk-stats-notify [2] http://docs.openstack.org/admin-guide-cloud/content/section_telemetry-compute-meters.html -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
