Re: [openstack-dev] [Openstack] [Ceilometer] [API] Batch alarm creation
Yes Ryan, that's exactly what I'm thinking. Glad to know that we have the same
opinion :)
BR
Kurt
-邮件原件-
发件人: Ryan Brown [mailto:rybr...@redhat.com]
发送时间: 2014年12月12日 23:30
收件人: openstack-dev@lists.openstack.org
主题: Re: [openstack-dev] [Openstack] [Ceilometer] [API] Batch alarm creation
On 12/12/2014 03:37 AM, Rao Dingyuan wrote:
Hi Eoghan and folks,
I'm thinking of adding an API to create multiple alarms in a batch.
I think adding an API to create multiple alarms is a good option to solve the
problem that once an *alarm target* (a vm or a new group of vms) is created,
multiple requests will be fired because multiple alarms are to be created.
In the our current project, this requiement is specially urgent since our
alarm target is one VM, and 6 alarms are to be created when one VM is created.
What do you guys think?
Best Regards,
Kurt Rao
Allowing batch operations is definitely a good idea, though it may not be a
solution to all of the problems you outlined.
One way to batch object creations would be to give clients the option to POST a
collection of alarms instead of a single alarm. Currently your API looks
like[1]:
POST /v2/alarms
BODY:
{
alarm_actions: ...
...
}
For batches you could modify your API to accept a body like:
{
alarms: [
{alarm_actions: ...},
{alarm_actions: ...},
{alarm_actions: ...},
{alarm_actions: ...}
]
}
It could (pretty easily) be a backwards-compatible change since the schemata
don't conflict, and you can even add some kind of a batch:true flag to make
it explicit that the user wants to create a collection. The API-WG has a
spec[2] out right now explaining the rationale behind collection
representations.
[1]:
http://docs.openstack.org/developer/ceilometer/webapi/v2.html#post--v2-alarms
[2]:
https://review.openstack.org/#/c/133660/11/guidelines/representation_structure.rst,unified
- Original -
发件人: Eoghan Glynn [mailto:egl...@redhat.com]
发送时间: 2014年12月3日 20:34
收件人: Rao Dingyuan
抄送: openst...@lists.openstack.org
主题: Re: [Openstack] [Ceilometer] looking for alarm best practice -
please help
Hi folks,
I wonder if anyone could share some best practice regarding to the
usage of ceilometer alarm. We are using the alarm
evaluation/notification of ceilometer and we don’t feel very well of
the way we use it. Below is our
problem:
Scenario:
When cpu usage or memory usage above a certain threshold, alerts
should be displayed on admin’s web page. There should be a 3 level
alerts according to meter value, namely notice, warning, fatal.
Notice means the meter value is between 50% ~ 70%, warning means
between 70% ~ 85% and fatal means above 85%
For example:
* when one vm’s cpu usage is 72%, an alert message should be
displayed saying
“Warning: vm[d9b7018b-06c4-4fba-8221-37f67f6c6b8c] cpu usage is above 70%”.
* when one vm’s memory usage is 90%, another alert message should be
created saying “Fatal: vm[d9b7018b-06c4-4fba-8221-37f67f6c6b8c]
memory usage is above 85%”
Our current Solution:
We used ceilometer alarm evaluation/notification to implement this.
To distinguish which VM and which meter is above what value, we’ve
created one alarm for each VM by each condition. So, to monitor 1 VM,
6 alarms will be created because there are 2 meters and for each meter there
are 3 levels.
That means, if there are 100 VMs to be monitored, 600 alarms will be
created.
Problems:
* The first problem is, when the number of meters increases, the
number of alarms will be multiplied. For example, customer may want
alerts on disk and network IO rates, and if we do that, there will be
4*3=12 alarms for each VM.
* The second problem is, when one VM is created, multiple alarms will
be created, meaning multiple http requests will be fired. In the case
above, 6 HTTP requests will be needed once a VM is created. And this
number also increases as the number of meters goes up.
One way of reducing both the number of alarms and the volume of notifications
would be to group related VMs, if such a concept exists in your use-case.
This is effectively how Heat autoscaling uses ceilometer, alarming on the
average of some statistic over a set of instances (as opposed to triggering
on individual instances).
The VMs could be grouped by setting user-metadata of form:
nova boot ... --meta metering.my_server_group=foobar
Any user-metadata prefixed with 'metering.' will be preserved by ceilometer
in the resource_metadata.user_metedata stored for each sample, so that it can
used to select the statistics on which the alarm is based, e.g.
ceilometer alarm-threshold-create --name cpu_high_foobar \
--description 'warning: foobar instance group running hot' \
--meter-name cpu_util --threshold 70.0 \
--comparison-operator gt --statistic avg \
...
--query
[openstack-dev] [Openstack] [Ceilometer] [API] Batch alarm creation
Hi Eoghan and folks, I'm thinking of adding an API to create multiple alarms in a batch. I think adding an API to create multiple alarms is a good option to solve the problem that once an *alarm target* (a vm or a new group of vms) is created, multiple requests will be fired because multiple alarms are to be created. In the our current project, this requiement is specially urgent since our alarm target is one VM, and 6 alarms are to be created when one VM is created. What do you guys think? Best Regards, Kurt Rao - Original - 发件人: Eoghan Glynn [mailto:egl...@redhat.com] 发送时间: 2014年12月3日 20:34 收件人: Rao Dingyuan 抄送: openst...@lists.openstack.org 主题: Re: [Openstack] [Ceilometer] looking for alarm best practice - please help Hi folks, I wonder if anyone could share some best practice regarding to the usage of ceilometer alarm. We are using the alarm evaluation/notification of ceilometer and we don’t feel very well of the way we use it. Below is our problem: Scenario: When cpu usage or memory usage above a certain threshold, alerts should be displayed on admin’s web page. There should be a 3 level alerts according to meter value, namely notice, warning, fatal. Notice means the meter value is between 50% ~ 70%, warning means between 70% ~ 85% and fatal means above 85% For example: * when one vm’s cpu usage is 72%, an alert message should be displayed saying “Warning: vm[d9b7018b-06c4-4fba-8221-37f67f6c6b8c] cpu usage is above 70%”. * when one vm’s memory usage is 90%, another alert message should be created saying “Fatal: vm[d9b7018b-06c4-4fba-8221-37f67f6c6b8c] memory usage is above 85%” Our current Solution: We used ceilometer alarm evaluation/notification to implement this. To distinguish which VM and which meter is above what value, we’ve created one alarm for each VM by each condition. So, to monitor 1 VM, 6 alarms will be created because there are 2 meters and for each meter there are 3 levels. That means, if there are 100 VMs to be monitored, 600 alarms will be created. Problems: * The first problem is, when the number of meters increases, the number of alarms will be multiplied. For example, customer may want alerts on disk and network IO rates, and if we do that, there will be 4*3=12 alarms for each VM. * The second problem is, when one VM is created, multiple alarms will be created, meaning multiple http requests will be fired. In the case above, 6 HTTP requests will be needed once a VM is created. And this number also increases as the number of meters goes up. One way of reducing both the number of alarms and the volume of notifications would be to group related VMs, if such a concept exists in your use-case. This is effectively how Heat autoscaling uses ceilometer, alarming on the average of some statistic over a set of instances (as opposed to triggering on individual instances). The VMs could be grouped by setting user-metadata of form: nova boot ... --meta metering.my_server_group=foobar Any user-metadata prefixed with 'metering.' will be preserved by ceilometer in the resource_metadata.user_metedata stored for each sample, so that it can used to select the statistics on which the alarm is based, e.g. ceilometer alarm-threshold-create --name cpu_high_foobar \ --description 'warning: foobar instance group running hot' \ --meter-name cpu_util --threshold 70.0 \ --comparison-operator gt --statistic avg \ ... --query metadata.user_metedata.my_server_group=foobar This approach is of course predicated on the there being some natural grouping relation between instances in your environment. Cheers, Eoghan = Do anyone have any suggestions? Best Regards! Kurt Rao ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openst...@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[Openstack] [Ceilometer] looking for alarm best practice - please help
Hi folks, I wonder if anyone could share some best practice regarding to the usage of ceilometer alarm. We are using the alarm evaluation/notification of ceilometer and we don't feel very well of the way we use it. Below is our problem: Scenario: When cpu usage or memory usage above a certain threshold, alerts should be displayed on admin's web page. There should be a 3 level alerts according to meter value, namely notice, warning, fatal. Notice means the meter value is between 50% ~ 70%, warning means between 70% ~ 85% and fatal means above 85% For example: * when one vm's cpu usage is 72%, an alert message should be displayed saying Warning: vm[d9b7018b-06c4-4fba-8221-37f67f6c6b8c] cpu usage is above 70%. * when one vm's memory usage is 90%, another alert message should be created saying Fatal: vm[d9b7018b-06c4-4fba-8221-37f67f6c6b8c] memory usage is above 85% Our current Solution: We used ceilometer alarm evaluation/notification to implement this. To distinguish which VM and which meter is above what value, we've created one alarm for each VM by each condition. So, to monitor 1 VM, 6 alarms will be created because there are 2 meters and for each meter there are 3 levels. That means, if there are 100 VMs to be monitored, 600 alarms will be created. Problems: * The first problem is, when the number of meters increases, the number of alarms will be multiplied. For example, customer may want alerts on disk and network IO rates, and if we do that, there will be 4*3=12 alarms for each VM. * The second problem is, when one VM is created, multiple alarms will be created, meaning multiple http requests will be fired. In the case above, 6 HTTP requests will be needed once a VM is created. And this number also increases as the number of meters goes up. = Do anyone have any suggestions? Best Regards! Kurt Rao _ ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[openstack-dev] 答复: Ceilometer memory.usage can not get info from libvirt
As described in the document: http://docs.openstack.org/developer/ceilometer/measurements.html#measurements “”” Note To enable the libvirt memory.usage supporting, you need libvirt version 1.1.1+, qemu version 1.5+, and you need to prepare suitable balloon driver in the image, particularly for Windows guests, most modern Linuxes have it built in. The memory.usage meters can’t be fetched without image balloon driver. “”” :) _ E_mail: mailto:raodingy...@chinacloud.com.cn raodingy...@chinacloud.com.cn 发件人: Du Jun [mailto:dj199...@gmail.com] 发送时间: 2014年11月17日 16:57 收件人: OpenStack Development Mailing List (not for usage questions) 主题: [openstack-dev] Ceilometer memory.usage can not get info from libvirt Hi all, 2014-11-17 16:04:01.563 5162 INFO ceilometer.agent [-] Polling pollster memory.usage in the context of meter_source 14 2014-11-17 16:04:01.564 5162 DEBUG ceilometer.compute.pollsters.memory [-] Checking memory usage for instance 7e53172c-f05f-4fda-9855-af6775c1f4a8 get_samples /opt/stack/ceilometer/ceilometer/compute/pollsters/memory.py:31 140002 2014-11-17 16:04:01.573 5162 WARNING ceilometer.compute.virt.libvirt.inspector [-] Failed to inspect memory usage of instance-0002, can not get info from libvirt 140003 2014-11-17 16:04:01.574 5162 ERROR ceilometer.compute.pollsters.memory [-] Could not get Memory Usage for 7e53172c-f05f-4fda-9855-af6775c1f4a8: 'NoneType' object has no attribute 'usage' 140004 2014-11-17 16:04:01.574 5162 TRACE ceilometer.compute.pollsters.memory Traceback (most recent call last): 140005 2014-11-17 16:04:01.574 5162 TRACE ceilometer.compute.pollsters.memory File /opt/stack/ceilometer/ceilometer/compute/pollsters/memory.py, line 37, in get_samples 140006 2014-11-17 16:04:01.574 5162 TRACE ceilometer.compute.pollsters.memory 'usage': memory_info.usage})) 140007 2014-11-17 16:04:01.574 5162 TRACE ceilometer.compute.pollsters.memory AttributeError: 'NoneType' object has no attribute 'usage' When ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] Compute agent local VM inspector - potential enhancement
Hi folks, Is there anybody working on this? In most of our cloud environments, business networks are isolated from management network. So, we are thinking about making *an agent in guest machine to send metrics to compute node using virtual serial port*. And then, compute node could send those data to ceilometer. That seems a general solution for all kinds of network topologies, and can send metrics without knowing any credentials. BR Kurt Rao -Original Message- 发件人: boden [mailto:bo...@linux.vnet.ibm.com] 发送时间: 2014年8月1日 20:37 收件人: openstack-dev@lists.openstack.org 主题: Re: [openstack-dev] [ceilometer] Compute agent local VM inspector - potential enhancement On 8/1/2014 4:37 AM, Eoghan Glynn wrote: Heat cfntools is based on SSH, so I assume it requires TCP/IP connectivity between VM and the central agent(or collector). But in the cloud, some networks are isolated from infrastructure layer network, because of security reasons. Some of our customers even explicitly require such security protection. Does it mean those isolated VMs cannot be monitored by this proposed-VM-agent? Yes, that sounds plausible to me. My understanding is that this VM agent for ceilometer would need connectivity to nova API as well as to the AMQP broker. IMHO the infrastructure requirements from a network topology POV will differ from provider to provider and based on customer reqs / env. Cheers, Eoghan I really wish we can figur out how it could work for all VMs but with no security issues. I'm not familiar with heat-cfntools, so, correct me if I am wrong :) Best regards! Kurt -邮件原件- 发件人: Eoghan Glynn [mailto:egl...@redhat.com] 发送时间: 2014年8月1日 14:46 收件人: OpenStack Development Mailing List (not for usage questions) 主题: Re: [openstack-dev] [ceilometer] Compute agent local VM inspector - potential enhancement Disclaimer: I'm not fully vested on ceilometer internals, so bear with me. For consumers wanting to leverage ceilometer as a telemetry service atop non-OpenStack Clouds or infrastructure they don't own, some edge cases crop up. Most notably the consumer may not have access to the hypervisor host and therefore cannot leverage the ceilometer compute agent on a per host basis. Yes, currently such access to the hypervisor host is required, least in the case of the libvirt-based inspector. In such scenarios it's my understanding the main option is to employ the central agent to poll measurements from the monitored resources (VMs, etc.). Well, the ceilometer central agent is not generally concerned with with polling related *directly* to VMs - rather it handles acquiring data from RESTful API (glance, neutron etc.) that are not otherwise available in the form of notifications, and also from host-level interfaces such as SNMP. Thanks for additional clarity. Perhaps this proposed local VM agent fills additional use cases whereupon ceilometer is being used without openstack proper (e.g. not a full set of openstack complaint services like neutron, glance, etc.). However this approach requires Cloud APIs (or other mechanisms) which allow the polling impl to obtain the desired measurements (VM memory, CPU, net stats, etc.) and moreover the polling approach has it's own set of pros / cons from a arch / topology perspective. Indeed. The other potential option is to setup the ceilometer compute agent within the VM and have each VM publish measurements to the collector -- a local VM agent / inspector if you will. With respect to this local VM agent approach: (a) I haven't seen this documented to date; is there any desire / reqs to support this topology? (b) If yes to #a, I whipped up a crude PoC here: http://tinyurl.com/pqjgotv Are folks willing to consider a BP for this approach? So in a sense this is similar to the Heat cfn-push-stats utility[1] and seems to suffer from the same fundamental problem, i.e. the need for injection of credentials (user/passwds, keys, whatever) into the VM in order to allow the metric datapoints be pushed up to the infrastructure layer (e.g. onto the AMQP bus, or to a REST API endpoint). How would you propose to solve that credentialing issue? My initial approximation would be to target use cases where end users do not have direct guest access or have limited guest access such that their UID / GID cannot access the conf file. For example instances which only provide app access provisioned using heat SoftwareDeployments (http://tinyurl.com/qxmh2of) or trove database instances. In general I don't see this approach from a security POV much different than whats done with the trove guest agent (http://tinyurl.com/ohvtmtz). Longer term perhaps credentials could be mitigated using Barbican as suggested here: https://bugs.launchpad.net/nova/+bug/1158328 Cheers, Eoghan [1] https://github.com/openstack/heat-cfntools/blob/master/bin/cfn-push-s tats
Re: [openstack-dev] [ceilometer] Compute agent local VM inspector - potential enhancement
Heat cfntools is based on SSH, so I assume it requires TCP/IP connectivity between VM and the central agent(or collector). But in the cloud, some networks are isolated from infrastructure layer network, because of security reasons. Some of our customers even explicitly require such security protection. Does it mean those isolated VMs cannot be monitored by this proposed-VM-agent? I really wish we can figur out how it could work for all VMs but with no security issues. I'm not familiar with heat-cfntools, so, correct me if I am wrong :) Best regards! Kurt -邮件原件- 发件人: Eoghan Glynn [mailto:egl...@redhat.com] 发送时间: 2014年8月1日 14:46 收件人: OpenStack Development Mailing List (not for usage questions) 主题: Re: [openstack-dev] [ceilometer] Compute agent local VM inspector - potential enhancement Disclaimer: I'm not fully vested on ceilometer internals, so bear with me. For consumers wanting to leverage ceilometer as a telemetry service atop non-OpenStack Clouds or infrastructure they don't own, some edge cases crop up. Most notably the consumer may not have access to the hypervisor host and therefore cannot leverage the ceilometer compute agent on a per host basis. Yes, currently such access to the hypervisor host is required, least in the case of the libvirt-based inspector. In such scenarios it's my understanding the main option is to employ the central agent to poll measurements from the monitored resources (VMs, etc.). Well, the ceilometer central agent is not generally concerned with with polling related *directly* to VMs - rather it handles acquiring data from RESTful API (glance, neutron etc.) that are not otherwise available in the form of notifications, and also from host-level interfaces such as SNMP. However this approach requires Cloud APIs (or other mechanisms) which allow the polling impl to obtain the desired measurements (VM memory, CPU, net stats, etc.) and moreover the polling approach has it's own set of pros / cons from a arch / topology perspective. Indeed. The other potential option is to setup the ceilometer compute agent within the VM and have each VM publish measurements to the collector -- a local VM agent / inspector if you will. With respect to this local VM agent approach: (a) I haven't seen this documented to date; is there any desire / reqs to support this topology? (b) If yes to #a, I whipped up a crude PoC here: http://tinyurl.com/pqjgotv Are folks willing to consider a BP for this approach? So in a sense this is similar to the Heat cfn-push-stats utility[1] and seems to suffer from the same fundamental problem, i.e. the need for injection of credentials (user/passwds, keys, whatever) into the VM in order to allow the metric datapoints be pushed up to the infrastructure layer (e.g. onto the AMQP bus, or to a REST API endpoint). How would you propose to solve that credentialing issue? Cheers, Eoghan [1] https://github.com/openstack/heat-cfntools/blob/master/bin/cfn-push-stats ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
