[Bug 1971753] [NEW] New resources are unable to be created in gnocchi after upgrade to Yoga
Public bug reported:
After upgrade from Ussuri to Yoga, ceilometer doesn't work properly.
Metric measures are generated for old servers. But for new created
servers, metric measures are unable to be published to gnocchi.
The issue happens in batch_measures function:
def batch_measures(self, measures, resource_infos):
# NOTE(sileht): We don't care about error here, we want
# resources metadata always been updated
try:
self._gnocchi.metric.batch_resources_metrics_measures(
measures, create_metrics=True)
except gnocchi_exc.BadRequest as e:
if not isinstance(e.message, dict):
LOG.debug("Tristan: if not isinstance ")
raise
if e.message.get('cause') != 'Unknown resources':
LOG.debug("Tristan: if cause != Unknown resources ")
raise
Here we test if e.message is a dict. In my case ceilometer gets a message with
a dict. But for some reason the character "'" in the exception message from
gnocchi-api is converted to "", that breaks this part of code.
Errors in notification logs:
2022-05-04 17:02:45.762 20 ERROR ceilometer.publisher.gnocchi [-]
400 Bad Request
400 Bad Request
The server could not comply with the request since it is either malformed or
otherwise incorrect.
{cause: Unknown resources, detail:
[{resource_id: 23c43227-0f8c-5da1-b367-9abd5be48a00,
original_resource_id:36f2c1be-50b6-4b58-b91f-b9153e28855a-hda},
{resource_id: e8e
033d7-f774-5026-a0d2-a06153b99ec3, original_resource_id:
36f2c1be-50b6-4b58-b91f-b9153e28855a-vda},
{resource_id: 87feb8f9-cff5-5118-9e85-ec3747111cfc,
original_resource_id:
67171b6e-1147-4084-8ba3-ed352d4f4238-vda}]}
** Affects: ceilometer (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1971753
Title:
New resources are unable to be created in gnocchi after upgrade to
Yoga
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceilometer/+bug/1971753/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1946793] Re: aodh uses deprecated gnocchi api to aggregate metrics and doesn't work properly
@James I don't think there are any errors in logs. The deprecated api works, it just doesn't give us a proper aggregation metric. You can try both these API with openstack commands as I pasted above. # openstack metric aggregates # openstack metric measures aggregation For only 1 running instance, they give the same value. But for 2 instances, they value are different, and the one from old method is wrong. Or you can just call the old and new gnocchi api with a curl command, you will see the difference. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1946793 Title: aodh uses deprecated gnocchi api to aggregate metrics and doesn't work properly To manage notifications about this bug go to: https://bugs.launchpad.net/aodh/+bug/1946793/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833526] Re: An instance libvirtError will add resources to the blacklist
We met the same issue. Sometimes ceilometer gets an exception caused by libvirt, For example, 1. qemuMonitorIORead:610 : Unable to read from monitor: Connection reset by peer 2. virNetSocketReadWire:1811 : End of file while reading data: Input/output error Most time these are just random and temporary issues, not fatal and permanent. If ceilometer tries again the next minute, it can get data from libvirt. But since ceilometer add the group of resources to blacklist, it will not poll data from libvirt anymore. The only way to make it work again is to restart ceilometer-compute agent. Below is the code causing issues, ceilometer/polling/manager.py ``` except plugin_base.PollsterPermanentError as err: LOG.error( 'Prevent pollster %(name)s from ' 'polling %(res_list)s on source %(source)s anymore!', dict(name=pollster.name, res_list=str(err.fail_res_list), source=source_name)) self.resources[key].blacklist.extend(err.fail_res_list) ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833526 Title: An instance libvirtError will add resources to the blacklist To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceilometer/+bug/1833526/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
