The HostState.failed_builds change was introduced in Rocky and backported through to Pike:
https://review.opendev.org/#/q/I71c56fe770f8c3f66db97fa542fdfdf2b9865fb8 ** Tags added: scheduler ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/pike Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Changed in: nova/pike Status: New => Triaged ** Changed in: nova/stein Importance: Undecided => Medium ** Changed in: nova/queens Status: New => Triaged ** Changed in: nova/queens Importance: Undecided => Medium ** Changed in: nova/rocky Importance: Undecided => Medium ** Changed in: nova/pike Importance: Undecided => Medium ** Changed in: nova/rocky Status: New => Triaged ** Changed in: nova/stein Status: New => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1834691 Title: AttributeError: 'HostState' object has no attribute 'failed_builds' Status in OpenStack Compute (nova): Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Compute (nova) queens series: Triaged Status in OpenStack Compute (nova) rocky series: Triaged Status in OpenStack Compute (nova) stein series: Triaged Bug description: I'm not sure how this happens, but I hit this in a devstack environment with 35 nova-compute services using the fake virt driver, one scheduler worker and then trying to schedule a server: Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts = self._get_sorted_hosts(spec_obj, hosts, num) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server spec_obj) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts, spec_obj) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weights = weigher.weigh_objects(weighed_objs, weighing_properties) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weight = self._weigh_object(obj.obj, weight_properties) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return host_state.failed_builds Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds' Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server The HostState.failed_builds attribute gets set outside of __init__ in _update_from_compute_node here: https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261 And I'd expect the _update_from_compute_node to get called from update() which is called from here: https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787 Ah bingo if the compute node hasn't had resource usage updated yet we don't set the failed_builds attribute: https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1834691/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp