Public bug reported:
Description
===========
nova compute service is down.
caompute Unable to execute any server related requests.
Steps to reproduce
==================
not sure, this has happend 2wice in newly deployed VM on 5th-day-operation,
there is enough disk available
stack@devstack-jan:~/nova$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-lv--0 52G 19G 32G 37% /
Expected result
===============
Compute service should stay active or able to restart.
Placement should be able to update resource provider
Actual result
=============
Compute service status stays down
Compute service can't be restarted.
Resource provider inventory does not get updated
stack@devstack-jan:~/nova$ openstack resource provider inventory list
5cdfba85-3122-49ea-b7b1-1fd0af461588
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved |
step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| VCPU | 4.0 | 1 | 6 | 0 |
1 | 6 | 1 |
| MEMORY_MB | 1.0 | 1 | 8812 | 512 |
1 | 8812 | 2048 |
| DISK_GB | 1.0 | 1 | 29 | 0 |
1 | 29 | 20 |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
If we recreate resource provider, then new once does not have DISK_GB
resource-class
Environment
===========
Single node devstack from master branch
Logs & Configs
==============
Compute-logs
-----------------
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
[None req-de66825f-768a-4951-a57b-c92f9f035255 None None] Error starting
thread.: nova.exception.PlacementPciException: Failed to gather or report PCI
resources to Placement: Th
ere was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: update conflict: Inventory
for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in
use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Traceback (most recent call last):
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1360, in
_update_to_placement
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.reportclient.update_from_provider_tree(
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/scheduler/client/report.py", line 1502, in
update_from_provider_tree
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.set_inventory_for_provider(
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/scheduler/client/report.py", line 1005, in
set_inventory_for_provider
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise exception.InventoryInUse(err['detail'])
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
nova.exception.InventoryInUse: There was a conflict when trying to complete
your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
update conflict: Inventory for 'DISK_GB' on resource provider
'5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
During handling of the above exception, another exception occurred:
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Traceback (most recent call last):
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File
"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py",
line 810, in run_service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
service.start()
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/service.py", line 181, in start
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.manager.pre_start_hook(self.service_ref)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 1793, in pre_start_hook
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.update_available_resource(nova.context.get_admin_context(),
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 11227, in
update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update_available_resource_for_node(context, nodename,
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 11102, in
_update_available_resource_for_node
Jan 13 07:33:20 devstack-jan
nova-compute[65544]: ERROR oslo_service.service
self.rt.update_available_resource(context, nodename,
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 935, in
update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]:
ERROR oslo_service.service self._update_available_resource(context,
resources, startup=startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File
"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py",
line 412, in inner
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return f(*args, **kwargs)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1066, in
_update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update(context, cn, startup=startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1375, in _update
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update_to_placement(context, compute_node, startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 56,
in wrapped_f
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return Retrying(*dargs, **dkw).call(f, *args, **kw)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 257,
in call
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return attempt.get(self._wrap_exception)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 301,
in get
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
six.reraise(self.value[0], self.value[1], self.value[2])
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/six.py", line 724, in
reraise
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise value
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 251,
in call
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1370, in
_update_to_placement
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise exception.PlacementPciException(error=str(e))
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
nova.exception.PlacementPciException: Failed to gather or report PCI resources
to Placement: There was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
update conflict: Inventory for 'DISK_GB' on resource provider
'5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Placement-logs
---------------
Jan 13 08:35:00 devstack-jan [email protected][2127]: DEBUG
placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 None None] Starting request:
192.168.122.183 "PUT
/placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories"
{{(pid=2127) __call__ /opt/stack/placement/placement/requestlog.py:55}}
Jan 13 08:35:00 devstack-jan [email protected][2127]: DEBUG
placement.wsgi_wrapper [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] Placement API returning
an error response: update conflict: Inventory for 'DISK_GB' on resource
provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use. {{(pid=2127) call_func
/opt/stack/placement/placement/wsgi_wrapper.py:31}}
Jan 13 08:35:00 devstack-jan [email protected][2127]: INFO
placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] 192.168.122.183 "PUT
/placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories"
status: 409 len: 332 microversion: 1.26
** Affects: nova
Importance: High
Status: New
** Changed in: nova
Importance: Undecided => High
** Description changed:
-
Description
===========
nova compute service is down.
- Unable to execute any API request.
-
+ caompute Unable to execute any server related requests.
Steps to reproduce
==================
not sure, this has happend 2wice in newly deployed VM on 5th-day-operation,
there is enough disk available
stack@devstack-jan:~/nova$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-lv--0 52G 19G 32G 37% /
-
Expected result
===============
Compute service should stay active or able to restart.
Placement should be able to update resource provider
Actual result
=============
Compute service status stays down
Compute service can't be restarted.
Resource provider inventory does not get updated
- stack@devstack-jan:~/nova$ openstack resource provider inventory list
5cdfba85-3122-49ea-b7b1-1fd0af461588
+ stack@devstack-jan:~/nova$ openstack resource provider inventory list
5cdfba85-3122-49ea-b7b1-1fd0af461588
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved |
step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| VCPU | 4.0 | 1 | 6 | 0 |
1 | 6 | 1 |
| MEMORY_MB | 1.0 | 1 | 8812 | 512 |
1 | 8812 | 2048 |
| DISK_GB | 1.0 | 1 | 29 | 0 |
1 | 29 | 20 |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
If we recreate resource provider, then new once does not have DISK_GB
resource-class
-
Environment
===========
Single node devstack from master branch
Logs & Configs
==============
Compute-logs
-----------------
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
[None req-de66825f-768a-4951-a57b-c92f9f035255 None None] Error starting
thread.: nova.exception.PlacementPciException: Failed to gather or report PCI
resources to Placement: Th
- ere was a conflict when trying to complete your request.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: update conflict: Inventory
for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in
use.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Traceback (most recent call last):
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1360, in
_update_to_placement
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.reportclient.update_from_provider_tree(
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/scheduler/client/report.py", line 1502, in
update_from_provider_tree
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.set_inventory_for_provider(
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/scheduler/client/report.py", line 1005, in
set_inventory_for_provider
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise exception.InventoryInUse(err['detail'])
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
nova.exception.InventoryInUse: There was a conflict when trying to complete
your request.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
update conflict: Inventory for 'DISK_GB' on resource provider
'5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
During handling of the above exception, another exception occurred:
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Traceback (most recent call last):
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File
"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py",
line 810, in run_service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
service.start()
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/service.py", line 181, in start
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.manager.pre_start_hook(self.service_ref)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 1793, in pre_start_hook
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.update_available_resource(nova.context.get_admin_context(),
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 11227, in
update_available_resource
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update_available_resource_for_node(context, nodename,
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 11102, in
_update_available_resource_for_node
Jan 13 07:33:20 devstack-jan
nova-compute[65544]: ERROR oslo_service.service
self.rt.update_available_resource(context, nodename,
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 935, in
update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]:
ERROR oslo_service.service self._update_available_resource(context,
resources, startup=startup)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File
"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py",
line 412, in inner
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return f(*args, **kwargs)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1066, in
_update_available_resource
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update(context, cn, startup=startup)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1375, in _update
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update_to_placement(context, compute_node, startup)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 56,
in wrapped_f
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return Retrying(*dargs, **dkw).call(f, *args, **kw)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
257, in call
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return attempt.get(self._wrap_exception)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
301, in get
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
six.reraise(self.value[0], self.value[1], self.value[2])
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/six.py", line 724, in
reraise
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise value
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
251, in call
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1370, in
_update_to_placement
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise exception.PlacementPciException(error=str(e))
+ ere was a conflict when trying to complete your request.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: update conflict: Inventory
for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in
use.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Traceback (most recent call last):
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1360, in
_update_to_placement
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.reportclient.update_from_provider_tree(
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/scheduler/client/report.py", line 1502, in
update_from_provider_tree
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.set_inventory_for_provider(
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/scheduler/client/report.py", line 1005, in
set_inventory_for_provider
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise exception.InventoryInUse(err['detail'])
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
nova.exception.InventoryInUse: There was a conflict when trying to complete
your request.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
update conflict: Inventory for 'DISK_GB' on resource provider
'5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
During handling of the above exception, another exception occurred:
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Traceback (most recent call last):
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File
"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py",
line 810, in run_service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
service.start()
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/service.py", line 181, in start
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.manager.pre_start_hook(self.service_ref)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 1793, in pre_start_hook
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.update_available_resource(nova.context.get_admin_context(),
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 11227, in
update_available_resource
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update_available_resource_for_node(context, nodename,
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 11102, in
_update_available_resource_for_node
Jan 13 07:33:20 devstack-jan
nova-compute[65544]: ERROR oslo_service.service
self.rt.update_available_resource(context, nodename,
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 935, in
update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]:
ERROR oslo_service.service self._update_available_resource(context,
resources, startup=startup)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File
"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py",
line 412, in inner
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return f(*args, **kwargs)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1066, in
_update_available_resource
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update(context, cn, startup=startup)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1375, in _update
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update_to_placement(context, compute_node, startup)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 56,
in wrapped_f
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return Retrying(*dargs, **dkw).call(f, *args, **kw)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
257, in call
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return attempt.get(self._wrap_exception)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
301, in get
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
six.reraise(self.value[0], self.value[1], self.value[2])
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/six.py", line 724, in
reraise
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise value
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
251, in call
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1370, in
_update_to_placement
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise exception.PlacementPciException(error=str(e))
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
nova.exception.PlacementPciException: Failed to gather or report PCI resources
to Placement: There was a conflict when trying to complete your request.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
update conflict: Inventory for 'DISK_GB' on resource provider
'5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
-
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
update conflict: Inventory for 'DISK_GB' on resource provider
'5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Placement-logs
---------------
Jan 13 08:35:00 devstack-jan [email protected][2127]: DEBUG
placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 None None] Starting request:
192.168.122.183 "PUT
/placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories"
{{(pid=2127) __call__ /opt/stack/placement/placement/requestlog.py:55}}
Jan 13 08:35:00 devstack-jan [email protected][2127]: DEBUG
placement.wsgi_wrapper [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] Placement API returning
an error response: update conflict: Inventory for 'DISK_GB' on resource
provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use. {{(pid=2127) call_func
/opt/stack/placement/placement/wsgi_wrapper.py:31}}
Jan 13 08:35:00 devstack-jan [email protected][2127]: INFO
placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] 192.168.122.183 "PUT
/placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories"
status: 409 len: 332 microversion: 1.26
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2093869
Title:
update conflict: Inventory for 'DISK_GB' on resource provider 'r_p-id'
in use
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
nova compute service is down.
caompute Unable to execute any server related requests.
Steps to reproduce
==================
not sure, this has happend 2wice in newly deployed VM on 5th-day-operation,
there is enough disk available
stack@devstack-jan:~/nova$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-lv--0 52G 19G 32G 37% /
Expected result
===============
Compute service should stay active or able to restart.
Placement should be able to update resource provider
Actual result
=============
Compute service status stays down
Compute service can't be restarted.
Resource provider inventory does not get updated
stack@devstack-jan:~/nova$ openstack resource provider inventory list
5cdfba85-3122-49ea-b7b1-1fd0af461588
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved |
step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| VCPU | 4.0 | 1 | 6 | 0 |
1 | 6 | 1 |
| MEMORY_MB | 1.0 | 1 | 8812 | 512 |
1 | 8812 | 2048 |
| DISK_GB | 1.0 | 1 | 29 | 0 |
1 | 29 | 20 |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
If we recreate resource provider, then new once does not have DISK_GB
resource-class
Environment
===========
Single node devstack from master branch
Logs & Configs
==============
Compute-logs
-----------------
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
[None req-de66825f-768a-4951-a57b-c92f9f035255 None None] Error starting
thread.: nova.exception.PlacementPciException: Failed to gather or report PCI
resources to Placement: Th
ere was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: update conflict: Inventory
for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in
use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Traceback (most recent call last):
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1360, in
_update_to_placement
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.reportclient.update_from_provider_tree(
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/scheduler/client/report.py", line 1502, in
update_from_provider_tree
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.set_inventory_for_provider(
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/scheduler/client/report.py", line 1005, in
set_inventory_for_provider
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise exception.InventoryInUse(err['detail'])
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
nova.exception.InventoryInUse: There was a conflict when trying to complete
your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
update conflict: Inventory for 'DISK_GB' on resource provider
'5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
During handling of the above exception, another exception occurred:
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Traceback (most recent call last):
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File
"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py",
line 810, in run_service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
service.start()
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/service.py", line 181, in start
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.manager.pre_start_hook(self.service_ref)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 1793, in pre_start_hook
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self.update_available_resource(nova.context.get_admin_context(),
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 11227, in
update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update_available_resource_for_node(context, nodename,
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/manager.py", line 11102, in
_update_available_resource_for_node
Jan 13 07:33:20 devstack-jan
nova-compute[65544]: ERROR oslo_service.service
self.rt.update_available_resource(context, nodename,
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 935, in
update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]:
ERROR oslo_service.service self._update_available_resource(context,
resources, startup=startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File
"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py",
line 412, in inner
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return f(*args, **kwargs)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1066, in
_update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update(context, cn, startup=startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1375, in _update
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
self._update_to_placement(context, compute_node, startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 56,
in wrapped_f
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return Retrying(*dargs, **dkw).call(f, *args, **kw)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
257, in call
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
return attempt.get(self._wrap_exception)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
301, in get
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
six.reraise(self.value[0], self.value[1], self.value[2])
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/six.py", line 724, in
reraise
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise value
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line
251, in call
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1370, in
_update_to_placement
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
raise exception.PlacementPciException(error=str(e))
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
nova.exception.PlacementPciException: Failed to gather or report PCI resources
to Placement: There was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
update conflict: Inventory for 'DISK_GB' on resource provider
'5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Placement-logs
---------------
Jan 13 08:35:00 devstack-jan [email protected][2127]: DEBUG
placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 None None] Starting request:
192.168.122.183 "PUT
/placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories"
{{(pid=2127) __call__ /opt/stack/placement/placement/requestlog.py:55}}
Jan 13 08:35:00 devstack-jan [email protected][2127]: DEBUG
placement.wsgi_wrapper [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] Placement API returning
an error response: update conflict: Inventory for 'DISK_GB' on resource
provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use. {{(pid=2127) call_func
/opt/stack/placement/placement/wsgi_wrapper.py:31}}
Jan 13 08:35:00 devstack-jan [email protected][2127]: INFO
placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1
req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] 192.168.122.183 "PUT
/placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories"
status: 409 len: 332 microversion: 1.26
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2093869/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
