Public bug reported: Hi,
While developing a loadtest on Openstack, we noticed that whenever we continue to send a create request even though the quota is full, we cannot run the openstack server list command for the relevant project. root@ansible:/home/ansible# openstack server list --project project_1 HttpException: 500: Server Error for url: https://xxxxxxr:8774/v2.1/servers/detail?project_id=cebcf25ed92841c19071d97b3eb0a971&deleted=False&all_tenants=True, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'KeyError'> openstack server show 14acce64-d571-4fe1-ba24-238527bef5b4 +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hostname | test-instance-140477 | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | instance-00000607 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-076z19yn | | OS-EXT-SRV-ATTR:root_device_name | None | | | == | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | config_drive | | | created | 2025-04-22T15:07:52Z | | description | test-instance-140477 | | fault | {'code': 413, 'created': '2025-04-22T15:07:53Z', 'message': 'Quota exceeded for ram: Requested 2048, but already used 8192 of 8192 ram'} | | flavor | description=, disk='0', ephemeral='0', extra_specs.aggregate_instance_extra_specs:gpu='false', id='g1.small', is_disabled=, is_public='True', location=, | | | name='g1.small', original_name='g1.small', ram='2048', rxtx_factor=, swap='0', vcpus='1' | | hostId | nova-api.log 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] Unexpected exception in API method: KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4' 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/wsgi.py", line 678, in wrapped 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times] 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi response = self._view_builder.detail( 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 475, in detail 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers_dict = self._list_view(self.show, request, instances, 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 535, in _list_view 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi provided_az=req_specs_dict[server.uuid])["server"] 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ~~~~~~~~~~~~~~^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4' 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi 2025-04-22 15:08:16.729 24 INFO nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] HTTP except ion thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'KeyError'> ** Affects: nova Importance: Undecided Status: New ** Description changed: Hi, While developing a loadtest on Openstack, we noticed that whenever we continue to send a create request even though the quota is full, we cannot run the openstack server list command for the relevant project. - root@ansible:/home/ansible# openstack server list --project project_1 - HttpException: 500: Server Error for url: https://int-api-osdev.portvmind.com.tr:8774/v2.1/servers/detail?project_id=cebcf25ed92841c19071d97b3eb0a971&deleted=False&all_tenants=True, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. + HttpException: 500: Server Error for url: https://xxxxxxr:8774/v2.1/servers/detail?project_id=cebcf25ed92841c19071d97b3eb0a971&deleted=False&all_tenants=True, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'KeyError'> openstack server show 14acce64-d571-4fe1-ba24-238527bef5b4 +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hostname | test-instance-140477 | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | instance-00000607 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-076z19yn | | OS-EXT-SRV-ATTR:root_device_name | None | | | == | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | config_drive | | | created | 2025-04-22T15:07:52Z | | description | test-instance-140477 | | fault | {'code': 413, 'created': '2025-04-22T15:07:53Z', 'message': 'Quota exceeded for ram: Requested 2048, but already used 8192 of 8192 ram'} | | flavor | description=, disk='0', ephemeral='0', extra_specs.aggregate_instance_extra_specs:gpu='false', id='g1.small', is_disabled=, is_public='True', location=, | | | name='g1.small', original_name='g1.small', ram='2048', rxtx_factor=, swap='0', vcpus='1' | | hostId | - - nova-api.log 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] Unexpected - exception in API method: KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4' + exception in API method: KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4' 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/wsgi.py", line 678, in wrapped 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times] 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi response = self._view_builder.detail( 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 475, in detail 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers_dict = self._list_view(self.show, request, instances, 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 535, in _list_view 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi provided_az=req_specs_dict[server.uuid])["server"] 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ~~~~~~~~~~~~~~^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4' 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi 2025-04-22 15:08:16.729 24 INFO nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] HTTP except ion thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'KeyError'> -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2107968 Title: Openstack cli not working right when nova compute quota reached Status in OpenStack Compute (nova): New Bug description: Hi, While developing a loadtest on Openstack, we noticed that whenever we continue to send a create request even though the quota is full, we cannot run the openstack server list command for the relevant project. root@ansible:/home/ansible# openstack server list --project project_1 HttpException: 500: Server Error for url: https://xxxxxxr:8774/v2.1/servers/detail?project_id=cebcf25ed92841c19071d97b3eb0a971&deleted=False&all_tenants=True, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'KeyError'> openstack server show 14acce64-d571-4fe1-ba24-238527bef5b4 +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hostname | test-instance-140477 | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | instance-00000607 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-076z19yn | | OS-EXT-SRV-ATTR:root_device_name | None | | | == | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | config_drive | | | created | 2025-04-22T15:07:52Z | | description | test-instance-140477 | | fault | {'code': 413, 'created': '2025-04-22T15:07:53Z', 'message': 'Quota exceeded for ram: Requested 2048, but already used 8192 of 8192 ram'} | | flavor | description=, disk='0', ephemeral='0', extra_specs.aggregate_instance_extra_specs:gpu='false', id='g1.small', is_disabled=, is_public='True', location=, | | | name='g1.small', original_name='g1.small', ram='2048', rxtx_factor=, swap='0', vcpus='1' | | hostId | nova-api.log 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] Unexpected exception in API method: KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4' 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/wsgi.py", line 678, in wrapped 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times] 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True) 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi response = self._view_builder.detail( 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 475, in detail 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers_dict = self._list_view(self.show, request, instances, 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 535, in _list_view 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi provided_az=req_specs_dict[server.uuid])["server"] 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ~~~~~~~~~~~~~~^^^^^^^^^^^^^ 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4' 2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi 2025-04-22 15:08:16.729 24 INFO nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] HTTP except ion thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'KeyError'> To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2107968/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
