Public bug reported:
When running rally scenarios the test fails with PortNotFoundClient exception.
>From the logs we see the following:
Failure:
-------
▼ 11 GetResourceFailure Failed to get the resource <Server:
s_rally_41596539_fMbobPdS>: The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID:
req-a9471015-e72e-42cf-acc3-8b361415d91e)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/rally/task/runner.py", line 71,
in _run_scenario_once
getattr(scenario_inst, method_name)(**scenario_kwargs)
File
"/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/security_group.py",
line 167, in run
self._delete_server(server)
File
"/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/utils.py",
line 451, in _delete_server
check_interval=CONF.openstack.nova_server_delete_poll_interval
File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line 214,
in wait_for_status
resource = update_resource(resource)
File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line 80,
in _get_from_manager
raise exceptions.GetResourceFailure(resource=resource, err=e)
GetResourceFailure: Failed to get the resource <Server:
s_rally_41596539_fMbobPdS>: The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID:
req-a9471015-e72e-42cf-acc3-8b361415d91e)
Neutron:
-------
port-creation:
2018-04-22 09:18:31.984 32139 DEBUG vmware_nsxlib.v3.client
[req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default] REST call: POST
api/v1/logical-ports. Headers: {'X-NSX-EUSER':
u'f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default
default', 'Content-Type': 'application/json', 'X-NSX-EREQID':
'req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b', 'Accept': 'application/json'}.
Body: {"address_bindings": [{"ip_address": "100.1.124.13", "mac_address":
"fa:16:3e:d4:3a:8f"}], "admin_state": "UP", "attachment": null, "description":
"", "display_name": "", "logical_switch_id":
"dfc6fa69-6df7-44db-ba43-68152104103f", "switching_profile_ids": [{"key":
"SpoofGuardSwitchingProfile", "value":
"0bb5dd95-1e3d-464e-a9d5-bd040bd91d9d"}], "tags": [{"scope":
"os-neutron-port-id", "tag": "1954b052-bc2c-48ba-aa24-e8c96fac1723"}, {"scope":
"os-project-id", "tag": "c87b5a2bc4e649069e9cfa4bbc874663"}, {"scope":
"os-project-name", "tag": "c_rally_41596539_tkcSTCuv"}, {"scope":
"os-api-version", "tag": "12.0.0.dev8294691"}, {"scope": "os-security-group",
"tag": "a4d155d5-285a-4e9a-8982-04f6a242d3ed"}, {"scope": "os-security-group",
"tag": "OS-Default-Section"}]} _rest_call
/usr/lib/python2.7/dist-packages/vmware_nsxlib/v3/client.py:201
.
.
2018-04-22 09:18:32.278 32139 INFO neutron.wsgi
[req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
10.127.108.239,10.127.108.237 "POST /v2.0/ports HTTP/1.1" status: 201 len: 983
time: 2.8035491
port-deletion:
2018-04-22 09:18:55.552 32135 INFO neutron.wsgi [req-774309da-
6fc6-4521-aed2-37f2eb01b5b4 f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
10.127.108.239,10.127.108.237 "DELETE /v2.0/ports/1954b052-bc2c-48ba-
aa24-e8c96fac1723 HTTP/1.1" status: 204 len: 168 time: 2.4448841
Nova:
----
2018-04-22 09:18:55.539 19221 DEBUG oslo_concurrency.lockutils
[req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f
69fad6d3240645b5854b52bca7e9b85e - default default] Lock
"9ab78fdd-5798-4426-baf2-2706d0d69345" acqu
ired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited
0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:276
2018-04-22 09:18:55.541 19221 DEBUG oslo_concurrency.lockutils
[req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f
69fad6d3240645b5854b52bca7e9b85e - default default] Lock
"9ab78fdd-5798-4426-baf2-2706d0d69345" rele
ased by "nova.context.get_or_set_cached_cell_and_set_connections" :: held
0.002s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:288
2018-04-22 09:18:55.662 19221 DEBUG nova.policy
[req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f
69fad6d3240645b5854b52bca7e9b85e - default default] Policy check for
os_compute_api:os-extended-server-attributes f
ailed with credentials {'service_roles': [], 'user_id':
u'8d478ca1fa274601a50f8d2a38c5e70f', 'roles': [u'_member_'], 'user_domain_id':
u'default', 'service_project_id': None, 'service_user_id': None,
'service_user_domain_id': None, 'servi
ce_project_domain_id': None, 'is_admin_project': False, 'is_admin': False,
'project_id': u'69fad6d3240645b5854b52bca7e9b85e', 'project_domain_id':
u'default'} authorize /usr/lib/python2.7/dist-packages/nova/policy.py:168
2018-04-22 09:18:55.774 19225 DEBUG neutronclient.v2_0.client
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default] Error message:
{"NeutronError": {"message": "Port
1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.", "type":
"PortNotFound", "detail": ""}} _handle_fault_response
/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:259
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default] Caught error: Port
1954b052-bc2c-48ba-aa24-e8c96fac1723
could not be found.
Neutron server returns request_ids:
['req-35436010-4faa-42de-ab84-a783f48b4f92']: PortNotFoundClient: Port
1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Traceback (most recent
call last):
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 82, in
__call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
req.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application,
catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in
call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter =
application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/requestlog.py", line 92,
in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack self._log_req(req,
res, start)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack self.force_reraise()
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
six.reraise(self.type_, self.value, self.tb)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/requestlog.py", line 87,
in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack res =
req.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application,
catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in
call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter =
application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return resp(environ,
start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 112, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
request.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application,
catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in
call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter =
application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py",
line 337, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack response =
req.get_response(self._app)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application,
catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in
call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter =
application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return resp(environ,
start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return resp(environ,
start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack response =
self.app(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return resp(environ,
start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 580, in
__call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack content_type, body,
accept)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 654, in
_process_stack
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack request, action_args)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 526, in
process_extensions
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack **action_args)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py",
line 534, in show
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self._show(req, resp_obj)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py",
line 530, in _show
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
self._extend_servers(req, [resp_obj.obj['server']])
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py",
line 512, in _extend_servers
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack servers))
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py",
line 376, in get_instances_security_groups_bindings
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ports =
self._get_ports_from_server_list(servers, neutron)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py",
line 335, in _get_ports_from_server_list
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
ports.extend(neutron.list_ports(**search_opts).get('ports'))
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 786, in
list_ports
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack **_params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 369, in
list
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack for r in
self._pagination(collection, path, **params):
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 384, in
_pagination
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack res = self.get(path,
params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in
get
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack headers=headers,
params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in
retry_request
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack headers=headers,
params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 294, in
do_request
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
self._handle_fault_response(status_code, replybody, resp)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 269, in
_handle_fault_response
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
exception_handler_v20(status_code, error_body)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 93, in
exception_handler_v20
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
request_ids=request_ids)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack PortNotFoundClient: Port
1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Neutron server returns
request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
2018-04-22 09:18:55.819 19225 INFO nova.api.openstack
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
http://192.168.20.101:8774/v2.1/c87b5a2bc4e649069e9cfa4bbc874663/servers/3e714b48-5193-4b13-b6c9-3374488b2e50
returned with HTTP 500
2018-04-22 09:18:55.822 19225 DEBUG nova.api.openstack.wsgi
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default] Returning 500 to user: The
server has either erred or is incapable of performing the requested operation.
__call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1064
At a deeper looks we see:
After looking more into the logs and the flow, here are the relevant log lines
that explain the issue, and indicate this is an exception not being handled in
Nova (see here
https://github.com/openstack/nova/blob/master/nova/network/security_group/neutron_driver.py#L378),
note the timestamps:
Port ID: 1954b052-bc2c-48ba-aa24-e8c96fac1723
Nova instance UUID: 3e714b48-5193-4b13-b6c9-3374488b2e50
Neutron logs
=============
Port deletion:
2018-04-22 09:18:55.552 32135 INFO neutron.wsgi
[req-774309da-6fc6-4521-aed2-37f2eb01b5b4 f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
10.127.108.239,10.127.108.237 "DELETE
/v2.0/ports/1954b052-bc2c-48ba-aa24-e8c96fac1723 HTTP/1.1" status: 204 len:
168 time: 2.4448841
GET ports failure:
2018-04-22 09:18:55.770 31303 INFO neutron.wsgi
[req-35436010-4faa-42de-ab84-a783f48b4f92 f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
10.127.108.236,10.127.108.237 "GET
/v2.0/ports?device_id=3e714b48-5193-4b13-b6c9-3374488b2e50 HTTP/1.1" status:
404 len: 333 time: 0.5505300
Nova log
========
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack PortNotFoundClient: Port
1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Neutron server returns
request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
2018-04-22 09:18:55.819 19225 INFO nova.api.openstack
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
http://192.168.20.101:8774/v2.1/c87b5a2bc4e649069e9cfa4bbc874663/servers/3e714b48-5193-4b13-b6c9-3374488b2e50
returned with HTTP 500
If there is a exception in self._get_secgroups_from_port_list should the
security groups returned be []? Or a better exception raised. This is due to a
race in the request in the retrieing the security groups and the deletion of
the instance. The instance does not exist at this stage...
The problem is discovered in stable queens
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1767058
Title:
Rally test boot_and_delete_server_with_secgroups fails
Status in OpenStack Compute (nova):
New
Bug description:
When running rally scenarios the test fails with PortNotFoundClient exception.
From the logs we see the following:
Failure:
-------
▼ 11 GetResourceFailure Failed to get the resource <Server:
s_rally_41596539_fMbobPdS>: The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID:
req-a9471015-e72e-42cf-acc3-8b361415d91e)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/rally/task/runner.py", line
71, in _run_scenario_once
getattr(scenario_inst, method_name)(**scenario_kwargs)
File
"/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/security_group.py",
line 167, in run
self._delete_server(server)
File
"/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/utils.py",
line 451, in _delete_server
check_interval=CONF.openstack.nova_server_delete_poll_interval
File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line
214, in wait_for_status
resource = update_resource(resource)
File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line 80,
in _get_from_manager
raise exceptions.GetResourceFailure(resource=resource, err=e)
GetResourceFailure: Failed to get the resource <Server:
s_rally_41596539_fMbobPdS>: The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID:
req-a9471015-e72e-42cf-acc3-8b361415d91e)
Neutron:
-------
port-creation:
2018-04-22 09:18:31.984 32139 DEBUG vmware_nsxlib.v3.client
[req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default] REST call: POST
api/v1/logical-ports. Headers: {'X-NSX-EUSER':
u'f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default
default', 'Content-Type': 'application/json', 'X-NSX-EREQID':
'req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b', 'Accept': 'application/json'}.
Body: {"address_bindings": [{"ip_address": "100.1.124.13", "mac_address":
"fa:16:3e:d4:3a:8f"}], "admin_state": "UP", "attachment": null, "description":
"", "display_name": "", "logical_switch_id":
"dfc6fa69-6df7-44db-ba43-68152104103f", "switching_profile_ids": [{"key":
"SpoofGuardSwitchingProfile", "value":
"0bb5dd95-1e3d-464e-a9d5-bd040bd91d9d"}], "tags": [{"scope":
"os-neutron-port-id", "tag": "1954b052-bc2c-48ba-aa24-e8c96fac1723"}, {"scope":
"os-project-id", "tag": "c87b5a2bc4e649069e9cfa4bbc874663"}, {"scope":
"os-project-name", "tag": "c_rally_41596539_tkcSTCuv"}, {"scope":
"os-api-version", "tag": "12.0.0.dev8294691"}, {"scope": "os-security-group",
"tag": "a4d155d5-285a-4e9a-8982-04f6a242d3ed"}, {"scope": "os-security-group",
"tag": "OS-Default-Section"}]} _rest_call
/usr/lib/python2.7/dist-packages/vmware_nsxlib/v3/client.py:201
.
.
2018-04-22 09:18:32.278 32139 INFO neutron.wsgi
[req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
10.127.108.239,10.127.108.237 "POST /v2.0/ports HTTP/1.1" status: 201 len: 983
time: 2.8035491
port-deletion:
2018-04-22 09:18:55.552 32135 INFO neutron.wsgi [req-774309da-
6fc6-4521-aed2-37f2eb01b5b4 f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
10.127.108.239,10.127.108.237 "DELETE /v2.0/ports/1954b052-bc2c-48ba-
aa24-e8c96fac1723 HTTP/1.1" status: 204 len: 168 time: 2.4448841
Nova:
----
2018-04-22 09:18:55.539 19221 DEBUG oslo_concurrency.lockutils
[req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f
69fad6d3240645b5854b52bca7e9b85e - default default] Lock
"9ab78fdd-5798-4426-baf2-2706d0d69345" acqu
ired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited
0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:276
2018-04-22 09:18:55.541 19221 DEBUG oslo_concurrency.lockutils
[req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f
69fad6d3240645b5854b52bca7e9b85e - default default] Lock
"9ab78fdd-5798-4426-baf2-2706d0d69345" rele
ased by "nova.context.get_or_set_cached_cell_and_set_connections" :: held
0.002s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:288
2018-04-22 09:18:55.662 19221 DEBUG nova.policy
[req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f
69fad6d3240645b5854b52bca7e9b85e - default default] Policy check for
os_compute_api:os-extended-server-attributes f
ailed with credentials {'service_roles': [], 'user_id':
u'8d478ca1fa274601a50f8d2a38c5e70f', 'roles': [u'_member_'], 'user_domain_id':
u'default', 'service_project_id': None, 'service_user_id': None,
'service_user_domain_id': None, 'servi
ce_project_domain_id': None, 'is_admin_project': False, 'is_admin': False,
'project_id': u'69fad6d3240645b5854b52bca7e9b85e', 'project_domain_id':
u'default'} authorize /usr/lib/python2.7/dist-packages/nova/policy.py:168
2018-04-22 09:18:55.774 19225 DEBUG neutronclient.v2_0.client
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default] Error message:
{"NeutronError": {"message": "Port
1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.", "type":
"PortNotFound", "detail": ""}} _handle_fault_response
/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:259
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default] Caught error: Port
1954b052-bc2c-48ba-aa24-e8c96fac1723
could not be found.
Neutron server returns request_ids:
['req-35436010-4faa-42de-ab84-a783f48b4f92']: PortNotFoundClient: Port
1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
Neutron server returns request_ids:
['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Traceback (most recent
call last):
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 82, in
__call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
req.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application,
catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in
call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter =
application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/requestlog.py", line 92,
in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack self._log_req(req,
res, start)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
self.force_reraise()
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
six.reraise(self.type_, self.value, self.tb)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/requestlog.py", line 87,
in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack res =
req.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application,
catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in
call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter =
application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
resp(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 112, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
request.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application,
catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in
call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter =
application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py",
line 337, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack response =
req.get_response(self._app)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application,
catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in
call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter =
application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
resp(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
resp(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack response =
self.app(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
resp(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 580, in
__call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack content_type,
body, accept)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 654, in
_process_stack
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack request,
action_args)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 526, in
process_extensions
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack **action_args)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py",
line 534, in show
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return
self._show(req, resp_obj)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py",
line 530, in _show
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
self._extend_servers(req, [resp_obj.obj['server']])
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py",
line 512, in _extend_servers
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack servers))
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py",
line 376, in get_instances_security_groups_bindings
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ports =
self._get_ports_from_server_list(servers, neutron)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py",
line 335, in _get_ports_from_server_list
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
ports.extend(neutron.list_ports(**search_opts).get('ports'))
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 786, in
list_ports
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack **_params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 369, in
list
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack for r in
self._pagination(collection, path, **params):
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 384, in
_pagination
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack res =
self.get(path, params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in
get
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack headers=headers,
params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in
retry_request
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack headers=headers,
params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 294, in
do_request
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
self._handle_fault_response(status_code, replybody, resp)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in
wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args,
**kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 269, in
_handle_fault_response
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
exception_handler_v20(status_code, error_body)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 93, in
exception_handler_v20
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
request_ids=request_ids)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack PortNotFoundClient:
Port 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Neutron server returns
request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
2018-04-22 09:18:55.819 19225 INFO nova.api.openstack
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
http://192.168.20.101:8774/v2.1/c87b5a2bc4e649069e9cfa4bbc874663/servers/3e714b48-5193-4b13-b6c9-3374488b2e50
returned with HTTP 500
2018-04-22 09:18:55.822 19225 DEBUG nova.api.openstack.wsgi
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default] Returning 500 to user: The
server has either erred or is incapable of performing the requested operation.
__call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1064
At a deeper looks we see:
After looking more into the logs and the flow, here are the relevant log
lines that explain the issue, and indicate this is an exception not being
handled in Nova (see here
https://github.com/openstack/nova/blob/master/nova/network/security_group/neutron_driver.py#L378),
note the timestamps:
Port ID: 1954b052-bc2c-48ba-aa24-e8c96fac1723
Nova instance UUID: 3e714b48-5193-4b13-b6c9-3374488b2e50
Neutron logs
=============
Port deletion:
2018-04-22 09:18:55.552 32135 INFO neutron.wsgi
[req-774309da-6fc6-4521-aed2-37f2eb01b5b4 f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
10.127.108.239,10.127.108.237 "DELETE
/v2.0/ports/1954b052-bc2c-48ba-aa24-e8c96fac1723 HTTP/1.1" status: 204 len:
168 time: 2.4448841
GET ports failure:
2018-04-22 09:18:55.770 31303 INFO neutron.wsgi
[req-35436010-4faa-42de-ab84-a783f48b4f92 f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
10.127.108.236,10.127.108.237 "GET
/v2.0/ports?device_id=3e714b48-5193-4b13-b6c9-3374488b2e50 HTTP/1.1" status:
404 len: 333 time: 0.5505300
Nova log
========
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack PortNotFoundClient:
Port 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Neutron server returns
request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
2018-04-22 09:18:55.819 19225 INFO nova.api.openstack
[req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b
c87b5a2bc4e649069e9cfa4bbc874663 - default default]
http://192.168.20.101:8774/v2.1/c87b5a2bc4e649069e9cfa4bbc874663/servers/3e714b48-5193-4b13-b6c9-3374488b2e50
returned with HTTP 500
If there is a exception in self._get_secgroups_from_port_list should the
security groups returned be []? Or a better exception raised. This is due to a
race in the request in the retrieing the security groups and the deletion of
the instance. The instance does not exist at this stage...
The problem is discovered in stable queens
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1767058/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
