[Yahoo-eng-team] [Bug 1704975] Re: Nova API floating IP delete fails (race with concurrent server delete)
Okay, we will do it that way.
** Changed in: nova
Status: Confirmed => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1704975
Title:
Nova API floating IP delete fails (race with concurrent server delete)
Status in OpenStack Compute (nova):
Invalid
Bug description:
This happens with OpenStack Newton.
We use some outdated openstack clients, that do not provide the "floating ip
delete" command. Therefore we use the "ip floating delete" command.
The floating IP is allocated using the "ip floating create" command. The
floating IP is then associated to a newly started virtual machine.
After some automated tests, the command to delete the virtual machine is
sent. To keep the project clean, the floating IP is deleted afterwards using
the "ip floating delete" command. This command fails from time to time due to
some race condition.
This race condition seems to be the following. Nova API requests
information about the floating IP from neutron:
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5",
"status": "ACTIVE", "description": "", "updated_at":
"2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id":
"f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address":
"192.168.3.10", "floating_ip_address": "10.50.0.62",
"revision_number": 2, "port_id":
"4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id":
"7829521236b143d2a6778e09ba588ec0", "project_id":
"7829521236b143d2a6778e09ba588ec0"}}
In the response the floating IP still seems to be associated to the
virtual machine - but the request to delete the virtual machine was
already sent.
Nova API requests information about the port, too:
GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"NeutronError": {"message": "Port
4b082eae-a527-45e6-8603-0d0882098e39 could not be found.", "type":
"PortNotFound", "detail": ""}}
But the virtual machine and its floating IP association seems to be
deleted in the meantime. This makes Nova API fail to delete the
floating IP:
2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found
for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 404 len: 466 time: 0.2575810
openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server
[req-dafa9bec-fe87-47f6-8c44-4420b074da4d
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 200 len: 475 time: 0.2409530
2017-07-18 11:32:51.654 7171 INFO nova.osapi_compute.wsgi.server
[req-8f93afcb-40ff-4d8e-9f2f-c6fa860a8931
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "DELETE
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 202 len: 337 time: 0.7844548
GET
[Yahoo-eng-team] [Bug 1704975] Re: Nova API floating IP delete fails
Reopening as this still affects our test automation.
** Changed in: nova
Status: Expired => New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1704975
Title:
Nova API floating IP delete fails
Status in OpenStack Compute (nova):
New
Bug description:
This happens with OpenStack Newton.
We use some outdated openstack clients, that do not provide the "floating ip
delete" command. Therefore we use the "ip floating delete" command.
The floating IP is allocated using the "ip floating create" command. The
floating IP is then associated to a newly started virtual machine.
After some automated tests, the command to delete the virtual machine is
sent. To keep the project clean, the floating IP is deleted afterwards using
the "ip floating delete" command. This command fails from time to time due to
some race condition.
This race condition seems to be the following. Nova API requests
information about the floating IP from neutron:
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5",
"status": "ACTIVE", "description": "", "updated_at":
"2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id":
"f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address":
"192.168.3.10", "floating_ip_address": "10.50.0.62",
"revision_number": 2, "port_id":
"4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id":
"7829521236b143d2a6778e09ba588ec0", "project_id":
"7829521236b143d2a6778e09ba588ec0"}}
In the response the floating IP still seems to be associated to the
virtual machine - but the request to delete the virtual machine was
already sent.
Nova API requests information about the port, too:
GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"NeutronError": {"message": "Port
4b082eae-a527-45e6-8603-0d0882098e39 could not be found.", "type":
"PortNotFound", "detail": ""}}
But the virtual machine and its floating IP association seems to be
deleted in the meantime. This makes Nova API fail to delete the
floating IP:
2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found
for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 404 len: 466 time: 0.2575810
openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server
[req-dafa9bec-fe87-47f6-8c44-4420b074da4d
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 200 len: 475 time: 0.2409530
2017-07-18 11:32:51.654 7171 INFO nova.osapi_compute.wsgi.server
[req-8f93afcb-40ff-4d8e-9f2f-c6fa860a8931
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "DELETE
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 202 len: 337 time: 0.7844548
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host:
[Yahoo-eng-team] [Bug 1704975] Re: Nova API floating IP delete fails
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]
** Changed in: nova
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1704975
Title:
Nova API floating IP delete fails
Status in OpenStack Compute (nova):
Expired
Bug description:
This happens with OpenStack Newton.
We use some outdated openstack clients, that do not provide the "floating ip
delete" command. Therefore we use the "ip floating delete" command.
The floating IP is allocated using the "ip floating create" command. The
floating IP is then associated to a newly started virtual machine.
After some automated tests, the command to delete the virtual machine is
sent. To keep the project clean, the floating IP is deleted afterwards using
the "ip floating delete" command. This command fails from time to time due to
some race condition.
This race condition seems to be the following. Nova API requests
information about the floating IP from neutron:
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5",
"status": "ACTIVE", "description": "", "updated_at":
"2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id":
"f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address":
"192.168.3.10", "floating_ip_address": "10.50.0.62",
"revision_number": 2, "port_id":
"4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id":
"7829521236b143d2a6778e09ba588ec0", "project_id":
"7829521236b143d2a6778e09ba588ec0"}}
In the response the floating IP still seems to be associated to the
virtual machine - but the request to delete the virtual machine was
already sent.
Nova API requests information about the port, too:
GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"NeutronError": {"message": "Port
4b082eae-a527-45e6-8603-0d0882098e39 could not be found.", "type":
"PortNotFound", "detail": ""}}
But the virtual machine and its floating IP association seems to be
deleted in the meantime. This makes Nova API fail to delete the
floating IP:
2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found
for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 404 len: 466 time: 0.2575810
openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server
[req-dafa9bec-fe87-47f6-8c44-4420b074da4d
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 200 len: 475 time: 0.2409530
2017-07-18 11:32:51.654 7171 INFO nova.osapi_compute.wsgi.server
[req-8f93afcb-40ff-4d8e-9f2f-c6fa860a8931
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "DELETE
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 202 len: 337 time: 0.7844548
GET
[Yahoo-eng-team] [Bug 1704975] Re: Nova API floating IP delete fails
Reopen as this still happens.
The steps to reproduce are described in detail in the bug description.
Reading the floating IP from Neutron by Nova needs to happen while the
VM is still alive and Nova did not start to delete it. The following
request to delete the floating IP needs to happen after Nova removed the
VM.
This happens for us in production using automated tests with Kitchen.
kitchen destroy schedules the deletion of the VM in Nova. After kitchen
destroy we issue the delete command to the floating IP using old
openstack clients still requesting this to the Nova daemon and not to
the Neutron daemon.
Maybe this is easily reproducible if you add breakpoints in Nova between
fetching the floating IP information from Neutron and before sending the
delete request. The other breakpoint needs to be right before the VM in
Nova is destroyed. Steps to pass the breakpoints have now been described
multiple times.
** Changed in: nova
Status: Expired => Incomplete
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1704975
Title:
Nova API floating IP delete fails
Status in OpenStack Compute (nova):
Incomplete
Bug description:
This happens with OpenStack Newton.
We use some outdated openstack clients, that do not provide the "floating ip
delete" command. Therefore we use the "ip floating delete" command.
The floating IP is allocated using the "ip floating create" command. The
floating IP is then associated to a newly started virtual machine.
After some automated tests, the command to delete the virtual machine is
sent. To keep the project clean, the floating IP is deleted afterwards using
the "ip floating delete" command. This command fails from time to time due to
some race condition.
This race condition seems to be the following. Nova API requests
information about the floating IP from neutron:
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5",
"status": "ACTIVE", "description": "", "updated_at":
"2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id":
"f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address":
"192.168.3.10", "floating_ip_address": "10.50.0.62",
"revision_number": 2, "port_id":
"4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id":
"7829521236b143d2a6778e09ba588ec0", "project_id":
"7829521236b143d2a6778e09ba588ec0"}}
In the response the floating IP still seems to be associated to the
virtual machine - but the request to delete the virtual machine was
already sent.
Nova API requests information about the port, too:
GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"NeutronError": {"message": "Port
4b082eae-a527-45e6-8603-0d0882098e39 could not be found.", "type":
"PortNotFound", "detail": ""}}
But the virtual machine and its floating IP association seems to be
deleted in the meantime. This makes Nova API fail to delete the
floating IP:
2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found
for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 404 len: 466 time: 0.2575810
openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server
[Yahoo-eng-team] [Bug 1704975] Re: Nova API floating IP delete fails
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]
** Changed in: nova
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1704975
Title:
Nova API floating IP delete fails
Status in OpenStack Compute (nova):
Expired
Bug description:
This happens with OpenStack Newton.
We use some outdated openstack clients, that do not provide the "floating ip
delete" command. Therefore we use the "ip floating delete" command.
The floating IP is allocated using the "ip floating create" command. The
floating IP is then associated to a newly started virtual machine.
After some automated tests, the command to delete the virtual machine is
sent. To keep the project clean, the floating IP is deleted afterwards using
the "ip floating delete" command. This command fails from time to time due to
some race condition.
This race condition seems to be the following. Nova API requests
information about the floating IP from neutron:
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5",
"status": "ACTIVE", "description": "", "updated_at":
"2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id":
"f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address":
"192.168.3.10", "floating_ip_address": "10.50.0.62",
"revision_number": 2, "port_id":
"4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id":
"7829521236b143d2a6778e09ba588ec0", "project_id":
"7829521236b143d2a6778e09ba588ec0"}}
In the response the floating IP still seems to be associated to the
virtual machine - but the request to delete the virtual machine was
already sent.
Nova API requests information about the port, too:
GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"NeutronError": {"message": "Port
4b082eae-a527-45e6-8603-0d0882098e39 could not be found.", "type":
"PortNotFound", "detail": ""}}
But the virtual machine and its floating IP association seems to be
deleted in the meantime. This makes Nova API fail to delete the
floating IP:
2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found
for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 404 len: 466 time: 0.2575810
openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server
[req-dafa9bec-fe87-47f6-8c44-4420b074da4d
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 200 len: 475 time: 0.2409530
2017-07-18 11:32:51.654 7171 INFO nova.osapi_compute.wsgi.server
[req-8f93afcb-40ff-4d8e-9f2f-c6fa860a8931
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "DELETE
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 202 len: 337 time: 0.7844548
GET
