Reopen as this still happens. The steps to reproduce are described in detail in the bug description. Reading the floating IP from Neutron by Nova needs to happen while the VM is still alive and Nova did not start to delete it. The following request to delete the floating IP needs to happen after Nova removed the VM.
This happens for us in production using automated tests with Kitchen. kitchen destroy schedules the deletion of the VM in Nova. After kitchen destroy we issue the delete command to the floating IP using old openstack clients still requesting this to the Nova daemon and not to the Neutron daemon. Maybe this is easily reproducible if you add breakpoints in Nova between fetching the floating IP information from Neutron and before sending the delete request. The other breakpoint needs to be right before the VM in Nova is destroyed. Steps to pass the breakpoints have now been described multiple times. ** Changed in: nova Status: Expired => Incomplete -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1704975 Title: Nova API floating IP delete fails Status in OpenStack Compute (nova): Incomplete Bug description: This happens with OpenStack Newton. We use some outdated openstack clients, that do not provide the "floating ip delete" command. Therefore we use the "ip floating delete" command. The floating IP is allocated using the "ip floating create" command. The floating IP is then associated to a newly started virtual machine. After some automated tests, the command to delete the virtual machine is sent. To keep the project clean, the floating IP is deleted afterwards using the "ip floating delete" command. This command fails from time to time due to some race condition. This race condition seems to be the following. Nova API requests information about the floating IP from neutron: GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1 Host: controller:9696 Connection: keep-alive Accept-Encoding: gzip, deflate Accept: application/json User-Agent: python-neutronclient X-Auth-Token: gAAAAABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8 {"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5", "status": "ACTIVE", "description": "", "updated_at": "2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id": "f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address": "192.168.3.10", "floating_ip_address": "10.50.0.62", "revision_number": 2, "port_id": "4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e- ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at": "2017-07-18T03:37:04Z", "tenant_id": "7829521236b143d2a6778e09ba588ec0", "project_id": "7829521236b143d2a6778e09ba588ec0"}} In the response the floating IP still seems to be associated to the virtual machine - but the request to delete the virtual machine was already sent. Nova API requests information about the port, too: GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1 Host: controller:9696 Connection: keep-alive Accept-Encoding: gzip, deflate Accept: application/json User-Agent: python-neutronclient X-Auth-Token: gAAAAABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8 {"NeutronError": {"message": "Port 4b082eae-a527-45e6-8603-0d0882098e39 could not be found.", "type": "PortNotFound", "detail": ""}} But the virtual machine and its floating IP association seems to be deleted in the meantime. This makes Nova API fail to delete the floating IP: 2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi [req-b8f7414b-6398-4738-b660-fac693a187ab 807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173 7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141 27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6 2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server [req-b8f7414b-6398-4738-b660-fac693a187ab 807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173 7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141 27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET /v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6 HTTP/1.1" status: 404 len: 466 time: 0.2575810 openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6 2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server [req-dafa9bec-fe87-47f6-8c44-4420b074da4d 807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173 7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141 27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET /v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6 HTTP/1.1" status: 200 len: 475 time: 0.2409530 2017-07-18 11:32:51.654 7171 INFO nova.osapi_compute.wsgi.server [req-8f93afcb-40ff-4d8e-9f2f-c6fa860a8931 807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173 7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141 27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "DELETE /v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6 HTTP/1.1" status: 202 len: 337 time: 0.7844548 GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1 Host: controller:9696 Connection: keep-alive Accept-Encoding: gzip, deflate Accept: application/json User-Agent: python-neutronclient X-Auth-Token: gAAAAABZbdXC9iHhy48rN_gmdR8mrhlJy9d5sLLMQS6Vj79oxcaa23yX6xzKzKix9Fqbd7vo-5jH_JW1IzaCDxp7HO0tIfHX1sc1-I0vLCZq7_w5_Ga50-k1Ra_-9_adi1GzqxKRIjE7Gfp_Fe7z4pIn1XNgur99McLgBaoBP4cyJ0nPoC05p0QSV1oMj8Yjb6dNrdCtmjSQNt_Els7ELq_9SMaSVN3zGi4BLlddAY9TcDqTop0yu-A {"floatingip": {"router_id": null, "status": "DOWN", "description": "", "updated_at": "2017-07-18T03:38:41Z", "dns_domain": "", "floating_network_id": "f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address": null, "floating_ip_address": "10.50.0.62", "revision_number": 3, "port_id": null, "id": "3aeb2a7e- ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at": "2017-07-18T03:37:04Z", "tenant_id": "7829521236b143d2a6778e09ba588ec0", "project_id": "7829521236b143d2a6778e09ba588ec0"}} - some other requests to networks and the IP address 10.50.0.62 DELETE /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1 Host: controller:9696 Connection: keep-alive Accept-Encoding: gzip, deflate Accept: application/json User-Agent: python-neutronclient X-Auth-Token: gAAAAABZbdXC9iHhy48rN_gmdR8mrhlJy9d5sLLMQS6Vj79oxcaa23yX6xzKzKix9Fqbd7vo-5jH_JW1IzaCDxp7HO0tIfHX1sc1-I0vLCZq7_w5_Ga50-k1Ra_-9_adi1GzqxKRIjE7Gfp_Fe7z4pIn1XNgur99McLgBaoBP4cyJ0nPoC05p0QSV1oMj8Yjb6dNrdCtmjSQNt_Els7ELq_9SMaSVN3zGi4BLlddAY9TcDqTop0yu-A Content-Length: 0 HTTP/1.1 204 No Content Content-Length: 0 X-Openstack-Request-Id: req-5aee2081-b069-42c7-9c21-57b58bcb6fc5 Date: Tue, 18 Jul 2017 09:32:51 GMT Connection: keep-alive To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1704975/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp