Fixed in 1.5.0 in Pike: https://review.openstack.org/#/c/438272/
** Also affects: os-vif/newton Importance: Undecided Status: New ** Also affects: os-vif/ocata Importance: Undecided Status: New ** Changed in: os-vif Status: Confirmed => Fix Released ** Changed in: os-vif Assignee: (unassigned) => Kevin Benton (kevinbenton) ** Changed in: os-vif/newton Status: New => In Progress ** Changed in: os-vif/ocata Status: New => In Progress ** Changed in: os-vif/ocata Assignee: (unassigned) => Kevin Benton (kevinbenton) ** Changed in: os-vif/newton Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka) ** Changed in: os-vif Importance: Undecided => High ** Changed in: os-vif/newton Importance: Undecided => High ** Changed in: os-vif/ocata Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1694769 Title: Nova fails to plug port because of missing ipset when calling iptables-restore Status in neutron: Won't Fix Status in os-vif: Fix Released Status in os-vif newton series: In Progress Status in os-vif ocata series: In Progress Bug description: This is Ocata, linuxbridge. http://logs.openstack.org/95/466395/3/gate/gate-tempest-dsvm-neutron- linuxbridge-ubuntu-xenial/e5923b4/logs/testr_results.html.gz File "tempest/common/compute.py", line 188, in create_test_server clients.servers_client, server['id'], wait_until) File "tempest/common/waiters.py", line 76, in wait_for_server_status server_id=server_id) tempest.exceptions.BuildErrorException: Server 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d failed to build and is in ERROR status Details: {u'created': u'2017-05-27T03:00:23Z', u'code': 500, u'message': u'No valid host was found. There are not enough hosts available.'} The failure in nova-cpu log: http://logs.openstack.org/95/466395/3/gate/gate-tempest-dsvm-neutron- linuxbridge-ubuntu- xenial/e5923b4/logs/screen-n-cpu.txt.gz#_2017-05-27_03_00_21_716 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [req-06c29149-80d9-4923-b9c4-54591a3f5e7e tempest-ServerActionsTestJSON-1792219232 tempest-ServerActionsTestJSON-1792219232] [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Instance failed to spawn 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Traceback (most recent call last): 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/opt/stack/new/nova/nova/compute/manager.py", line 2124, in _build_resources 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] yield resources 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/opt/stack/new/nova/nova/compute/manager.py", line 1930, in _build_and_run_instance 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] block_device_info=block_device_info) 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2698, in spawn 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] destroy_disks_on_failure=True) 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5114, in _create_domain_and_network 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] destroy_disks_on_failure) 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] self.force_reraise() 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] six.reraise(self.type_, self.value, self.tb) 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5077, in _create_domain_and_network 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] self.plug_vifs(instance, network_info) 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 749, in plug_vifs 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] self.vif_driver.plug(instance, vif) 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 786, in plug 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] self._plug_os_vif(instance, vif_obj) 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 766, in _plug_os_vif 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] raise exception.InternalError(msg) 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:16:2c:4d,bridge_name='brq9c933655-e1',has_traffic_filtering=True,id=416d65ee-709e-4b50-a0f1-23d988773b9f,network=Network(9c933655-e176-41b2-9b3a-8e46b13450ca),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap416d65ee-70'). Got error: Unexpected error while running command. 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Command: iptables-restore -c 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Exit code: 2 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Stdout: u'' 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Stderr: u"iptables-restore v1.6.0: Set NIPv47e2555da-67d1-4e12-9317- doesn't exist.\n\nError occurred at line: 202\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n" 2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] I see neutron l2 agent logging destroying/creating the ipset in span of the job run, including around the time Nova failed to plug: 2017-05-27 03:00:18.250 30306 DEBUG neutron.agent.linux.utils [req- 1a4f80b9-372f-4eb4-be1c-f88cadc300ef - -] Running command (rootwrap daemon): ['ipset', 'destroy', 'NIPv47e2555da-67d1-4e12-9317-'] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:113 I suspect there is some race between linuxbridge agent and nova compute. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1694769/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp