Public bug reported: On a juno-2 TripleO based system, after performing a node upgrade via `nova rebuild`, we encountered an issue where some virtual machines were unreachable. Upon further investigation, we found that the OVS VIF was missing, and the root cause was that the re-creation failed as seen in the log below. The flow was never re-asserted as the logic in nova/virt/libvirt/vif.py function plug_ovs_hybrid only executes the steps that would result in the VIF being recreated if the interface did not already exist. As the interface already existed upon attempting to restart, it appears to have never been re-asserted hence requiring the command to be manually executed to restore network connectivity for the virtual machine.
# cat nova-compute.log Traceback (most recent call last): File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 455, in fire_timers timer() File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__ cb(*args, **kw) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main result = function(*args, **kwargs) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/service.py", line 490, in run_service service.start() File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/service.py", line 164, in start self.manager.init_host() File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1059, in init_host self._init_instance(context, instance) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 918, in _init_instance self.driver.plug_vifs(instance, net_info) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 820, in plug_vifs self.vif_driver.plug(instance, vif) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 627, in plug self.plug_ovs(instance, vif) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 492, in plug_ovs self.plug_ovs_hybrid(instance, vif) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 488, in plug_ovs_hybrid instance['uuid']) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/linux_net.py", line 1320, in create_ovs_vif_port 'external-ids:vm-uuid=%s' % instance_id]) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/linux_net.py", line 1310, in _ovs_vsctl raise exception.AgentError(method=full_args) AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if-exists', 'del-port', u'qvo8670d393-d4', '--', 'add-port', u'br-int', u'qvo8670d393-d4', '--', 'set', 'Interface', u'qvo8670d393-d4', u'external-ids:iface-id=8670d393-d47e-41e1-b2b5-b94819266934', 'external-ids:iface-status=active', u'external-ids:attached-mac=fa:16:3e:b1:23:d1', 'external-ids:vm-uuid=26f03afc-e198-41d0-b7c4-77616f9bdfbe'] ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1391699 Title: nova vm networking failed to initialize as step after interface creation failed. Status in OpenStack Compute (Nova): New Bug description: On a juno-2 TripleO based system, after performing a node upgrade via `nova rebuild`, we encountered an issue where some virtual machines were unreachable. Upon further investigation, we found that the OVS VIF was missing, and the root cause was that the re-creation failed as seen in the log below. The flow was never re-asserted as the logic in nova/virt/libvirt/vif.py function plug_ovs_hybrid only executes the steps that would result in the VIF being recreated if the interface did not already exist. As the interface already existed upon attempting to restart, it appears to have never been re-asserted hence requiring the command to be manually executed to restore network connectivity for the virtual machine. # cat nova-compute.log Traceback (most recent call last): File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 455, in fire_timers timer() File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__ cb(*args, **kw) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main result = function(*args, **kwargs) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/service.py", line 490, in run_service service.start() File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/service.py", line 164, in start self.manager.init_host() File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1059, in init_host self._init_instance(context, instance) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 918, in _init_instance self.driver.plug_vifs(instance, net_info) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 820, in plug_vifs self.vif_driver.plug(instance, vif) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 627, in plug self.plug_ovs(instance, vif) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 492, in plug_ovs self.plug_ovs_hybrid(instance, vif) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 488, in plug_ovs_hybrid instance['uuid']) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/linux_net.py", line 1320, in create_ovs_vif_port 'external-ids:vm-uuid=%s' % instance_id]) File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/linux_net.py", line 1310, in _ovs_vsctl raise exception.AgentError(method=full_args) AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if-exists', 'del-port', u'qvo8670d393-d4', '--', 'add-port', u'br-int', u'qvo8670d393-d4', '--', 'set', 'Interface', u'qvo8670d393-d4', u'external-ids:iface-id=8670d393-d47e-41e1-b2b5-b94819266934', 'external-ids:iface-status=active', u'external-ids:attached-mac=fa:16:3e:b1:23:d1', 'external-ids:vm-uuid=26f03afc-e198-41d0-b7c4-77616f9bdfbe'] To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1391699/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp