Private bug reported: This is Openstack Queens on Bionic. The main difference from templates is no neutron-gateway (provider network only) and use of DPDK.
Bundle used in deployment is at: https://git.launchpad.net/cpe-deployments/tree/?h=2019-05-27-Telefonica-OCS-OP-152907 There are other issues under investigation about dpdk and checksumming but they don't seem related to this at first look. https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1833713 - Instances cannot be started once they are shutdown - It's happening to every instance after the problem first appeared - It's happening on different hosts - Any try to start will timeout with errors in nova log (bellow) - Nothing new appears in openvswitch logs with normal debugging level - Nothing new appears on libvirt logs for the instance (last status is from last boot) 2019-07-10 13:40:42.013 19975 ERROR oslo_messaging.rpc.server InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFVHostUser(active=True,address=fa:16:3e:8e:8f:9b,has_traffic_filtering=False,id=ab6225f4-1cd8-43c7-8777-52c99ae80f67,mode='server',network=Network (d8249c3d-03d9-44ac-8eae-fa967993c73d),path='/run/libvirt-vhost- user/vhuab6225f4-1c',plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='vhuab6225f4-1c'). Got error: Error during following call to agent: ['ovs-vsctl', '-- timeout=120', '--', '--if-exists', 'del-port', u'vhuab6225f4-1c', '--', 'add-port', u'br-int', u'vhuab6225f4-1c', '--', 'set', 'Interface', u'vhuab6225f4-1c', u'external-ids:iface- id=ab6225f4-1cd8-43c7-8777-52c99ae80f67', 'external-ids:iface- status=active', u'external-ids:attached-mac=fa:16:3e:8e:8f:9b', u 'external-ids:vm-uuid=5e46868f-8a52-4d70-b08a-9a320dc9821b', 'type=dpdkvhostuserclient', u'options:vhost-server-path=/run/libvirt- vhost-user/vhuab6225f4-1c'] 2019-07-10 13:43:05.511 19975 ERROR os_vif AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if- exists', 'del-port', u'vhuab6225f4-1c', '--', 'add-port', u'br-int', u'vhuab6225f4-1c', '--', 'set', 'Interface', u'vhuab6225f4-1c', u 'external-ids:iface-id=ab6225f4-1cd8-43c7-8777-52c99ae80f67', 'external- ids:iface-status=active', u'external-ids:attached- mac=fa:16:3e:8e:8f:9b', u'external-ids:vm-uuid=5e46868f-8a52-4d70-b08a- 9a320dc9821b', 'type=dpdkvhostuserclient', u'options:vhost-server- path=/run/libvirt-vhost-user/vhuab6225f4-1c'] Complete logs will follow. ** Affects: nova Importance: Undecided Status: New ** Information type changed from Public to Private ** Description changed: - - This is Openstack Queens on Bionic. The main difference from templates is no neutron-gateway (provider network only) and use of DPDK. + This is Openstack Queens on Bionic. The main difference from templates + is no neutron-gateway (provider network only) and use of DPDK. Bundle used in deployment is at: https://git.launchpad.net/cpe-deployments/tree/?h=2019-05-27-Telefonica-OCS-OP-152907 There are other issues under investigation about dpdk and checksumming but they don't seem related to this at first look. https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1833713 - Instances cannot be started once they are shutdown + - It's happening to every instance after the problem first appeared - Any try to start will timeout with errors in nova log (bellow) - Nothing new appears in openvswitch logs with normal debugging level - Nothing new appears on libvirt logs for the instance (last status is from last boot) 2019-07-10 13:40:42.013 19975 ERROR oslo_messaging.rpc.server InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFVHostUser(active=True,address=fa:16:3e:8e:8f:9b,has_traffic_filtering=False,id=ab6225f4-1cd8-43c7-8777-52c99ae80f67,mode='server',network=Network (d8249c3d-03d9-44ac-8eae-fa967993c73d),path='/run/libvirt-vhost- user/vhuab6225f4-1c',plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='vhuab6225f4-1c'). Got error: Error during following call to agent: ['ovs-vsctl', '-- timeout=120', '--', '--if-exists', 'del-port', u'vhuab6225f4-1c', '--', 'add-port', u'br-int', u'vhuab6225f4-1c', '--', 'set', 'Interface', u'vhuab6225f4-1c', u'external-ids:iface- id=ab6225f4-1cd8-43c7-8777-52c99ae80f67', 'external-ids:iface- status=active', u'external-ids:attached-mac=fa:16:3e:8e:8f:9b', u 'external-ids:vm-uuid=5e46868f-8a52-4d70-b08a-9a320dc9821b', 'type=dpdkvhostuserclient', u'options:vhost-server-path=/run/libvirt- vhost-user/vhuab6225f4-1c'] 2019-07-10 13:43:05.511 19975 ERROR os_vif AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if- exists', 'del-port', u'vhuab6225f4-1c', '--', 'add-port', u'br-int', u'vhuab6225f4-1c', '--', 'set', 'Interface', u'vhuab6225f4-1c', u 'external-ids:iface-id=ab6225f4-1cd8-43c7-8777-52c99ae80f67', 'external- ids:iface-status=active', u'external-ids:attached- mac=fa:16:3e:8e:8f:9b', u'external-ids:vm-uuid=5e46868f-8a52-4d70-b08a- 9a320dc9821b', 'type=dpdkvhostuserclient', u'options:vhost-server- path=/run/libvirt-vhost-user/vhuab6225f4-1c'] Complete logs will follow. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1836105 Title: Instance does not start - Error during following call to agent: ovs- vsctl Status in OpenStack Compute (nova): New Bug description: This is Openstack Queens on Bionic. The main difference from templates is no neutron-gateway (provider network only) and use of DPDK. Bundle used in deployment is at: https://git.launchpad.net/cpe-deployments/tree/?h=2019-05-27-Telefonica-OCS-OP-152907 There are other issues under investigation about dpdk and checksumming but they don't seem related to this at first look. https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1833713 - Instances cannot be started once they are shutdown - It's happening to every instance after the problem first appeared - It's happening on different hosts - Any try to start will timeout with errors in nova log (bellow) - Nothing new appears in openvswitch logs with normal debugging level - Nothing new appears on libvirt logs for the instance (last status is from last boot) 2019-07-10 13:40:42.013 19975 ERROR oslo_messaging.rpc.server InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFVHostUser(active=True,address=fa:16:3e:8e:8f:9b,has_traffic_filtering=False,id=ab6225f4-1cd8-43c7-8777-52c99ae80f67,mode='server',network=Network (d8249c3d-03d9-44ac-8eae-fa967993c73d),path='/run/libvirt-vhost- user/vhuab6225f4-1c',plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='vhuab6225f4-1c'). Got error: Error during following call to agent: ['ovs-vsctl', '-- timeout=120', '--', '--if-exists', 'del-port', u'vhuab6225f4-1c', '--', 'add-port', u'br-int', u'vhuab6225f4-1c', '--', 'set', 'Interface', u'vhuab6225f4-1c', u'external-ids:iface- id=ab6225f4-1cd8-43c7-8777-52c99ae80f67', 'external-ids:iface- status=active', u'external-ids:attached-mac=fa:16:3e:8e:8f:9b', u 'external-ids:vm-uuid=5e46868f-8a52-4d70-b08a-9a320dc9821b', 'type=dpdkvhostuserclient', u'options:vhost-server-path=/run/libvirt- vhost-user/vhuab6225f4-1c'] 2019-07-10 13:43:05.511 19975 ERROR os_vif AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if- exists', 'del-port', u'vhuab6225f4-1c', '--', 'add-port', u'br-int', u'vhuab6225f4-1c', '--', 'set', 'Interface', u'vhuab6225f4-1c', u 'external-ids:iface-id=ab6225f4-1cd8-43c7-8777-52c99ae80f67', 'external-ids:iface-status=active', u'external-ids:attached- mac=fa:16:3e:8e:8f:9b', u'external-ids:vm-uuid=5e46868f-8a52-4d70 -b08a-9a320dc9821b', 'type=dpdkvhostuserclient', u'options:vhost- server-path=/run/libvirt-vhost-user/vhuab6225f4-1c'] Complete logs will follow. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1836105/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp