** Changed in: neutron Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1379663
Title: After upgrading - ovs-vswitchd cannot add existing ports Status in neutron: Won't Fix Status in OpenStack Compute (nova): Expired Bug description: Hi there, After upgrading (stop all services, yum upgrade, db sync) from older Icehouse build to latest Icehouse build, my compute node (specifically openstack-nova-compute) cannot be started. I deployed a number of instances before upgrade, and after upgrading openstack-nova-compute refuses to start up. The logs seem to point to some issue with ovs- vswitch unable to bind ports of the existing instances. All other services at controller and network nodes seem to be running fine. And before upgrading, everything was working fine. # rpm -qa | grep openstack-nova openstack-nova-compute-2014.1.2-1.el6.noarch openstack-nova-common-2014.1.2-1.el6.noarch At compute.log: 2014-10-10 14:37:39.372 24897 ERROR nova.openstack.common.threadgroup [-] Unexpected vif_type=binding_failed 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup Traceback (most recent call last): 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup x.wait() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup return self.thread.wait() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 173, in wait 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup return self._exit_event.wait() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 121, in wait 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 293, in switch 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup return self.greenlet.switch() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 486, in run_service 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup service.start() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/service.py", line 163, in start 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup self.manager.init_host() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1044, in init_host 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup self._init_instance(context, instance) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 902, in _init_instance 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup self.driver.plug_vifs(instance, net_info) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 860, in plug_vifs 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup self.vif_driver.plug(instance, vif) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py", line 616, in plug 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup _("Unexpected vif_type=%s") % vif_type) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup NovaException: Unexpected vif_type=binding_failed 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup At ovs-vswitchd.log: 2014-10-10T06:37:38Z|00377|dpif|WARN|Dropped 9 log messages in last 17953 seconds (most recently, 17952 seconds ago) due to excessive rate 2014-10-10T06:37:38Z|00378|dpif|WARN|system@ovs-system: port_del failed (Invalid argument) 2014-10-10T06:37:38Z|00379|netdev_linux|WARN|Dropped 8 log messages in last 17953 seconds (most recently, 17952 seconds ago) due to excessive rate 2014-10-10T06:37:38Z|00380|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:38Z|00381|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device 2014-10-10T06:37:38Z|00382|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:38Z|00383|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device 2014-10-10T06:37:38Z|00384|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:38Z|00385|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device 2014-10-10T06:37:44Z|00386|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:44Z|00387|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device 2014-10-10T06:37:44Z|00388|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:44Z|00389|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device 2014-10-10T06:37:44Z|00390|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:44Z|00391|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device 2014-10-10T06:37:44Z|00392|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:44Z|00393|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device 2014-10-10T06:37:44Z|00394|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:44Z|00395|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device 2014-10-10T06:37:44Z|00396|bridge|INFO|bridge br-tun: using datapath ID 00003666e785084d 2014-10-10T06:37:44Z|00397|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" 2014-10-10T06:37:44Z|00398|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:44Z|00399|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device 2014-10-10T06:37:44Z|00400|bridge|INFO|bridge br-tun: added interface br-tun on port 65534 2014-10-10T06:37:44Z|00401|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:44Z|00402|bridge|INFO|bridge br-int: added interface patch-tun on port 1 2014-10-10T06:37:44Z|00403|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:44Z|00404|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:44Z|00405|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:44Z|00406|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:44Z|00407|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:44Z|00408|bridge|INFO|bridge br-tun: added interface patch-int on port 1 2014-10-10T06:37:45Z|00409|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:45Z|00410|dpif|WARN|Dropped 6 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate 2014-10-10T06:37:45Z|00411|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device 2014-10-10T06:37:45Z|00412|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:45Z|00413|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:45Z|00414|bridge|INFO|bridge br-tun: added interface gre-c0a80547 on port 2 2014-10-10T06:37:48Z|00415|ofproto|INFO|br-int: 3 flow_mods 10 s ago (2 adds, 1 deletes) 2014-10-10T06:37:54Z|00416|ofproto|INFO|br-tun: 13 flow_mods in the 1 s starting 10 s ago (11 adds, 1 deletes, 1 modifications) But then these ports are already running under br-int - # ovs-vsctl show 8fa6592e-b814-4c90-8ce5-8ee2f2e61d39 Bridge br-int fail_mode: secure Port "qvoeda2db74-94" tag: 4095 Interface "qvoeda2db74-94" Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "qvo2beb3ffb-6d" tag: 1 Interface "qvo2beb3ffb-6d" Port "qvo1a5fea4a-6e" tag: 1 Interface "qvo1a5fea4a-6e" Port "qvo684ea2ea-63" tag: 1 Interface "qvo684ea2ea-63" Port "qvoeae48b5f-9b" tag: 1 Interface "qvoeae48b5f-9b" Port br-int Interface br-int type: internal Bridge br-tun Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port br-tun Interface br-tun type: internal Port "gre-c0a80547" Interface "gre-c0a80547" type: gre options: {in_key=flow, local_ip="192.168.5.72", out_key=flow, remote_ip="192.168.5.71"} ovs_version: "1.11.0" Regards, Boon Lee To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1379663/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp