This is an automated cleanup. This bug report has been closed because it is older than 18 months and there is no open code change to fix this. After this time it is unlikely that the circumstances which lead to the observed issue can be reproduced.
If you can reproduce the bug, please: * reopen the bug report (set to status "New") * AND add the detailed steps to reproduce the issue (if applicable) * AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>" Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON). Valid example: CONFIRMED FOR: LIBERTY ** Changed in: nova Status: Confirmed => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1379663 Title: After upgrading - ovs-vswitchd cannot add existing ports Status in neutron: Incomplete Status in OpenStack Compute (nova): Expired Bug description: Hi there, After upgrading (stop all services, yum upgrade, db sync) from older Icehouse build to latest Icehouse build, my compute node (specifically openstack-nova-compute) cannot be started. I deployed a number of instances before upgrade, and after upgrading openstack-nova-compute refuses to start up. The logs seem to point to some issue with ovs- vswitch unable to bind ports of the existing instances. All other services at controller and network nodes seem to be running fine. And before upgrading, everything was working fine. # rpm -qa | grep openstack-nova openstack-nova-compute-2014.1.2-1.el6.noarch openstack-nova-common-2014.1.2-1.el6.noarch At compute.log: 2014-10-10 14:37:39.372 24897 ERROR nova.openstack.common.threadgroup [-] Unexpected vif_type=binding_failed 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup Traceback (most recent call last): 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup x.wait() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup return self.thread.wait() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 173, in wait 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup return self._exit_event.wait() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 121, in wait 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 293, in switch 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup return self.greenlet.switch() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 486, in run_service 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup service.start() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/service.py", line 163, in start 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup self.manager.init_host() 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1044, in init_host 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup self._init_instance(context, instance) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 902, in _init_instance 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup self.driver.plug_vifs(instance, net_info) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 860, in plug_vifs 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup self.vif_driver.plug(instance, vif) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py", line 616, in plug 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup _("Unexpected vif_type=%s") % vif_type) 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup NovaException: Unexpected vif_type=binding_failed 2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup At ovs-vswitchd.log: 2014-10-10T06:37:38Z|00377|dpif|WARN|Dropped 9 log messages in last 17953 seconds (most recently, 17952 seconds ago) due to excessive rate 2014-10-10T06:37:38Z|00378|dpif|WARN|system@ovs-system: port_del failed (Invalid argument) 2014-10-10T06:37:38Z|00379|netdev_linux|WARN|Dropped 8 log messages in last 17953 seconds (most recently, 17952 seconds ago) due to excessive rate 2014-10-10T06:37:38Z|00380|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:38Z|00381|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device 2014-10-10T06:37:38Z|00382|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:38Z|00383|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device 2014-10-10T06:37:38Z|00384|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:38Z|00385|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device 2014-10-10T06:37:44Z|00386|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:44Z|00387|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device 2014-10-10T06:37:44Z|00388|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:44Z|00389|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device 2014-10-10T06:37:44Z|00390|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:44Z|00391|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device 2014-10-10T06:37:44Z|00392|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:44Z|00393|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device 2014-10-10T06:37:44Z|00394|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:44Z|00395|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device 2014-10-10T06:37:44Z|00396|bridge|INFO|bridge br-tun: using datapath ID 00003666e785084d 2014-10-10T06:37:44Z|00397|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" 2014-10-10T06:37:44Z|00398|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:44Z|00399|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device 2014-10-10T06:37:44Z|00400|bridge|INFO|bridge br-tun: added interface br-tun on port 65534 2014-10-10T06:37:44Z|00401|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:44Z|00402|bridge|INFO|bridge br-int: added interface patch-tun on port 1 2014-10-10T06:37:44Z|00403|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:44Z|00404|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:44Z|00405|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:44Z|00406|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:44Z|00407|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:44Z|00408|bridge|INFO|bridge br-tun: added interface patch-int on port 1 2014-10-10T06:37:45Z|00409|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device 2014-10-10T06:37:45Z|00410|dpif|WARN|Dropped 6 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate 2014-10-10T06:37:45Z|00411|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device 2014-10-10T06:37:45Z|00412|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device 2014-10-10T06:37:45Z|00413|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device 2014-10-10T06:37:45Z|00414|bridge|INFO|bridge br-tun: added interface gre-c0a80547 on port 2 2014-10-10T06:37:48Z|00415|ofproto|INFO|br-int: 3 flow_mods 10 s ago (2 adds, 1 deletes) 2014-10-10T06:37:54Z|00416|ofproto|INFO|br-tun: 13 flow_mods in the 1 s starting 10 s ago (11 adds, 1 deletes, 1 modifications) But then these ports are already running under br-int - # ovs-vsctl show 8fa6592e-b814-4c90-8ce5-8ee2f2e61d39 Bridge br-int fail_mode: secure Port "qvoeda2db74-94" tag: 4095 Interface "qvoeda2db74-94" Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "qvo2beb3ffb-6d" tag: 1 Interface "qvo2beb3ffb-6d" Port "qvo1a5fea4a-6e" tag: 1 Interface "qvo1a5fea4a-6e" Port "qvo684ea2ea-63" tag: 1 Interface "qvo684ea2ea-63" Port "qvoeae48b5f-9b" tag: 1 Interface "qvoeae48b5f-9b" Port br-int Interface br-int type: internal Bridge br-tun Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port br-tun Interface br-tun type: internal Port "gre-c0a80547" Interface "gre-c0a80547" type: gre options: {in_key=flow, local_ip="192.168.5.72", out_key=flow, remote_ip="192.168.5.71"} ovs_version: "1.11.0" Regards, Boon Lee To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1379663/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp