Reviewed: https://review.openstack.org/325370 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d2508163cfcede20df641239be081fe63c79150b Submitter: Jenkins Branch: master
commit d2508163cfcede20df641239be081fe63c79150b Author: Brian Haley <brian.ha...@hpe.com> Date: Fri Jun 3 11:34:35 2016 -0400 OVS: don't throw KeyError when duplicate VLAN tags exist In _restore_local_vlan_map() we can have two ports with the same VLAN tag, but trying to remove the second will throw a KeyError, causing the agent to not start. Use discard() instead so we only remove an entry if it's there. Closes-bug: #1526974 Change-Id: I479c693f490c704c5b6c1462e9ab236684e9c259 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1526974 Title: KeyError prevents openvswitch agent from starting Status in neutron: Fix Released Bug description: On Liberty I ran into a situation where the openvswitch agent won't start and fails with the following stack trace: 2015-12-16 16:01:42.852 10772 CRITICAL neutron [req-afb4e123-1940-48df-befc-9319516152b5 - - - - -] KeyError: 8 2015-12-16 16:01:42.852 10772 ERROR neutron Traceback (most recent call last): 2015-12-16 16:01:42.852 10772 ERROR neutron File "/opt/neutron/bin/neutron-openvswitch-agent", line 11, in <module> 2015-12-16 16:01:42.852 10772 ERROR neutron sys.exit(main()) 2015-12-16 16:01:42.852 10772 ERROR neutron File "/opt/neutron/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main 2015-12-16 16:01:42.852 10772 ERROR neutron agent_main.main() 2015-12-16 16:01:42.852 10772 ERROR neutron File "/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main 2015-12-16 16:01:42.852 10772 ERROR neutron mod.main() 2015-12-16 16:01:42.852 10772 ERROR neutron File "/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 36, in main 2015-12-16 16:01:42.852 10772 ERROR neutron ovs_neutron_agent.main(bridge_classes) 2015-12-16 16:01:42.852 10772 ERROR neutron File "/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1913, in main 2015-12-16 16:01:42.852 10772 ERROR neutron agent = OVSNeutronAgent(bridge_classes, **agent_config) 2015-12-16 16:01:42.852 10772 ERROR neutron File "/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 302, in __init__ 2015-12-16 16:01:42.852 10772 ERROR neutron self._restore_local_vlan_map() 2015-12-16 16:01:42.852 10772 ERROR neutron File "/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 358, in _restore_local_vlan_map 2015-12-16 16:01:42.852 10772 ERROR neutron self.available_local_vlans.remove(local_vlan) 2015-12-16 16:01:42.852 10772 ERROR neutron KeyError: 8 2015-12-16 16:01:42.852 10772 ERROR neutron Somehow the ovs table ended up with 2 ports with the same local vlan tag. # ovs-vsctl -- --columns=name,tag,other_config list Port | grep -E 'qvob7ba561c-e5|qvod3e1f984-0e' -A 2 name : "qvob7ba561c-e5" tag : 8 other_config : {net_uuid="fb33e234-714d-44f8-8728-1a466ef5aca0", network_type=vxlan, physical_network=None, segmentation_id="5969"} -- name : "qvod3e1f984-0e" tag : 8 other_config : {net_uuid="47e0f11c-7aa4-4eb4-97dc-0ef4e064680c", network_type=vxlan, physical_network=None, segmentation_id="5836"} Additionally, I noticed the ofport for one of them was -1. # ovs-vsctl -- --columns=name,ofport,external_ids list Interface | grep -E 'qvob7ba561c-e5|qvod3e1f984-0e' -A 2 name : "qvod3e1f984-0e" ofport : 20 external_ids : {attached-mac="fa:16:3e:d7:eb:05", iface-id="d3e1f984-0e4f-4d39-a074-1c0809ad864c", iface-status=active, vm-uuid="a00032c8-f516-42e3-865e-1988768bab84"} -- name : "qvob7ba561c-e5" ofport : -1 external_ids : {attached-mac="fa:16:3e:a9:c3:69", iface-id="b7ba561c-e5a2-4128-b36c-9484a763f4de", iface-status=active, vm-uuid="71873533-a4ab-4af6-8ace-e75c60b828f9"} I'm not sure if this is relevant, but the VM that has -1 ofport is in the following state +--------------------------------------+------------------------------------------------------+----------------------------------+-----------+------------+-------------+-------------------------------------------------------+ | ID | Name | Tenant ID | Status | Task State | Power State | Networks | +--------------------------------------+------------------------------------------------------+----------------------------------+-----------+------------+-------------+-------------------------------------------------------+ | 71873533-a4ab-4af6-8ace-e75c60b828f9 | test-instance-1 | 99e641ee27434c36b4f83fbee0599e67 | SHUTOFF | - | Shutdown | | +--------------------------------------+------------------------------------------------------+----------------------------------+-----------+------------+-------------+-------------------------------------------------------+ ------------------------------------------------------------------------------------------------------------------------------------ Neutron Version: 69d531565dcd180f6f1141bad143b3ea4dcd7ade Operating System: CentOS Linux 7 (Core) Kernel: Linux 3.10.0-229.11.1.el7.x86_64 Architecture: x86_64 ovs-vsctl (Open vSwitch) 2.3.1 Compiled Dec 26 2014 15:35:14 DB Schema 7.6.2 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1526974/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp