[Yahoo-eng-team] [Bug 1526974] [NEW] KeyError prevents openvswitch agent from starting

2015-12-16 Thread Edgar Cantu 
Public bug reported:

On Liberty I ran into a situation where the openvswitch agent won't
start and fails with the following stack trace:


2015-12-16 16:01:42.852 10772 CRITICAL neutron 
[req-afb4e123-1940-48df-befc-9319516152b5 - - - - -] KeyError: 8
2015-12-16 16:01:42.852 10772 ERROR neutron Traceback (most recent call last):
2015-12-16 16:01:42.852 10772 ERROR neutron   File 
"/opt/neutron/bin/neutron-openvswitch-agent", line 11, in 
2015-12-16 16:01:42.852 10772 ERROR neutron sys.exit(main())
2015-12-16 16:01:42.852 10772 ERROR neutron   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py",
 line 20, in main
2015-12-16 16:01:42.852 10772 ERROR neutron agent_main.main()
2015-12-16 16:01:42.852 10772 ERROR neutron   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py",
 line 49, in main
2015-12-16 16:01:42.852 10772 ERROR neutron mod.main()
2015-12-16 16:01:42.852 10772 ERROR neutron   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py",
 line 36, in main
2015-12-16 16:01:42.852 10772 ERROR neutron 
ovs_neutron_agent.main(bridge_classes)
2015-12-16 16:01:42.852 10772 ERROR neutron   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 1913, in main
2015-12-16 16:01:42.852 10772 ERROR neutron agent = 
OVSNeutronAgent(bridge_classes, **agent_config)
2015-12-16 16:01:42.852 10772 ERROR neutron   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 302, in __init__
2015-12-16 16:01:42.852 10772 ERROR neutron self._restore_local_vlan_map()
2015-12-16 16:01:42.852 10772 ERROR neutron   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 358, in _restore_local_vlan_map
2015-12-16 16:01:42.852 10772 ERROR neutron 
self.available_local_vlans.remove(local_vlan)
2015-12-16 16:01:42.852 10772 ERROR neutron KeyError: 8
2015-12-16 16:01:42.852 10772 ERROR neutron


Somehow the ovs table ended up with 2 ports with the same local vlan tag.

# ovs-vsctl -- --columns=name,tag,other_config list Port | grep -E
'qvob7ba561c-e5|qvod3e1f984-0e' -A 2

name: "qvob7ba561c-e5"
tag : 8
other_config: {net_uuid="fb33e234-714d-44f8-8728-1a466ef5aca0", 
network_type=vxlan, physical_network=None, segmentation_id="5969"}
--
name: "qvod3e1f984-0e"
tag : 8
other_config: {net_uuid="47e0f11c-7aa4-4eb4-97dc-0ef4e064680c", 
network_type=vxlan, physical_network=None, segmentation_id="5836"}


Additionally, I noticed the ofport for one of them was -1.

# ovs-vsctl -- --columns=name,ofport,external_ids list Interface | grep
-E 'qvob7ba561c-e5|qvod3e1f984-0e' -A 2

name: "qvod3e1f984-0e"
ofport  : 20
external_ids: {attached-mac="fa:16:3e:d7:eb:05", 
iface-id="d3e1f984-0e4f-4d39-a074-1c0809ad864c", iface-status=active, 
vm-uuid="a00032c8-f516-42e3-865e-1988768bab84"}
--
name: "qvob7ba561c-e5"
ofport  : -1
external_ids: {attached-mac="fa:16:3e:a9:c3:69", 
iface-id="b7ba561c-e5a2-4128-b36c-9484a763f4de", iface-status=active, 
vm-uuid="71873533-a4ab-4af6-8ace-e75c60b828f9"}


I'm not sure if this is relevant, but the VM that has -1 ofport is in the 
following state

+--+--+--+---++-+---+
| ID   | Name   
  | Tenant ID| Status| Task State | 
Power State | Networks  |
+--+--+--+---++-+---+
| 71873533-a4ab-4af6-8ace-e75c60b828f9 | test-instance-1
| 99e641ee27434c36b4f83fbee0599e67 | SHUTOFF   | -  | Shutdown  
  |   |
+--+--+--+---++-+---+



Neutron Version: 69d531565dcd180f6f1141bad143b3ea4dcd7ade

Operating System: CentOS Linux 7 (Core)
Kernel: Linux 3.10.0-229.11.1.el7.x86_64
Architecture: x86_64

ovs-vsctl (Open vSwitch) 2.3.1
Compiled

[Yahoo-eng-team] [Bug 1513242] [NEW] StaleDataError in disassociate_floatingips

2015-11-04 Thread Edgar Cantu 
Public bug reported:

If a VM with a floating IP is deleted while concurrently deleting the
floating IP (i.e. "nova delete" and "neutron floatingip-delete") the VM
may go into error state.

Nova receives a 500 error code from neutron because neutron fails to
delete the port and the following stack is observed:

2015-11-04 21:48:37.447 17275 ERROR neutron.api.v2.resource 
[req-764172bc-4938-440d-9e6a-dd4c0f120f1b ] delete failed
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource Traceback (most 
recent call last):
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 83, 
in resource
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource result = 
method(request=request, **args)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/api/v2/base.py", line 490, in 
delete
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
obj_deleter(request.context, id, **kwargs)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/oslo_db/api.py", line 131, in wrapper
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource return f(*args, 
**kwargs)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 
1272, in delete_port
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource context, id, 
do_notify=False)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 274, 
in disassociate_floatingips
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
do_notify=do_notify)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1361, in 
disassociate_floatingips
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource context, 
port_id)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1088, in 
disassociate_floatingips
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 'router_id': 
None})
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 483, 
in __exit__
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource self.rollback()
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 
60, in __exit__
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
compat.reraise(exc_type, exc_value, exc_tb)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 480, 
in __exit__
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource self.commit()
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 385, 
in commit
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
self._prepare_impl()
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 365, 
in _prepare_impl
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
self.session.flush()
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
1986, in flush
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
self._flush(objects)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2104, in _flush
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
transaction.rollback(_capture_exception=True)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 
60, in __exit__
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
compat.reraise(exc_type, exc_value, exc_tb)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
2068, in _flush
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
flush_context.execute()
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File 
"/opt/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 
372, in execute
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource 
rec.execute(self)
2015-11-04 21:48:37.447 17275 TRACE neutron.api.v2.resource   File