Public bug reported: Filing this on ubuntu/neutron package, as neutron itself is EOL'd for Icehouse.
FYI this is a nonHA icehouse/trusty deploy using serverteam's juju charms. On one of our production environments with a rather high rate of API calls, (sp for transient VMs from CI), we frequently get neutron OVS breakage on compute nodes¹, which we've been able to more or less correlate with the following alike errors at /var/log/neutron /openvswitch-agent.log: 2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 399, in _del_fdb_flow 2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp lvm.tun_ofports.remove(ofport) 2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp KeyError: '13' Detailed log: http://paste.ubuntu.com/14431656/ - note the same time of occurrence on the 3 diff compute nodes shown there. ¹ What we then observe are missing are missing tun_ids from ovs-ofctl dump-flows br-tun ie provider:segmentation_id not present at the compute node for a VM with a neutron network that has it. Afaics this had been fixed upstream at ( lp#1421105 ): https://git.openstack.org/cgit/openstack/neutron/commit/?id=841b2f58f375df53b380cf5796bb31c82cd09260 , please consider backporting it to Icehouse, it's a pretty trivial fix. ** Affects: neutron (Ubuntu) Importance: Undecided Status: New ** Tags: canonical-bootstack ** Description changed: Filing this on ubuntu/neutron package, as neutron itself is EOL'd for Icehouse. FYI this is a nonHA icehouse/trusty deploy using serverteam's juju charms. - On one of our production environments with a rather high rate of API calls, - (sp for transient VMs from CI), we frequently get neutron OVS breakage - on compute nodes¹, which we've been able to more or less correlate with - the following alike errors at /var/log/neutron/openvswitch-agent.log: + On one of our production environments with a rather high rate of API + calls, (sp for transient VMs from CI), we frequently get neutron OVS + breakage on compute nodes¹, which we've been able to more or less + correlate with the following alike errors at /var/log/neutron + /openvswitch-agent.log: 2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 399, in _del_fdb_flow 2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp lvm.tun_ofports.remove(ofport) 2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp KeyError: '13' - Detailed log: http://paste.ubuntu.com/14431656/ - note the same time - of occurrence on the 3 diff compute nodes shown there. + Detailed log: http://paste.ubuntu.com/14431656/ - note the same time of + occurrence on the 3 diff compute nodes shown there. - ¹ What we then observe are missing are missing tun_ids from + ¹ What we then observe are missing are missing tun_ids from ovs-ofctl dump-flows br-tun - ie provider:segmentation_id not present at the compute node - for a VM with a neutron network that has it. + ie provider:segmentation_id not present at the compute node for a VM with a neutron network that has it. Afaics this had been fixed upstream at ( lp#1421105 ): https://git.openstack.org/cgit/openstack/neutron/commit/?id=841b2f58f375df53b380cf5796bb31c82cd09260 , please consider backporting it to Icehouse, it's a pretty trivial fix. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531963 Title: trusty/icehouse neutron-plugin-openvswitch-agent: lvm.tun_ofports.remove crashes with KeyError To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1531963/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
