Reviewed: https://review.openstack.org/272986 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8dcf39aae7a099e01bd322891526c134e87e6b1b Submitter: Jenkins Branch: master
commit 8dcf39aae7a099e01bd322891526c134e87e6b1b Author: Kevin Benton <blak...@gmail.com> Date: Wed Jan 27 02:17:01 2016 -0800 Unmarshall portinfo on update_fdb_entries calls The unmarshalling function was not aware of the data structure used by update_fdb_entries, so it would not setup PortInfo named tuples in the 'before' and 'after' fields. This would break the fdb_chg_ip_tun function which expected to be able to use named attributes. This patch adjusts the unmarshalling function to be aware of this datastrucure. This has likely been broken since the change that added named tuples here: I7f8c93b0e12ee0179bb23dfbb3a3d814615b1c2e It probably went undetected for so long because the exception will only be observed when the updated entry does not have an agent IP that matches the local agent's (i.e. not single-node). Even in a multi-node environment, this would only trigger an error when the fixed_ips of a port changed so it wouldn't show up in a normal port wiring life-cycle. Closes-Bug: #1538387 Change-Id: I0aacb3af9ebd160ebfb801f77b186075303c3df5 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1538387 Title: fdb_chg_ip_tun throwing exception because fdb_entries not in correct format Status in neutron: Fix Released Bug description: I've been trying to track down failures in the DVR multinode job. I'm now tripping over this one. For context I've been focusing on a single change, but if you see a failure in the gate-tempest-dsvm-neutron-dvr-multinode-full job you'll probably be able to find similar info. This is the change: http://logs.openstack.org/77/177777/4/check/gate-tempest-dsvm-neutron- dvr-multinode-full/5abca7b/logs/ The screen-q-agt log shows a traceback here: http://logs.openstack.org/77/177777/4/check/gate-tempest-dsvm-neutron- dvr-multinode- full/5abca7b/logs/screen-q-agt.txt.gz#_2016-01-18_10_11_29_715 <snip> 2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/l2pop/rpc_manager/l2population_rpc.py", line 312, in fdb_chg_ip_tun 2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher mac_ip.mac_address, 2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher AttributeError: 'list' object has no attribute 'mac_address' 2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher The info passed to fdb_chg_ip_tun() should have a "PortInfo" namedtuple as data, but from the line before we can see it doesn't: DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-671e8634-c753-4002-acfd-68515dd44f29 None None] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method fdb_chg_ip_tun called with arguments (<neutron.context.Context object at 0x7fa2ee6988d0>, <neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.br_tun.DeferredOVSTunnelBridge object at 0x7fa2ee79abd0>, {u'c4ae0757-e3e5-419c-b2ba-4d7817964237': {u'10.209.192.28': {u'before': [[u'fa:16:3e:8d:2e:48', u'2003:0:0:1::1']]}}}, '10.208.193.94', {u'7ca0dcf2-fb63-4959-92ee- cc757da8d120': So from this it's clear that _unmarshall_fdb_entries() either hasn't been called, or didn't do anything. Looking over in screen-q-svc.log for the info before the RPC call finds: DEBUG neutron.plugins.ml2.drivers.l2pop.rpc [req- f32790a5-0160-47b9-89b4-763b9c23bc08 tempest- TestGettingAddress-2071048693 tempest-TestGettingAddress-1817548879] Fanout notify l2population agents at q-agent-notifier the message update_fdb_entries with {'chg_ip': {u'c4ae0757-e3e5-419c-b2ba- 4d7817964237': {u'10.208.193.94': {'before': [PortInfo(mac_address=u'fa:16:3e:8d:2e:48', ip_address=u'2003:0:0:1::1')]}}}} _notification_fanout /opt/stack/new/neutron/neutron/plugins/ml2/drivers/l2pop/rpc.py:47 This is the message right before _marshall_fdb_entries() was called to convert the PortInfo into [<mac>, <ip>] pairs, and from the above it looks like it did. I'm just starting to look at this now, but maybe someone more familiar with l2pop has a guess at what's broken. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1538387/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp