Public bug reported: A race condition can sometimes occur in l-3 agent when a dvr based floatingip is being deleted from one router and another dvr based floatingip is being configured on another router in the same node. Especially if the floatingip being deleted was the last floatingip on the node. Although fix for Bug # 1373100 [1] eliminated frequent observation of this behavior in upstream tests, it still shows up. Couple of recent examples:
http://logs.openstack.org/88/128288/1/check/check-tempest-dsvm-neutron- dvr/8fdd1de/ http://logs.openstack.org/03/123403/7/check/check-tempest-dsvm-neutron- dvr/859534a/ Relevant log messages: 2014-10-14 16:06:15.803 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-82fb2751-30ba-4015-a5da-6c8563064db9', 'ip', 'link', 'del', 'fpr-7ed86ca6-b'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46 2014-10-14 16:06:15.838 22303 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', 'addr', 'show', 'rfp-7ed86ca6-b'] Exit code: 0 Stdout: '2: rfp-7ed86ca6-b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\n link/ether c6:88:ee:71:a7:51 brd ff:ff:ff:ff:ff:ff\n inet 169.254.30.212/31 scope global rfp-7ed86ca6-b\n valid_lft forever preferred_lft forever\n inet6 fe80::c488:eeff:fe71:a751/64 scope link \n valid_lft forever preferred_lft forever\n' Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:81 2014-10-14 16:06:15.839 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', '-4', 'addr', 'add', '172.24.4.91/32', 'brd', '172.24.4.91', 'scope', 'global', 'dev', 'rfp-7ed86ca6-b'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46 2014-10-14 16:06:16.221 22303 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-82fb2751-30ba-4015-a5da-6c8563064db9', 'ip', 'link', 'del', 'fpr-7ed86ca6-b'] Exit code: 0 Stdout: '' Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:81 2014-10-14 16:06:16.222 22303 DEBUG neutron.agent.l3_agent [-] DVR: unplug: fg-f04e25ef-e3 _destroy_fip_namespace /opt/stack/new/neutron/neutron/agent/l3_agent.py:679 2014-10-14 16:06:16.222 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-ex'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46 2014-10-14 16:06:16.251 22303 ERROR neutron.agent.linux.utils [-] Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', '-4', 'addr', 'add', '172.24.4.91/32', 'brd', '172.24.4.91', 'scope', 'global', 'dev', 'rfp-7ed86ca6-b'] Exit code: 1 Stdout: '' Stderr: 'Cannot find device "rfp-7ed86ca6-b"\n' [1] https://bugs.launchpad.net/neutron/+bug/1373100 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1381238 Title: Race condition on processing DVR floating IPs Status in OpenStack Neutron (virtual network service): New Bug description: A race condition can sometimes occur in l-3 agent when a dvr based floatingip is being deleted from one router and another dvr based floatingip is being configured on another router in the same node. Especially if the floatingip being deleted was the last floatingip on the node. Although fix for Bug # 1373100 [1] eliminated frequent observation of this behavior in upstream tests, it still shows up. Couple of recent examples: http://logs.openstack.org/88/128288/1/check/check-tempest-dsvm- neutron-dvr/8fdd1de/ http://logs.openstack.org/03/123403/7/check/check-tempest-dsvm- neutron-dvr/859534a/ Relevant log messages: 2014-10-14 16:06:15.803 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-82fb2751-30ba-4015-a5da-6c8563064db9', 'ip', 'link', 'del', 'fpr-7ed86ca6-b'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46 2014-10-14 16:06:15.838 22303 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', 'addr', 'show', 'rfp-7ed86ca6-b'] Exit code: 0 Stdout: '2: rfp-7ed86ca6-b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\n link/ether c6:88:ee:71:a7:51 brd ff:ff:ff:ff:ff:ff\n inet 169.254.30.212/31 scope global rfp-7ed86ca6-b\n valid_lft forever preferred_lft forever\n inet6 fe80::c488:eeff:fe71:a751/64 scope link \n valid_lft forever preferred_lft forever\n' Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:81 2014-10-14 16:06:15.839 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', '-4', 'addr', 'add', '172.24.4.91/32', 'brd', '172.24.4.91', 'scope', 'global', 'dev', 'rfp-7ed86ca6-b'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46 2014-10-14 16:06:16.221 22303 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-82fb2751-30ba-4015-a5da-6c8563064db9', 'ip', 'link', 'del', 'fpr-7ed86ca6-b'] Exit code: 0 Stdout: '' Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:81 2014-10-14 16:06:16.222 22303 DEBUG neutron.agent.l3_agent [-] DVR: unplug: fg-f04e25ef-e3 _destroy_fip_namespace /opt/stack/new/neutron/neutron/agent/l3_agent.py:679 2014-10-14 16:06:16.222 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-ex'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46 2014-10-14 16:06:16.251 22303 ERROR neutron.agent.linux.utils [-] Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', '-4', 'addr', 'add', '172.24.4.91/32', 'brd', '172.24.4.91', 'scope', 'global', 'dev', 'rfp-7ed86ca6-b'] Exit code: 1 Stdout: '' Stderr: 'Cannot find device "rfp-7ed86ca6-b"\n' [1] https://bugs.launchpad.net/neutron/+bug/1373100 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1381238/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
