[Yahoo-eng-team] [Bug 1528977] Re: Neutron router not working with latest iproute2 package included in CentOS-7.2-1511
*** This bug is a duplicate of bug 1497309 *** https://bugs.launchpad.net/bugs/1497309 Closing as a duplicate of bug 1497309. ** This bug has been marked a duplicate of bug 1497309 l3-agent unable to parse output from ip netns list (iproute2 >= 4.0) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1528977 Title: Neutron router not working with latest iproute2 package included in CentOS-7.2-1511 Status in neutron: Confirmed Bug description: Seems that something has been changed in the new iproute version and now attempts to add more than one interface to router cause errors posted at the bottom. This affects neutron-l3-agent on CentOS-7.2-1511 and possible on RedHat (cannot check this). Quick solution is to simple downgrade the package: # wget http://mirror.centos.org/centos/7.1.1503/os/x86_64/Packages/iproute-3.10.0-21.el7.x86_64.rpm # yum -y downgrade ./iproute-3.10.0-21.el7.x86_64.rpm Part of the /var/log/neutron/l3-agent.log: 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.linux.utils [-] Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a'] Exit code: 1 Stdin: Stdout: Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info [-] Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a'] Exit code: 1 Stdin: Stdout: Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Traceback (most recent call last): 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 356, in call 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info return func(*args, **kwargs) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 692, in process 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 396, in _process_internal_ports 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self.internal_network_added(p) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 328, in internal_network_added 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info INTERNAL_DEV_PREFIX) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 303, in _internal_network_added 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info prefix=prefix) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 252, in plug 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info bridge, namespace, prefix) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 483, in plug_new 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info namespace2=namespace) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 144, in add_veth 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self.ensure_namespace(namespace2) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 163, in ensure_namespace 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info ip = self.netns.add(name) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 793, in add 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self._as_root([], ('add', name), use_root_namespace=True) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 280, in _as_root 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 80, in _as_root 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info
[Yahoo-eng-team] [Bug 1528977] Re: Neutron router not working with latest iproute2 package included in CentOS-7.2-1511
** Also affects: centos Importance: Undecided Status: New ** No longer affects: centos -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1528977 Title: Neutron router not working with latest iproute2 package included in CentOS-7.2-1511 Status in neutron: New Bug description: Seems that something has been changed in the new iproute version and now attempts to add more than one interface to router cause errors posted at the bottom. This affects neutron-l3-agent on CentOS-7.2-1511 and possible on RedHat (cannot check this). Quick solution is to simple downgrade the package: # wget http://mirror.centos.org/centos/7.1.1503/os/x86_64/Packages/iproute-3.10.0-21.el7.x86_64.rpm # yum -y downgrade ./iproute-3.10.0-21.el7.x86_64.rpm Part of the /var/log/neutron/l3-agent.log: 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.linux.utils [-] Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a'] Exit code: 1 Stdin: Stdout: Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info [-] Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a'] Exit code: 1 Stdin: Stdout: Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Traceback (most recent call last): 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 356, in call 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info return func(*args, **kwargs) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 692, in process 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 396, in _process_internal_ports 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self.internal_network_added(p) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 328, in internal_network_added 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info INTERNAL_DEV_PREFIX) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 303, in _internal_network_added 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info prefix=prefix) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 252, in plug 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info bridge, namespace, prefix) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 483, in plug_new 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info namespace2=namespace) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 144, in add_veth 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self.ensure_namespace(namespace2) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 163, in ensure_namespace 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info ip = self.netns.add(name) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 793, in add 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self._as_root([], ('add', name), use_root_namespace=True) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 280, in _as_root 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 80, in _as_root 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error) 2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File