Public bug reported:

when dhcp namespace is corrupted somehow, ovs bridge will be blasted
with thousands of dangling ports which are created by dhcp agent.

the corrupted namespace will cause following exception:

2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp 
[req-db1e4f25-2263-49e9-ba5b-308ea9ccfdec - - - - -] Unable to plug DHCP port 
for network 0c59667a-433a-4e97-9568-07ee6210c98b. Releasing port.
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp Traceback (most recent 
call last):
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", 
line 1407, in setup
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     self.plug(network, 
port, interface_name)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", 
line 1375, in plug
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
mtu=network.get('mtu'))
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/interface.py",
 line 268, in plug
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     bridge, namespace, 
prefix, mtu)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/interface.py",
 line 389, in plug_new
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
namespace_obj.add_device_to_namespace(ns_dev)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 232, in add_device_to_namespace
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
device.link.set_netns(self.namespace)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 516, in set_netns
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     self._as_root([], 
('set', self.name, 'netns', namespace))
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 364, in _as_root
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
use_root_namespace=use_root_namespace)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 100, in _as_root
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
log_fail_as_error=self.log_fail_as_error)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 109, in _execute
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
log_fail_as_error=log_fail_as_error)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", 
line 156, in execute
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     raise 
ProcessExecutionError(msg, returncode=returncode)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp ProcessExecutionError: 
Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp
2017-10-30 14:12:35.479 6 ERROR neutron.agent.linux.utils 
[req-29d446ad-eed5-47a0-bfc7-496dad2d35f2 - - - - -] Exit code: 2; Stdin: ; 
Stdout: ; Stderr: RTNETLINK answers: Invalid argument

** Affects: neutron
     Importance: Undecided
     Assignee: yong sheng gong (gongysh)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1728642

Title:
  corrupted namespace blasted ovs bridge with thousands of dangling port

Status in neutron:
  In Progress

Bug description:
  when dhcp namespace is corrupted somehow, ovs bridge will be blasted
  with thousands of dangling ports which are created by dhcp agent.

  the corrupted namespace will cause following exception:

  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp 
[req-db1e4f25-2263-49e9-ba5b-308ea9ccfdec - - - - -] Unable to plug DHCP port 
for network 0c59667a-433a-4e97-9568-07ee6210c98b. Releasing port.
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp Traceback (most 
recent call last):
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", 
line 1407, in setup
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
self.plug(network, port, interface_name)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", 
line 1375, in plug
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
mtu=network.get('mtu'))
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/interface.py",
 line 268, in plug
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     bridge, 
namespace, prefix, mtu)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/interface.py",
 line 389, in plug_new
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
namespace_obj.add_device_to_namespace(ns_dev)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 232, in add_device_to_namespace
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
device.link.set_netns(self.namespace)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 516, in set_netns
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
self._as_root([], ('set', self.name, 'netns', namespace))
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 364, in _as_root
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
use_root_namespace=use_root_namespace)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 100, in _as_root
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
log_fail_as_error=self.log_fail_as_error)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
 line 109, in _execute
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     
log_fail_as_error=log_fail_as_error)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", 
line 156, in execute
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     raise 
ProcessExecutionError(msg, returncode=returncode)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp 
ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK 
answers: Invalid argument
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp
  2017-10-30 14:12:35.479 6 ERROR neutron.agent.linux.utils 
[req-29d446ad-eed5-47a0-bfc7-496dad2d35f2 - - - - -] Exit code: 2; Stdin: ; 
Stdout: ; Stderr: RTNETLINK answers: Invalid argument

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1728642/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to