Public bug reported: In neutron-tempest-dvr-multinode-ha-full I found errors in L3 agent's logs. Error example is on: http://logs.openstack.org/79/633979/11/check/neutron-tempest-dvr-ha-multinode-full/415f056/controller/logs/screen-q-l3.txt.gz?level=ERROR#_Feb_22_12_00_34_232612
Error looks like: Feb 22 12:00:34.219810 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port Feb 22 12:00:34.219810 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent [-] Error while deleting router 1f90a750-d3af-454a-9031-29f399cf6457: AttributeError: 'NoneType' object has no attribute 'disable' Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Traceback (most recent call last): Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 447, in _router_added Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent ri.delete() Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 461, in delete Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent super(HaRouter, self).delete() Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 431, in delete Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self.disable_radvd() Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 547, in disable_radvd Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self.radvd.disable() Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'disable' Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 1f90a750-d3af-454a-9031-29f399cf6457: Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Traceback (most recent call last): Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 687, in _process_routers_if_compatible Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 562, in _process_router_if_compatible Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self._process_added_router(router) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 567, in _process_added_router Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self._router_added(router['id'], router) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 450, in _router_added Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent router_id) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self.force_reraise() Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 439, in _router_added Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent ri.initialize(self.process_monitor) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 126, in initialize Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent raise Exception(msg) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Funny thing is that I found it on my patch https://review.openstack.org/#/c/633979/ which is migrating job definition to zuulv3 syntax. There is no such error in existing legacy job because it is not creates HA routers by default. So this is another issue that neutron-tempest-dvr-HA-multinode-full job is actually not HA but that can be fixed in https://review.openstack.org/#/c/633979/ after this issue reported here will be fixed. ** Affects: neutron Importance: Medium Assignee: Slawek Kaplonski (slaweq) Status: Confirmed ** Tags: gate-failure l3-dvr-backlog -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1817435 Title: Error while stoping radvd on dvr ha router Status in neutron: Confirmed Bug description: In neutron-tempest-dvr-multinode-ha-full I found errors in L3 agent's logs. Error example is on: http://logs.openstack.org/79/633979/11/check/neutron-tempest-dvr-ha-multinode-full/415f056/controller/logs/screen-q-l3.txt.gz?level=ERROR#_Feb_22_12_00_34_232612 Error looks like: Feb 22 12:00:34.219810 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port Feb 22 12:00:34.219810 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent [-] Error while deleting router 1f90a750-d3af-454a-9031-29f399cf6457: AttributeError: 'NoneType' object has no attribute 'disable' Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Traceback (most recent call last): Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 447, in _router_added Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent ri.delete() Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 461, in delete Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent super(HaRouter, self).delete() Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 431, in delete Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self.disable_radvd() Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 547, in disable_radvd Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self.radvd.disable() Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'disable' Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 1f90a750-d3af-454a-9031-29f399cf6457: Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Traceback (most recent call last): Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 687, in _process_routers_if_compatible Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 562, in _process_router_if_compatible Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self._process_added_router(router) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 567, in _process_added_router Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self._router_added(router['id'], router) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 450, in _router_added Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent router_id) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent self.force_reraise() Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 439, in _router_added Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent ri.initialize(self.process_monitor) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 126, in initialize Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent raise Exception(msg) Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Funny thing is that I found it on my patch https://review.openstack.org/#/c/633979/ which is migrating job definition to zuulv3 syntax. There is no such error in existing legacy job because it is not creates HA routers by default. So this is another issue that neutron-tempest-dvr-HA-multinode-full job is actually not HA but that can be fixed in https://review.openstack.org/#/c/633979/ after this issue reported here will be fixed. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1817435/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
