Public bug reported:
Following seen in l3 agent logs:
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent [-] Failed to
process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent Traceback (most
recent call last):
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in
_process_router_update
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
self._process_router_if_compatible(router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 434, in
_process_router_if_compatible
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
self._process_added_router(router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 439, in
_process_added_router
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
self._router_added(router['id'], router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 340, in
_router_added
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent ri =
self._create_router(router_id, router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 337, in
_create_router
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent return
legacy_router.LegacyRouter(*args, **kwargs)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 61, in
__init__
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
DEFAULT_ADDRESS_SCOPE: ADDRESS_SCOPE_MARK_IDS.pop()}
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent KeyError: 'pop from
an empty set'
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
2016-04-05 09:30:09.034 24216 DEBUG neutron.agent.l3.agent [-] Starting router
update for e341e0e2-5089-46e9-91f9-2099a156b27f, action None, priority 1
_process_router_update
/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:463
2016-04-05 09:30:09.035 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL
msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 exchange 'neutron' topic 'q-l3-plugin'
_send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-04-05 09:30:09.417 24216 DEBUG oslo_messaging._drivers.amqpdriver [-]
received reply msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 __call__
/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:302
2016-04-05 09:30:09.418 24216 ERROR neutron.agent.l3.agent [-] Failed to
process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
So agent is constantly resyncing (causing load on neutron server) and
unable to handle new routers.
I believe that set "ADDRESS_SCOPE_MARK_IDS = set(range(1024, 2048))"
from router_info.py should not be agent global but it should be
ADDRESS_SCOPE_MARK_IDS per router. Or at least need to return values
back to the set when router is deleted.
** Affects: neutron
Importance: Undecided
Assignee: Oleg Bondarev (obondarev)
Status: New
** Tags: l3-ipam-dhcp
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1566291
Title:
L3 agent: at some point an agent becomes unable to handle new routers
Status in neutron:
New
Bug description:
Following seen in l3 agent logs:
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent [-] Failed to
process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent Traceback (most
recent call last):
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in
_process_router_update
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
self._process_router_if_compatible(router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 434, in
_process_router_if_compatible
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
self._process_added_router(router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 439, in
_process_added_router
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
self._router_added(router['id'], router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 340, in
_router_added
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent ri =
self._create_router(router_id, router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 337, in
_create_router
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent return
legacy_router.LegacyRouter(*args, **kwargs)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 61, in
__init__
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
DEFAULT_ADDRESS_SCOPE: ADDRESS_SCOPE_MARK_IDS.pop()}
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent KeyError: 'pop
from an empty set'
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
2016-04-05 09:30:09.034 24216 DEBUG neutron.agent.l3.agent [-] Starting
router update for e341e0e2-5089-46e9-91f9-2099a156b27f, action None, priority 1
_process_router_update
/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:463
2016-04-05 09:30:09.035 24216 DEBUG oslo_messaging._drivers.amqpdriver [-]
CALL msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 exchange 'neutron' topic
'q-l3-plugin' _send
/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-04-05 09:30:09.417 24216 DEBUG oslo_messaging._drivers.amqpdriver [-]
received reply msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 __call__
/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:302
2016-04-05 09:30:09.418 24216 ERROR neutron.agent.l3.agent [-] Failed to
process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
So agent is constantly resyncing (causing load on neutron server) and
unable to handle new routers.
I believe that set "ADDRESS_SCOPE_MARK_IDS = set(range(1024, 2048))"
from router_info.py should not be agent global but it should be
ADDRESS_SCOPE_MARK_IDS per router. Or at least need to return values
back to the set when router is deleted.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1566291/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
