Public bug reported:
The neutron-dhcp-agent has a resync_throttle mechanism to ensure a minimum
event interval if events occur too frequently [0].
However, this mechanism doesn't work because a new throttler object is created
each time.
I conducted a simple experiment by forcing a failure within the
'sync_state' call and observed no throttling of sync_state:
--- a/neutron/agent/dhcp/agent.py
+++ b/neutron/agent/dhcp/agent.py
@@ -303,6 +303,7 @@ class DhcpAgent(manager.Manager):
known_network_ids = set(self.cache.get_network_ids())
try:
+ raise Exception('test sync_state error')
active_networks = self.plugin_rpc.get_active_networks_info(
enable_dhcp_filter=False)
LOG.info('All active networks have been fetched through RPC.')
Hundreds attempts to call 'Synchronizing state' per second:
Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent [-] Unable to sync network
state.: Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Traceback (most recent call
last):
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent File
"/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 306, in sync_state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent raise Exception('test
sync_state error')
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] Resync event has been
scheduled {{(pid=798097) _periodic_resync_helper
/opt/stack/neutron/neutron/agent/dhcp/agent.py:385}}
Jan 23 07:04:39 DEBUG neutron.common.utils [-] Calling throttled function clear
{{(pid=798097) wrapper /opt/stack/neutron/neutron/common/utils.py:117}}
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] resync (*): [Exception('test
sync_state error')] {{(pid=798097) _periodic_resync_helper
/opt/stack/neutron/neutron/agent/dhcp/agent.py:401}}
Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent [-] Unable to sync network
state.: Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Traceback (most recent call
last):
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent File
"/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 306, in sync_state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent raise Exception('test
sync_state error')
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] Resync event has been
scheduled {{(pid=798097) _periodic_resync_helper
/opt/stack/neutron/neutron/agent/dhcp/agent.py:385}}
Jan 23 07:04:39 DEBUG neutron.common.utils [-] Calling throttled function clear
{{(pid=798097) wrapper /opt/stack/neutron/neutron/common/utils.py:117}}
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] resync (*): [Exception('test
sync_state error')] {{(pid=798097) _periodic_resync_helper
/opt/stack/neutron/neutron/agent/dhcp/agent.py:401}}
Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
[0] https://bugs.launchpad.net/neutron/+bug/1780370
** Affects: neutron
Importance: Undecided
Assignee: Anton Kurbatov (akurbatov)
Status: New
** Changed in: neutron
Assignee: (unassigned) => Anton Kurbatov (akurbatov)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2095595
Title:
The neutron-dhcp-agent resync throttling mechanism doesn't work
Status in neutron:
New
Bug description:
The neutron-dhcp-agent has a resync_throttle mechanism to ensure a minimum
event interval if events occur too frequently [0].
However, this mechanism doesn't work because a new throttler object is
created each time.
I conducted a simple experiment by forcing a failure within the
'sync_state' call and observed no throttling of sync_state:
--- a/neutron/agent/dhcp/agent.py
+++ b/neutron/agent/dhcp/agent.py
@@ -303,6 +303,7 @@ class DhcpAgent(manager.Manager):
known_network_ids = set(self.cache.get_network_ids())
try:
+ raise Exception('test sync_state error')
active_networks = self.plugin_rpc.get_active_networks_info(
enable_dhcp_filter=False)
LOG.info('All active networks have been fetched through RPC.')
Hundreds attempts to call 'Synchronizing state' per second:
Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent [-] Unable to sync network
state.: Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Traceback (most recent call
last):
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent File
"/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 306, in sync_state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent raise Exception('test
sync_state error')
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Exception: test sync_state
error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] Resync event has been
scheduled {{(pid=798097) _periodic_resync_helper
/opt/stack/neutron/neutron/agent/dhcp/agent.py:385}}
Jan 23 07:04:39 DEBUG neutron.common.utils [-] Calling throttled function
clear {{(pid=798097) wrapper /opt/stack/neutron/neutron/common/utils.py:117}}
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] resync (*):
[Exception('test sync_state error')] {{(pid=798097) _periodic_resync_helper
/opt/stack/neutron/neutron/agent/dhcp/agent.py:401}}
Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent [-] Unable to sync network
state.: Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Traceback (most recent call
last):
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent File
"/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 306, in sync_state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent raise Exception('test
sync_state error')
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Exception: test sync_state
error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] Resync event has been
scheduled {{(pid=798097) _periodic_resync_helper
/opt/stack/neutron/neutron/agent/dhcp/agent.py:385}}
Jan 23 07:04:39 DEBUG neutron.common.utils [-] Calling throttled function
clear {{(pid=798097) wrapper /opt/stack/neutron/neutron/common/utils.py:117}}
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] resync (*):
[Exception('test sync_state error')] {{(pid=798097) _periodic_resync_helper
/opt/stack/neutron/neutron/agent/dhcp/agent.py:401}}
Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
[0] https://bugs.launchpad.net/neutron/+bug/1780370
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2095595/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
