Public bug reported:
When hosting a large amount of routers and/or networks the RPC calls
from the agents can take a long time which requires us to increase the
rpc_response_timeout from the default of 60 seconds to a higher value
for the agents to not timeout.
This has the side effect that if a rabbitmq or neutron-server is
restarted all agents that is currently reporting there will hang for a
long time until report_state times out, during this time neutron-server
has not got any reports causing it to set the agent as down.
When it times out and tries again the reporting will succeed but a full
sync will be triggered for all agents that was previously dead. This in
itself can cause a very high load on the control plane.
Consider the fact that a configuration change is deployed using tooling
to all neutron-server nodes which is restarted, all agents will die,
when they either 1) come back after rpc_response_timeout is reached and
tries again or 2) is restarted manually all of them will do a full sync.
We should have a configuration option that only applies to the rpc
timeout for the report_state RPC call from agents because that could be
lowered to be within the bounds of the agent not being seen as down.
The old behavior can be kept by simply falling back to
rpc_response_timeout by default instead of introducing a new default in
this override.
** Affects: neutron
Importance: Undecided
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1948676
Title:
rpc response timeout for agent report_state is not possible
Status in neutron:
In Progress
Bug description:
When hosting a large amount of routers and/or networks the RPC calls
from the agents can take a long time which requires us to increase the
rpc_response_timeout from the default of 60 seconds to a higher value
for the agents to not timeout.
This has the side effect that if a rabbitmq or neutron-server is
restarted all agents that is currently reporting there will hang for a
long time until report_state times out, during this time neutron-
server has not got any reports causing it to set the agent as down.
When it times out and tries again the reporting will succeed but a
full sync will be triggered for all agents that was previously dead.
This in itself can cause a very high load on the control plane.
Consider the fact that a configuration change is deployed using
tooling to all neutron-server nodes which is restarted, all agents
will die, when they either 1) come back after rpc_response_timeout is
reached and tries again or 2) is restarted manually all of them will
do a full sync.
We should have a configuration option that only applies to the rpc
timeout for the report_state RPC call from agents because that could
be lowered to be within the bounds of the agent not being seen as
down.
The old behavior can be kept by simply falling back to
rpc_response_timeout by default instead of introducing a new default
in this override.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1948676/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
