Public bug reported: After rabbitmq cluster is in split brain state because of some network issues, neutron ovs agents in compute nodes are always reporting error info, some open flow rules are removed, so VMs can't be accessed normally.
Here are some important error info: 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:35:05.586 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec 2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:05.413 21720 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : e9e7407c173a438096eb1fad7e22d9fb 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:36:05.600 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent failed_devices['added'] |= self._bind_devices(need_binding_devices) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.conf.host, agent_restarted=agent_restarted) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent agent_restarted=agent_restarted) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent time.sleep(wait) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.force_reraise() 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent six.reraise(self.type_, self.value, self.tb) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._original_context.call(ctxt, method, **kwargs) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent failed_devices['added'] |= self._bind_devices(need_binding_devices) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.conf.host, agent_restarted=agent_restarted) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent agent_restarted=agent_restarted) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent time.sleep(wait) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.force_reraise() 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent six.reraise(self.type_, self.value, self.tb) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._original_context.call(ctxt, method, **kwargs) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944 2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944 2020-04-16 08:36:24.575 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Loop iteration exceeded interval (2 vs. 137.142431974)! loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1951 ** Affects: neutron Importance: Undecided Status: New ** Tags: rocky ** Summary changed: - VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors and after rabbitmq is in split-brain state + VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors after rabbitmq is in split-brain state -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1873265 Title: VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors after rabbitmq is in split-brain state Status in neutron: New Bug description: After rabbitmq cluster is in split brain state because of some network issues, neutron ovs agents in compute nodes are always reporting error info, some open flow rules are removed, so VMs can't be accessed normally. Here are some important error info: 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3 2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:35:05.586 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec 2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:05.413 21720 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : e9e7407c173a438096eb1fad7e22d9fb 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939 2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:36:05.600 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent failed_devices['added'] |= self._bind_devices(need_binding_devices) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.conf.host, agent_restarted=agent_restarted) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent agent_restarted=agent_restarted) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent time.sleep(wait) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.force_reraise() 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent six.reraise(self.type_, self.value, self.tb) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._original_context.call(ctxt, method, **kwargs) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent failed_devices['added'] |= self._bind_devices(need_binding_devices) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.conf.host, agent_restarted=agent_restarted) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent agent_restarted=agent_restarted) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent time.sleep(wait) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.force_reraise() 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent six.reraise(self.type_, self.value, self.tb) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._original_context.call(ctxt, method, **kwargs) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a 2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944 2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944 2020-04-16 08:36:24.575 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Loop iteration exceeded interval (2 vs. 137.142431974)! loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1951 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1873265/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp