** Also affects: ovsdbapp
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1881424
Title:
Neutron ovs agent fails on rpc_loop iteration:1
Status in kolla-ansible:
Triaged
Status in kolla-ansible victoria series:
Triaged
Status in neutron:
New
Status in ovsdbapp:
New
Bug description:
Hi Neutrinos!
This is from Kolla-Ansible CI, it started happening in Victoria on May 28.
It affects all distros deb-ubu-centos and makes the jobs fail (ovs agent is
dead).
It does *not* affect OVN though.
br-ex exists before iteration:0 and acts fine in iteration:0 but not
in iteration:1.
The only quasi-relevant change in neutron seems to be
https://review.opendev.org/721554 but still it should only affect DVR
which we are not running. Odd.
Full logs with config and debug are attached.
The relevant logs look like this (bottom of the file):
2020-05-30 15:27:59.003 7 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent rpc_loop -
iteration:1 started
2020-05-30 15:27:59.006 7 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] ofctl request
version=0x4,msg_type=0x12,msg_len=0x38,xid=0x15f04540,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=23,type=1)
result
[OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=371283074374527098,duration_nsec=718000000,duration_sec=5,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_count=0,priority=0,table_id=23)],flags=0,type=1)]
_send_msg
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:113
2020-05-30 15:27:59.008 7 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Physical bridge br-ex was
just re-created.
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name
autocreate_indices
/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name
autocreate_indices
/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name
autocreate_indices
/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.010 7 DEBUG ovsdbapp.backend.ovs_idl
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index target
autocreate_indices
/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.011 7 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Mapping physical network
physnet1 to bridge br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Error executing command:
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with
name=br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command Traceback
(most recent call last):
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py",
line 38, in execute
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command
self.run_idl(None)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py",
line 214, in run_idl
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command record =
self.api.lookup(self.table, self.record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py",
line 171, in lookup
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command return
self._lookup(table, record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py",
line 218, in _lookup
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command row =
idlutils.row_by_value(self, rl.table, rl.column, record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py",
line 95, in row_by_value
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command raise
RowNotFound(table=table, col=column, match=match)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with
name=br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command
2020-05-30 15:27:59.015 7 DEBUG neutron.agent.common.async_process
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Halting async process
[ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids
--format=json]. stop
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/common/async_process.py:135
2020-05-30 15:27:59.015 7 DEBUG neutron.agent.linux.utils
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Running command: ['kill',
'-9', '75'] create_process
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent main thread died of
an exception: TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
Traceback (most recent call last):
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py",
line 40, in agent_main_wrapper
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
ovs_agent.main(bridge_classes)
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2780, in main
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
agent.daemon_loop()
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2674, in daemon_loop
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
self.rpc_loop(polling_manager=pm)
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2545, in rpc_loop
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
added_bridges)
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1416, in _reconfigure_physical_bridges
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
sync = self._do_reconfigure_physical_bridges(bridges)
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1439, in _do_reconfigure_physical_bridges
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
self.setup_physical_bridges(bridge_mappings)
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1495, in setup_physical_bridges
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
self._check_bridge_datapath_id(br, datapath_ids_set)
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1457, in _check_bridge_datapath_id
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
dpid = int(bridge.get_datapath_id(), 16)
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.036 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
2020-05-30 15:27:59.048 7 DEBUG neutron.agent.common.async_process [-] Output
received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface
name,ofport,external_ids --format=json]: None _read_stdout
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/common/async_process.py:262
2020-05-30 15:27:59.048 7 ERROR neutron.agent.common.async_process [-] Error
received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface
name,ofport,external_ids --format=json]: None
2020-05-30 15:27:59.052 7 CRITICAL neutron [-] Unhandled error: TypeError:
int() can't convert non-string with explicit base
2020-05-30 15:27:59.052 7 ERROR neutron Traceback (most recent call last):
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/bin/neutron-openvswitch-agent", line 8, in <module>
2020-05-30 15:27:59.052 7 ERROR neutron sys.exit(main())
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py",
line 20, in main
2020-05-30 15:27:59.052 7 ERROR neutron agent_main.main()
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py",
line 37, in main
2020-05-30 15:27:59.052 7 ERROR neutron of_main.main()
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py",
line 35, in main
2020-05-30 15:27:59.052 7 ERROR neutron
'neutron.plugins.ml2.drivers.openvswitch.agent.'
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/base/app_manager.py",
line 375, in run_apps
2020-05-30 15:27:59.052 7 ERROR neutron hub.joinall(services)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 102,
in joinall
2020-05-30 15:27:59.052 7 ERROR neutron t.wait()
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line
181, in wait
2020-05-30 15:27:59.052 7 ERROR neutron return self._exit_event.wait()
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/event.py", line 132,
in wait
2020-05-30 15:27:59.052 7 ERROR neutron current.throw(*self._exc)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line
221, in main
2020-05-30 15:27:59.052 7 ERROR neutron result = function(*args, **kwargs)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 64,
in _launch
2020-05-30 15:27:59.052 7 ERROR neutron raise e
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 59,
in _launch
2020-05-30 15:27:59.052 7 ERROR neutron return func(*args, **kwargs)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py",
line 43, in agent_main_wrapper
2020-05-30 15:27:59.052 7 ERROR neutron LOG.exception("Agent main thread
died of an exception")
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line
220, in __exit__
2020-05-30 15:27:59.052 7 ERROR neutron self.force_reraise()
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line
196, in force_reraise
2020-05-30 15:27:59.052 7 ERROR neutron six.reraise(self.type_,
self.value, self.tb)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-05-30 15:27:59.052 7 ERROR neutron raise value
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py",
line 40, in agent_main_wrapper
2020-05-30 15:27:59.052 7 ERROR neutron ovs_agent.main(bridge_classes)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2780, in main
2020-05-30 15:27:59.052 7 ERROR neutron agent.daemon_loop()
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2674, in daemon_loop
2020-05-30 15:27:59.052 7 ERROR neutron self.rpc_loop(polling_manager=pm)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2545, in rpc_loop
2020-05-30 15:27:59.052 7 ERROR neutron added_bridges)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1416, in _reconfigure_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron sync =
self._do_reconfigure_physical_bridges(bridges)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1439, in _do_reconfigure_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron
self.setup_physical_bridges(bridge_mappings)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1495, in setup_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron
self._check_bridge_datapath_id(br, datapath_ids_set)
2020-05-30 15:27:59.052 7 ERROR neutron File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1457, in _check_bridge_datapath_id
2020-05-30 15:27:59.052 7 ERROR neutron dpid =
int(bridge.get_datapath_id(), 16)
2020-05-30 15:27:59.052 7 ERROR neutron TypeError: int() can't convert
non-string with explicit base
2020-05-30 15:27:59.052 7 ERROR neutron
To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1881424/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
