Public bug reported:

It happens that tpi patch ports between trunk bridge and integration
bridge don't have internal vlan tag in other_config row in ovsdb. It
looks like a race between trunk handler and ovs agent.

Example of failure: http://logs.openstack.org/92/527992/2/check/neutron-
tempest-plugin-dvr-multinode-
scenario/166eee3/logs/subnode-2/screen-q-agt.txt.gz#_Dec_14_18_31_25_801432

Trace example: 
Dec 14 18:31:25.801432 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None 
req-412981e2-ac8a-4fe8-8ec4-288bbb63e2a7 None None] Error while processing VIF 
ports: OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its 
other_config: {}
Dec 14 18:31:25.801580 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most 
recent call last):
Dec 14 18:31:25.801708 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2080, in rpc_loop
Dec 14 18:31:25.801838 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, 
ovs_restarted)
Dec 14 18:31:25.801965 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 157, in 
wrapper
Dec 14 18:31:25.802089 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = 
f(*args, **kwargs)
Dec 14 18:31:25.802214 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 1676, in process_network_ports
Dec 14 18:31:25.802345 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
port_info.get('updated', set()))
Dec 14 18:31:25.802476 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 256, in 
setup_port_filters
Dec 14 18:31:25.802600 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
self.refresh_firewall(updated_devices)
Dec 14 18:31:25.802725 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 110, in 
decorated_function
Dec 14 18:31:25.802850 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, 
**kwargs)
Dec 14 18:31:25.802983 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 209, in 
refresh_firewall
Dec 14 18:31:25.803103 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
self._apply_port_filter(device_ids, update_filter=True)
Dec 14 18:31:25.803237 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 141, in 
_apply_port_filter
Dec 14 18:31:25.803366 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
self.firewall.update_port_filter(device)
Dec 14 18:31:25.803492 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 509, in update_port_filter
Dec 14 18:31:25.803612 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
self.prepare_port_filter(port)
Dec 14 18:31:25.803763 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 492, in prepare_port_filter
Dec 14 18:31:25.803894 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = 
self.get_or_create_ofport(port)
Dec 14 18:31:25.804021 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 464, in get_or_create_ofport
Dec 14 18:31:25.804152 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
Dec 14 18:31:25.804275 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 448, in _get_port_vlan_tag
Dec 14 18:31:25.804414 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return 
get_tag_from_other_config(self.int_br.br, port_name)
Dec 14 18:31:25.804548 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 74, in get_tag_from_other_config
Dec 14 18:31:25.804670 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
port_name=port_name, other_config=other_config)
Dec 14 18:31:25.804794 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its 
other_config: {}
Dec 14 18:31:25.804923 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

** Affects: neutron
     Importance: High
     Assignee: Jakub Libosvar (libosvar)
         Status: Confirmed


** Tags: ovs-fw sg-fw trunk

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1740885

Title:
  Trunk ports are sometimes not tagged with internal vlan

Status in neutron:
  Confirmed

Bug description:
  It happens that tpi patch ports between trunk bridge and integration
  bridge don't have internal vlan tag in other_config row in ovsdb. It
  looks like a race between trunk handler and ovs agent.

  Example of failure: http://logs.openstack.org/92/527992/2/check
  /neutron-tempest-plugin-dvr-multinode-
  scenario/166eee3/logs/subnode-2/screen-q-agt.txt.gz#_Dec_14_18_31_25_801432

  Trace example: 
  Dec 14 18:31:25.801432 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None 
req-412981e2-ac8a-4fe8-8ec4-288bbb63e2a7 None None] Error while processing VIF 
ports: OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its 
other_config: {}
  Dec 14 18:31:25.801580 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most 
recent call last):
  Dec 14 18:31:25.801708 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2080, in rpc_loop
  Dec 14 18:31:25.801838 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, 
ovs_restarted)
  Dec 14 18:31:25.801965 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 157, in 
wrapper
  Dec 14 18:31:25.802089 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = 
f(*args, **kwargs)
  Dec 14 18:31:25.802214 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 1676, in process_network_ports
  Dec 14 18:31:25.802345 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
port_info.get('updated', set()))
  Dec 14 18:31:25.802476 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 256, in 
setup_port_filters
  Dec 14 18:31:25.802600 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
self.refresh_firewall(updated_devices)
  Dec 14 18:31:25.802725 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 110, in 
decorated_function
  Dec 14 18:31:25.802850 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, 
**kwargs)
  Dec 14 18:31:25.802983 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 209, in 
refresh_firewall
  Dec 14 18:31:25.803103 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
self._apply_port_filter(device_ids, update_filter=True)
  Dec 14 18:31:25.803237 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 141, in 
_apply_port_filter
  Dec 14 18:31:25.803366 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
self.firewall.update_port_filter(device)
  Dec 14 18:31:25.803492 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 509, in update_port_filter
  Dec 14 18:31:25.803612 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
self.prepare_port_filter(port)
  Dec 14 18:31:25.803763 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 492, in prepare_port_filter
  Dec 14 18:31:25.803894 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     of_port = 
self.get_or_create_ofport(port)
  Dec 14 18:31:25.804021 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 464, in get_or_create_ofport
  Dec 14 18:31:25.804152 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
  Dec 14 18:31:25.804275 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 448, in _get_port_vlan_tag
  Dec 14 18:31:25.804414 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return 
get_tag_from_other_config(self.int_br.br, port_name)
  Dec 14 18:31:25.804548 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File 
"/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", 
line 74, in get_tag_from_other_config
  Dec 14 18:31:25.804670 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     
port_name=port_name, other_config=other_config)
  Dec 14 18:31:25.804794 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its 
other_config: {}
  Dec 14 18:31:25.804923 ubuntu-xenial-rax-ord-0001444835 
neutron-openvswitch-agent[17015]: ERROR 
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1740885/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to