Sanjeev N created CLOUDSTACK-6715:
-------------------------------------
Summary: [SDN] Inconsistency in ovs-flow table after vm migration
from one host to another
Key: CLOUDSTACK-6715
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6715
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server, Network Controller
Affects Versions: 4.4.0
Environment: Latest build from 4.4 with commit
e6961fd21bb6d793302c234d0f409f66dc498072
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.4.0
[SDN] Inconsistency in ovs-flow table after vm migration from one host to
another
Steps to reproduce:
================
1.Bring up CS in advanced zone with two xen hosts in a cluster
2.Create physical network with GRE isolation
3.Create isolated network with OVS provider
4.Deploy few vms in the network and make sure that all the vms(including VR)
are deployed on only one host.
5.Now migrate one vm to another host in the cluster and verify flow tables for
this isolated network bridge on both the xen hosts
Result:
=======
Inconsistency in flow tables on both the xen hosts
Following it the flow table from the host before vm migration (All the vms and
VR were only on this host before vm migration):
[root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=1173.14s, table=0, n_packets=0, n_bytes=0,
priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:1,output:2
cookie=0x0, duration=1173.15s, table=0, n_packets=0, n_bytes=0,
priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
cookie=0x0, duration=1203.276s, table=0, n_packets=0, n_bytes=0,
priority=1200,ip,in_port=1,nw_dst=224.0.0.0/24 actions=NORMAL
cookie=0x0, duration=1226.258s, table=0, n_packets=901, n_bytes=85612,
priority=0 actions=NORMAL
cookie=0x0, duration=1173.129s, table=0, n_packets=0, n_bytes=0,
priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:1,output:2
cookie=0x0, duration=1203.286s, table=0, n_packets=0, n_bytes=0,
priority=1200,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
cookie=0x0, duration=1173.167s, table=0, n_packets=7, n_bytes=1494,
priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
Flow table after vm migration on the same host:
[root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=992.789s, table=0, n_packets=0, n_bytes=0, priority=0
actions=NORMAL
[root@xen-host-14 ~]#
Ports info on the bridge after vm migration:
[root@xen-host-14 ~]# ovs-vsctl list-ports xapi4
t986-2-1
vif11.0
vif12.0
Flow table on the target host where vm was migrated to:
[root@xen-host-13 ~]# ovs-ofctl dump-flows xapi4
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=1025.129s, table=0, n_packets=0, n_bytes=0,
priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:2
cookie=0x0, duration=1025.139s, table=0, n_packets=0, n_bytes=0,
priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
cookie=0x0, duration=1032.932s, table=0, n_packets=0, n_bytes=0,
priority=1000,ip,in_port=1,nw_dst=224.0.0.0/24 actions=drop
cookie=0x0, duration=1033.247s, table=0, n_packets=0, n_bytes=0, priority=0
actions=NORMAL
cookie=0x0, duration=1025.119s, table=0, n_packets=0, n_bytes=0,
priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:2
cookie=0x0, duration=1032.942s, table=0, n_packets=0, n_bytes=0,
priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=drop
cookie=0x0, duration=1025.148s, table=0, n_packets=1, n_bytes=42,
priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
[root@xen-host-13 ~]# ovs-vsctl list-ports xapi4
t986-1-2
vif17.0
Following is the log snippet from both the hosts during vm migration (tunnel
creation during vm migration):
2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter create_tunnel ####
2014-05-20 11:31:21 DEBUG [root] Creating tunnel from host 2 to host 1 with
GRE key 986
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge',
'xapi4', 'name']
2014-05-20 11:31:21 DEBUG [root] bridge xapi4 for creating tunnel - VERIFIED
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'add-port', 'xapi4', 't986-2-1', '--', 'set', 'interface', 't986-2-1',
'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.13']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'port', 't986-2-1', 'interfaces']
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:key']
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
2014-05-20 11:31:22 DEBUG [root] Tunnel interface
validated:['/usr/bin/ovs-vsctl', 'get', 'interface',
'01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'ofport']
2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
2014-05-20 11:31:22 DEBUG [root] The command exited with the error code: 1
(stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
)
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,ip,nw_dst=224.0.0.0/24,actions=drop']
2014-05-20 11:31:22 DEBUG [root] Broadcast drop rules added
2014-05-20 11:31:22 DEBUG [root] Successfully created tunnel from host 2 to
host 1 with GRE key 986
2014-05-20 11:31:22 DEBUG [root] #### VMOPS exit create_tunnel ####
2014-05-20 11:31:29 DEBUG [root] Executing:['cat',
'/etc/xensource/network.conf']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'iface-to-br', 'vif19.0']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
2014-05-20 11:31:29 DEBUG [root] The command exited with the error code: 1
(stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
)
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'br-to-vlan', 'xapi4']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'list-ports', 'xapi4']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'Interface', 't986-2-1', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'Interface', 'vif11.0', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'Interface', 'vif12.0', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'Interface', 'vif19.0', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'del-flows', 'xapi4', ',in_port=3']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:1,output:2']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:1,output:2']
ovstunnel log from the host where vm has been migrated to:
2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter setup_ovs_bridge ####
2014-05-20 11:31:21 DEBUG [root] About to manually create the bridge:xapi4
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--',
'--may-exist', 'add-br', 'xapi4', '--', 'set', 'bridge', 'xapi4',
'other_config:gre_key=OVSTunnel986']
2014-05-20 11:31:21 DEBUG [root] Bridge has been manually created:
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set',
'Bridge', 'xapi4',
'external_ids:xs-network-uuid=98636962-280c-bb8f-1e57-d79bcc295c1f']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set',
'Bridge', 'xapi4', 'stp_enable=true']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'bridge', 'xapi4', 'other_config:gre_key']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-set', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'other-config:is-ovs-tun-network=True']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=ovs-host-setup']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-set', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'other-config:ovs-host-setup=1,2']
2014-05-20 11:31:21 DEBUG [root] Setup_ovs_bridge completed with
result:SUCCESS:xapi4
2014-05-20 11:31:21 DEBUG [root] #### VMOPS exit setup_ovs_bridge ####
2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter create_tunnel ####
2014-05-20 11:31:21 DEBUG [root] Creating tunnel from host 1 to host 2 with
GRE key 986
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge',
'xapi4', 'name']
2014-05-20 11:31:21 DEBUG [root] bridge xapi4 for creating tunnel - VERIFIED
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'add-port', 'xapi4', 't986-1-2', '--', 'set', 'interface', 't986-1-2',
'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.14']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'port', 't986-1-2', 'interfaces']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:key']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
2014-05-20 11:31:21 DEBUG [root] Tunnel interface
validated:['/usr/bin/ovs-vsctl', 'get', 'interface',
'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'ofport']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
2014-05-20 11:31:21 DEBUG [root] The command exited with the error code: 1
(stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
)
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,ip,nw_dst=224.0.0.0/24,actions=drop']
2014-05-20 11:31:21 DEBUG [root] Broadcast drop rules added
2014-05-20 11:31:21 DEBUG [root] Successfully created tunnel from host 1 to
host 2 with GRE key 986
2014-05-20 11:31:21 DEBUG [root] #### VMOPS exit create_tunnel ####
2014-05-20 11:31:29 DEBUG [root] Executing:['cat',
'/etc/xensource/network.conf']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'iface-to-br', 'vif17.0']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe',
'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f',
'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
2014-05-20 11:31:29 DEBUG [root] The command exited with the error code: 1
(stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
)
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'br-to-vlan', 'xapi4']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl',
'list-ports', 'xapi4']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'Interface', 't986-1-2', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get',
'Interface', 'vif17.0', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff,actions=NORMAL']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,ip,nw_dst=224.0.0.0/24,actions=NORMAL']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:2']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl',
'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:2']
~
--
This message was sent by Atlassian JIRA
(v6.2#6252)
