Thx for the info, I'll try reproduce it on my local setup using active-backup bond. And use it as managment interface,
Will update if I run into the same issue~ Thanks, Alex Wang, On Wed, Jun 3, 2015 at 11:52 PM, Chris <[email protected]> wrote: > Hello Alex, > > > > I will do the gdb debug when it happens again. > > > > Here the output, in this case there are 4 VMs running on the host. The > physical interfaces eth2 & eth3 are in active-passive mode bonded to bond0. > mgmt0 is used for the management of the host system. Anything else is > created by OpenStack. > > The failure happens independent from the number of VMs running on the > host: > > > > *ovs-vsctl show* > > fbbaf640-ed82-4735-99d2-fbe09f4041f1 > > Bridge "br-bond0" > > Port "mgmt0" > > Interface "mgmt0" > > type: internal > > Port "br-bond0" > > Interface "br-bond0" > > type: internal > > Port "phy-br-bond0" > > Interface "phy-br-bond0" > > Port "bond0" > > Interface "eth2" > > Interface "eth3" > > Bridge br-int > > fail_mode: secure > > Port br-int > > Interface br-int > > type: internal > > Port "qvo4166dc0a-69" > > tag: 1 > > Interface "qvo4166dc0a-69" > > Port "qvo6d8b70de-9c" > > tag: 1 > > Interface "qvo6d8b70de-9c" > > Port "qvoe75237cc-7f" > > tag: 1 > > Interface "qvoe75237cc-7f" > > Port "qvoe3b9e1fc-a5" > > tag: 1 > > Interface "qvoe3b9e1fc-a5" > > Port "int-br-bond0" > > Interface "int-br-bond0" > > ovs_version: "2.3.0" > > > > *ip a* > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > inet 127.0.0.1/8 scope host lo > > 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 > > link/ether 9c:b6:54:b3:67:34 brd ff:ff:ff:ff:ff:ff > > 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 > > link/ether 9c:b6:54:b3:67:35 brd ff:ff:ff:ff:ff:ff > > 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen > 1000 > > link/ether 64:51:06:f0:85:98 brd ff:ff:ff:ff:ff:ff > > 5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen > 1000 > > link/ether 64:51:06:f0:85:9c brd ff:ff:ff:ff:ff:ff > > 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > > link/ether de:76:ca:6d:57:48 brd ff:ff:ff:ff:ff:ff > > 7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > > link/ether ee:f8:2f:ef:94:42 brd ff:ff:ff:ff:ff:ff > > 8: br-bond0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > > link/ether 64:51:06:f0:85:98 brd ff:ff:ff:ff:ff:ff > > 15: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state > UNKNOWN > > link/ether fa:e1:51:33:5d:dc brd ff:ff:ff:ff:ff:ff > > inet 10.201.195.75/24 brd 10.201.195.255 scope global mgmt0 > > 16: qbr4166dc0a-69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UNKNOWN > > link/ether be:9a:00:23:b1:f3 brd ff:ff:ff:ff:ff:ff > > 17: qvo4166dc0a-69: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 > qdisc pfifo_fast state UP qlen 1000 > > link/ether ae:27:12:55:09:0c brd ff:ff:ff:ff:ff:ff > > 18: qvb4166dc0a-69: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 > qdisc pfifo_fast state UP qlen 1000 > > link/ether be:9a:00:23:b1:f3 brd ff:ff:ff:ff:ff:ff > > 19: tap4166dc0a-69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > pfifo_fast state UNKNOWN qlen 500 > > link/ether fe:16:3e:d6:34:82 brd ff:ff:ff:ff:ff:ff > > 20: qbre75237cc-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UNKNOWN > > link/ether c2:f2:1a:30:a2:84 brd ff:ff:ff:ff:ff:ff > > 21: qvoe75237cc-7f: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 > qdisc pfifo_fast state UP qlen 1000 > > link/ether 1e:7f:f4:57:aa:82 brd ff:ff:ff:ff:ff:ff > > 22: qvbe75237cc-7f: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 > qdisc pfifo_fast state UP qlen 1000 > > link/ether c2:f2:1a:30:a2:84 brd ff:ff:ff:ff:ff:ff > > 23: tape75237cc-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > pfifo_fast state UNKNOWN qlen 500 > > link/ether fe:16:3e:02:a4:d7 brd ff:ff:ff:ff:ff:ff > > 24: qbre3b9e1fc-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UNKNOWN > > link/ether ce:c7:ec:59:fd:1a brd ff:ff:ff:ff:ff:ff > > 25: qvoe3b9e1fc-a5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 > qdisc pfifo_fast state UP qlen 1000 > > link/ether c6:5f:19:25:7c:be brd ff:ff:ff:ff:ff:ff > > 26: qvbe3b9e1fc-a5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 > qdisc pfifo_fast state UP qlen 1000 > > link/ether ce:c7:ec:59:fd:1a brd ff:ff:ff:ff:ff:ff > > 27: tape3b9e1fc-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > pfifo_fast state UNKNOWN qlen 500 > > link/ether fe:16:3e:bb:a7:19 brd ff:ff:ff:ff:ff:ff > > 28: qbr6d8b70de-9c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UNKNOWN > > link/ether c2:45:27:31:f8:d1 brd ff:ff:ff:ff:ff:ff > > 29: qvo6d8b70de-9c: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 > qdisc pfifo_fast state UP qlen 1000 > > link/ether 46:23:9e:4b:2a:fd brd ff:ff:ff:ff:ff:ff > > 30: qvb6d8b70de-9c: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 > qdisc pfifo_fast state UP qlen 1000 > > link/ether c2:45:27:31:f8:d1 brd ff:ff:ff:ff:ff:ff > > 31: tap6d8b70de-9c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > pfifo_fast state UNKNOWN qlen 500 > > link/ether fe:16:3e:40:9e:44 brd ff:ff:ff:ff:ff:ff > > 32: bond0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > > link/ether f6:83:30:bc:4c:7c brd ff:ff:ff:ff:ff:ff > > 35: phy-br-bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > pfifo_fast state UP qlen 1000 > > link/ether 76:51:9d:bd:24:83 brd ff:ff:ff:ff:ff:ff > > 36: int-br-bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > pfifo_fast state UP qlen 1000 > > link/ether 5e:fa:05:2a:1a:ef brd ff:ff:ff:ff:ff:ff > > > > > > Cheers, > > Chris > > > > > > *From:* Alex Wang [mailto:[email protected]] > *Sent:* Thursday, June 04, 2015 13:29 > *To:* Chris > *Cc:* [email protected]; [email protected]; Soputhi Sea > *Subject:* Re: [ovs-discuss] Openvswitch network disconnect > > > > > > > > On Wed, Jun 3, 2015 at 11:16 PM, Chris <[email protected]> wrote: > > Hello, > > We are using Openvswitch in our Openstack setup. > > ovs-vswitchd --version > ovs-vswitchd (Open vSwitch) 2.3.0 > Compiled Oct 28 2014 17:48:05 > OpenFlow versions 0x1:0x1 > > We experience openvswitch failures from time to time. It seems to happen > random, no network traffic spikes for example. > The VM ports and the openvswitch port for the host management just stop > working. But the openvswitch services (ovsdb-server/ ovs-vswitchd) are > still running > > For debug purpose the following commands has been executed: > > - "ovs-vsctl show" the interfaces are still listed > The ones below didn't show any result and just hang after the execution: > - ovs-appctl bond/show bond0 > - ovs-appctl vlog/list > - ovs-ofctl dump-flows br-bond0 > > > > > > This seems to indicate that the ovs-vswitchd process is dead locked... > > I could not find comment in branch-2.3 that relates to dead lock. It > would be > > helpful if you can gdb into the running ovs-vswitchd process and provide > > the backtraces when there is a failure. > > > > Or could I know what your setup looks like (ovs-vsctl show output)? > > > > Thanks, > > Alex Wang, > > > > > > > > A "service openvswitch restart" fix it, the connection from the VMs and the > host are back immediately. > > Any help appreciated! > > Cheers, > Chris > > > > _______________________________________________ > discuss mailing list > [email protected] > http://openvswitch.org/mailman/listinfo/discuss > > >
_______________________________________________ discuss mailing list [email protected] http://openvswitch.org/mailman/listinfo/discuss
