On 6/22/2018 9:13 PM, [email protected] wrote:
Hi all,
I am trying to use OVS (2.5.4) with DPSK on Ubuntu server 16.04.3 (KVM
host) purely for inter-VM communication.
I have been following this guide very closely:
https://help.ubuntu.com/lts/serverguide/DPDK.html
Hi,
Is there a hard requirement why you are using OVS 2.5.4?
DPDK support was very early at that stage. There has been many bug
fixes/new features & performance improvements for OVS with DPDK since
then. (The file descriptor error is fixed in a later release for sure).
If you can then I'd recommend moving to the latest OVS 2.9 and testing
your usecase there to see if you see the segfault issue .
Ian
While I don't have any issue at all with using a normal OVS bridge for
the VMs running in KVM host:
$ sudo ovs-vsctl add-br br-MGT
I cannot succeed in using OVS with DPDK bridges:
$ sudo ovs-vsctl add-br br-LAN1 -- set bridge br-LAN1 datapath_type=netdev
$ sudo ovs-vsctl add-br br-LAN2 -- set bridge br-LAN2 datapath_type=netdev
I have created 2 ports as follows:
$ sudo ovs-vsctl add-port br-LAN1 vhost-user1 -- set Interface
vhost-user1 type=dpdkvhostuser
$ sudo ovs-vsctl add-port br-LAN2 vhost-user2 -- set Interface
vhost-user2 type=dpdkvhostuser
The VM is defined with one interface on each bridges
<interface type='bridge'>
<source bridge='br-MGT'/>
<virtualport type='openvswitch'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<interface type='vhostuser'>
<source type='unix' path='/var/run/openvswitch/vhost-user1'
mode='client'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</interface>
<interface type='vhostuser'>
<source type='unix' path='/var/run/openvswitch/vhost-user2'
mode='client'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</interface>
After the VM is started, I observe the following errors for both
vhost-user1 and vhost-user2:
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: fail to bind
fd:66, remove file:/var/run/openvswitch/vhost-user2 and try again.
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]:
ovs|00023|dpdk|ERR|vhost-user socket device setup failure for socket
/var/run/openvswitch/vhost-user2
Also, I observe a segmentation fault crash:
Jun 23 03:35:14 ubt-ovs kernel: [ 272.156977] vhost_thread2[1540]:
segfault at 18 ip 00007f379d9b395f sp 00007f379c619740 error 4 in
libdpdk.so.0[7f379d979000+1ea000]
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[1531]:
ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid 1532 died, killed
(Segmentation fault), core dumped, restarting
I have tried to remove files /var/run/openvswitch/vhost-user*, delete
dpdkvhostuser ports, restart the KVM host...
But these 2 kind of issues occur again systematically.
What investigation would you suggest to help me understand the cause(s)
of these issues?
More complete logs below.
From the file /var/log/openvswitch/ovs-vswitchd.log:
2018-06-22T19:30:58.531Z|00002|vlog|INFO|opened log file
/var/log/openvswitch/ovs-vswitchd.log
2018-06-22T19:30:58.638Z|00003|ovs_numa|INFO|Discovered 6 CPU cores on
NUMA node 0
2018-06-22T19:30:58.638Z|00004|ovs_numa|INFO|Discovered 1 NUMA nodes and
6 CPU cores
2018-06-22T19:30:58.638Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connecting...
2018-06-22T19:30:58.640Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connected
2018-06-22T19:30:58.645Z|00007|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath supports recirculation
2018-06-22T19:30:58.645Z|00008|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS
label stack length probed as 3
2018-06-22T19:30:58.645Z|00009|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath supports unique flow ids
2018-06-22T19:30:58.645Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_state
2018-06-22T19:30:58.645Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_zone
2018-06-22T19:30:58.645Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_mark
2018-06-22T19:30:58.645Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_label
2018-06-22T19:30:58.659Z|00014|ofproto_dpif|INFO|system@ovs-system:
Datapath supports recirculation
2018-06-22T19:30:58.659Z|00015|ofproto_dpif|INFO|system@ovs-system: MPLS
label stack length probed as 1
2018-06-22T19:30:58.659Z|00016|ofproto_dpif|INFO|system@ovs-system:
Datapath supports unique flow ids
2018-06-22T19:30:58.659Z|00017|ofproto_dpif|INFO|system@ovs-system:
Datapath supports ct_state
2018-06-22T19:30:58.659Z|00018|ofproto_dpif|INFO|system@ovs-system:
Datapath supports ct_zone
2018-06-22T19:30:58.659Z|00019|ofproto_dpif|INFO|system@ovs-system:
Datapath supports ct_mark
2018-06-22T19:30:58.659Z|00020|ofproto_dpif|INFO|system@ovs-system:
Datapath supports ct_label
2018-06-22T19:30:58.672Z|00001|ofproto_dpif_upcall(handler12)|INFO|received
packet on unassociated datapath port 0
2018-06-22T19:30:58.673Z|00021|bridge|INFO|bridge br-LAN2: added
interface br-LAN2 on port 65534
2018-06-22T19:30:58.674Z|00022|bridge|INFO|bridge br-MGT: added
interface ens160 on port 1
2018-06-22T19:30:58.702Z|00023|bridge|INFO|bridge br-MGT: added
interface br-MGT on port 65534
2018-06-22T19:30:58.702Z|00001|ofproto_dpif_upcall(handler15)|INFO|received
packet on unassociated datapath port 1
2018-06-22T19:30:58.702Z|00001|ofproto_dpif_upcall(handler11)|INFO|received
packet on unassociated datapath port 1
2018-06-22T19:30:58.702Z|00002|ofproto_dpif_upcall(handler11)|INFO|received
packet on unassociated datapath port 1
2018-06-22T19:30:58.702Z|00003|ofproto_dpif_upcall(handler11)|INFO|received
packet on unassociated datapath port 1
2018-06-22T19:30:58.702Z|00024|bridge|INFO|bridge br-LAN1: added
interface br-LAN1 on port 65534
2018-06-22T19:30:58.703Z|00025|bridge|INFO|bridge br-LAN2: using
datapath ID 0000968152e9a440
2018-06-22T19:30:58.703Z|00026|connmgr|INFO|br-LAN2: added service
controller "punix:/var/run/openvswitch/br-LAN2.mgmt"
2018-06-22T19:30:58.707Z|00027|bridge|INFO|bridge br-MGT: using datapath
ID 0000000c295e757f
2018-06-22T19:30:58.707Z|00028|connmgr|INFO|br-MGT: added service
controller "punix:/var/run/openvswitch/br-MGT.mgmt"
2018-06-22T19:30:58.712Z|00029|bridge|INFO|bridge br-LAN1: using
datapath ID 0000761b373c734e
2018-06-22T19:30:58.712Z|00030|connmgr|INFO|br-LAN1: added service
controller "punix:/var/run/openvswitch/br-LAN1.mgmt"
2018-06-22T19:30:58.725Z|00031|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.5.4
2018-06-22T19:31:06.714Z|00032|memory|INFO|154292 kB peak resident set
size after 10.0 seconds
2018-06-22T19:31:06.714Z|00033|memory|INFO|handlers:8 ports:4
revalidators:4 rules:15 udpif keys:5
2018-06-22T19:32:49.234Z|00034|dpdk|INFO|Socket
/var/run/openvswitch/vhost-user1 created for vhost-user port vhost-user1
2018-06-22T19:32:49.351Z|00035|dpif_netdev|INFO|Created 1 pmd threads on
numa node 0
2018-06-22T19:32:49.351Z|00036|bridge|INFO|bridge br-LAN1: added
interface vhost-user1 on port 1
2018-06-22T19:33:00.097Z|00037|dpdk|INFO|Socket
/var/run/openvswitch/vhost-user2 created for vhost-user port vhost-user2
2018-06-22T19:33:00.097Z|00038|bridge|INFO|bridge br-LAN2: added
interface vhost-user2 on port 1
2018-06-22T19:35:01.151Z|00039|bridge|INFO|bridge br-MGT: added
interface vnet0 on port 2
2018-06-22T19:35:13.998Z|00001|dpdk(vhost_thread2)|INFO|State of queue 0
( tx_qid 0 ) of vhost device '/var/run/openvswitch/vhost-user1' 0
changed to 'enabled'
2018-06-22T19:35:14.910Z|00003|daemon_unix(monitor)|ERR|1 crashes: pid
1532 died, killed (Segmentation fault), core dumped, restarting
2018-06-22T19:35:14.918Z|00004|ovs_numa|INFO|Discovered 6 CPU cores on
NUMA node 0
2018-06-22T19:35:14.918Z|00005|ovs_numa|INFO|Discovered 1 NUMA nodes and
6 CPU cores
2018-06-22T19:35:14.918Z|00006|memory|INFO|37220 kB peak resident set
size after 258.2 seconds
2018-06-22T19:35:14.918Z|00007|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connecting...
2018-06-22T19:35:14.920Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connected
2018-06-22T19:35:14.926Z|00009|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath supports recirculation
2018-06-22T19:35:14.926Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS
label stack length probed as 3
2018-06-22T19:35:14.926Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath supports unique flow ids
2018-06-22T19:35:14.926Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_state
2018-06-22T19:35:14.926Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_zone
2018-06-22T19:35:14.926Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_mark
2018-06-22T19:35:14.926Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_label
2018-06-22T19:35:14.940Z|00016|ofproto_dpif|INFO|system@ovs-system:
Datapath supports recirculation
2018-06-22T19:35:14.940Z|00017|ofproto_dpif|INFO|system@ovs-system: MPLS
label stack length probed as 1
2018-06-22T19:35:14.940Z|00018|ofproto_dpif|INFO|system@ovs-system:
Datapath supports unique flow ids
2018-06-22T19:35:14.940Z|00019|ofproto_dpif|INFO|system@ovs-system:
Datapath supports ct_state
2018-06-22T19:35:14.940Z|00020|ofproto_dpif|INFO|system@ovs-system:
Datapath supports ct_zone
2018-06-22T19:35:14.940Z|00021|ofproto_dpif|INFO|system@ovs-system:
Datapath supports ct_mark
2018-06-22T19:35:14.940Z|00022|ofproto_dpif|INFO|system@ovs-system:
Datapath supports ct_label
2018-06-22T19:35:14.963Z|00001|ofproto_dpif_upcall(handler10)|INFO|received
packet on unassociated datapath port 0
2018-06-22T19:35:14.964Z|00023|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhost-user2
2018-06-22T19:35:14.964Z|00024|bridge|WARN|could not open network device
vhost-user2 (Unknown error -1)
2018-06-22T19:35:14.965Z|00025|bridge|INFO|bridge br-LAN2: added
interface br-LAN2 on port 65534
2018-06-22T19:35:14.965Z|00026|bridge|INFO|bridge br-MGT: added
interface ens160 on port 1
2018-06-22T19:35:14.965Z|00027|bridge|INFO|bridge br-MGT: added
interface vnet0 on port 2
2018-06-22T19:35:14.965Z|00028|bridge|INFO|bridge br-MGT: added
interface br-MGT on port 65534
2018-06-22T19:35:14.965Z|00029|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhost-user1
2018-06-22T19:35:14.965Z|00030|bridge|WARN|could not open network device
vhost-user1 (Unknown error -1)
2018-06-22T19:35:14.965Z|00031|bridge|INFO|bridge br-LAN1: added
interface br-LAN1 on port 65534
2018-06-22T19:35:14.965Z|00032|bridge|INFO|bridge br-LAN2: using
datapath ID 0000968152e9a440
2018-06-22T19:35:14.965Z|00033|connmgr|INFO|br-LAN2: added service
controller "punix:/var/run/openvswitch/br-LAN2.mgmt"
2018-06-22T19:35:14.977Z|00034|bridge|INFO|bridge br-MGT: using datapath
ID 0000000c295e757f
2018-06-22T19:35:14.977Z|00035|connmgr|INFO|br-MGT: added service
controller "punix:/var/run/openvswitch/br-MGT.mgmt"
2018-06-22T19:35:14.981Z|00036|bridge|INFO|bridge br-LAN1: using
datapath ID 0000761b373c734e
2018-06-22T19:35:14.981Z|00037|connmgr|INFO|br-LAN1: added service
controller "punix:/var/run/openvswitch/br-LAN1.mgmt"
2018-06-22T19:35:14.990Z|00038|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhost-user2
2018-06-22T19:35:14.990Z|00039|bridge|WARN|could not open network device
vhost-user2 (Unknown error -1)
2018-06-22T19:35:14.990Z|00040|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhost-user1
2018-06-22T19:35:14.990Z|00041|bridge|WARN|could not open network device
vhost-user1 (Unknown error -1)
2018-06-22T19:35:14.992Z|00042|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.5.4
2018-06-22T19:35:24.995Z|00043|memory|INFO|peak resident set size grew
315% in last 10.1 seconds, from 37220 kB to 154308 kB
2018-06-22T19:35:24.995Z|00044|memory|INFO|handlers:8 ports:5
revalidators:4 rules:15 udpif keys:7
From the file /var/log/syslog:
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_VRING_ENABLE
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: set queue
enable: 1 to qp idx: 0
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_VRING_ENABLE
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: set queue
enable: 1 to qp idx: 1
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_FEATURES
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_MEM_TABLE
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: mapped region
0 fd:91 to:0x7f3780400000 sz:0x200000 off:0x0 align:0x200000
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: mapped region
1 fd:92 to:0xffffffffffffffff sz:0x80000000 off:0xc0000 align:0x200000
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: mmap qemu
guest failed.
Jun 23 03:35:13 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_VRING_NUM
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_VRING_ENABLE
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: set queue
enable: 1 to qp idx: 0
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_VRING_BASE
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_VRING_ENABLE
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: set queue
enable: 1 to qp idx: 1
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[1532]: VHOST_CONFIG: read message
VHOST_USER_SET_VRING_ADDR
Jun 23 03:35:14 ubt-ovs kernel: [ 272.156977] vhost_thread2[1540]:
segfault at 18 ip 00007f379d9b395f sp 00007f379c619740 error 4 in
libdpdk.so.0[7f379d979000+1ea000]
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.8455]
devices removed (path: /sys/devices/virtual/net/br-LAN1, iface: br-LAN1)
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.8681]
devices removed (path: /sys/devices/virtual/net/br-LAN2, iface: br-LAN2)
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.8909]
devices removed (path: /sys/devices/virtual/net/ovs-netdev, iface:
ovs-netdev)
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[1531]:
ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid 1532 died, killed
(Segmentation fault), core dumped, restarting
Jun 23 03:35:14 ubt-ovs systemd-udevd[2099]: Could not generate
persistent MAC address for ovs-netdev: No such file or directory
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9283]
manager: (ovs-netdev): new Tun device
(/org/freedesktop/NetworkManager/Devices/8)
Jun 23 03:35:14 ubt-ovs kernel: [ 273.083664] device ovs-netdev entered
promiscuous mode
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9526]
devices added (path: /sys/devices/virtual/net/ovs-netdev, iface: ovs-netdev)
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9526]
device added (path: /sys/devices/virtual/net/ovs-netdev, iface:
ovs-netdev): no ifupdown configuration found.
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: socket
created, fd:66
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: fail to bind
fd:66, remove file:/var/run/openvswitch/vhost-user2 and try again.
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]:
ovs|00023|dpdk|ERR|vhost-user socket device setup failure for socket
/var/run/openvswitch/vhost-user2
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: socket
created, fd:67
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: fail to bind
fd:67, remove file:/var/run/openvswitch/vhost-user1 and try again.
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]:
ovs|00029|dpdk|ERR|vhost-user socket device setup failure for socket
/var/run/openvswitch/vhost-user1
Jun 23 03:35:14 ubt-ovs kernel: [ 273.122363] device br-LAN2 entered
promiscuous mode
Jun 23 03:35:14 ubt-ovs kernel: [ 273.123015] device br-LAN1 entered
promiscuous mode
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9663]
manager: (br-LAN2): new Tun device
(/org/freedesktop/NetworkManager/Devices/9)
Jun 23 03:35:14 ubt-ovs systemd-udevd[2148]: Could not generate
persistent MAC address for br-LAN1: No such file or directory
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9688]
manager: (br-LAN1): new Tun device
(/org/freedesktop/NetworkManager/Devices/10)
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: socket
created, fd:30
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: fail to bind
fd:30, remove file:/var/run/openvswitch/vhost-user2 and try again.
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]:
ovs|00038|dpdk|ERR|vhost-user socket device setup failure for socket
/var/run/openvswitch/vhost-user2
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: socket
created, fd:30
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9910]
devices added (path: /sys/devices/virtual/net/br-LAN2, iface: br-LAN2)
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]: VHOST_CONFIG: fail to bind
fd:30, remove file:/var/run/openvswitch/vhost-user1 and try again.
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9910]
device added (path: /sys/devices/virtual/net/br-LAN2, iface: br-LAN2):
no ifupdown configuration found.
Jun 23 03:35:14 ubt-ovs ovs-vswitchd[2095]:
ovs|00040|dpdk|ERR|vhost-user socket device setup failure for socket
/var/run/openvswitch/vhost-user1
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9938]
devices added (path: /sys/devices/virtual/net/br-LAN1, iface: br-LAN1)
Jun 23 03:35:14 ubt-ovs NetworkManager[1170]: <info> [1529696114.9938]
device added (path: /sys/devices/virtual/net/br-LAN1, iface: br-LAN1):
no ifupdown configuration found.
Thank you.
Krog
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
