On 18 Jun 2019, at 12:17, Ilya Maximets wrote:
On 18.06.2019 12:45, Eelco Chaudron wrote:
On 17 Jun 2019, at 22:32, William Tu wrote:
<SNIP>
1000,463875,419996,402010
However I can not restart OVS (see other email on how I restart),
even if I clear the XDP programs before a restart it fails, and
cores.
The only way to recover is to reboot the box and start from scratch:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f455919a9b5 in xsk_clear_bpf_maps (xsk=0x21) at xsk.c:462
462 bpf_map_update_elem(xsk->qidconf_map_fd,
&xsk->queue_id, &qid, 0);
[Current thread is 1 (Thread 0x7f4559f1c000 (LWP 4898))]
Missing separate debuginfos, use: dnf debuginfo-install
elfutils-libelf-0.174-6.el8.x86_64 glibc-2.28-42.el8_0.1.x86_64
libatomic-8.2.1-3.5.el8.x86_64 libcap-ng-0.7.9-4.el8.x86_64
numactl-libs-2.0.12-2.el8.x86_64 openssl-libs-1.1.1-8.el8.x86_64
zlib-1.2.11-10.el8.x86_64
(gdb) bt
#0 0x00007f455919a9b5 in xsk_clear_bpf_maps (xsk=0x21) at xsk.c:462
#1 0x00007f455919b278 in xsk_socket__delete (xsk=0x21) at xsk.c:711
#2 0x00000000009b3af1 in xsk_destroy (xsk_info=<optimized out>) at
lib/netdev-afxdp.c:313
#3 xsk_destroy_all (netdev=0x1df49a0) at lib/netdev-afxdp.c:313
#4 0x00000000009b4fe9 in netdev_afxdp_destruct (netdev_=0x1df49a0)
at lib/netdev-afxdp.c:845
#5 0x0000000000906e53 in netdev_unref (dev=0x1df49a0) at
lib/netdev.c:573
#6 0x00000000008739b1 in iface_do_create (errp=0x7ffe4fc5b588,
netdevp=0x7ffe4fc5b580, ofp_portp=0x7ffe4fc5b578,
iface_cfg=0x1cde5d0, br=0x1ce1690) at vswitchd/bridge.c:1825
#7 iface_create (port_cfg=0x1cb3690, iface_cfg=0x1cde5d0,
br=0x1ce1690) at vswitchd/bridge.c:1848
#8 bridge_add_ports__ (br=br@entry=0x1ce1690,
wanted_ports=wanted_ports@entry=0x1ce1770,
with_requested_port=with_requested_port@entry=false) at
vswitchd/bridge.c:936
#9 0x0000000000875ef7 in bridge_add_ports (wanted_ports=0x1ce1770,
br=0x1ce1690) at vswitchd/bridge.c:952
#10 bridge_reconfigure (ovs_cfg=ovs_cfg@entry=0x1cb4b90) at
vswitchd/bridge.c:666
#11 0x0000000000879521 in bridge_run () at vswitchd/bridge.c:3043
#12 0x00000000004ef545 in main (argc=<optimized out>, argv=<optimized
out>) at vswitchd/ovs-vswitchd.c:127
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ovs-vswitchd[5861]: ovs|00051|netdev_afxdp|ERR|xsk_socket__create
failed (Device or resource busy) mode: SKB qid: 0
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ovs-vswitchd[5861]: ovs|00052|netdev_afxdp|ERR|failed to create
AF_XDP socket on queue 0
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ovs-vswitchd[5861]: ovs|00055|netdev_afxdp|ERR|AF_XDP device tapVM
reconfig fails
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ovs-vswitchd[5861]: ovs|00056|dpif_netdev|ERR|Failed to set interface
tapVM new configuration
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ovs-vswitchd[5861]: ovs|00062|netdev_afxdp|ERR|xsk_socket__create
failed (Device or resource busy) mode: DRV qid: 0
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ovs-vswitchd[5861]: ovs|00063|netdev_afxdp|ERR|failed to create
AF_XDP socket on queue 0
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ovs-vswitchd[5861]: ovs|00066|netdev_afxdp|ERR|AF_XDP device eno1
reconfig fails
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ovs-vswitchd[5861]: ovs|00067|dpif_netdev|ERR|Failed to set interface
eno1 new configuration
Jun 18 03:52:06 wsfd-netdev76.ntdv.lab.eng.bos.redhat.com kernel:
ovs-vswitchd[5861]: segfault at 123 ip 00000000009b3afd sp
00007ffff954a770 error 4 in ovs-vswitchd[400000+899000]
I guess, this crash caused by trying to destroy unallocated queue.
Following change could help:
---
diff --git a/lib/netdev-afxdp.c b/lib/netdev-afxdp.c
index a6543e8f5..6e1431dce 100644
--- a/lib/netdev-afxdp.c
+++ b/lib/netdev-afxdp.c
@@ -249,7 +249,7 @@ xsk_configure_all(struct netdev *netdev)
ifindex = linux_get_ifindex(netdev_get_name(netdev));
n_rxq = netdev_n_rxq(netdev);
- dev->xsks = xmalloc(n_rxq * sizeof(struct xsk_socket_info *));
+ dev->xsks = xzalloc(n_rxq * sizeof(struct xsk_socket_info *));
/* configure each queue */
for (i = 0; i < n_rxq; i++) {
---
This should prevent OVS from crash, however, I don't know why socket
creation fails in your case.
I know William is preparing a v12, so will do some more testing once
it’s out, and update my kernel to the latest bpf-next just to make
sure I’m not running into a known/fixed issue.
//Eelco
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
