Hi,
We have a severe problem with VPP – it’s cyclic restarting due to the following
error:
Dec 21 09:46:32 843V0N3 vpp[3846]: vlib_worker_thread_barrier_sync_int: worker
thread deadlock
This happens on both servers of the setup and it cannot recover. Can you please
help us to debug this issue?
VPP version:
# vppctl show version
vpp v22.02.0-1~g0d1b46707-dirty built by suse on SUSE at 2022-05-02T09:46:05
which is a built of version 22.02.-1 on SLES 15 SP3 including the dpdk-patch
0001-add-patch-to-disable-source-pruning-in-i40e-driver.patch
The startup.conf:
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
coredump-size unlimited
cli-listen /run/vpp/cli.sock
gid vpp
startup-config /etc/vpp/vpp-static-config.txt
}
api-trace {
on
}
api-segment {
gid vpp
}
socksvr {
socket-name /var/run/vpp/ic-api.sock
}
memory {
main-heap-page-size 1G
}
cpu {
main-core 2
corelist-workers 4,6,42,44,46
}
buffers {
buffers-per-numa 128000
}
dpdk {
dev default {
num-rx-queues 5
}
uio-driver vfio-pci
dev 0000:3b:00.0 {
name Radio-0
}
dev 0000:3b:00.1 {
name Ext-0
}
dev 0000:5e:02.1 {
name NCIC-1-v1
}
}
plugins {
plugin default { disable }
plugin dpdk_plugin.so { enable }
plugin ioam_plugin.so { enable }
plugin perfmon_plugin.so { enable }
plugin tracedump_plugin.so { enable }
plugin l3xc_plugin.so { enable }
plugin ping_plugin.so { enable }
plugin avf_plugin.so { enable }
plugin acl_plugin.so { enable }
plugin svs_plugin.so { enable }
plugin vrrp_plugin.so { enable }
plugin dhcp_plugin.so { enable }
plugin nat_plugin.so { enable }
plugin abf_plugin.so { enable }
plugin lacp_plugin.so { enable }
plugin flowprobe_plugin.so { enable }
}
The log gives:
Dec 21 10:33:27 hostname systemd[1]: Starting Vector Packet Processing
Process...
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: +
dpdk_devbind=/usr/local/bin/dpdk-devbind.py
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: +
VPP_CONF=/etc/vpp/ic-startup.conf
Dec 21 10:33:27 hostname ic-vpp-service.sh[64760]: ++ grep -v '#'
/etc/vpp/ic-startup.conf
Dec 21 10:33:27 hostname ic-vpp-service.sh[64761]: ++ grep dev
Dec 21 10:33:27 hostname ic-vpp-service.sh[64762]: ++ grep -v default
Dec 21 10:33:27 hostname ic-vpp-service.sh[64763]: ++ sed 's/.*dev //'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64764]: ++ cut '-d ' -f1
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + DEVICES='0000:3b:00.0
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: 0000:3b:00.1
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: 0000:5e:02.1'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64766]: ++ grep uio-driver
/etc/vpp/ic-startup.conf
Dec 21 10:33:27 hostname ic-vpp-service.sh[64767]: ++ sed 's/.*uio-driver //'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64768]: ++ cut '-d ' -f1
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + DPDK_DRV=vfio-pci
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
Dec 21 10:33:27 hostname ic-vpp-service.sh[64770]: ++
/usr/local/bin/dpdk-devbind.py -s
Dec 21 10:33:27 hostname ic-vpp-service.sh[64771]: ++ grep 0000:3b:00.0
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + cdev='0000:3b:00.0
'\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Radio-0 drv=i40e
unused=vfio-pci '
Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ get_drv '0000:3b:00.0
'\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Radio-0 drv=i40e
unused=vfio-pci '
Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ [[ -z 0000:3b:00.0
'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=Radio-0 drv=i40e
unused=vfio-pci ]]
Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ echo i40e
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + drv=i40e
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + check_intf --start
0000:3b:00.0 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\'''
if=Radio-0 drv=i40e unused=vfio-pci
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + action=--start
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + shift
Dec 21 10:33:27 hostname ic-vpp-service.sh[64954]: ++ echo 0000:3b:00.0
''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Radio-0 drv=i40e
unused=vfio-pci
Dec 21 10:33:27 hostname ic-vpp-service.sh[64955]: ++ sed 's/.* if=//'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64956]: ++ cut '-d ' -f1
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + name=Radio-0
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + [[ Radio-0 == *\-\v* ]]
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + ip l s Radio-0 down
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: +
/usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:3b:00.0
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
Dec 21 10:33:30 hostname ic-vpp-service.sh[66019]: ++ grep 0000:3b:00.1
Dec 21 10:33:30 hostname ic-vpp-service.sh[66018]: ++
/usr/local/bin/dpdk-devbind.py -s
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + cdev='0000:3b:00.1
'\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Ext-0 drv=i40e
unused=vfio-pci '
Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ get_drv '0000:3b:00.1
'\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Ext-0 drv=i40e
unused=vfio-pci '
Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ [[ -z 0000:3b:00.1
'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=Ext-0 drv=i40e
unused=vfio-pci ]]
Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ echo i40e
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + drv=i40e
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + check_intf --start
0000:3b:00.1 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\'''
if=Ext-0 drv=i40e unused=vfio-pci
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + action=--start
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + shift
Dec 21 10:33:30 hostname ic-vpp-service.sh[66368]: ++ echo 0000:3b:00.1
''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Ext-0 drv=i40e
unused=vfio-pci
Dec 21 10:33:30 hostname ic-vpp-service.sh[66369]: ++ sed 's/.* if=//'
Dec 21 10:33:30 hostname ic-vpp-service.sh[66370]: ++ cut '-d ' -f1
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + name=Ext-0
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + [[ Ext-0 == *\-\v* ]]
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']'
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + ip l s Ext-0 down
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: +
/usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:3b:00.1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
Dec 21 10:33:33 hostname ic-vpp-service.sh[66986]: ++
/usr/local/bin/dpdk-devbind.py -s
Dec 21 10:33:33 hostname ic-vpp-service.sh[66987]: ++ grep 0000:5e:02.1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + cdev='0000:5e:02.1
'\''Ethernet Virtual Function 700 Series 154c'\'' if=NCIC-1-v1 drv=iavf
unused=vfio-pci '
Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ get_drv '0000:5e:02.1
'\''Ethernet Virtual Function 700 Series 154c'\'' if=NCIC-1-v1 drv=iavf
unused=vfio-pci '
Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ [[ -z '' ]]
Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ echo iavf
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + drv=iavf
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + check_intf --start
0000:5e:02.1 ''\''Ethernet' Virtual Function 700 Series '154c'\''' if=NCIC-1-v1
drv=iavf unused=vfio-pci
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + action=--start
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + shift
Dec 21 10:33:33 hostname ic-vpp-service.sh[67201]: ++ echo 0000:5e:02.1
''\''Ethernet' Virtual Function 700 Series '154c'\''' if=NCIC-1-v1 drv=iavf
unused=vfio-pci
Dec 21 10:33:33 hostname ic-vpp-service.sh[67202]: ++ sed 's/.* if=//'
Dec 21 10:33:33 hostname ic-vpp-service.sh[67203]: ++ cut '-d ' -f1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + name=NCIC-1-v1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + [[ NCIC-1-v1 == *\-\v* ]]
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + parent=NCIC-1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + vf=1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']'
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + ip l s NCIC-1 vf 1 spoof
off state enable trust on
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: +
/usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:5e:02.1
Dec 21 10:33:34 hostname systemd[1]: Started Vector Packet Processing Process.
Dec 21 10:33:37 hostname vnet[67475]: dpdk/cryptodev: dpdk_cryptodev_init: Not
enough cryptodev resources
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: adl
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
arping plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
builtinurl plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: cdp
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: ct6
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: dns
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
geneve plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: gtpu
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
http_static plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: ikev2
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: l2tp
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: lb
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register_gpe:
lisp_gpe plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: lldp
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
mactime plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: mdata
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: memif
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: nsh
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: nsim
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
oddbuf plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: pppoe
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: rdma
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: stn
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
tls_openssl plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
vmxnet3 plugin not loaded...
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [0]
sw_if_index 2 VR ID 11 IPv4 transitioning to Backup
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting
virtual MAC address 00:00:5e:00:01:0b on hardware interface 2
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 1 VRs configured,
1 VRs running
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [1]
sw_if_index 24 VR ID 207 IPv4 transitioning to Backup
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting
virtual MAC address 00:00:5e:00:01:cf on hardware interface 2
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 2 VRs configured,
2 VRs running
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [2]
sw_if_index 25 VR ID 202 IPv4 transitioning to Backup
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting
virtual MAC address 00:00:5e:00:01:ca on hardware interface 2
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 3 VRs configured,
3 VRs running
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [3]
sw_if_index 26 VR ID 201 IPv4 transitioning to Backup
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting
virtual MAC address 00:00:5e:00:01:c9 on hardware interface 2
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 4 VRs configured,
4 VRs running
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [0]
sw_if_index 2 VR ID 11 IPv4 transitioning to Master
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR
addresses on sw_if_index 2
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
virtual MAC address 00:00:5e:00:01:0b on hardware interface 2
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [1]
sw_if_index 24 VR ID 207 IPv4 transitioning to Master
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR
addresses on sw_if_index 24
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
virtual MAC address 00:00:5e:00:01:cf on hardware interface 2
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [2]
sw_if_index 25 VR ID 202 IPv4 transitioning to Master
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR
addresses on sw_if_index 25
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
virtual MAC address 00:00:5e:00:01:ca on hardware interface 2
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [3]
sw_if_index 26 VR ID 201 IPv4 transitioning to Master
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR
addresses on sw_if_index 26
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
virtual MAC address 00:00:5e:00:01:c9 on hardware interface 2
Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to
start already started VR ([0] sw_if_index 2 VR ID 11 IPv4)
Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to
start already started VR ([1] sw_if_index 24 VR ID 207 IPv4)
Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to
start already started VR ([2] sw_if_index 25 VR ID 202 IPv4)
Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to
start already started VR ([3] sw_if_index 26 VR ID 201 IPv4)
Dec 21 10:34:36 hostname vpp[67475]: vlib_worker_thread_barrier_sync_int:
worker thread deadlock
Dec 21 10:34:36 hostname vnet[67475]: received signal SIGABRT, PC 0x7f7207c5acdb
Dec 21 10:34:36 hostname vnet[67475]: #0 0x00007f7208ea3234
unix_signal_handler + 0x124
Dec 21 10:34:36 hostname vnet[67475]: #1 0x00007f72087e68c0 0x7f72087e68c0
Dec 21 10:34:36 hostname vnet[67475]: #2 0x00007f7207c5acdb gsignal + 0x10d
Dec 21 10:34:36 hostname vnet[67475]: #3 0x00007f7207c5c375 abort + 0x177
Dec 21 10:34:36 hostname vnet[67475]: #4 0x0000000000407643 0x407643
Dec 21 10:34:36 hostname vnet[67475]: #5 0x00007f7208e89541
vlib_worker_thread_barrier_sync_int + 0x451
Dec 21 10:34:36 hostname vnet[67475]: #6 0x00007f720a5aa3af
vl_api_rpc_call_main_thread + 0x3cf
Dec 21 10:34:36 hostname vnet[67475]: #7 0x00007f7209a95c1b arp_reply + 0x7eb
Dec 21 10:34:36 hostname vnet[67475]: #8 0x00007f7208e50b65
dispatch_pending_node + 0x135
Dec 21 10:34:36 hostname vnet[67475]: #9 0x00007f7208e55214 vlib_main + 0xd34
Dec 21 10:34:36 hostname vnet[67475]: #10 0x00007f7208ea2288 thread0 + 0x28
Dec 21 10:34:36 hostname vnet[67475]: #11 0x00007f72083b600c 0x7f72083b600c
Dec 21 10:34:49 hostname systemd[1]: vpp.service: Main process exited,
code=dumped, status=6/ABRT
We have our own controller which uses the API to do some dynamic configuration
of VPP – apart from different routes in different tables, it configures VRRP as
follows:
# vppctl show vrrp vr
[0] sw_if_index 2 VR ID 11 IPv4
state Master flags: preempt yes accept yes unicast no
priority: configured 200 adjusted 200
timers: adv interval 100 master adv 100 skew 21 master down 321
virtual MAC 00:00:5e:00:01:0b
addresses 10.95.255.2
peer addresses
tracked interfaces
[1] sw_if_index 24 VR ID 207 IPv4
state Master flags: preempt yes accept yes unicast no
priority: configured 200 adjusted 200
timers: adv interval 100 master adv 100 skew 21 master down 321
virtual MAC 00:00:5e:00:01:cf
addresses 10.94.131.34
peer addresses
tracked interfaces
[2] sw_if_index 25 VR ID 202 IPv4
state Master flags: preempt yes accept yes unicast no
priority: configured 200 adjusted 200
timers: adv interval 100 master adv 100 skew 21 master down 321
virtual MAC 00:00:5e:00:01:ca
addresses 10.94.29.90
peer addresses
tracked interfaces
[3] sw_if_index 26 VR ID 201 IPv4
state Master flags: preempt yes accept yes unicast no
priority: configured 200 adjusted 200
timers: adv interval 100 master adv 100 skew 21 master down 321
virtual MAC 00:00:5e:00:01:c9
addresses 10.95.250.3
peer addresses
tracked interfaces
The IP addresses of the VRRP master are:
# vppctl show int addr
Ext-0 (up):
L3 10.95.255.3/29 ip4 table-id 1 fib-idx 4
L3 10.95.255.2/29 ip4 table-id 1 fib-idx 4
Ext-0.507 (up):
L3 10.94.131.35/29 ip4 table-id 2 fib-idx 6
L3 10.94.131.34/29 ip4 table-id 2 fib-idx 6
Ext-0.506 (up):
L3 10.94.29.91/29 ip4 table-id 3 fib-idx 7
L3 10.94.29.90/29 ip4 table-id 3 fib-idx 7
Ext-0.501 (up):
L3 10.95.250.4/29 ip4 table-id 4 fib-idx 8
L3 10.95.250.3/29 ip4 table-id 4 fib-idx 8
I don’t have gdb on the affected system. On the build server, I only see:
Core was generated by `/usr/bin/vpp -c /etc/vpp/ic-startup.conf'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f18c5b23cdb in ?? ()
[Current thread is 1 (LWP 14555)]
(gdb) list
91 vat_plugin_hash_create ();
92 #endif
93
94 if (!vlib_plugin_path)
95 vpp_find_plugin_path ();
96 }
97
98 /*
99 * Default path for runtime data
100 */
(gdb) bt
#0 0x00007f18c5b23cdb in ?? ()
Backtrace stopped: Cannot access memory at address 0x7f18c116cac0
Any ideas on what to check or change to get rid of the cyclic vpp restart?
Any help is appreciated,
Thank you,
BR/Mechthild
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22352): https://lists.fd.io/g/vpp-dev/message/22352
Mute This Topic: https://lists.fd.io/mt/95806883/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
