Hi Mechthild, Your issue is caused by the main thread waiting too long for a worker to finish. You may examine these lines in src/vlib/threads.h:
171 /* Check for a barrier sync request every 30ms */ 172 #define BARRIER_SYNC_DELAY (0.030000) 173 174 #if CLIB_DEBUG > 0 175 /* long barrier timeout, for gdb... */ 176 #define BARRIER_SYNC_TIMEOUT (600.1) 177 #else 178 #define BARRIER_SYNC_TIMEOUT (1.0) 179 #endif Your restart is caused by the timeout defined in these lines. You may increase it to investigate your issue (of course it is not a fix). After increasing timeout and being in interactive mode you can issue command "show run" it will tell you in which node you are spending most of your time and potentially identify source of your problem. Alternatively, you may use perf tool for that task. BR, Piotr From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Mechthild Buescher via lists.fd.io Sent: Wednesday, December 21, 2022 3:24 PM To: vpp-dev@lists.fd.io Subject: [vpp-dev] VPP crashes with vlib_worker_thread_barrier_syn_int:thread deadlock Hi, We have a severe problem with VPP - it's cyclic restarting due to the following error: Dec 21 09:46:32 843V0N3 vpp[3846]: vlib_worker_thread_barrier_sync_int: worker thread deadlock This happens on both servers of the setup and it cannot recover. Can you please help us to debug this issue? VPP version: # vppctl show version vpp v22.02.0-1~g0d1b46707-dirty built by suse on SUSE at 2022-05-02T09:46:05 which is a built of version 22.02.-1 on SLES 15 SP3 including the dpdk-patch 0001-add-patch-to-disable-source-pruning-in-i40e-driver.patch The startup.conf: unix { nodaemon log /var/log/vpp/vpp.log full-coredump coredump-size unlimited cli-listen /run/vpp/cli.sock gid vpp startup-config /etc/vpp/vpp-static-config.txt } api-trace { on } api-segment { gid vpp } socksvr { socket-name /var/run/vpp/ic-api.sock } memory { main-heap-page-size 1G } cpu { main-core 2 corelist-workers 4,6,42,44,46 } buffers { buffers-per-numa 128000 } dpdk { dev default { num-rx-queues 5 } uio-driver vfio-pci dev 0000:3b:00.0 { name Radio-0 } dev 0000:3b:00.1 { name Ext-0 } dev 0000:5e:02.1 { name NCIC-1-v1 } } plugins { plugin default { disable } plugin dpdk_plugin.so { enable } plugin ioam_plugin.so { enable } plugin perfmon_plugin.so { enable } plugin tracedump_plugin.so { enable } plugin l3xc_plugin.so { enable } plugin ping_plugin.so { enable } plugin avf_plugin.so { enable } plugin acl_plugin.so { enable } plugin svs_plugin.so { enable } plugin vrrp_plugin.so { enable } plugin dhcp_plugin.so { enable } plugin nat_plugin.so { enable } plugin abf_plugin.so { enable } plugin lacp_plugin.so { enable } plugin flowprobe_plugin.so { enable } } The log gives: Dec 21 10:33:27 hostname systemd[1]: Starting Vector Packet Processing Process... Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + dpdk_devbind=/usr/local/bin/dpdk-devbind.py Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + VPP_CONF=/etc/vpp/ic-startup.conf Dec 21 10:33:27 hostname ic-vpp-service.sh[64760]: ++ grep -v '#' /etc/vpp/ic-startup.conf Dec 21 10:33:27 hostname ic-vpp-service.sh[64761]: ++ grep dev Dec 21 10:33:27 hostname ic-vpp-service.sh[64762]: ++ grep -v default Dec 21 10:33:27 hostname ic-vpp-service.sh[64763]: ++ sed 's/.*dev //' Dec 21 10:33:27 hostname ic-vpp-service.sh[64764]: ++ cut '-d ' -f1 Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + DEVICES='0000:3b:00.0 Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: 0000:3b:00.1 Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: 0000:5e:02.1' Dec 21 10:33:27 hostname ic-vpp-service.sh[64766]: ++ grep uio-driver /etc/vpp/ic-startup.conf Dec 21 10:33:27 hostname ic-vpp-service.sh[64767]: ++ sed 's/.*uio-driver //' Dec 21 10:33:27 hostname ic-vpp-service.sh[64768]: ++ cut '-d ' -f1 Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + DPDK_DRV=vfio-pci Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']' Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES Dec 21 10:33:27 hostname ic-vpp-service.sh[64770]: ++ /usr/local/bin/dpdk-devbind.py -s Dec 21 10:33:27 hostname ic-vpp-service.sh[64771]: ++ grep 0000:3b:00.0 Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + cdev='0000:3b:00.0 '\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Radio-0 drv=i40e unused=vfio-pci ' Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ get_drv '0000:3b:00.0 '\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Radio-0 drv=i40e unused=vfio-pci ' Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ [[ -z 0000:3b:00.0 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=Radio-0 drv=i40e unused=vfio-pci ]] Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ echo i40e Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + drv=i40e Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + check_intf --start 0000:3b:00.0 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Radio-0 drv=i40e unused=vfio-pci Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + action=--start Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + shift Dec 21 10:33:27 hostname ic-vpp-service.sh[64954]: ++ echo 0000:3b:00.0 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Radio-0 drv=i40e unused=vfio-pci Dec 21 10:33:27 hostname ic-vpp-service.sh[64955]: ++ sed 's/.* if=//' Dec 21 10:33:27 hostname ic-vpp-service.sh[64956]: ++ cut '-d ' -f1 Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + name=Radio-0 Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + [[ Radio-0 == *\-\v* ]] Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']' Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + ip l s Radio-0 down Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + /usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:3b:00.0 Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES Dec 21 10:33:30 hostname ic-vpp-service.sh[66019]: ++ grep 0000:3b:00.1 Dec 21 10:33:30 hostname ic-vpp-service.sh[66018]: ++ /usr/local/bin/dpdk-devbind.py -s Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + cdev='0000:3b:00.1 '\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Ext-0 drv=i40e unused=vfio-pci ' Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ get_drv '0000:3b:00.1 '\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Ext-0 drv=i40e unused=vfio-pci ' Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ [[ -z 0000:3b:00.1 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=Ext-0 drv=i40e unused=vfio-pci ]] Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ echo i40e Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + drv=i40e Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + check_intf --start 0000:3b:00.1 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Ext-0 drv=i40e unused=vfio-pci Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + action=--start Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + shift Dec 21 10:33:30 hostname ic-vpp-service.sh[66368]: ++ echo 0000:3b:00.1 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Ext-0 drv=i40e unused=vfio-pci Dec 21 10:33:30 hostname ic-vpp-service.sh[66369]: ++ sed 's/.* if=//' Dec 21 10:33:30 hostname ic-vpp-service.sh[66370]: ++ cut '-d ' -f1 Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + name=Ext-0 Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + [[ Ext-0 == *\-\v* ]] Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']' Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + ip l s Ext-0 down Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + /usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:3b:00.1 Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES Dec 21 10:33:33 hostname ic-vpp-service.sh[66986]: ++ /usr/local/bin/dpdk-devbind.py -s Dec 21 10:33:33 hostname ic-vpp-service.sh[66987]: ++ grep 0000:5e:02.1 Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + cdev='0000:5e:02.1 '\''Ethernet Virtual Function 700 Series 154c'\'' if=NCIC-1-v1 drv=iavf unused=vfio-pci ' Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ get_drv '0000:5e:02.1 '\''Ethernet Virtual Function 700 Series 154c'\'' if=NCIC-1-v1 drv=iavf unused=vfio-pci ' Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ [[ -z '' ]] Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ echo iavf Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + drv=iavf Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + check_intf --start 0000:5e:02.1 ''\''Ethernet' Virtual Function 700 Series '154c'\''' if=NCIC-1-v1 drv=iavf unused=vfio-pci Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + action=--start Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + shift Dec 21 10:33:33 hostname ic-vpp-service.sh[67201]: ++ echo 0000:5e:02.1 ''\''Ethernet' Virtual Function 700 Series '154c'\''' if=NCIC-1-v1 drv=iavf unused=vfio-pci Dec 21 10:33:33 hostname ic-vpp-service.sh[67202]: ++ sed 's/.* if=//' Dec 21 10:33:33 hostname ic-vpp-service.sh[67203]: ++ cut '-d ' -f1 Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + name=NCIC-1-v1 Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + [[ NCIC-1-v1 == *\-\v* ]] Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + parent=NCIC-1 Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + vf=1 Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']' Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + ip l s NCIC-1 vf 1 spoof off state enable trust on Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + /usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:5e:02.1 Dec 21 10:33:34 hostname systemd[1]: Started Vector Packet Processing Process. Dec 21 10:33:37 hostname vnet[67475]: dpdk/cryptodev: dpdk_cryptodev_init: Not enough cryptodev resources Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: adl plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: arping plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: builtinurl plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: cdp plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: ct6 plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: dns plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: geneve plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: gtpu plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: http_static plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: ikev2 plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: l2tp plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: lb plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register_gpe: lisp_gpe plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: lldp plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: mactime plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: mdata plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: memif plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: nsh plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: nsim plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: oddbuf plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: pppoe plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: rdma plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: stn plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: tls_openssl plugin not loaded... Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: vmxnet3 plugin not loaded... Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [0] sw_if_index 2 VR ID 11 IPv4 transitioning to Backup Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting virtual MAC address 00:00:5e:00:01:0b on hardware interface 2 Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 1 VRs configured, 1 VRs running Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [1] sw_if_index 24 VR ID 207 IPv4 transitioning to Backup Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting virtual MAC address 00:00:5e:00:01:cf on hardware interface 2 Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 2 VRs configured, 2 VRs running Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [2] sw_if_index 25 VR ID 202 IPv4 transitioning to Backup Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting virtual MAC address 00:00:5e:00:01:ca on hardware interface 2 Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 3 VRs configured, 3 VRs running Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address: vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported for interface index 0 Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [3] sw_if_index 26 VR ID 201 IPv4 transitioning to Backup Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting virtual MAC address 00:00:5e:00:01:c9 on hardware interface 2 Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 4 VRs configured, 4 VRs running Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [0] sw_if_index 2 VR ID 11 IPv4 transitioning to Master Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR addresses on sw_if_index 2 Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding virtual MAC address 00:00:5e:00:01:0b on hardware interface 2 Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [1] sw_if_index 24 VR ID 207 IPv4 transitioning to Master Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR addresses on sw_if_index 24 Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding virtual MAC address 00:00:5e:00:01:cf on hardware interface 2 Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [2] sw_if_index 25 VR ID 202 IPv4 transitioning to Master Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR addresses on sw_if_index 25 Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding virtual MAC address 00:00:5e:00:01:ca on hardware interface 2 Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [3] sw_if_index 26 VR ID 201 IPv4 transitioning to Master Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR addresses on sw_if_index 26 Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding virtual MAC address 00:00:5e:00:01:c9 on hardware interface 2 Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to start already started VR ([0] sw_if_index 2 VR ID 11 IPv4) Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to start already started VR ([1] sw_if_index 24 VR ID 207 IPv4) Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to start already started VR ([2] sw_if_index 25 VR ID 202 IPv4) Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to start already started VR ([3] sw_if_index 26 VR ID 201 IPv4) Dec 21 10:34:36 hostname vpp[67475]: vlib_worker_thread_barrier_sync_int: worker thread deadlock Dec 21 10:34:36 hostname vnet[67475]: received signal SIGABRT, PC 0x7f7207c5acdb Dec 21 10:34:36 hostname vnet[67475]: #0 0x00007f7208ea3234 unix_signal_handler + 0x124 Dec 21 10:34:36 hostname vnet[67475]: #1 0x00007f72087e68c0 0x7f72087e68c0 Dec 21 10:34:36 hostname vnet[67475]: #2 0x00007f7207c5acdb gsignal + 0x10d Dec 21 10:34:36 hostname vnet[67475]: #3 0x00007f7207c5c375 abort + 0x177 Dec 21 10:34:36 hostname vnet[67475]: #4 0x0000000000407643 0x407643 Dec 21 10:34:36 hostname vnet[67475]: #5 0x00007f7208e89541 vlib_worker_thread_barrier_sync_int + 0x451 Dec 21 10:34:36 hostname vnet[67475]: #6 0x00007f720a5aa3af vl_api_rpc_call_main_thread + 0x3cf Dec 21 10:34:36 hostname vnet[67475]: #7 0x00007f7209a95c1b arp_reply + 0x7eb Dec 21 10:34:36 hostname vnet[67475]: #8 0x00007f7208e50b65 dispatch_pending_node + 0x135 Dec 21 10:34:36 hostname vnet[67475]: #9 0x00007f7208e55214 vlib_main + 0xd34 Dec 21 10:34:36 hostname vnet[67475]: #10 0x00007f7208ea2288 thread0 + 0x28 Dec 21 10:34:36 hostname vnet[67475]: #11 0x00007f72083b600c 0x7f72083b600c Dec 21 10:34:49 hostname systemd[1]: vpp.service: Main process exited, code=dumped, status=6/ABRT We have our own controller which uses the API to do some dynamic configuration of VPP - apart from different routes in different tables, it configures VRRP as follows: # vppctl show vrrp vr [0] sw_if_index 2 VR ID 11 IPv4 state Master flags: preempt yes accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:0b addresses 10.95.255.2 peer addresses tracked interfaces [1] sw_if_index 24 VR ID 207 IPv4 state Master flags: preempt yes accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:cf addresses 10.94.131.34 peer addresses tracked interfaces [2] sw_if_index 25 VR ID 202 IPv4 state Master flags: preempt yes accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:ca addresses 10.94.29.90 peer addresses tracked interfaces [3] sw_if_index 26 VR ID 201 IPv4 state Master flags: preempt yes accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:c9 addresses 10.95.250.3 peer addresses tracked interfaces The IP addresses of the VRRP master are: # vppctl show int addr Ext-0 (up): L3 10.95.255.3/29 ip4 table-id 1 fib-idx 4 L3 10.95.255.2/29 ip4 table-id 1 fib-idx 4 Ext-0.507 (up): L3 10.94.131.35/29 ip4 table-id 2 fib-idx 6 L3 10.94.131.34/29 ip4 table-id 2 fib-idx 6 Ext-0.506 (up): L3 10.94.29.91/29 ip4 table-id 3 fib-idx 7 L3 10.94.29.90/29 ip4 table-id 3 fib-idx 7 Ext-0.501 (up): L3 10.95.250.4/29 ip4 table-id 4 fib-idx 8 L3 10.95.250.3/29 ip4 table-id 4 fib-idx 8 I don't have gdb on the affected system. On the build server, I only see: Core was generated by `/usr/bin/vpp -c /etc/vpp/ic-startup.conf'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f18c5b23cdb in ?? () [Current thread is 1 (LWP 14555)] (gdb) list 91 vat_plugin_hash_create (); 92 #endif 93 94 if (!vlib_plugin_path) 95 vpp_find_plugin_path (); 96 } 97 98 /* 99 * Default path for runtime data 100 */ (gdb) bt #0 0x00007f18c5b23cdb in ?? () Backtrace stopped: Cannot access memory at address 0x7f18c116cac0 Any ideas on what to check or change to get rid of the cyclic vpp restart? Any help is appreciated, Thank you, BR/Mechthild -------------------------------------------------------------- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22354): https://lists.fd.io/g/vpp-dev/message/22354 Mute This Topic: https://lists.fd.io/mt/95806883/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-