We recently started running VPP on Debian 12 with dual-port Mellanox CX414A-GCAT NICs, and encountered an issue where the LACP bond flaps randomly while the physical links remain up. I noticed a previous thread where this was reported ( https://lists.fd.io/g/vpp-dev/topic/mellanox_connectx5_lacp_not/113443571) and attempted the same fix: changing the LACP timer on the switch-side (Arista DCS-7050QX-32S-R) to fast using "lacp timer fast" on the physical ports. This temporarily stabilized the connection until LACP resumed flapping randomly again, leaving me at a standstill trying to determine what is occurring.
I've attached my startup config and a redacted version of my vpp bootstrap config to provide better context on the configuration used, as well as the output from "show err" and switch logs for the port channel. For reference here, devices "0000:87:00.0" and "0000:87:00.1" are the NIC ports with the LACP members. VPP version is v26.06-rc0~519-gf206434c7, on Debian 12 Bookworm with Linux kernel version 6.1.0-44-amd64. Hardware is a dual Intel Xeon E5-2637v4 CPU with 64GB DDR4 ECC memory. Hyperthreading is disabled, and the VPP workers are pinned to the CPU cores connected to the network cards to avoid motherboard PCIe bridge bottlenecks. VPP does not seem to crash; it stays running without any errors, and I don't see any errors in the logs. Any insight is appreciated. Kind regards, Peter Potvin
Apr 27 10:12:08 switch Lag: %LAG-5-MEMBER_REMOVED: Interface Ethernet15/1 has
left Port-Channel15 ("RU25") due to: unknown reason
Apr 27 10:13:08 switch Lag: %LACP-4-PARTNER_CHURN: LACP Partner Churn Detected
on Ethernet15/1
Apr 27 10:13:11 switch Lag: %LACP-4-ACTOR_CHURN: LACP Actor Churn Detected on
Ethernet15/1
Apr 27 10:14:06 switch Lag: %LAG-5-MEMBER_ADDED: Interface Ethernet15/1 has
joined Port-Channel15 ("RU25")
Apr 27 16:07:18 switch Lag: %LAG-5-MEMBER_REMOVED: Interface Ethernet15/1 has
left Port-Channel15 ("RU25") due to: unknown reason
bootstrap.conf
Description: Binary data
startup.conf
Description: Binary data
vpp# show err
Count Node Reason
Severity
1 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h
pa error
4 ip6-icmp-input router advertisements sent
info
15822 BondEthernet1-output interface is down
error
1 null-node blackholed packets
error
301 dpdk-input no error
error
62443 arp-reply ARP replies sent
info
85 arp-reply IP4 source address not local to
sub error
470 arp-reply ARP request IP4 source address
lear info
16 ip6-glean neighbor solicitations sent
info
16 ip4-glean ARP requests sent
info
25 ip6-input ip6 ttl <= 1
error
22 ip6-input ip6 source lookup miss
error
954 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h
pa error
33 ip4-input ip4 ttl <= 1
error
1 ip4-local ip4 source lookup miss
error
25 ip6-icmp-error hop limit exceeded response sent
info
491 ip6-icmp-input valid packets
info
215 ip6-icmp-input echo replies sent
info
76887 ip6-icmp-input neighbor solicitations for unknown
error
564 ip6-icmp-input neighbor advertisements sent
info
491 ip6-icmp-input router advertisements received
info
33 ip4-icmp-error hop limit exceeded response sent
info
4 ip4-icmp-input echo replies sent
info
835 snap-input unknown oui/snap protocol
error
159 llc-input unknown llc ssap/dsap
error
4 ethernet-input no error
error
1FiftySixGigabitEthernet86/0/1-outpu interface is down
error
4 null-node blackholed packets
error
5262 dpdk-input no error
error
49889 arp-reply ARP replies sent
info
980 arp-reply IP4 source address not local to
sub error
15587 arp-reply IP4 destination address not local
t error
881 arp-reply ARP request IP4 source address
lear info
290 ip6-glean throttled
info
903 ip6-glean neighbor solicitations sent
info
8959 ip4-glean ARP requests throttled
info
9368 ip4-glean ARP requests sent
info
4299 ip6-input ip6 ttl <= 1
error
44 ip6-input ip6 source lookup miss
error
1336 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h
pa error
63 ip4-input ip4 ttl <= 1
error
6 ip4-local ip4 source lookup miss
error
4299 ip6-icmp-error hop limit exceeded response sent
info
314 ip6-icmp-input valid packets
info
2755 ip6-icmp-input echo replies sent
info
48459 ip6-icmp-input neighbor solicitations for unknown
error
981 ip6-icmp-input neighbor advertisements sent
info
314 ip6-icmp-input router advertisements received
info
63 ip4-icmp-error hop limit exceeded response sent
info
2993 snap-input unknown oui/snap protocol
error
701 llc-input unknown llc ssap/dsap
error
1 ethernet-input no error
error
14 ethernet-input unknown ethernet type
error
16 ethernet-input unknown vlan
error
11 ethernet-input subinterface down
error
1FiftySixGigabitEthernet86/0/1-outpu interface is down
error
295 null-node blackholed packets
error
49 lldp-input lldp packets received on disabled
i error
6 lacp-input good lacp packets -- consumed
error
1399 lacp-input good lacp packets -- cache hit
error
20844 arp-reply ARP replies sent
info
168 arp-reply ARP request IP4 source address
lear info
372 arp-reply RX interface is unnumbered to
diffe error
3 ip6-glean throttled
info
62 ip6-glean neighbor solicitations sent
info
7 ip4-glean ARP requests throttled
info
2790 ip4-glean ARP requests sent
info
15 ip4-arp ARP requests throttled
info
2 ip4-arp ARP requests sent
info
1808 ip6-input ip6 ttl <= 1
error
8 ip6-input ip6 source lookup miss
error
6574 ip4-input ip4 ttl <= 1
error
39 ip4-local ip4 source lookup miss
error
1799 ip6-icmp-error hop limit exceeded response sent
info
9 ip6-icmp-error error message dropped
error
33 ip6-icmp-input valid packets
info
244 ip6-icmp-input echo replies sent
info
8236 ip6-icmp-input neighbor solicitations for unknown
error
438 ip6-icmp-input neighbor advertisements sent
info
13 ip6-icmp-input router advertisements received
info
1622 ip4-icmp-error destination unreachable response
se info
6572 ip4-icmp-error hop limit exceeded response sent
info
2 ip4-icmp-error error message dropped
error
90 ip4-icmp-input echo replies sent
info
497991110 bond-input no error
error
1454 bond-input pass through (CDP, LLDP, slow
proto error
221931 ethernet-input no error
error
9 ethernet-input unknown vlan
error
1FiftySixGigabitEthernet86/0/1-outpu interface is down
error
214 null-node blackholed packets
error
49 lldp-input lldp packets received on disabled
i error
7 lacp-input good lacp packets -- consumed
error
1396 lacp-input good lacp packets -- cache hit
error
51795 arp-reply ARP replies sent
info
4 arp-reply IP4 source address not local to
sub error
1817 arp-reply IP4 destination address not local
t error
248 arp-reply ARP request IP4 source address
lear info
1 arp-reply ARP hw addr does not match L2
frame error
1 ip6-discover-neighbor neighbor solicitations sent
info
7 ip6-glean throttled
info
93 ip6-glean neighbor solicitations sent
info
8 ip4-glean ARP requests throttled
info
3489 ip4-glean ARP requests sent
info
21 ip4-arp ARP requests throttled
info
5 ip4-arp ARP requests sent
info
1390 ip6-input ip6 ttl <= 1
error
6 ip6-input ip6 source lookup miss
error
2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h
pa error
5631 ip4-input ip4 ttl <= 1
error
247 ip4-local ip4 source lookup miss
error
1377 ip6-icmp-error hop limit exceeded response sent
info
13 ip6-icmp-error error message dropped
error
50 ip6-icmp-input valid packets
info
231 ip6-icmp-input echo replies sent
info
15260 ip6-icmp-input neighbor solicitations for unknown
error
595 ip6-icmp-input neighbor advertisements sent
info
34 ip6-icmp-input router advertisements received
info
1345 ip4-icmp-error destination unreachable response
se info
5631 ip4-icmp-error hop limit exceeded response sent
info
586 ip4-icmp-input echo replies sent
info
482171651 bond-input no error
error
1452 bond-input pass through (CDP, LLDP, slow
proto error
685 snap-input unknown oui/snap protocol
error
700 llc-input unknown llc ssap/dsap
error
224042 ethernet-input no error
error
2 ethernet-input unknown vlan
error-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#26999): https://lists.fd.io/g/vpp-dev/message/26999 Mute This Topic: https://lists.fd.io/mt/119066888/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
