Hi Ben,
We have a few new boxes that have a connectX 6 fitted (Mellanox ConnectX-6 Dx
100GbE QSFP56 2-port PCIe 4 Ethernet Adapter) and we run into an issue with
LACP (seems a popular topic these days.. :-)). We are not able to get LACP up,
while running this:
create int rdma host-if ens3f0 name rdma0
create int rdma host-if ens3f1 name rdma1
set interface state rdma1 up
set interface state rdma0 up
create bond mode lacp
bond add BondEthernet0 rdma0
bond add BondEthernet0 rdma1
set int state BondEthernet0 up
On the switch side (Mellanox sn2700) everything is the same as with our
Mellanox CX5 NICs (also on RDMA, with same lacp configuration as above). CX5
stuff all works a charm.
For the CX6, we receive the BDPU’s from RDMA, and they seem to be processed:
Packet 1
00:00:52:494905: rdma-input
rdma: rdma0 (1) next-node bond-input
00:00:52:494920: bond-input
src b8:59:9f:67:fa:ba, dst 01:80:c2:00:00:02, rdma0 -> rdma0
00:00:52:494926: ethernet-input
SLOW_PROTOCOLS: b8:59:9f:67:fa:ba -> 01:80:c2:00:00:02
00:00:52:494930: lacp-input
rdma0:
Length: 110
LACPv1
Actor Information TLV: length 20
System b8:59:9f:67:fa:80
System priority 32768
Key 13834
Port priority 32768
Port number 25
State 0x45
LACP_STATE_LACP_ACTIVITY (0)
LACP_STATE_AGGREGATION (2)
LACP_STATE_DEFAULTED (6)
Partner Information TLV: length 20
System 00:00:00:00:00:00
System priority 0
Key 0
Port priority 0
Port number 0
State 0x7c
LACP_STATE_AGGREGATION (2)
LACP_STATE_SYNCHRONIZATION (3)
LACP_STATE_COLLECTIING (4)
LACP_STATE_DISTRIBUTING (5)
LACP_STATE_DEFAULTED (6)
0x0000: 0101 0114 8000 b859 9f67 fa80 360a 8000
0x0010: 0019 4500 0000 0214 0000 0000 0000 0000
0x0020: 0000 0000 0000 7c00 0000 0310 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000
0x0050: 0000 0000 0000 0000 0000 0000 0000 0000
0x0060: 0000 0000 0000 0000 0000 0000 0000
00:00:52:494936: error-drop
rx:rdma0
00:00:52:494937: drop
lacp-input: good lacp packets — consumed
vpp# sh lacp actor state
partner state
interface name sw_if_index bond interface
exp/def/dis/col/syn/agg/tim/act exp/def/dis/col/syn/agg/tim/act
rdma0 1 BondEthernet0 0 1 0 0 1 1
1 1 0 0 0 0 0 0 0 0
LAG ID: [(ffff,02-fe-c6-4c-c3-62,0003,00ff,0001),
(ffff,00-00-00-00-00-00,0003,00ff,0001)]
RX-state: DEFAULTED, TX-state: TRANSMIT, MUX-state: ATTACHED, PTX-state:
PERIODIC_TX
rdma1 2 BondEthernet0 0 1 0 0 0 1
1 1 0 0 0 0 0 0 0 0
LAG ID: [(ffff,02-fe-c6-4c-c3-62,0003,00ff,0002),
(ffff,00-00-00-00-00-00,0003,00ff,0002)]
RX-state: DEFAULTED, TX-state: TRANSMIT, MUX-state: DETACHED, PTX-state:
PERIODIC_TX
vpp# sh bond details
BondEthernet0
mode: lacp
load balance: l2
number of active members: 0
number of members: 2
rdma0
rdma1
device instance: 0
interface id: 0
sw_if_index: 3
hw_if_index: 3
vpp# sh log
2022/12/05 12:43:14:840 notice plugin/load Loaded plugin: abf_plugin.so
(Access Control List (ACL) Based Forwarding)
2022/12/05 12:43:14:841 notice plugin/load Loaded plugin: acl_plugin.so
(Access Control Lists (ACL))
2022/12/05 12:43:14:842 notice plugin/load Loaded plugin: adl_plugin.so
(Allow/deny list plugin)
2022/12/05 12:43:14:842 notice plugin/load Loaded plugin:
af_xdp_plugin.so (AF_XDP Device Plugin)
2022/12/05 12:43:14:842 notice plugin/load Loaded plugin:
arping_plugin.so (Arping (arping))
2022/12/05 12:43:14:843 notice plugin/load Loaded plugin: avf_plugin.so
(Intel Adaptive Virtual Function (AVF) Device Driver)
2022/12/05 12:43:14:843 notice plugin/load Loaded plugin:
builtinurl_plugin.so (vpp built-in URL support)
2022/12/05 12:43:14:843 notice plugin/load Loaded plugin: cdp_plugin.so
(Cisco Discovery Protocol (CDP))
2022/12/05 12:43:14:844 notice plugin/load Loaded plugin: cnat_plugin.so
(CNat Translate)
2022/12/05 12:43:14:860 notice plugin/load Loaded plugin:
crypto_ipsecmb_plugin.so (Intel IPSEC Multi-buffer Crypto Engine)
2022/12/05 12:43:14:860 notice plugin/load Loaded plugin:
crypto_native_plugin.so (Intel IA32 Software Crypto Engine)
2022/12/05 12:43:14:860 notice plugin/load Loaded plugin:
crypto_openssl_plugin.so (OpenSSL Crypto Engine)
2022/12/05 12:43:14:861 notice plugin/load Loaded plugin:
crypto_sw_scheduler_plugin.so (SW Scheduler Crypto Async Engine plugin)
2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: ct6_plugin.so
(IPv6 Connection Tracker)
2022/12/05 12:43:14:861 notice plugin/load Loaded plugin:
det44_plugin.so (Deterministic NAT (CGN))
2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: dhcp_plugin.so
(Dynamic Host Configuration Protocol (DHCP))
2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: dns_plugin.so
(Simple DNS name resolver)
2022/12/05 12:43:14:861 notice plugin/load Loaded plugin:
dslite_plugin.so (Dual-Stack Lite)
2022/12/05 12:43:14:861 notice plugin/load Loaded plugin:
flowprobe_plugin.so (Flow per Packet)
2022/12/05 12:43:14:862 notice plugin/load Loaded plugin:
geneve_plugin.so (GENEVE Tunnels)
2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: gtpu_plugin.so
(GPRS Tunnelling Protocol, User Data (GTPv1-U))
2022/12/05 12:43:14:862 notice plugin/load Loaded plugin:
hs_apps_plugin.so (Host Stack Applications)
2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: hsi_plugin.so
(Host Stack Intercept (HSI))
2022/12/05 12:43:14:862 notice plugin/load Loaded plugin:
http_static_plugin.so (HTTP Static Server)
2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: igmp_plugin.so
(Internet Group Management Protocol (IGMP))
2022/12/05 12:43:14:862 notice plugin/load Loaded plugin:
ikev2_plugin.so (Internet Key Exchange (IKEv2) Protocol)
2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: ila_plugin.so
(Identifier Locator Addressing (ILA) for IPv6)
2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: ioam_plugin.so
(Inbound Operations, Administration, and Maintenance (OAM))
2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: l2tp_plugin.so
(Layer 2 Tunneling Protocol v3 (L2TP))
2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: l3xc_plugin.so
(L3 Cross-Connect (L3XC))
2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: lacp_plugin.so
(Link Aggregation Control Protocol (LACP))
2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: lb_plugin.so
(Load Balancer (LB))
2022/12/05 12:43:14:863 notice plugin/load Plugin disabled (default):
linux_cp_plugin.so
2022/12/05 12:43:14:863 notice plugin/load Plugin disabled (default):
linux_cp_unittest_plugin.so
2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: lisp_plugin.so
(Locator ID Separation Protocol (LISP))
2022/12/05 12:43:14:864 notice plugin/load Plugin disabled (default):
lisp_unittest_plugin.so
2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: lldp_plugin.so
(Link Layer Discovery Protocol (LLDP))
2022/12/05 12:43:14:864 notice plugin/load Loaded plugin:
mactime_plugin.so (Time-based MAC Source Address Filter)
2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: map_plugin.so
(Mapping of Address and Port (MAP))
2022/12/05 12:43:14:864 notice plugin/load Loaded plugin:
mdata_plugin.so (Buffer metadata change tracker.)
2022/12/05 12:43:14:864 notice plugin/load Loaded plugin:
memif_plugin.so (Packet Memory Interface (memif) -- Experimental)
2022/12/05 12:43:14:864 notice plugin/load Loaded plugin:
mss_clamp_plugin.so (TCP MSS clamping plugin)
2022/12/05 12:43:14:865 notice plugin/load Loaded plugin:
nat44_ei_plugin.so (IPv4 Endpoint-Independent NAT (NAT44 EI))
2022/12/05 12:43:14:865 notice plugin/load Loaded plugin:
nat64_plugin.so (NAT64)
2022/12/05 12:43:14:865 notice plugin/load Loaded plugin:
nat66_plugin.so (NAT66)
2022/12/05 12:43:14:865 notice plugin/load Loaded plugin: nat_plugin.so
(Network Address Translation (NAT))
2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: nsh_plugin.so
(Network Service Header (NSH))
2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: nsim_plugin.so
(Network Delay Simulator)
2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: ping_plugin.so
(Ping (ping))
2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: pnat_plugin.so
(Policy 1:1 NAT)
2022/12/05 12:43:14:866 notice plugin/load Loaded plugin:
pppoe_plugin.so (PPP over Ethernet (PPPoE))
2022/12/05 12:43:14:866 notice plugin/load Plugin disabled (default):
quic_plugin.so
2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: rdma_plugin.so
(RDMA IBverbs Device Driver)
2022/12/05 12:43:14:867 notice plugin/load Loaded plugin:
srv6ad_plugin.so (Dynamic Segment Routing for IPv6 (SRv6) Proxy)
2022/12/05 12:43:14:867 notice plugin/load Loaded plugin:
srv6adflow_plugin.so (Dynamic Segment Routing for IPv6 (SRv6) Proxy)
2022/12/05 12:43:14:867 notice plugin/load Loaded plugin:
srv6am_plugin.so (Masquerading Segment Routing for IPv6 (SRv6) Proxy)
2022/12/05 12:43:14:867 notice plugin/load Loaded plugin:
srv6as_plugin.so (Static Segment Routing for IPv6 (SRv6) Proxy)
2022/12/05 12:43:14:867 notice plugin/load Loaded plugin:
srv6mobile_plugin.so (SRv6 GTP Endpoint Functions)
2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: stn_plugin.so
(VPP Steals the NIC (STN) for Container Integration)
2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: svs_plugin.so
(Source Virtual Routing and Forwarding (VRF) Select)
2022/12/05 12:43:14:868 notice plugin/load Loaded plugin:
tlsmbedtls_plugin.so (Transport Layer Security (TLS) Engine, Mbedtls Based)
2022/12/05 12:43:14:868 notice plugin/load Loaded plugin:
tlsopenssl_plugin.so (Transport Layer Security (TLS) Engine, OpenSSL Based)
2022/12/05 12:43:14:868 notice plugin/load Loaded plugin:
tlspicotls_plugin.so (Transport Layer Security (TLS) Engine, Picotls Based)
2022/12/05 12:43:14:868 notice plugin/load Loaded plugin: urpf_plugin.so
(Unicast Reverse Path Forwarding (uRPF))
2022/12/05 12:43:14:869 notice plugin/load Loaded plugin:
vmxnet3_plugin.so (VMWare Vmxnet3 Device Driver)
2022/12/05 12:43:14:869 notice plugin/load Loaded plugin: vrrp_plugin.so
(VRRP v3 (RFC 5798))
2022/12/05 12:43:14:869 notice plugin/load Loaded plugin:
wireguard_plugin.so (Wireguard Protocol)
2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin:
arping_test_plugin.so
2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin:
vrrp_test_plugin.so
2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin:
ikev2_test_plugin.so
2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin:
lacp_test_plugin.so
2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin:
acl_test_plugin.so
2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin:
mactime_test_plugin.so
2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin:
vnet_ip6_nd_test_plugin.so
2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin:
ct6_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
lisp_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
lb_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
mdata_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
vnet_ip_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
vnet_l2_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
vpp_api_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
ioam_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
stn_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
flowprobe_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
avf_test_plugin.so
2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin:
http_static_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
gtpu_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
dns_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
memif_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
nsim_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
vnet_interface_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
adl_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
geneve_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
tlsopenssl_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
rdma_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
vlib_vlibapi_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
vmxnet3_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
builtinurl_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
vnet_session_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
dhcp_test_plugin.so
2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin:
af_xdp_test_plugin.so
2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin:
vnet_arp_test_plugin.so
2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin:
nsh_test_plugin.so
2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin:
pppoe_test_plugin.so
2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin:
vnet_sr_mpls_test_plugin.so
2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin:
vnet_ipsec_test_plugin.so
2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin:
l2tp_test_plugin.so
2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin:
lldp_test_plugin.so
2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin:
cdp_test_plugin.so
MELLANOX-SN2700 [standalone: master] (config) # show lacp interfaces ethernet
1/4
Port: 1/4
Port State: Suspend
Channel Group: 9
Pseudo port-channel: Po9
LACP port-priority: 32768
LACP Rate: Slow
LACP Activity: Active
LACP Timeout: Long
Aggregation State: Aggregation, Defaulted,
--------------------------------------------------------------
LACP Port Admin Oper Port Port
Port State Priority Key Key Number State
--------------------------------------------------------------
1/4 Suspend 32768 13834 13834 0x4 0x45
Any idea?
I was able to create a LACP bond on the Mellanox CX6 interfaces in Linux
itself..
Thanks,
Eyle
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22279): https://lists.fd.io/g/vpp-dev/message/22279
Mute This Topic: https://lists.fd.io/mt/95468251/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
