I'm no expert in LACP so maybe someone can jump in to confirm whether it looks good on VPP side, but as we're receiving LACP packets and VPP seems to be happy, I wonder if the issue could be VPP failing to send LACP packets back to the switch? Eg. because the NIC eat them up for some reason (security...)? Could you share the output of 'sh err' and 'sh hard', and also check whether the switch is receiving any LACP packets?
Best ben > -----Original Message----- > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Eyle > Brinkhuis > Sent: Monday, December 5, 2022 14:23 > To: Benoit Ganne (bganne) <bga...@cisco.com> > Cc: vpp-dev <vpp-dev@lists.fd.io> > Subject: [vpp-dev] LACP issues w/ cdma/connectX 6 > > Hi Ben, > > We have a few new boxes that have a connectX 6 fitted (Mellanox ConnectX-6 > Dx 100GbE QSFP56 2-port PCIe 4 Ethernet Adapter) and we run into an issue > with LACP (seems a popular topic these days.. :-)). We are not able to get > LACP up, while running this: > > > > create int rdma host-if ens3f0 name rdma0 > > create int rdma host-if ens3f1 name rdma1 > > set interface state rdma1 up > > set interface state rdma0 up > > create bond mode lacp > > bond add BondEthernet0 rdma0 > > bond add BondEthernet0 rdma1 > > set int state BondEthernet0 up > > > > > On the switch side (Mellanox sn2700) everything is the same as with our > Mellanox CX5 NICs (also on RDMA, with same lacp configuration as above). > CX5 stuff all works a charm. > > > > > For the CX6, we receive the BDPU’s from RDMA, and they seem to be > processed: > > > > > Packet 1 > > > > > 00:00:52:494905: rdma-input > > rdma: rdma0 (1) next-node bond-input > > 00:00:52:494920: bond-input > > src b8:59:9f:67:fa:ba, dst 01:80:c2:00:00:02, rdma0 -> rdma0 > > 00:00:52:494926: ethernet-input > > SLOW_PROTOCOLS: b8:59:9f:67:fa:ba -> 01:80:c2:00:00:02 > > 00:00:52:494930: lacp-input > > rdma0: > > Length: 110 > > LACPv1 > > Actor Information TLV: length 20 > > System b8:59:9f:67:fa:80 > > System priority 32768 > > Key 13834 > > Port priority 32768 > > Port number 25 > > State 0x45 > > LACP_STATE_LACP_ACTIVITY (0) > > LACP_STATE_AGGREGATION (2) > > LACP_STATE_DEFAULTED (6) > > Partner Information TLV: length 20 > > System 00:00:00:00:00:00 > > System priority 0 > > Key 0 > > Port priority 0 > > Port number 0 > > State 0x7c > > LACP_STATE_AGGREGATION (2) > > LACP_STATE_SYNCHRONIZATION (3) > > LACP_STATE_COLLECTIING (4) > > LACP_STATE_DISTRIBUTING (5) > > LACP_STATE_DEFAULTED (6) > > 0x0000: 0101 0114 8000 b859 9f67 fa80 360a 8000 > > 0x0010: 0019 4500 0000 0214 0000 0000 0000 0000 > > 0x0020: 0000 0000 0000 7c00 0000 0310 0000 0000 > > 0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 > > 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 > > 0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 > > 0x0060: 0000 0000 0000 0000 0000 0000 0000 > > 00:00:52:494936: error-drop > > rx:rdma0 > > 00:00:52:494937: drop > > lacp-input: good lacp packets — consumed > > > > > vpp# sh lacp actor > state partner state > > interface name sw_if_index bond interface > exp/def/dis/col/syn/agg/tim/act exp/def/dis/col/syn/agg/tim/act > > rdma0 1 BondEthernet0 0 1 0 0 > 1 1 1 1 0 0 0 0 0 0 0 0 > > LAG ID: [(ffff,02-fe-c6-4c-c3-62,0003,00ff,0001), (ffff,00-00-00-00-00- > 00,0003,00ff,0001)] > > RX-state: DEFAULTED, TX-state: TRANSMIT, MUX-state: ATTACHED, PTX-state: > PERIODIC_TX > > rdma1 2 BondEthernet0 0 1 0 0 > 0 1 1 1 0 0 0 0 0 0 0 0 > > LAG ID: [(ffff,02-fe-c6-4c-c3-62,0003,00ff,0002), (ffff,00-00-00-00-00- > 00,0003,00ff,0002)] > > RX-state: DEFAULTED, TX-state: TRANSMIT, MUX-state: DETACHED, PTX-state: > PERIODIC_TX > > > > > vpp# sh bond details > > BondEthernet0 > > mode: lacp > > load balance: l2 > > number of active members: 0 > > number of members: 2 > > rdma0 > > rdma1 > > device instance: 0 > > interface id: 0 > > sw_if_index: 3 > > hw_if_index: 3 > > > > > > > > vpp# sh log > > 2022/12/05 12:43:14:840 notice plugin/load Loaded plugin: > abf_plugin.so (Access Control List (ACL) Based Forwarding) > > 2022/12/05 12:43:14:841 notice plugin/load Loaded plugin: > acl_plugin.so (Access Control Lists (ACL)) > > 2022/12/05 12:43:14:842 notice plugin/load Loaded plugin: > adl_plugin.so (Allow/deny list plugin) > > 2022/12/05 12:43:14:842 notice plugin/load Loaded plugin: > af_xdp_plugin.so (AF_XDP Device Plugin) > > 2022/12/05 12:43:14:842 notice plugin/load Loaded plugin: > arping_plugin.so (Arping (arping)) > > 2022/12/05 12:43:14:843 notice plugin/load Loaded plugin: > avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Driver) > > 2022/12/05 12:43:14:843 notice plugin/load Loaded plugin: > builtinurl_plugin.so (vpp built-in URL support) > > 2022/12/05 12:43:14:843 notice plugin/load Loaded plugin: > cdp_plugin.so (Cisco Discovery Protocol (CDP)) > > 2022/12/05 12:43:14:844 notice plugin/load Loaded plugin: > cnat_plugin.so (CNat Translate) > > 2022/12/05 12:43:14:860 notice plugin/load Loaded plugin: > crypto_ipsecmb_plugin.so (Intel IPSEC Multi-buffer Crypto Engine) > > 2022/12/05 12:43:14:860 notice plugin/load Loaded plugin: > crypto_native_plugin.so (Intel IA32 Software Crypto Engine) > > 2022/12/05 12:43:14:860 notice plugin/load Loaded plugin: > crypto_openssl_plugin.so (OpenSSL Crypto Engine) > > 2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: > crypto_sw_scheduler_plugin.so (SW Scheduler Crypto Async Engine plugin) > > 2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: > ct6_plugin.so (IPv6 Connection Tracker) > > 2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: > det44_plugin.so (Deterministic NAT (CGN)) > > 2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: > dhcp_plugin.so (Dynamic Host Configuration Protocol (DHCP)) > > 2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: > dns_plugin.so (Simple DNS name resolver) > > 2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: > dslite_plugin.so (Dual-Stack Lite) > > 2022/12/05 12:43:14:861 notice plugin/load Loaded plugin: > flowprobe_plugin.so (Flow per Packet) > > 2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: > geneve_plugin.so (GENEVE Tunnels) > > 2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: > gtpu_plugin.so (GPRS Tunnelling Protocol, User Data (GTPv1-U)) > > 2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: > hs_apps_plugin.so (Host Stack Applications) > > 2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: > hsi_plugin.so (Host Stack Intercept (HSI)) > > 2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: > http_static_plugin.so (HTTP Static Server) > > 2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: > igmp_plugin.so (Internet Group Management Protocol (IGMP)) > > 2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: > ikev2_plugin.so (Internet Key Exchange (IKEv2) Protocol) > > 2022/12/05 12:43:14:862 notice plugin/load Loaded plugin: > ila_plugin.so (Identifier Locator Addressing (ILA) for IPv6) > > 2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: > ioam_plugin.so (Inbound Operations, Administration, and Maintenance (OAM)) > > 2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: > l2tp_plugin.so (Layer 2 Tunneling Protocol v3 (L2TP)) > > 2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: > l3xc_plugin.so (L3 Cross-Connect (L3XC)) > > 2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: > lacp_plugin.so (Link Aggregation Control Protocol (LACP)) > > 2022/12/05 12:43:14:863 notice plugin/load Loaded plugin: > lb_plugin.so (Load Balancer (LB)) > > 2022/12/05 12:43:14:863 notice plugin/load Plugin disabled > (default): linux_cp_plugin.so > > 2022/12/05 12:43:14:863 notice plugin/load Plugin disabled > (default): linux_cp_unittest_plugin.so > > 2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: > lisp_plugin.so (Locator ID Separation Protocol (LISP)) > > 2022/12/05 12:43:14:864 notice plugin/load Plugin disabled > (default): lisp_unittest_plugin.so > > 2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: > lldp_plugin.so (Link Layer Discovery Protocol (LLDP)) > > 2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: > mactime_plugin.so (Time-based MAC Source Address Filter) > > 2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: > map_plugin.so (Mapping of Address and Port (MAP)) > > 2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: > mdata_plugin.so (Buffer metadata change tracker.) > > 2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: > memif_plugin.so (Packet Memory Interface (memif) -- Experimental) > > 2022/12/05 12:43:14:864 notice plugin/load Loaded plugin: > mss_clamp_plugin.so (TCP MSS clamping plugin) > > 2022/12/05 12:43:14:865 notice plugin/load Loaded plugin: > nat44_ei_plugin.so (IPv4 Endpoint-Independent NAT (NAT44 EI)) > > 2022/12/05 12:43:14:865 notice plugin/load Loaded plugin: > nat64_plugin.so (NAT64) > > 2022/12/05 12:43:14:865 notice plugin/load Loaded plugin: > nat66_plugin.so (NAT66) > > 2022/12/05 12:43:14:865 notice plugin/load Loaded plugin: > nat_plugin.so (Network Address Translation (NAT)) > > 2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: > nsh_plugin.so (Network Service Header (NSH)) > > 2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: > nsim_plugin.so (Network Delay Simulator) > > 2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: > ping_plugin.so (Ping (ping)) > > 2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: > pnat_plugin.so (Policy 1:1 NAT) > > 2022/12/05 12:43:14:866 notice plugin/load Loaded plugin: > pppoe_plugin.so (PPP over Ethernet (PPPoE)) > > 2022/12/05 12:43:14:866 notice plugin/load Plugin disabled > (default): quic_plugin.so > > 2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: > rdma_plugin.so (RDMA IBverbs Device Driver) > > 2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: > srv6ad_plugin.so (Dynamic Segment Routing for IPv6 (SRv6) Proxy) > > 2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: > srv6adflow_plugin.so (Dynamic Segment Routing for IPv6 (SRv6) Proxy) > > 2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: > srv6am_plugin.so (Masquerading Segment Routing for IPv6 (SRv6) Proxy) > > 2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: > srv6as_plugin.so (Static Segment Routing for IPv6 (SRv6) Proxy) > > 2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: > srv6mobile_plugin.so (SRv6 GTP Endpoint Functions) > > 2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: > stn_plugin.so (VPP Steals the NIC (STN) for Container Integration) > > 2022/12/05 12:43:14:867 notice plugin/load Loaded plugin: > svs_plugin.so (Source Virtual Routing and Forwarding (VRF) Select) > > 2022/12/05 12:43:14:868 notice plugin/load Loaded plugin: > tlsmbedtls_plugin.so (Transport Layer Security (TLS) Engine, Mbedtls > Based) > > 2022/12/05 12:43:14:868 notice plugin/load Loaded plugin: > tlsopenssl_plugin.so (Transport Layer Security (TLS) Engine, OpenSSL > Based) > > 2022/12/05 12:43:14:868 notice plugin/load Loaded plugin: > tlspicotls_plugin.so (Transport Layer Security (TLS) Engine, Picotls > Based) > > 2022/12/05 12:43:14:868 notice plugin/load Loaded plugin: > urpf_plugin.so (Unicast Reverse Path Forwarding (uRPF)) > > 2022/12/05 12:43:14:869 notice plugin/load Loaded plugin: > vmxnet3_plugin.so (VMWare Vmxnet3 Device Driver) > > 2022/12/05 12:43:14:869 notice plugin/load Loaded plugin: > vrrp_plugin.so (VRRP v3 (RFC 5798)) > > 2022/12/05 12:43:14:869 notice plugin/load Loaded plugin: > wireguard_plugin.so (Wireguard Protocol) > > 2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin: > arping_test_plugin.so > > 2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin: > vrrp_test_plugin.so > > 2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin: > ikev2_test_plugin.so > > 2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin: > lacp_test_plugin.so > > 2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin: > acl_test_plugin.so > > 2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin: > mactime_test_plugin.so > > 2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin: > vnet_ip6_nd_test_plugin.so > > 2022/12/05 12:43:15:269 notice vat-plug/load Loaded plugin: > ct6_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > lisp_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > lb_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > mdata_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > vnet_ip_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > vnet_l2_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > vpp_api_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > ioam_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > stn_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > flowprobe_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > avf_test_plugin.so > > 2022/12/05 12:43:15:270 notice vat-plug/load Loaded plugin: > http_static_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > gtpu_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > dns_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > memif_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > nsim_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > vnet_interface_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > adl_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > geneve_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > tlsopenssl_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > rdma_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > vlib_vlibapi_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > vmxnet3_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > builtinurl_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > vnet_session_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > dhcp_test_plugin.so > > 2022/12/05 12:43:15:271 notice vat-plug/load Loaded plugin: > af_xdp_test_plugin.so > > 2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin: > vnet_arp_test_plugin.so > > 2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin: > nsh_test_plugin.so > > 2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin: > pppoe_test_plugin.so > > 2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin: > vnet_sr_mpls_test_plugin.so > > 2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin: > vnet_ipsec_test_plugin.so > > 2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin: > l2tp_test_plugin.so > > 2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin: > lldp_test_plugin.so > > 2022/12/05 12:43:15:272 notice vat-plug/load Loaded plugin: > cdp_test_plugin.so > > > > > MELLANOX-SN2700 [standalone: master] (config) # show lacp interfaces > ethernet 1/4 > > Port: 1/4 > > Port State: Suspend > > Channel Group: 9 > > Pseudo port-channel: Po9 > > LACP port-priority: 32768 > > LACP Rate: Slow > > LACP Activity: Active > > LACP Timeout: Long > > Aggregation State: Aggregation, Defaulted, > > > > > -------------------------------------------------------------- > > LACP Port Admin Oper Port Port > > Port State Priority Key Key Number State > > -------------------------------------------------------------- > > 1/4 Suspend 32768 13834 13834 0x4 0x45 > > > > > > > > Any idea? > > > > > I was able to create a LACP bond on the Mellanox CX6 interfaces in Linux > itself.. > > > > > Thanks, > > > > > Eyle > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22280): https://lists.fd.io/g/vpp-dev/message/22280 Mute This Topic: https://lists.fd.io/mt/95468251/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
