> -----Original Message----- > From: Dariusz Sosnowski <[email protected]> > Sent: Monday, October 26, 2026 5:27 PM > To: Slava Ovsiienko <[email protected]>; Bing Zhao > <[email protected]>; Ori Kam <[email protected]>; Suanming Mou > <[email protected]>; Matan Azrad <[email protected]> > Cc: [email protected] > Subject: [PATCH 00/13] net/mlx5: support nested E-Switch > > NVIDIA NICs and DPUs have an embedded switch (E-Switch) to which all > associated functions (PFs, VFs, SFs) and physical ports are connected through > virtual ports (vports). > When E-Switch is in switchdev mode, it is exposed as a single IB device with > multiple ports, owned by the PF (E-Switch manager). > Uplink and each of the representor netdevs are bound to one of the IB device > ports. > mlx5 PMD probes the PF device and spawns ethdev ports for the uplink and > for representors selected through representor devargs. > Type and location of each port in the E-Switch topology are derived from > phys_port_name of the related netdev. > > Future FW and Linux kernel releases will introduce nested E-Switch. > Nested E-Switch groups a set of VFs or SFs into a new E-Switch instance. > Its architecture is similar to the regular E-Switch: > there is a single IB device with multiple ports, owned by one of the grouped > functions, which serves as the E-Switch manager (transfer proxy) of the > nested E-Switch. > Remaining grouped functions are bound to other IB device ports and serve as > representors. > > However, nested E-Switch differs from the regular one in ways which break > assumptions made by mlx5 PMD: > > - Ports of nested E-Switch are netdevs of real VFs/SFs, > not representor netdevs created on the PF. > Each of these functions has its own PCI or auxiliary device, > but the IB device can be opened only through the device of E-Switch > manager. > mlx5 PMD assumes a one-to-one relation between rte_device and mlx5 > common device, > and that all ethdev ports in a single E-Switch are spawned on the same > rte_device. > - These netdevs do not have phys_port_name. > Instead, they are related to devlink ports with VIRTUAL flavour. > > Goal of this patchset is to allow mlx5 PMD to probe ports of nested E-Switch. > Each of the grouped VFs/SFs is probed using its own PCI or auxiliary device, > which mlx5 PMD associates with the IB device of the nested E-Switch. > Representor devargs are not required for such ports and ethdev port name is > the name of the probed device. > Probing of ports not related to VIRTUAL devlink ports is unchanged. > Other mlx5 class drivers (compress, crypto, regex, vdpa) still allow probing > only through the IB device owner. > > - Patches 1-5 - Rework Netlink helpers in common/mlx5, > so that they can be used in common driver code, before any ethdev port is > spawned. > IB port number and IB port to ifindex queries no longer require > the IB port to netdev cache, generic netdev query is separated from switch > info query, > and parent device (bus and device name) of a netdev is queried. > Also adds a check whether given device is an auxiliary device. > - Patch 6 - When mlx5 common device is created, gathers a list of associated > devices. > For each IB device port, the parent device of related netdev is added to the > list. > - Patch 7 - Uses associated devices list to find mlx5 common device for the > probed rte_device > and passes the probed rte_device to mlx5 class drivers. > Only class drivers which explicitly opt in can be probed > through a device other than the IB device owner. > - Patches 8-9 - Add Netlink queries for the devlink port related to a netdev > (through IFLA_DEVLINK_PORT attribute, available since Linux 6.2) > and for flavour of that devlink port. > - Patch 10 - Adds a common check whether given device is managed by any > mlx5 driver. > - Patch 11 - Adds MLX5_ETH_FOREACH_SIBLING macro, which iterates over > mlx5 ethdev ports > sharing the same mlx5 common device. It replaces iteration over ports > sharing the same rte_device, which is no longer valid for nested E-Switch. > - Patch 12 - Allows probing mlx5 networking driver through associated PCI > devices (VFs). > Ports related to VIRTUAL devlink ports are recognized as nested E-Switch > ports: > port of IB device owner is the E-Switch manager, others are representors. > Only the port related to the probed device is spawned. > - Patch 13 - Allows probing mlx5 networking driver through associated > auxiliary devices (SFs). > Auxiliary device with multiport IB device is treated as nested E-Switch > port. > > Dariusz Sosnowski (13): > common/mlx5: rework IB port number query > common/mlx5: allow ifindex query without cache > common/mlx5: separate netdev info from switch info > common/mlx5: query parent device over Netlink > common/mlx5: add check for auxiliary device > common/mlx5: gather associated devices > common/mlx5: always use EAL device for probing > common/mlx5: query devlink port for netdevs > common/mlx5: get devlink port by port index > common/mlx5: add check for managed device > net/mlx5: add sibling device iteration macro > net/mlx5: allow associated devices > net/mlx5: allow multiport auxiliary device probe > > drivers/common/mlx5/linux/meson.build | 6 + > drivers/common/mlx5/linux/mlx5_common_os.c | 164 ++++++ > drivers/common/mlx5/linux/mlx5_common_os.h | 4 + > drivers/common/mlx5/linux/mlx5_nl.c | 578 ++++++++++++++----- > drivers/common/mlx5/linux/mlx5_nl.h | 67 ++- > drivers/common/mlx5/mlx5_common.c | 108 +++- > drivers/common/mlx5/mlx5_common.h | 54 ++ > drivers/common/mlx5/windows/mlx5_common_os.c | 19 + > drivers/common/mlx5/windows/mlx5_common_os.h | 1 + > drivers/compress/mlx5/mlx5_compress.c | 1 + > drivers/crypto/mlx5/mlx5_crypto.c | 1 + > drivers/net/mlx5/linux/mlx5_ethdev_os.c | 24 +- > drivers/net/mlx5/linux/mlx5_os.c | 280 +++++++-- > drivers/net/mlx5/linux/mlx5_os.h | 2 +- > drivers/net/mlx5/mlx5.c | 81 ++- > drivers/net/mlx5/mlx5.h | 8 + > drivers/net/mlx5/mlx5_ethdev.c | 40 +- > drivers/net/mlx5/mlx5_flow.c | 24 +- > drivers/net/mlx5/mlx5_mac.c | 8 +- > drivers/net/mlx5/mlx5_trigger.c | 10 +- > drivers/net/mlx5/windows/mlx5_os.c | 25 +- > drivers/regex/mlx5/mlx5_regex.c | 1 + > drivers/vdpa/mlx5/mlx5_vdpa.c | 1 + > 23 files changed, 1221 insertions(+), 286 deletions(-) > > -- > 2.43.0
Relevant changes did not make it into v7.4 Linux kernel merge window and most probably will be picked up for v7.5. This series will be resent for 27.03. I'll mark this series as deferred.
