Hello all,
I am looking for guidance on using the mlx5 PMD in switchdev mode with the DPDK
primary/secondary process model.
Environment
*
DPDK 26.03
*
mlx5 NIC
*
NIC configured in switchdev mode
*
One DPDK primary process and one DPDK secondary process
*
Both applications use the same PCI allow-list containing the PF and VFs
*
The PF and VFs remain bound to the mlx5 kernel driver
*
Both processes use the same DPDK multi-process configuration and shared-memory
prefix
The primary successfully probes the PF and available representor ports with the
'representor' devarg. However, the secondary does not successfully probe the PF
or representor devices, even though the PF and VFs are included in the
allow-list. Consequently, those devices are absent from the secondary’s ethdev
list and do not appear when iterating with RTE_FOREACH_DEV after initialization.
The server we are running has other applications that require the NIC to be
switchdev mode. The PF is the E-Switch master, while the secondary processes
traffic through a VF ethdev. The main question is how the PF, VF, and
representor ethdevs are expected to be managed when some devices are visible
only to the primary.
Our current understanding is that:
*
The primary owns the PF and the E-Switch master.
*
The PF may not be independently attachable or visible to the secondary.
*
PF-specific operations, such as creating or destroying E-Switch transfer rules,
should be performed by the primary.
*
The secondary may process traffic through a VF ethdev, while control-plane
operations involving the PF are handled by the primary.
Could someone confirm whether this is the expected architecture for mlx5
switchdev with DPDK primary/secondary processes?
Primary/secondary hotplug behavior
I reviewed the DPDK 26.03 hotplug implementation and confirmed that
primary/secondary synchronization is performed for both hotplug add and remove
operations.
For hotplug add, the primary probes the device locally and then sends an attach
request to all registered secondary processes. If communication fails or a
secondary cannot attach the device, the primary rolls back the local attach and
notifies the secondaries of the rollback.
For hotplug remove, the primary first sends a synchronous remove request
through the EAL multiprocess channel to all registered secondaries. Only after
that coordination does it proceed with the local bus-specific device removal.
This generic synchronization is part of the common EAL hotplug path and is not
dependent on the bus or PMD.
The mlx5 PMD also performs its own multi-process coordination during device
stop and close operations. Therefore, removing an mlx5 device may involve both
mlx5-specific notifications and the generic EAL hotplug notification.
This creates a problem in our configuration:
*
The primary can probe the PF and create representors.
*
The secondary cannot probe or attach to the PF or representors.
*
A primary hotplug add still attempts to attach the device in every registered
secondary.
*
A primary hotplug remove still attempts to remove the device from every
registered secondary.
*
The secondary cannot successfully mirror these operations because the device
does not exist in its ethdev view.
*
As a result, the primary hotplug operation can fail or be rolled back because
the secondary cannot attach to or remove the device.
This appears to be a mismatch between the generic primary/secondary hotplug
model and the mlx5 switchdev model, where the PF and representors may be
primary-only resources. Managing ethdevs in the primary application always has
implications in secondary, so how is this supposed to be handled in switchdev
mode when secondary cannot even probe the PF ports?
Telemetry issue
There is also a related issue involving the DPDK telemetry thread.
The telemetry thread sends an ethdev/info request for port 0. Port 0 (PF
E-switch master) has a valid MAC address in our application. However, in the
failure path we observed, format_ether_addr() is called with a NULL pointer
while formatting the telemetry response. The helper dereferences or formats the
address pointer without first checking whether it is NULL, resulting in a
segmentation fault.
Would it be appropriate for format_ether_addr() to handle a NULL pointer
defensively—for example, by formatting the address as all zeroes—or should the
caller guarantee that the pointer is always valid and return an error if it is
not?
A possible defensive behavior would be conceptually similar to:
if (addr == NULL) {
/* Format as 00:00:00:00:00:00,
* or report an invalid address.
*/
}
I would appreciate feedback on:
* Whether a NULL MAC-address pointer is expected to be possible in the
ethdev/info telemetry path
* Whether a NULL check and all-zero MAC formatting would be acceptable
upstream behavior
Regards,
Dylan
