When a VM has been assigned a number of PCIe devices behind a common PCIe switch and the VM intends to use peer-to-peer (P2P) transfers between the devices, it is useful to be able to more report the apparent PCIe bandwidth so it reflects the real handwidth of the underlying hardware. This allows software in the VM (such as NCCL[1]) to make better decisions about whether to use P2P or not.
The existing emulated PCIe switch is a vintage TI XIO3130 and only support 2.5 GT/s and x1, so we introduces new generic upstream and downstream port devices (pcie-upstream-port and pcie-downstream-port), inspired by the existing generic pcie-root-port device. The existing experimental "x-speed" and "x-width" options are made non-experimental, allowing the pcie-root-port and pcie-downstream-port capabilities to be configured to match the hardware. The pcie-root-port device is updated to default to a Gen6 capable device. Finally, where a root or upstream port's bandwidth has been limited, we also ensure the other end of the link reports the same link status. Note that the device IDs for the new port devices are currently placeholders and will need to be allocated. Some open questions: - The ergonomics of the speed/width options doesn't seem great as they're enum properties. In particular, 2.5 GT/s is requested with the string "2_5". Should we introduce the ability to have floating-point properties? Or should the units for "speed" be MT/s (renaming the option to "speed_mts") instead and both properties should be integers? - PCIe 6.0 removed support for the x32 link width. Should the generic root and downstream ports default to 64 GT/s and x16 instead? David [1] https://developer.nvidia.com/nccl David Vrabel (4): hw/pci-bridge: Add generic PCIe switch upstream and downstream ports hw/pci-bridge: Make speed and width for PCIe ports non-experimental hw/pci-bridge: Default the generic PCIe root port as Gen6 capable hw/pci: Ensure PCIe link speed/width is consistent between both ends docs/pcie.txt | 6 +- hw/pci-bridge/gen_pcie_downstream_port.c | 206 ++++++++++++++++++++ hw/pci-bridge/gen_pcie_root_port.c | 6 +- hw/pci-bridge/gen_pcie_upstream_port.c | 182 +++++++++++++++++ hw/pci-bridge/meson.build | 3 +- hw/pci/pcie.c | 26 ++- tests/functional/x86_64/meson.build | 1 + tests/functional/x86_64/test_pcie_switch.py | 98 ++++++++++ 8 files changed, 517 insertions(+), 11 deletions(-) create mode 100644 hw/pci-bridge/gen_pcie_downstream_port.c create mode 100644 hw/pci-bridge/gen_pcie_upstream_port.c create mode 100755 tests/functional/x86_64/test_pcie_switch.py -- 2.43.0
