This RFC v2 is a follow-up to RFC v1 [1].
On some platforms, peer-to-peer (P2P) DMA between PCIe devices requires
guest physical addresses (GPAs) assigned to device BARs to match their
corresponding host physical addresses (HPAs).
RFC v1 proposed moving PCI enumeration and BAR assignment into QEMU
before firmware execution, with EDK2 operating in discovery-only mode
via PcdPciDisableBusEnumeration.
The feedback was that PCI enumeration and resource assignment should
remain in firmware rather than being performed by the VMM. The suggested
EDK2 mechanism was EFI_INCOMPATIBLE_PCI_DEVICE_SUPPORT_PROTOCOL, which
provides a per-device interception point for PCI resource assignment.
Following the feedback, RFC v2 keeps PCI enumeration and resource
assignment in firmware. QEMU only validates the user-provided fixed
BAR configuration and provides the required metadata to firmware
through the "etc/fixed-bars" fw_cfg file.
Two new PCI properties are introduced:
* fixed-bar=on on a PCIe root port marks its subordinate hierarchy for
fixed BAR placement. Every device with memory BARs in that hierarchy
must provide a complete pci-bars= configuration.
* pci-bars=barN@<addr>[,barM@<addr>]... on a PCI endpoint specifies the
required address for each memory BAR. All memory BARs on the device
must have an explicitly assigned address.
QEMU validates the user-specified configuration before passing it to
firmware. Validation checks BAR alignment, verifies that addresses are
within the configured PCIe MMIO apertures, and ensures that fixed BAR
ranges do not overlap. QEMU performs neither PCI enumeration nor BAR
allocation.
On the firmware side, a new DXE driver, QemuFixedBarsDxe, installs
EFI_INCOMPATIBLE_PCI_DEVICE_SUPPORT_PROTOCOL before PciBusDxe starts.
When PciBusDxe calls CheckDevice() for a discovered PCI function, the
driver returns ACPI address descriptors with _MIF|_MAF set for fixed
BARs. Two small changes to PciBusDxe preserve these fixed addresses and
program them into the BAR registers during BAR programming.
After PciEnumerationComplete, QemuFixedBarsDxe walks each fixed
root-port hierarchy and programs the bridge memory windows to cover
the fixed BAR ranges assigned to endpoint devices.
Example 1: VFIO passthrough devices under a fixed BAR root port.
-device
pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1,io-reserve=0,fixed-bar=on
-device x3130-upstream,id=upstream1,bus=pcie.port1
-device xio3130-downstream,id=downstream1_1,bus=upstream1,chassis=1,slot=2
-device vfio-pci-nohotplug,host=0018:06:00.0,bus=downstream1_1,id=dev0
-set
device.dev0.pci-bars=bar0@0x48000000000,bar2@0x50000000000,bar4@0x58000000000
Example 2: Extending the PCIe MMIO aperture with highmem-mmio-base
(and highmem-mmio-size), using a mix of emulated and VFIO devices.
-machine
virt,...,highmem-mmio-base=0x400000000000,highmem-mmio-size=0x400000000000
-device e1000,netdev=net0,bus=pcie.0,pci-bars=bar0@0x10000000
-device
pcie-root-port,id=pcie.port9,bus=pcie.9,chassis=4,io-reserve=0,fixed-bar=on
-device x3130-upstream,id=upstream9,bus=pcie.port9
-device xio3130-downstream,id=downstream9_1,bus=upstream9,chassis=4,slot=1
-device
vfio-pci,host=0012:03:00.1,bus=downstream9_1,id=nic1,pci-bars=bar0@0x7000c0000000
-device xio3130-downstream,id=downstream9_2,bus=upstream9,chassis=4,slot=2
-device vfio-pci-nohotplug,host=0019:06:00.0,bus=downstream9_2,id=dev1
-set
device.dev1.pci-bars=bar0@0x6d4000000000,bar2@0x6d8000000000,bar4@0x6e0000000000
The pci-bars= property is generic and may be used with any PCI endpoint
device, including emulated devices and devices with 32-bit memory BARs.
The optional highmem-mmio-base machine property allows the operator to
reposition the high PCIe MMIO window so that the requested BAR
addresses can be placed within the configured address space.
Limitations:
- I/O BARs and the expansion ROM BAR are not covered by pci-bars=.
- This has only been tested on the AArch64 virt machine.
- SR-IOV VF BARs are not covered.
Testing:
The series was tested on the AArch64 virt machine using multiple PCI
topologies, including emulated devices, VFIO passthrough devices, PCIe
switches, and fixed BAR root ports.
A git branch with this series applied is available at:
https://github.com/tdavenvidia/upstream-qemu/tree/RFC-v2-fixed-bar-upstream
[1] RFC v1 upstream thread:
https://lore.kernel.org/qemu-devel/[email protected]/
Tushar Dave (5):
hw/pci: add fixed-bar and pci-bars properties
pci: add validation for fixed BAR configuration
pci: add fixed BAR fw_cfg blob export
hw/arm/virt: export fixed BAR metadata via fw_cfg
hw/arm/virt: add highmem-mmio-base property
hw/arm/virt.c | 78 ++++++++-
hw/pci-bridge/pcie_root_port.c | 1 +
hw/pci/meson.build | 2 +
hw/pci/pci-fixed-bar-blob.c | 291 ++++++++++++++++++++++++++++++++
hw/pci/pci-fixed-bar-validate.c | 279 ++++++++++++++++++++++++++++++
hw/pci/pci-fixed-bar-validate.h | 21 +++
hw/pci/pci-fixed-bar.h | 60 +++++++
hw/pci/pci.c | 129 ++++++++++++++
include/hw/pci/pci_device.h | 10 ++
include/hw/pci/pcie_port.h | 1 +
10 files changed, 871 insertions(+), 1 deletion(-)
create mode 100644 hw/pci/pci-fixed-bar-blob.c
create mode 100644 hw/pci/pci-fixed-bar-validate.c
create mode 100644 hw/pci/pci-fixed-bar-validate.h
create mode 100644 hw/pci/pci-fixed-bar.h
--
2.34.1
