This patch series introduces a mechanism to specify Guest Physical Addresses
(GPAs) for PCI BARs, allowing explicit placement of guest MMIO BAR addresses
to match host physical addresses for assigned devices.
On some platforms, P2P DMA between devices in the same IOMMU group is required
for the desired data path and performance. The PCI fabric ACS is configured
to permit direct P2P without routing traffic through the host bridge.
To support this multi-device IOMMU group P2P scenario in virtualization, the
VM may need to preserve the host PCI BAR addresses in the guest.
A new fixed-bar-N property on a PCI endpoint device specifies the required
guest address for BAR N (N = 0 through 5):
-device some-device,fixed-bar-0=<addr>[,fixed-bar-1=<addr>]...
Once any fixed-bar-N property is set on a device, every memory BAR on that
device must have a fixed address. Under a PCIe root port, fixed BAR placement
is all-or-nothing within that root port's subtree. If one endpoint below the
root port has a fixed BAR, every other endpoint with a memory BAR below that
same root port must also have all of its memory BARs fixed.
QEMU validates the configuration during device realization. Each address must
be aligned to its BAR size, fall within the appropriate PCIe MMIO aperture,
and not overlap any other fixed BAR. QEMU aborts on any validation failure.
QEMU performs neither PCI enumeration nor BAR allocation; the validated
configuration is exported to firmware through the "etc/fixed-bars" fw_cfg
blob, which carries each PCI function's vendor/device identification,
firmware device path, and, for fixed devices, the configured address, size,
and type of each BAR.
The fixed-bar-N property and its validation are generic and apply to any PCI
device. This series wires up the fw_cfg export on the virt machine only;
other machine types would need their own equivalent wiring to expose the
same mechanism to firmware.
On the firmware side, the existing IncompatiblePciDeviceSupportDxe driver is
extended to consume this blob. Before PCI enumeration, it reserves the fixed
BAR address ranges in the GCD memory space map. During device discovery, its
CheckDevice() implementation checks for a fixed-bars entry first, falling
back to the existing 64-bit-MMIO preference descriptor otherwise, returning
ACPI address descriptors with _MIF|_MAF set for fixed BARs.
PciBusDxe preserves and programs these fixed addresses when programming each
BAR, and sizes and programs each ancestor bridge's resource window to cover
the union of its fixed descendants' address ranges. This allows fixed BARs
behind one or more PCI bridges to be correctly routed.
Fixed BAR placement requires firmware to preserve QEMU's PCI configuration
across reset. This is requested through the PCI boot configuration
preservation mechanism using ACPI _DSM function 5. Fixed BAR configuration is
rejected when ACPI is disabled.
Example 1: VFIO passthrough device with fixed BARs and the PCIe MMIO
aperture adjusted to accommodate the fixed addresses.
-machine virt,...,highmem-mmio-base=4T,highmem-mmio-size=4T
-device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1,io-reserve=0
-device x3130-upstream,id=upstream1,bus=pcie.port1
-device xio3130-downstream,id=downstream1_1,bus=upstream1,chassis=1,slot=2
-device vfio-pci-nohotplug,host=0018:06:00.0,bus=downstream1_1,id=dev0,\
fixed-bar-0=0x48000000000,fixed-bar-2=0x50000000000,fixed-bar-4=0x58000000000
Example 2: VFIO and emulated devices under the same root port with fixed BARs
and the PCIe MMIO aperture adjusted using highmem-mmio-base.
-machine virt,...,highmem-mmio-base=64T,highmem-mmio-size=64T
-device pxb-pcie,id=pcie.1,bus_nr=1,bus=pcie.0
-device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1,io-reserve=0
-device x3130-upstream,id=upstream1,bus=pcie.port1
-device xio3130-downstream,id=downstream1_1,bus=upstream1,chassis=1,slot=1
-device
vfio-pci-nohotplug,host=0018:06:00.0,bus=downstream1_1,rombar=0,id=dev0,iommufd=iommufd0,\
fixed-bar-0=0x6b4000000000,fixed-bar-2=0x6b8000000000,fixed-bar-4=0x6c0000000000
-device xio3130-downstream,id=downstream1_2,bus=upstream1,chassis=1,slot=2
-device
vfio-pci,host=0012:03:00.1,bus=downstream1_2,id=nic1,iommufd=iommufd0,fixed-bar-0=0x6c8000000000
-device xio3130-downstream,id=downstream1_3,bus=upstream1,chassis=1,slot=3
-device
e1000,netdev=net0,bus=downstream1_3,id=nic_emu,fixed-bar-0=0x20000000
Limitations:
* I/O BARs and the expansion ROM BAR are not covered by fixed-bar-N.
* This has only been tested on the AArch64 virt machine.
* SR-IOV VF BARs are not covered.
Testing:
The series was tested on the AArch64 virt machine across a range of PCIe
topologies: emulated devices, VFIO passthrough devices, PCIe switches,
single and multiple fixed-BAR endpoints under a shared root port, and fixed
BARs behind nested bridges.
Changes since RFC v2:
* Replace pci-bars/fixed-bar with per-BAR fixed-bar-N properties; drop the
root-port fixed-bar=on property. Fixed-BAR handling is inferred from
devices in the root-port subtree.
* Drop the separate post-enumeration bridge-window reprogramming pass;
PciBusDxe's own CalculateResourceAperture()/ProgramPpbApperture() now
size and program bridge windows for fixed BARs directly, as part of
normal resource calculation and programming.
* Include the firmware device path in the fw_cfg blob.
* Integrate fixed-BAR handling into the existing
IncompatiblePciDeviceSupportDxe driver instead of a separate
QemuFixedBarsDxe driver.
RFC v2:
https://lore.kernel.org/qemu-devel/[email protected]/
RFC v1:
https://lore.kernel.org/qemu-devel/[email protected]/
A git branch with this series applied is available at:
https://github.com/tdavenvidia/upstream-qemu/tree/Upstream-fixed-bar-Sep-15-V1
The related EDK2 change is available at:
https://github.com/tdavenvidia/edk2/tree/Upstream-fixed-bar-Sep-15-V1
Tushar Dave (5):
hw/pci: introduce PCI fixed BAR properties
pci: add validation for fixed BAR configuration
pci: add fixed BAR fw_cfg blob export
hw/arm/virt: export fixed BAR metadata via fw_cfg
hw/arm/virt: add highmem-mmio-base property
hw/arm/virt.c | 78 +++++++-
hw/pci/meson.build | 2 +
hw/pci/pci-fixed-bar-blob.c | 305 ++++++++++++++++++++++++++++++
hw/pci/pci-fixed-bar-validate.c | 326 ++++++++++++++++++++++++++++++++
hw/pci/pci-fixed-bar-validate.h | 21 ++
hw/pci/pci-fixed-bar.h | 63 ++++++
hw/pci/pci.c | 43 +++++
include/hw/pci/pci_device.h | 7 +
8 files changed, 844 insertions(+), 1 deletion(-)
create mode 100644 hw/pci/pci-fixed-bar-blob.c
create mode 100644 hw/pci/pci-fixed-bar-validate.c
create mode 100644 hw/pci/pci-fixed-bar-validate.h
create mode 100644 hw/pci/pci-fixed-bar.h
--
2.34.1
