This patch series introduces a mechanism to specify Guest Physical Addresses
(GPAs) for PCI BARs, allowing explicit placement of guest MMIO BAR addresses
to match host physical addresses for assigned devices.
On some platforms, P2P DMA between devices in the same IOMMU group is required
for the desired data path and performance. The PCI fabric ACS is configured
to permit direct P2P without routing traffic through the host bridge.
To support this multi-device IOMMU group P2P scenario in virtualization, the
VM may need to preserve the host PCI BAR addresses in the guest.
A new fixed-bar-N property on a PCI endpoint device specifies the required
guest address for BAR N (N = 0 through 5):
-device some-device,fixed-bar-0=<addr>[,fixed-bar-1=<addr>]...
Once any fixed-bar-N property is set on a device, every memory BAR on that
device must have a fixed address. Under a PCIe root port, fixed BAR placement
is all-or-nothing within that root port's subtree. If one endpoint below the
root port has a fixed BAR, every other endpoint with a memory BAR below that
same root port must also have all of its memory BARs fixed.
QEMU validates the configuration when the machine is done initializing. Each
address must be aligned to its BAR size, fall within the appropriate PCIe MMIO
aperture, and not overlap any other fixed BAR. QEMU aborts on any validation
failure. QEMU performs neither PCI enumeration nor BAR allocation; the
validated configuration is exported to firmware through the "etc/fixed-bars"
fw_cfg blob, which carries each PCI function's vendor/device identification,
firmware device path, and, for fixed devices, the configured address, size,
and type of each BAR. The blob format is documented in
docs/specs/pci-fixed-bars.rst.
The fixed-bar-N property and its validation are generic and apply to any PCI
device. A machine type opts in by setting the new pci_fixed_bar_supported
MachineClass flag. Machine types that leave it unset reject fixed-bar-N. This
series enables it on the virt machine only, by setting the flag and wiring up
the fw_cfg export. Other machine types would need equivalent wiring to expose
the same mechanism to firmware.
On the firmware side, the existing IncompatiblePciDeviceSupportDxe driver is
extended to consume this blob. During device discovery, its CheckDevice()
implementation returns ACPI address descriptors with _MIF|_MAF set for the
fixed BARs found in the blob. PciBusDxe's resource allocator already supports
fixed BARs, including sizing and pinning the bridge windows above them. It now
treats a BAR as fixed when the descriptor carries _MIF|_MAF, and programs the
fixed address into the BAR register.
Fixed BAR placement requires the guest to preserve the PCI resource
configuration established by firmware. This is requested through the PCI boot
configuration preservation mechanism using ACPI _DSM function 5. Fixed BAR
configuration is rejected when firmware is not loaded or ACPI is disabled.
Example 1: VFIO passthrough device with fixed BARs and the PCIe MMIO
aperture adjusted to accommodate the fixed addresses.
-machine virt,...,highmem-mmio-base=4T,highmem-mmio-size=4T
-device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1,io-reserve=0
-device x3130-upstream,id=upstream1,bus=pcie.port1
-device xio3130-downstream,id=downstream1_1,bus=upstream1,chassis=1,slot=2
-device vfio-pci-nohotplug,host=0018:06:00.0,bus=downstream1_1,id=dev0,\
fixed-bar-0=0x48000000000,fixed-bar-2=0x50000000000,fixed-bar-4=0x58000000000
Example 2: VFIO and emulated devices under the same root port with fixed BARs
and the PCIe MMIO aperture adjusted using highmem-mmio-base.
-machine virt,...,highmem-mmio-base=64T,highmem-mmio-size=64T
-device pxb-pcie,id=pcie.1,bus_nr=1,bus=pcie.0
-device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1,io-reserve=0
-device x3130-upstream,id=upstream1,bus=pcie.port1
-device xio3130-downstream,id=downstream1_1,bus=upstream1,chassis=1,slot=1
-device
vfio-pci-nohotplug,host=0018:06:00.0,bus=downstream1_1,rombar=0,id=dev0,iommufd=iommufd0,\
fixed-bar-0=0x6b4000000000,fixed-bar-2=0x6b8000000000,fixed-bar-4=0x6c0000000000
-device xio3130-downstream,id=downstream1_2,bus=upstream1,chassis=1,slot=2
-device
vfio-pci,host=0012:03:00.1,bus=downstream1_2,id=nic1,iommufd=iommufd0,fixed-bar-0=0x6c8000000000
-device xio3130-downstream,id=downstream1_3,bus=upstream1,chassis=1,slot=3
-device
e1000,netdev=net0,bus=downstream1_3,id=nic_emu,fixed-bar-0=0x20000000
Limitations:
* I/O BARs and the expansion ROM BAR are not covered by fixed-bar-N.
* SR-IOV VF BARs are not covered.
Testing:
The series was tested on the AArch64 virt machine with a range of PCIe
configurations, including emulated and VFIO passthrough devices, PCIe
switches, single and multiple fixed-BAR endpoints behind a shared root port,
devices attached directly to the host bridge, and endpoints behind nested
bridges. All rejection paths were also exercised, with the corresponding
error messages verified. These included unsupported machine types,
hot-plugged devices, non-existent BARs, I/O BARs, unaligned, overlapping, or
out-of-aperture BARs, incomplete BAR configurations, highmem-mmio=off,
missing firmware, and highmem-mmio-base values that cause address-space
overflow.
Changes since v1:
* Add a pci_fixed_bar_supported MachineClass flag; fixed-bar-N is rejected
on machine types that do not set it. virt sets it.
* Keep the fixed BAR claims in FixedBarsInfo instead of a static array, and
report validation failures through Error ** to the machine code instead of
calling exit(1).
* Allow a 64-bit non-prefetchable BAR in either aperture when the device is
attached to the host bridge; behind a PCI bridge it must lie below 4 GiB.
A 64-bit prefetchable BAR must still lie in the 64-bit aperture.
* Treat the high PCIe MMIO aperture as absent with highmem-mmio=off, and
reject fixed BAR configuration unless firmware is loaded and ACPI is
enabled.
* Check highmem-mmio-base + highmem-mmio-size against the PA space without
overflow, and document highmem-mmio-base in docs/system/arm/virt.rst.
* Document the etc/fixed-bars format in docs/specs/pci-fixed-bars.rst, and
note that the firmware device path is carried for future matching.
* Move pci-fixed-bar.h to include/hw/pci/.
v1:
https://lore.kernel.org/qemu-arm/[email protected]/
A git branch with this series applied is available at:
https://github.com/tdavenvidia/upstream-qemu/tree/Upstream-fixed-bar-Oct-29-V2
The related EDK2 change is available at:
https://github.com/tianocore/edk2/pull/13183
Tushar Dave (5):
hw/pci: introduce PCI fixed BAR properties
hw/pci: add validation for fixed BAR configuration
hw/pci: add fixed BAR fw_cfg blob export
hw/arm/virt: export fixed BAR metadata via fw_cfg
hw/arm/virt: add highmem-mmio-base property
docs/specs/index.rst | 1 +
docs/specs/pci-fixed-bars.rst | 75 +++++++
docs/system/arm/virt.rst | 4 +
hw/arm/virt.c | 82 ++++++-
hw/pci/meson.build | 2 +
hw/pci/pci-fixed-bar-blob.c | 308 +++++++++++++++++++++++++++
hw/pci/pci-fixed-bar-validate.c | 365 ++++++++++++++++++++++++++++++++
hw/pci/pci-fixed-bar-validate.h | 22 ++
hw/pci/pci.c | 71 +++++++
include/hw/core/boards.h | 3 +
include/hw/pci/pci-fixed-bar.h | 63 ++++++
include/hw/pci/pci_device.h | 7 +
12 files changed, 1002 insertions(+), 1 deletion(-)
create mode 100644 docs/specs/pci-fixed-bars.rst
create mode 100644 hw/pci/pci-fixed-bar-blob.c
create mode 100644 hw/pci/pci-fixed-bar-validate.c
create mode 100644 hw/pci/pci-fixed-bar-validate.h
create mode 100644 include/hw/pci/pci-fixed-bar.h
--
2.34.1
