Overview
--------
This patch series enables AMD IOMMU to be attached to non-system root buses
(e. g. , pcie-pxb, pci-root-bus), allowing device isolation between VFIO
passthrough devices and emulated devices. This is a prerequisite for
hardware-accelerated virtual IOMMU (vIOMMU) support.
Problem Statement
-----------------
The current AMD IOMMU implementation has the following limitations:
1. Only supports attachment to the default pcie.0 system bus
2. Builds IVRS (I/O Virtualization Reporting Structure) by iterating over
all system buses, preventing device isolation
3. Cannot separate VFIO passthrough devices from emulated devices on
different buses
This creates a fundamental issue for hardware-accelerated vIOMMU, which
requires VFIO devices to be isolated from emulated devices, as the vIOMMU
can only operate on passthrough devices.
Solution Approach
-----------------
This series introduces the following changes:
1. Allow AMD IOMMU to be attached to arbitrary root buses using the "bus="
parameter, while Maintaining the backward compatibility by selecting the
pcie.0 by default if this parameter is missing.
2. Implement root bus discovery by traversing the parent hierarchy to find
the first root bus
3. Register IOMMU operations on the discovered root bus
4. Modify IVRS generation to build IVHD (I/O Virtualization Hardware
Definition) entries based on the specified root bus, traversing child
buses and devices in a top-down manner
5. Introduce x86_iommu_device_get() and the get_x86_iommu() callback so the
KVM layer can determine which IOMMU serves a given device.
kvm_arch_fixup_msi_route() and kvm_arch_add_msi_route_post() call
x86_iommu_get_default() for every MSI route. That returns the single
machine IOMMU and is correct only when that IOMMU is attached to pcie.0
and serves every device.
Once the IOMMU is attached below another root bus, it serves only the
devices under that bus. The KVM layer must program interrupt-remapping
routes through that IOMMU and leave devices outside its scope untranslated.
The PCI core can already find the bus that owns a device, but PCIIOMMUOps
cannot return the corresponding X86IOMMUState. get_x86_iommu() provides
that link. x86_iommu_device_get() uses it, and both AMD and Intel IOMMU
implement the callback because this KVM path is shared.
Command line
------------
qemu-system-x86_64 \
-enable-kvm -m 10G -smp 8 \
-netdev user,id=USER0,hostfwd=tcp::3333-:22 \
-cpu
EPYC-Genoa,x2apic=on,kvm-msi-ext-dest-id=on,+kvm-pv-unhalt,kvm-pv-tlb-flush,kvm-pv-ipi,kvm-pv-sched-yield
\
-machine q35,kernel_irqchip=split \
-global kvm-pit.lost_tick_policy=discard \
-nographic \
-blockdev node-name=drive0,driver=qcow2,file.driver=file,file.filename=<IMG> \
-device virtio-blk-pci,num-queues=8,drive=drive0 \
-device pxb-pcie,id=pcie.1,bus=pcie.0,bus_nr=0x50 \
-device pcie-root-port,id=rp1,bus=pcie.1,chassis=3,slot=1 \
-device AMDVI-PCI,id=iommupci1,bus=rp1 \
-device amd-iommu,intremap=on,dma-remap=on,xtsup=on,pci-id=iommupci1 \
-device pcie-root-port,id=rp2,bus=pcie.2,chassis=4,slot=1 \
-device
virtio-net-pci,id=vnet1,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0,bus=rp2
\
Example
-------
Example1, for below PCI configuration, IOMMU will serve all the devices
under pxb-pci (i.e. Device1 and Device2) since its a first root bus,
but will not serve Device3 and Device4, effectively isolating the
devices under pxb-pci bus.
pcie.0 -+- pxb-pci -+- pcie-root-port1 --- AMD IOMMU
| +- pcie-root-port2 --- Device1
| \- pcie-root-port3 --- Device2
+- Device3
\- Device4
Example2, for below PCI configuration, IOMMU will serve all the devices
Device[1-4], since first root bus in the parent hierarchy is pcie.0
pcie.0 -+- pcie-root-port1 --- AMD IOMMU
+- pcie-root-port2 --- Device1
+- pcie-root-port3 --- Device2
+- Device3
\- Device4
Example3, Below three PCI topologies will cause failure as AMD IOMMU can
only attach to **PCIE** hierarchy.
[A]
pcie.0 -+- pcie-pci-bridge --- AMD IOMMU
\- Device1
[B]
pcie.0 -+- pci-bridge --- AMD IOMMU
\- Device1
[C]
pcie.0 -+- pci-bridge --- pcie-root-port --- AMD IOMMU
\- Device1
Sairaj Kodilkar (8):
amd_iommu: Assign ioapic_as only when interrupt remapping is
supported.
x86-iommu: Introduce x86_iommu_device_get() for x86-iommu retrieval
amd_iommu: Add support for get_x86_iommu callback
intel_iommu: Add support for get_x86_iommu callback
i386/kvm: Use x86_iommu_device_get() for per-device IOMMU lookup
amd_iommu: Introduce root bus
acpi-build: Create PCI topology using amd iommu root bus
amd_iommu: Allow IOMMU to attachment on non-system (pcie.0) bus
hw/i386/acpi-build.c | 142 +++++++++++++++++++-----------------
hw/i386/amd_iommu.c | 75 +++++++++++++++++--
hw/i386/amd_iommu.h | 3 +
hw/i386/intel_iommu.c | 8 ++
hw/i386/x86-iommu-stub.c | 5 ++
hw/i386/x86-iommu.c | 16 ++++
hw/intc/ioapic.c | 3 +-
include/hw/i386/x86-iommu.h | 6 ++
include/hw/pci/pci.h | 12 +++
target/i386/kvm/kvm.c | 15 +++-
10 files changed, 208 insertions(+), 77 deletions(-)
base-commit: 562bae590f194fb590beb5c65da44fc35ab9f64a
--
2.34.1
