From: Manish Honap <[email protected]>
This series adds QEMU support for passing a CXL Type-2 device (an
accelerator with host-managed device memory, e.g. a GPU) to a guest via
vfio-pci. The guest drives its own virtual endpoint HDM decoder and QEMU
maps the device memory at the guest physical address the guest commits,
while the host owns the host physical placement.
v1 [1] was reviewed by Junjie Cao and Cedric Le Goater. v2 addresses
Junjie's comments; see "Changes since v1" below.
Base: qemu master, commit 28e7aad522.
Kernel dependency
-----------------
Pairs with the kernel "vfio/cxl: CXL Type-2 device passthrough" series [2].
The kernel exposes the device memory as an HPA-backed VFIO region, traps
the HDM decoder block and runs its lock-on-commit FSM, handles the CXL
DVSEC (including a guest-triggered reset), and reports two things through
VFIO:
- A device flag (VFIO_DEVICE_FLAGS_CXL), and
- The component-register geometry (VFIO_REGION_INFO_CAP_CXL_COMP_REGS).
That series in turn builds on the cxl_reset series [3].
Status of the kernel side: v5 is on-list [2]; rebased onto cxl_reset [3]
and addressed the v4 review comments. This QEMU series depends on the VFIO
uAPI above (patch 1 imports it) and on the kernel servicing fd read/write
on the HDM memory region, which QEMU uses as the fallback when the region
is not mmap'd. Both are part of the same vfio-cxl series and are stable
across those revisions.
Sample supported topology
-------------------------
Guest disk, network, and system-RAM lines are omitted:
-machine virt,accel=kvm,gic-version=3,hmat=on,cxl=on,ras=on, \
highmem-mmio-size=4T
-object iommufd,id=iommufd0
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1
-device cxl-rp,port=1,bus=cxl.1,id=rport0.1,chassis=4, \
pref64-reserve=2G,mem-reserve=1G
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=256G
-device arm-smmuv3,primary-bus=cxl.1,id=smmuv3.0,accel=on,ats=on, \
ril=on,ssidsize=8,oas=48
-device vfio-pci-nohotplug,host=<BDF>,bus=rport0.1,id=dev0, \
iommufd=iommufd0
-object acpi-generic-initiator,id=gi0,pci-dev=dev0,node=2
... (one acpi-generic-initiator per guest NUMA node the HDM memory backs)
Address model
-------------
The kernel fixes device memory to a host physical range before the guest
sees the device, and hardware presents a firmware-committed, locked
endpoint HDM decoder whose registers hold that host physical base.
QEMU never exposes the host physical base. It virtualizes the decoder base
registers in the trapped component-register read and returns the base of
the device's CFMWS window, a guest physical address, so the guest only
ever sees a GPA. QEMU maps the RAM-device region (backed by the fixed HPA)
at that CFMWS base. The kernel never learns the GPA and the guest never
learns the HPA.
Because the decoder is already committed at boot, no guest commit write
triggers the mapping. QEMU maps once the guest enables memory decoding
(the Command register Memory-Space bit) and re-checks on any decoder
control write, so the region enters the guest address space and the IOAS
while the device is live.
When the guest clears Memory-Space, QEMU withdraws the mapping, matching
the kernel's revoke of the backing PTEs on the same write, so a guest
access during the disabled interval cannot fault a zapped mapping and stop
the VM; the enable path re-installs it.
The pxb-cxl _DSM is here because an OS may treat PCI configuration as
reassignable, and a BAR move would break the CXL.mem mapping. It applies
only to the host bridge that carries the passed-through CXL device.
Reset
-----
There is no QEMU reset patch. A guest CXL reset is a DVSEC write that
lands in vfio config space and is handled by the host kernel, which runs
the CXL reset sequence and stamps the outcome into DVSEC STATUS2. The
kernel re-commits this firmware-fixed decoder across the reset, and the
guest reaches its memory through the mapping QEMU already installed.
Changes since v1
----------------
All from Junjie Cao's v1 review:
- The _DSM is emitted only when the machine asks OSPM to preserve the
firmware PCI configuration (preserve_config). x86 q35 passes false, so
its DSDT is unchanged and the bios-tables golden files stay valid.
- The endpoint count counts every present function, not one per slot, so
two functions cold-plugged at one slot no longer bind and map the same
window at the same base.
- The unrealize path (vfio_exitfn) drops the CXL mapping, so an ACPI
eject no longer leaves the vfio fd held through the region's owner
reference.
- The decoder-count read uses the shared cxl_decoder_count_dec() helper
with a floor of 1, instead of a local switch that stopped at 4 decoders.
- On commit, a guest decoder base that differs from the CFMWS base is
refused and logged rather than mapped at the window base regardless.
- The hotplug validation branch moved into the patch that registers the
machine-init-done notifier, so no intermediate commit can exit(1) on a
bad device_add.
AI assistance
-------------
Parts of this series were drafted with the help of an AI coding assistant:
the code was AI-prototyped, then reviewed, edited, and rewritten by me,
and the documentation was drafted the same way. The assistant was also
used to research the existing vfio and CXL APIs. I built the series and
functionally tested it against real CXL Type-2 hardware. I take
responsibility for the whole of every patch and certify it under the
DCO via Signed-off-by. The patches carry AI-used-for: trailers,
"code (prototype)" or "docs", per QEMU's code-provenance policy.
This series adds new passthrough code, which is outside the mechanical,
small-bug-fix, docs, and tests categories the policy covers without prior
maintainer agreement. I am flagging that here rather than assuming it is
in scope.
Validation
----------
- Every patch passes scripts/checkpatch.pl --codespell --strict (patch 1
carries the expected imported-from-Linux header warning).
- The series applies cleanly on the stated base.
- Built and functionally tested against a CXL Type-2 device: guest boot,
decoder commit and mapping, and guest-triggered CXL reset.
Pending items
-------------
- Multi-decoder, interleaved, and switched topologies are future work.
- Trapped CXL RAS registers are planned as a new VFIO region subtype that
the existing region-by-subtype detection already handles.
- The bios-tables test refresh is not needed now that the _DSM is gated on
preserve_config.
References
----------
[1] [PATCH 0/10] QEMU: CXL Type-2 device passthrough via vfio-pci
https://lore.kernel.org/qemu-devel/[email protected]/
[2] [PATCH v5 00/27] vfio/pci: Add CXL Type-2 device passthrough support
https://lore.kernel.org/linux-cxl/[email protected]/
[3] [PATCH v12 00/12] PCI/CXL: Add CXL reset support for Type 2 devices:
https://lore.kernel.org/linux-cxl/[email protected]/
Manish Honap (10):
linux-headers: Update vfio.h for CXL Type-2 passthrough
hw/vfio/region: Add vfio_region_setup_with_ops()
hw/vfio/pci: Detect a CXL Type-2 device and read its geometry
hw/vfio/pci: Enforce the passthrough topology for a CXL device
hw/vfio/pci: Back the CXL memory with a RAM-device region
hw/vfio/pci: Bind a CXL device to its fixed memory window
hw/vfio/pci: Map the CXL memory on the guest decoder commit
docs/cxl: Document CXL Type-2 device passthrough
hw/arm/smmu-common: Allow pxb-cxl as an SMMUv3 primary bus
hw/pci-host: Emit a _DSM on pxb-cxl to preserve firmware PCI config
docs/system/devices/cxl.rst | 45 ++
hw/acpi/Kconfig | 1 +
hw/acpi/cxl-stub.c | 2 +-
hw/acpi/cxl.c | 12 +-
hw/acpi/pci.c | 40 ++
hw/arm/smmu-common.c | 19 +-
hw/cxl/cxl-host-stubs.c | 5 +
hw/i386/acpi-build.c | 2 +-
hw/pci-bridge/pci_expander_bridge_stubs.c | 6 +
hw/pci-host/gpex-acpi.c | 42 +-
hw/vfio/pci.c | 745 ++++++++++++++++++++++
hw/vfio/pci.h | 27 +
hw/vfio/region.c | 27 +-
hw/vfio/vfio-region.h | 3 +
include/hw/acpi/cxl.h | 2 +-
include/hw/acpi/pci.h | 1 +
linux-headers/linux/vfio.h | 24 +
17 files changed, 947 insertions(+), 56 deletions(-)
--
2.25.1
