From: Manish Honap <[email protected]>
Hello,
This series adds cxl-type2, an emulated CXL Type-2 device with volatile
device memory. It is emulated in such a way that all the Linux checks
are satisfied so that CXL passthrough using vfio-pci (vfio-cxl) can be
developed and tested in a guest without Type-2 hardware.
This series builds on the Type-2 model from Zhi's series (carried in
Cédric's tree) and carries one fix from Shuai's tree. Shuai's tree also
set up the decoders through device properties and stubbed CXL Reset; the
firmware setup and CXL Reset here are new code.
Carried patches keep their authors, with a note for each change.
The device:
- PCI ID 1b36:0017 and the processing accelerator class 0x1200;
- PCIe DVSEC for CXL Devices with CXL.cache, CXL.mem, one HDM range,
cache write-back and invalidate, CXL Reset and Mem Clr; Register
Locator and Flex Bus Port DVSECs;
- CXL.cache advertised but not modelled. The device sends no CXL.cache
requests, and its memory is host RAM, which is coherent already. A
cache write-back and invalidate completes at once. The capability is
set because vfio-cxl takes a device as Type-2 only when it reports
both CXL.cache and CXL.mem;
- component registers with one HDM decoder;
- slot 0, function 0 below a CXL root port of a pxb-cxl, memory in one
fixed memory window selected with fmw-index, cold plug only (no
hotplug or unplug), not migratable.
No Linux driver claims the device on its own. cxl_pci does not bind to
the accelerator class, and sfc, the other driver that calls the CXL
core Type-2 interface, matches only its own devices. A guest binds
cxl-type2 to vfio-pci, which hands it to vfio-cxl. Testing the CXL core
Type-2 interface without vfio-cxl needs a driver for 1b36:0017 that
calls it. There is none yet.
QEMU has no firmware that commits the HDM decoders of a CXL route at
boot, and Linux attaches a Type-2 device only to a committed region
(devm_cxl_probe_mem()). So QEMU does that step itself. cxl-type2
registers its range with its pxb-cxl at realize (patch 13). At every
reset the pxb-cxl commits HDM decoder 0 for it (patch 12), and the
device commits its own decoder with Lock On Commit and enables
CXL.cache, CXL.mem and HDM decoding (patch 14). A pxb-cxl with a
registered range keeps its HDM decoder even with a single root port,
where it would otherwise pass accesses through. Type-3 devices, and
pxb-cxl without a registered range, are not affected.
A CXL Reset keeps the locked decoder committed, as RWL fields are not
affected by CXL Reset once locked (CXL r4.0 Table 8-1). DVSEC Control,
the range bases and HDM Decoder Global Control return to their
defaults, which leaves Mem_Enable and HDM Decoder Enable at 0. While
either is 0 the device drops CXL.mem, so guest accesses to its memory
fail until software sets both bits again. The CXL.io part of the reset
is not emulated. CONFIG_LOCK makes the RWL DVSEC fields read only until
a system reset.
Patches:
- 1-7: Shared CXL code: read-0 ops for the CXL.io range of the
component registers, helpers from Zhi, an HDM decoder count,
DVSEC bit names, a device-neutral view for the direct mapping
code, PCI IDs;
- 8-17: The device; patch 16 also exports the default component
register write handling, so the device can check the enable
bits after a write;
- 18: Docs;
Left for later series:
- The CXL device registers with the mailbox, event logs and MSI-X, and
the CDAT;
- Direct mapping lifecycle fixes for Type-3 and switch topologies, and
Lock On Commit in host bridges and switches;
- Fixes for guest-triggered aborts on 1 and 2 byte mailbox writes and a
repeated DOE Go;
- Software-programmed decoders, for Linux-created Type-2 regions;
- A Type-2 device below a CXL switch, and migration.
Testing:
- Build every commit for x86_64 and aarch64, checkpatch for style.
- vfio-cxl selftest (vfio_cxl_type2_test) in an Arm L1 guest on KVM,
with the device below a CXL root port and an SMMUv3: 19 of 19 pass
- Not tested: x86_64 L1, a build without CONFIG_CXL_TYPE2_DEVICE.
The Linux vfio-cxl series(v5) this was tested with:
https://lore.kernel.org/r/[email protected]
Sent as RFC to get comments on the series structure and to request
1b36:0017 devid for cxl-type2 device allocation.
References:
[PATCH RFC 0/5] hw/cxl: Type 2 Device RFC
https://lore.kernel.org/linux-cxl/[email protected]/
Type-2 emulation trees:
[1] Zhi Wang
https://github.com/zhiwang-nvidia/qemu/commits/zhi/qemu-type2-emulation-for-review
[2] Cédric Le Goater
https://github.com/legoater/qemu/tree/cxl-type2
[3] Shuai Xue
https://gitee.com/axiqia/qemu-kvm/tree/cxl-type2-test
Manish Honap (12):
hw/cxl: Make HDM decoder initialization count-aware
hw/cxl: Define the CXL Device DVSEC capability and control bits
hw/cxl: Find the memory of a CXL device through one helper
hw/pci: Add PCI IDs for the CXL Type-2 device
hw/mem: Require a CXL root port of a pxb-cxl for cxl-type2
hw/pci-bridge: Add firmware HDM decoder state to pxb-cxl
hw/mem: Bind cxl-type2 to a fixed memory window
hw/mem: Commit the cxl-type2 HDM decoder at reset
hw/mem: Add CXL.cache with cache write-back and invalidate to
cxl-type2
hw/mem: Implement CXL Reset for cxl-type2
hw/mem: Clear the cxl-type2 volatile memory on a CXL Reset request
docs/cxl: Document the CXL Type-2 device
Shuai Xue (1):
hw/cxl: Give the component register .io sub-region read-0 ops
Zhi Wang (5):
hw/cxl: Factor out cxl_host_addr_to_dpa()
hw/cxl: Introduce cxl_component_update_dvsec()
hw/cxl: Introduce CXL type-2 device emulation
hw/cxl: Add the CXL type-2 DVSECs and component registers
hw/cxl: Map the CXL type-2 device memory through the fixed windows
MAINTAINERS | 1 +
docs/specs/pci-ids.rst | 2 +
docs/system/devices/cxl.rst | 68 +++
hw/cxl/cxl-component-utils.c | 227 +++++++++-
hw/cxl/cxl-host.c | 130 ++++--
hw/mem/Kconfig | 5 +
hw/mem/cxl_type2.c | 580 ++++++++++++++++++++++++
hw/mem/cxl_type2_stubs.c | 20 +
hw/mem/cxl_type3.c | 68 +--
hw/mem/meson.build | 2 +
hw/pci-bridge/pci_expander_bridge_cxl.c | 27 +-
include/hw/cxl/cxl.h | 1 +
include/hw/cxl/cxl_component.h | 26 ++
include/hw/cxl/cxl_device.h | 27 ++
include/hw/cxl/cxl_pci.h | 31 ++
include/hw/pci/pci.h | 1 +
include/hw/pci/pci_bridge.h | 6 +
include/hw/pci/pci_ids.h | 3 +
18 files changed, 1102 insertions(+), 123 deletions(-)
create mode 100644 hw/mem/cxl_type2.c
create mode 100644 hw/mem/cxl_type2_stubs.c
base-commit: f71afb409a87c4ecd0f9d63a5963eb645dfab780
--
2.53.0
