From: Manivannan Sadhasivam <[email protected]> Document the PCIe Endpoint Controller (pcie-ep-ctrl) and the Endpoint Function (pcie-ep-generic), including how they share a single guest address space.
The documentation walks through the QEMU invocation and the configfs bring-up of an Endpoint Function. It also covers the kernel configuration needed for a single guest to act as both the Endpoint Controller and the PCI host that enumerates the Function. The current limitations are listed at the end. Signed-off-by: Manivannan Sadhasivam <[email protected]> --- MAINTAINERS | 5 ++ docs/system/device-emulation.rst | 1 + docs/system/devices/pcie-ep.rst | 140 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 146 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 2b5b581e17..e241f206bc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2170,6 +2170,11 @@ F: docs/pci* F: docs/specs/*pci* F: docs/system/sriov.rst +PCIe Endpoint Emulation +M: Manivannan Sadhasivam <[email protected]> +S: Maintained +F: docs/system/devices/pcie-ep.rst + ARM PCI Hotplug M: Gustavo Romero <[email protected]> L: [email protected] diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst index 40054bb7df..841ce35511 100644 --- a/docs/system/device-emulation.rst +++ b/docs/system/device-emulation.rst @@ -95,6 +95,7 @@ Emulated Devices devices/keyboard.rst devices/net.rst devices/nvme.rst + devices/pcie-ep.rst devices/scsi/index.rst devices/usb-u2f.rst devices/usb.rst diff --git a/docs/system/devices/pcie-ep.rst b/docs/system/devices/pcie-ep.rst new file mode 100644 index 0000000000..0e661234f7 --- /dev/null +++ b/docs/system/devices/pcie-ep.rst @@ -0,0 +1,140 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +PCIe Endpoint Emulation +----------------------- + +QEMU can emulate a PCIe Endpoint Controller (EPC) that works with the +Linux PCI Endpoint framework (``drivers/pci/endpoint/``). This allows +running a Linux Endpoint Function driver, such as ``pci-epf-test``, and +exercising it against the matching host-side PCI driver. + +Everything runs inside a single QEMU instance. One guest kernel acts as +both sides. It drives the Endpoint Controller through the +``pci-ep-generic`` platform driver and, when the Link comes up, it +enumerates the resulting Function as a PCI host. Because there is only +one guest, both sides share a single physical address space, so the +Endpoint BARs and the outbound DMA window are plain aliases of guest RAM +with no copying and no inter-process communication. + +Two devices cooperate: + +``pcie-ep-ctrl`` + A SysBus device on the ``virt`` machine's platform bus that exposes an + MMIO based Endpoint Controller (EPC) controlled by the ``pci-ep-generic`` + Linux EPC driver. The guest configures Endpoint Functions via configfs + and triggers operations (set_bar, raise_irq, map_addr, and so on) + through register writes. When the Link is brought up, the controller + creates a ``pcie-ep-generic`` Function and hotplugs it onto a Root Port. + +``pcie-ep-generic`` + The PCI Endpoint Function. It is created programmatically by + ``pcie-ep-ctrl`` and cannot be instantiated with ``-device``. Its PCI + identity, BAR layout and interrupt capabilities come from the register + file the EPC driver programmed. Each BAR aliases guest RAM at the + address the controller was told to use, so the host and the Function + share the same memory. + +Architecture +^^^^^^^^^^^^ + +:: + + +------------------------- one QEMU / one kernel -------------------------+ + | | + | EPF driver (pci-epf-test) Host driver (pci-endpoint-test)| + | | ^ | + | | configfs | /dev/ | + | v | | + | EPC driver (pci-ep-generic) PCIe subsystem (pciehp) | + | | ^ | + | | MMIO | enumerate/hotplug | + | v | | + | +----------------+ create + hotplug +---------------------+ | + | | pcie-ep-ctrl |--------------------->| pcie-ep-generic | | + | +----------------+ | (on Root Port) | | + | | +---------------------+ | + | | alias | alias | + | v v | + | +-------------------------------------------------------------+ | + | | guest RAM (single address space) | | + | | outbound DMA window aliases + Endpoint BAR backing | | + | +-------------------------------------------------------------+ | + +-------------------------------------------------------------------------+ + +Outbound maps add an alias of the target RAM into the controller's +outbound window at the offset the driver chose, and unmaps remove it. +Endpoint BAR accesses reach the same RAM the Function was given for its +BAR backing buffers. No data is copied and no synchronization points are +needed, because both sides address the same memory. + +Usage +^^^^^ + +A single invocation instantiates a Root Port and the Endpoint +Controller, pointing the controller at the Root Port with ``target-bus``: + +.. parsed-literal:: + + |qemu_system_aarch64| -machine virt -cpu cortex-a57 -m 1G \\ + -kernel Image -initrd rootfs.cpio.gz \\ + -device pcie-root-port,id=rp0,bus=pcie.0,chassis=1,slot=1,hotplug=on \\ + -device pcie-ep-ctrl,target-bus=rp0 \\ + -append "console=ttyAMA0" + +pcie-ep-ctrl properties +^^^^^^^^^^^^^^^^^^^^^^^^ + +``target-bus=ID`` + Device ID of the PCIe Root Port the Endpoint Function is hotplugged + onto when the Link comes up. Required. + +``outbound-size=SIZE`` + Size of the outbound DMA window (default 16 MiB). + +Guest configuration +^^^^^^^^^^^^^^^^^^^^ + +Configure the Endpoint Function via configfs (using ``pci-epf-test`` as an +example): + +.. code-block:: sh + + mount -t configfs configfs /sys/kernel/config + # The controller directory is named after the address the platform bus + # assigned to the device; discover it rather than hardcoding: + ctrl=$(ls /sys/kernel/config/pci_ep/controllers/ | head -n 1) + mkdir /sys/kernel/config/pci_ep/functions/pci_epf_test/func0 + echo 0x104c > /sys/kernel/config/pci_ep/functions/pci_epf_test/func0/vendorid + echo 0xb00d > /sys/kernel/config/pci_ep/functions/pci_epf_test/func0/deviceid + echo 1 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func0/msi_interrupts + echo 1 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func0/msix_interrupts + ln -s /sys/kernel/config/pci_ep/functions/pci_epf_test/func0 \ + /sys/kernel/config/pci_ep/controllers/$ctrl/ + echo 1 > /sys/kernel/config/pci_ep/controllers/$ctrl/start + +The ``start`` write brings the Link up and triggers the hotplug. The same +kernel then enumerates the Endpoint on the Root Port as a standard PCI +device, which the host driver binds to. + +Kernel requirements +^^^^^^^^^^^^^^^^^^^^ + +The single guest kernel needs both the Endpoint and the host support: + +- ``CONFIG_PCI_ENDPOINT=y`` +- ``CONFIG_PCI_EP_GENERIC=y`` (the ``pci-ep-generic`` platform driver) +- EPF driver for your use case (e.g. ``CONFIG_PCI_EPF_TEST=y``) +- Host driver matching the EPF's vendor/device ID (e.g. + ``CONFIG_PCI_ENDPOINT_TEST=y`` for ``pci-epf-test``) +- PCIe hotplug (``pciehp``) so the enumerated Endpoint appears at runtime + +Limitations +^^^^^^^^^^^ + +- No DMA engine emulation. Transfers use the CPU, so the Endpoint + framework's DMA-backed tests do not complete. +- Legacy INTx delivery is not routed on ARM ``virt``. Use MSI or MSI-X. +- The doorbell path is not implemented. +- Migration is not supported, as the controller mirrors Endpoint state + built at runtime. +- Currently wired only for the ARM ``virt`` machine. -- 2.43.0
