This series lets QEMU, when running as a Xen device model, drive
vhost-user backends that map guest memory through the Xen foreign
mapping interface, implementing the front-end side of
VHOST_USER_PROTOCOL_F_XEN_MMAP. The protocol extension itself is
already documented in docs/interop/vhost-user.rst (feature bit 17,
extended memory region description) and implemented by rust-vmm's
vhost / vm-memory crates and the vhost-device backends built on them.
The problem this solves: under Xen the guest's RAM is not allocated by
QEMU and is not backed by a file descriptor. memory_region_get_fd()
returns -1, so vhost_section() filters out every RAM section, the vhost
memory listener registers no regions, and starting any vhost-user
device fails with "Failed initializing vhost-user memory map". With
F_XEN_MMAP the backend maps guest memory itself.
The protocol requires one file descriptor per region in SET_MEM_TABLE.
Guest RAM under Xen has no backing fd, so the front-end opens
/dev/xen/privcmd per region purely to satisfy that requirement; the
backend derives the mapping from guest_phys_addr + domid and never
reads the fd. Each fd is closed once the message has been sent.
This patchset was rebased onto the new vhost_phys_vring_addr infrastructure
and extends vhost_user_gpa_addresses() so that negotiated
F_XEN_MMAP (bit 17), not F_GPA_ADDRESSES (bit 21, which the backend doesn't
advertise), drives GPA addressing for both rings and userspace_addr.
The two patches:
1/2 accept the Xen RAM section in vhost_section()
2/2 negotiate F_XEN_MMAP and build SET_MEM_TABLE from the extended
region layout.
Testing:
Tested on Xen/ARM64 with a DomU using virtio-mmio transports created
by the xenpvh machine, running vhost-device-sound (rust-vmm, built
with the "xen" feature) as the backend in dom0. The device negotiates,
receives the memory table and ring addresses, and the guest's
virtio-snd driver probes and operates.
Non-Xen / x86 KVM: vhost-user-snd backed by
vhost-device-sound (null backend) on a q35/KVM guest. The device
negotiates, the guest virtio-snd driver probes and runs the control and
PCM paths, and the SET_MEM_TABLE and vring-address traffic is identical
to a build without this series confirming the
non-Xen path is unchanged.
The control message exchange between the frontend and backend was
tracked using sockdump as was described in:
Making VirtIO sing - implementing virtio-sound in rust-vmm project
|-> at FOSDEM 2024
Setup:
The main part of the xl config this enables:
virtio = [
'backend=0,type=virtio,device,transport=mmio,grant_usage=false'
]
device_model_args = [
...
'-chardev', 'socket,id=snd_chardev,path=/tmp/snd.sock',
'-device', 'vhost-user-snd,chardev=snd_chardev,id=snd,iommu_platform=true',
...
]
Xen 4.22-unstable was used with:
-enable-IOREQ_SERVER
-enable-EXPERT
An extra patch was added to xen-tools.
Namely, xen tools will request a pv device drive type for ARM64 but
qemu expects pvh. This is a known issue:
github.com/Xilinx/xen/commit/5f669949c9ffdb1947cb47038956b5fb8eeb072a
Qemu master was used configured with the following flags:
--target-list=aarch64-softmmu \
--cross-prefix=aarch64-linux-gnu- \
--enable-xen \
--enable-vhost-user \
--extra-cflags="-I$XEN-TOOLS/usr/local/include" \
--extra-ldflags="-L$XEN-TOOLS/usr/local/lib -Wl,
-rpath-link,$XEN-TOOLS/usr/local/lib" \
Likewise for x86:
--target-list=aarch64-softmmu \
--enable-slirp \
--enable-xen \
--enable-vhost-user \
--enable-virtfs \
Linux version 6.11.7 was used with extra configuration flags:
* For enabling Xen Dom0/DomU support
* For enabling virtio (mmio, snd, etc.)
* For enabling sockdump features (BPF, IKHEADERS, KPROBE, etc.)
* Extra debug flags (DEBUG_FS, etc.)
vhost-device commit-id:
c3bb658ef4fe20a2f264dbbbbc6fa19f1c08c0c5
Was used built with:
--features alsa-backend,xen
Importantly in vhost-device-scmi/src/vhu_scmi.rs:
// QUEUE_SIZE must be apparently at least 1024 for MMIO.
// There is probably a maximum size per descriptor defined in the kernel.
const QUEUE_SIZE: usize = 1024;
A similar change was made to make mmio work in vhost-user-sound device:
```
diff --git a/vhost-device-sound/src/device.rs b/vhost-device-sound/src/device.rs
index 99e1a8f..1397076 100644
--- a/vhost-device-sound/src/device.rs
+++ b/vhost-device-sound/src/device.rs
@@ -603,7 +603,7 @@ impl VhostUserBackend for VhostUserSoundBackend {
// a queue is filled up. In this case, adding an element to the queue
// returns ENOSPC and the element is not queued for a later attempt and
// is lost. `64` is a "good enough" value from our observations.
- 64
+ 1024
}
fn features(&self) -> u64 {
```
Without this frontend and backend will fail to negotiate queue size.
Scope and known limitations:
* Foreign mappings only. Grant mappings are not supported: vhost's
section tracking derives a host pointer for each region, which is
invalid for the grant pseudo-region, and per-access grant mapping
needs a different region description (GRANT | no-advance-map). Patch
1 rejects the xen.grants region explicitly. Setting grant_usage=true
does not change the qemu<->backend vhost-user exchange.
* VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS is suppressed under Xen:
the ADD/REM_MEM_REG path has not been converted to the extended
region format, and Xen guests currently expose a single RAM region,
so SET_MEM_TABLE is sufficient. Multiple RAM regions are not yet
exercised. Postcopy is refused.
* Spec vs reference implementation: docs/interop/vhost-user.rst
describes the "can not be mapped in advance" xen-mmap flag as Bit 8
(value 0x100), whereas rust-vmm's vm-memory uses 0x8 (bit 3,
MmapXenFlags::NO_ADVANCE_MAP). This series uses neither, but the
discrepancy probably wants resolving in the spec. Viresh, which is
intended -- bit position 8 or value 0x8?
* userspace_addr is carried unchanged in the region descriptor; under
Xen it does not correspond to a mapping and backends do not
interpret it. An alternative would be to define it (e.g. mirror
guest_phys_addr).
Open questions:
- userspace_addr semantics under Xen: leave it unchanged, or define it?
- Multi-region support: convert ADD/REM_MEM_REG to the extended layout
rather than suppressing CONFIGURE_MEM_SLOTS?
- Grant-mapping support: worth pursuing, and what region-description
shape do backends expect?
- Updating vhost-device-sound to reflect the mmio support.
References:
- vhost-user spec, F_XEN_MMAP / extended memory region / xen mmap flags:
docs/interop/vhost-user.rst
- rust-vmm vm-memory MmapXenFlags (FOREIGN=0x1, GRANT=0x2,
NO_ADVANCE_MAP=0x8): src/mmap/xen.rs
- Making VirtIO sing - implementing virtio-sound in rust-vmm project
|-> at FOSDEM 2024
Signed-off-by: Dusan Stojkovic <[email protected]>
Signed-off-by: Nikola Jelic <[email protected]>
---
Dusan Stojkovic (2):
vhost: accept Xen guest RAM sections for vhost-user
vhost-user: implement VHOST_USER_PROTOCOL_F_XEN_MMAP
hw/virtio/trace-events | 2 +
hw/virtio/vhost-user.c | 120 +++++++++++++++++++++++++++++++++++++++--
hw/virtio/vhost.c | 18 +++++++
hw/xen/xen_stubs.c | 5 ++
include/hw/virtio/vhost-user.h | 2 +-
5 files changed, 143 insertions(+), 4 deletions(-)
---
base-commit: c7cf7c810153d6f5f31aa2d5c0dee9087f6b4dff
change-id: 20260618-vhost-xen-foreign-mapping-d023c85bb706
Best regards,
--
Dusan Stojkovic <[email protected]>
