Hi Gavin,
On Mon, Sep 21, 2026 at 02:05:19PM +1000, Gavin Shan wrote:
Hi Mathieu,
On 9/4/26 5:35 AM, Mathieu Poirier wrote:
This patchset provides minimal functionality to start a Realm VM
from an Arm RME capable host using the following command line:
qemu-system-aarch64 \
-M confidential-guest-support=rme0,memory-backend=ram0 \
-object rme-guest,id=rme0,convert-in-place=on \
-object memory-backend-guest-memfd,id=ram0,size=1G,share=on \
-cpu host -M virt -enable-kvm -M gic-version=3,its=on -nodefaults ..
It is a refactoring of Jean-Philippe Brucker's initial work dating from a while
back. It is compatible with Steven Price's v16 revision [1] of his work adding
CCA support to KVM.
This revision is based on Michael Roth's work on guest memfd in-place memory
conversion [2], rebased to QEMU master. I left out the AMD specific changes
from Michael's work because I can't test the result. Other than the
modications to the linux headers, this series is identical to V3.
It was tested on the QEMU SBSA machine. For convenience, a repository is hosted
here [3], along with the TF-A [4] and RMM [5] for the SBSA machine (compatible
with Steven's v16 patchset). The Repository for the kernel [6] is also
available. Note that for the latter, defines for
KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES and KVM_CAP_ARM_RMI have been modified to
follow QEMU master.
Instructions to compile and run the entire stack can be found here [7].
Device Assignment is not included.
Thanks,
Mathieu
[1]. https://lore.kernel.org/kvm/[email protected]/
[2]. https://lore.kernel.org/kvm/[email protected]/
[3]. https://gitlab.com/Linaro/cca-public/qemu/-/tree/upstream-v4?ref_type=heads
[4].
https://gitlab.com/Linaro/cca-public/tf-a/trusted-firmware-a/-/tree/cca/v13?ref_type=heads
[5]. https://gitlab.com/Linaro/cca-public/rmm/-/tree/cca/v16?ref_type=heads
[6].
https://gitlab.com/Linaro/cca-public/linux/-/tree/upstream-v4?ref_type=heads
[7]. https://gitlab.com/Linaro/cca-public/build-instructions
Give this series and (v19) arm/cca host series [1] a try, and found the
following
issues. I'm sharing the found issues that I need take chance to look at them
closely in future.
[1] https://git.gitlab.arm.com/linux-arm/linux-cca.git (branch:
cca/cca-host/kvm-v19/integration)
Note that the host is emulated using qemu (TCG mode) via 'virt' platform, using
the following command lines
laptop$ /home/gshan/sandbox/cca/host/qemu/build/qemu-system-aarch64
\
-M virt,virtualization=on,secure=on,gic-version=3,acpi=off
\
-cpu max,x-rme=on -m 4G -smp 8
\
-serial mon:stdio -monitor none -nographic -nodefaults
\
-bios /home/gshan/sandbox/cca/host/tf-a/flash.bin
\
-kernel /home/gshan/sandbox/cca/host/linux/arch/arm64/boot/Image
\
-initrd
/home/gshan/sandbox/cca/host/buildroot/output/images/rootfs.cpio.xz \
-append kvm-arm.mode=rmm
\
-device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1
\
-device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2
\
-device pcie-root-port,bus=pcie.0,chassis=3,id=pcie.3
\
-device pcie-root-port,bus=pcie.0,chassis=4,id=pcie.4
\
-device virtio-9p-device,fsdev=shr0,mount_tag=shr0
\
-fsdev
local,security_model=none,path=/home/gshan/sandbox/cca/guest,id=shr0 \
-netdev tap,id=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
\
-device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:80
The realm VM is started using the following command lines.
host$ qemu-system-aarch64 -enable-kvm -object
rme-guest,id=rme0,convert-in-place=on \
-machine virt,gic-version=3,confidential-guest-support=rme0 -m 2G -cpu
host \
-smp maxcpus=4,cpus=4,sockets=1,clusters=1,cores=2,threads=2
\
-object memory-backend-guest-memfd,id=mem0,size=2G,share=on
\
-numa node,nodeid=0,cpus=0-3,memdev=mem0
\
-serial mon:stdio -monitor none -nographic -nodefaults
\
-kernel /mnt/linux/arch/arm64/boot/Image
\
-initrd /mnt/buildroot/output/images/rootfs.cpio.xz
\
-append earlycon=pl011,mmio,0x10009000000
Found issues
============
I was able to do the same manipulations on my side, but using the SBSA machine
for system emulation, with results as described below:
(1) qemu is terminiated on qmp command 'dump-guest-memory'
(qemu) dump-guest-memory -p "/tmp/mem" 0x40000000 0x10000000
**
ERROR:../system/memory_mapping.c:215:guest_phys_block_add_section: assertion failed:
(predecessor->target_end <= target_start)
Bail out! ERROR:../system/memory_mapping.c:215:guest_phys_block_add_section:
assertion failed: (predecessor->target_end <= target_start)
On my side this command causes the Realm to abort, as I expected it to be.
Realm memory should not be observable from the host. There may be a debug mode
to enable this kind of operation but I will have to look at the documentation
for details.
(2) qemu emulation can't be stopped and resumed
(qemu) stop
(qemu) cont
Failed to set KVM_REG_ARM_TIMER_CNT
That too I was expecting. Anything that can comprimise a Realm will cause the
HW to abort. I'll investigate the "Failed to set KVM_REG_ARM_TIMER_CNT" error
message though.
(3) qemu is terminiated on qmp command 'system_reset'
(qemu) system_reset
cpus are not resettable, terminating
(4) 'reboot' from the guest doesn't work
root@guest:~# reboot
:
Requesting system reboot
[ 123.264893] reboot: Restarting system
qemu-system-aarch64: cpus are not resettable, terminating
I suspect 'system_reset' from the monitor and a 'reboot' from the command line
will, at some point, end up following the same sequence. I need to think about
the 'reboot' use case further.
(5) Incompatible to virtio-iommu-pci. The realm guest doesn't boot up if a
virtio-iommu-pci
device is specified in the command lines
host$ qemu-system-aarch64 -enable-kvm \
-object rme-guest,id=rme0,convert-in-place=on \
-machine virt,gic-version=3,confidential-guest-support=rme0 \
-m 2G -cpu host -smp maxcpus=4,cpus=4,sockets=1,clusters=1,cores=2,threads=2 \
-object memory-backend-guest-memfd,id=mem0,size=2G,share=on \
-numa node,nodeid=0,cpus=0-3,memdev=mem0 \
-serial mon:stdio -monitor none -nographic -nodefaults \
-kernel /mnt/linux/arch/arm64/boot/Image \
-initrd /mnt/buildroot/output/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x10009000000 \
-device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1 \
-device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2 \
-device virtio-iommu-pci,bus=pcie.0 \
-drive file=/mnt/disk.qcow2,if=none,id=drive0 \
-device virtio-blk-pci,id=virtblk0,bus=pcie.1,drive=drive0,num-queues=4 \
-netdev tap,id=tap1,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:c0
I added the following to my Realm's boot command parameters:
-device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1 \
-device virtio-iommu-pci,bus=pcie.0 \
:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
[ 0.000000] Linux version 7.2.0-rc5-gavin-gf5098b6bae76
([email protected]) (gcc (GCC) 14.3.1
20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-65.el10) #47 SMP PREEMPT Mon
Jul 27 02:29:03 EDT 2026
[ 0.000000] KASLR enabled
[ 0.000000] Machine model: linux,dummy-virt
[ 0.000000] earlycon: pl11 at MMIO 0x0000010009000000 (options '')
[ 0.000000] printk: legacy bootconsole [pl11] enabled
[ 0.000000] efi: UEFI not found.
[ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in
the DT
[ 0.000000] NODE_DATA(0) allocated [mem 0xbfbecb00-0xbfbf09bf]
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] psci: SMC Calling Convention v1.2
[ 0.000000] RME: Using RSI version 1.0
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000040000000-0x00000000bfffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
:
[ 6.664423] Serial: AMBA PL011 UART driver
[ 7.151312] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud =
0) is a PL011 rev1
[ 7.213181] printk: console [ttyAMA0] enabled
[ 8.536454] HugeTLB: registered 1.00 Gi
<...The realm guest stops here, no output since then...>
On my side I get to user space but virtio-serial and virtio-net don't work. I'm
guessing virtio-iommu-pci does the same thing as this patch [1], but virtio
devices are disrupted because it is not configured properly. I need to
investigate to find the best approach to follow, i.e implicitly use (and
configure) a virtio-iommu-pci or make sure that [1] and a virtio-iommu-pci
device can play nice together.
Thanks,
Mathieu
[1]. [RFC v4 21/24] target/arm/kvm-rme: Add DMA remapping for the shared memory
region
Thanks,
Gavin
