On Mon, May 04, 2026 at 04:30:19PM +0400, Marc-André Lureau wrote:
> In TDX guests, virtio-mem plug/unplug/re-plug fails because
> kvm_set_phys_mem() unconditionally sets KVM memory attributes to
> PRIVATE for all guest_memfd regions. On re-plug, the PRIVATE->PRIVATE
> transition is a no-op, so KVM doesn't re-AUG pages and the guest's
> TDG.MEM.PAGE.ACCEPT fails.
Know little on TDX, please bare with me..
I saw KVM does a seamcall to ADD or AUG whenever a new EPT pte is set, via
this path:
__tdp_mmu_set_spte_atomic
set_external_spte_present
tdx_sept_set_private_spte <------
On unplug, I'm expecting with your prior patches, gmem pages will be
truncated properly, so they'll be all gone.
Then, qemu does replug -> guest gets that event, start access page -> EPT
violation, KVM resolving page fault with __tdp_mmu_set_spte_atomic() (per
above) and a new page -> triggering AUG (not ADD, since it's post-boot).
Could you elaborate here why AUG is missing in the first place?
Thanks,
>
> Implement the "start-shared" approach: virtio-mem memory starts with
> shared KVM attributes. The guest converts shared->private on plug (via
> set_memory_encrypted -> MapGPA + ACCEPT), and back to shared on unplug
> (via set_memory_decrypted). This ensures every plug triggers a real
> SHARED->PRIVATE transition, causing KVM to AUG fresh pages.
>
> Add RAM_GUEST_MEMFD_START_SHARED flag and set it during virtio-mem
> realize for guest_memfd-backed regions. Use
> ram_block_attributes_state_change() to properly update the attributes
> bitmap through the API. Skip setting PRIVATE in kvm_set_phys_mem()
> when the flag is set. On unplug, explicitly reset KVM attributes to
> shared on the host side to handle the case where the guest skips
> set_memory_decrypted().
>
> See also virtio-comment "[PATCH RFC] virtio-mem: add shared/private memory
> property details".
>
> Signed-off-by: Marc-André Lureau <[email protected]>
> ---
> include/system/memory.h | 6 ++++++
> accel/kvm/kvm-all.c | 3 ++-
> hw/virtio/virtio-mem.c | 27 ++++++++++++++++++++++++++-
> 3 files changed, 34 insertions(+), 2 deletions(-)
>
> diff --git a/include/system/memory.h b/include/system/memory.h
> index 28a75dac4ae..9dbf67efe50 100644
> --- a/include/system/memory.h
> +++ b/include/system/memory.h
> @@ -277,6 +277,12 @@ typedef struct IOMMUTLBEvent {
> */
> #define RAM_PRIVATE (1 << 13)
>
> +/*
> + * RAM with guest_memfd that should start with shared KVM memory
> + * attributes. The guest converts to private on use.
> + */
> +#define RAM_GUEST_MEMFD_START_SHARED (1 << 14)
> +
> static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn,
> IOMMUNotifierFlag flags,
> hwaddr start, hwaddr end,
> diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
> index 97463a683f4..c034e74c8e5 100644
> --- a/accel/kvm/kvm-all.c
> +++ b/accel/kvm/kvm-all.c
> @@ -1737,7 +1737,8 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml,
> abort();
> }
>
> - if (memory_region_has_guest_memfd(mr)) {
> + if (memory_region_has_guest_memfd(mr) &&
> + !(mr->ram_block->flags & RAM_GUEST_MEMFD_START_SHARED)) {
> err = kvm_set_memory_attributes_private(start_addr, slot_size);
> if (err) {
> error_report("%s: failed to set memory attribute private:
> %s",
> diff --git a/hw/virtio/virtio-mem.c b/hw/virtio/virtio-mem.c
> index 35e03ed7599..b46efe21126 100644
> --- a/hw/virtio/virtio-mem.c
> +++ b/hw/virtio/virtio-mem.c
> @@ -19,6 +19,7 @@
> #include "system/memory.h"
> #include "system/numa.h"
> #include "system/system.h"
> +#include "system/kvm.h"
> #include "system/ramblock.h"
> #include "system/reset.h"
> #include "system/runstate.h"
> @@ -479,6 +480,11 @@ static int virtio_mem_set_block_state(VirtIOMEM *vmem,
> uint64_t start_gpa,
> if (vmem->dynamic_memslots) {
> virtio_mem_deactivate_unplugged_memslots(vmem, offset, size);
> }
> + if (rb->flags & RAM_GUEST_MEMFD_START_SHARED) {
> + kvm_set_memory_attributes_shared(start_gpa, size);
> + ram_block_attributes_state_change(rb->attributes,
> + offset, size, false);
> + }
> return 0;
> }
>
> @@ -606,10 +612,12 @@ static int virtio_mem_unplug_all(VirtIOMEM *vmem)
> RAMBlock *rb = vmem->memdev->mr.ram_block;
>
> if (vmem->size) {
> + uint64_t used = qemu_ram_get_used_length(rb);
> +
> if (virtio_mem_is_busy()) {
> return -EBUSY;
> }
> - if (ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb))) {
> + if (ram_block_discard_range(rb, 0, used)) {
> return -EBUSY;
> }
> virtio_mem_notify_unplug_all(vmem);
> @@ -622,6 +630,11 @@ static int virtio_mem_unplug_all(VirtIOMEM *vmem)
> if (vmem->dynamic_memslots) {
> virtio_mem_deactivate_unplugged_memslots(vmem, 0, region_size);
> }
> + if (rb->flags & RAM_GUEST_MEMFD_START_SHARED) {
> + kvm_set_memory_attributes_shared(vmem->addr, used);
> + ram_block_attributes_state_change(rb->attributes,
> + 0, used, false);
> + }
> }
>
> trace_virtio_mem_unplugged_all();
> @@ -859,6 +872,18 @@ static void virtio_mem_device_realize(DeviceState *dev,
> Error **errp)
> rb = vmem->memdev->mr.ram_block;
> page_size = qemu_ram_pagesize(rb);
>
> + /*
> + * For CoCo VMs with guest_memfd, use the "start-shared" model:
> + * memory starts as shared and the guest converts to private on
> + * plug.
> + */
> + if (rb->flags & RAM_GUEST_MEMFD) {
> + rb->flags |= RAM_GUEST_MEMFD_START_SHARED;
> + ram_block_attributes_state_change(rb->attributes, 0,
> + qemu_ram_get_used_length(rb),
> + false);
> + }
> +
> if (virtio_mem_has_legacy_guests()) {
> switch (vmem->unplugged_inaccessible) {
> case ON_OFF_AUTO_AUTO:
>
> --
> 2.54.0
>
>
--
Peter Xu
