> -----Original Message----- > From: Eric Auger <[email protected]> > Sent: 01 October 2026 17:41 > To: Shameer Kolothum Thodi <[email protected]>; qemu- > [email protected]; [email protected] > Cc: [email protected]; Nicolin Chen <[email protected]>; Nathan > Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An > <[email protected]>; [email protected]; Ankit Agrawal <[email protected]>; Aniket > Agashe <[email protected]>; [email protected]; > [email protected] > Subject: Re: [RFC PATCH 0/7] hw/arm/smmuv3: Live migration for accel=on > > External email: Use caution opening links or attachments > > > Hi Shameer, > > On 9/4/26 8:05 PM, Shameer Kolothum wrote: > > Hi, > > > > This series adds live migration support for an accelerated SMMUv3 > > (-device arm-smmuv3,accel=on) with vfio-pci endpoints on the iommufd > > backend. > > > > The destination replays rather than receives. The stream table, the > > STEs, the CDs and the Stage 1 page tables all live in guest RAM, so > > once that and the registers are restored, the destination walks the > > stream table again and rebuilds the vDEVICE and the nested HWPT for every > device. > > > > What the series does: > > > > - drains the kernel vEVENTQ into the guest event queue when the VM stops, > > so that pending fault events are not lost (1, 2) > > - migrates the resolved SMMUv3 feature set in a new smmuv3/accel > > subsection and fails the load if the two ends differ (3) > > - replays the guest stream table on the destination and restores the > > vEVENTQ and the GBPA HWPT (4, 5) > > - asks VFIO for dirty tracking on the Stage 2 nesting parent (6) > > - replaces the accel migration blocker with one scoped to CMDQV, which > > cannot be migrated for now (7) > > > > This depends on the host kernel supporting dirty tracking on the Stage > > 2 nesting parent, for assigned devices that cannot track dirty pages > > themselves. A branch based on Jason's iommu page table work is here [0]. > > > > Big caveat: > > > > This is not yet tested with a real device that supports migration as I > > do not have a test setup for that. Folks who do have such a setup have > > reached out and are copied here, so hopefully this RFC lets them give > > it a spin and provide some early feedback. > > Please could you clarify all the requirements for supporting this migration > scenario (at EP, SMMU and kernel level)?
For a minimum we need: EP: - A device with migration support and a vfio variant driver that implements VFIO migration, the same as for migration without a vIOMMU. Host SMMUv3: - Dirty tracking on the Stage 2 nesting parent. Stage 2 maps GPA to PA, so the dirty bits it sets are already in guest physical terms. That needs HTTU (IDR0.HTTU reporting HD) and coherent table walks, which is what arm_smmu_dbm_capable() checks and iommufd reports as IOMMU_HW_CAP_DIRTY_TRACKING. Without it VFIO adds its usual blocker. Host Kernel: - IOMMU_HWPT_ALLOC_DIRTY_TRACKING on a nesting parent domain. Upstream arm-smmu-v3 returns EOPNOTSUPP for that combination. The branch in [0] enables it on top of Jason's iommu page table work, by setting S2HA and S2HD in the STE and using the DBM support in the generic page table code. Thanks, Shameer > > > > QEMU branch is here: > > > > https://github.com/shamiali2008/qemu-master/tree/smmuv3-accel-live- > mig > > rn-rfcv1 > > > > Please let me know your feedback. > > > > Thanks, > > Shameer > > [0]: > > > https://github.com/shamiali2008/linux/commits/iommu_pt_arm64_s2_dbm > / > > > > Shameer Kolothum (7): > > hw/arm/smmuv3-accel: Factor out the single vEVENT read > > hw/arm/smmuv3-accel: Drain the vEVENTQ when the VM stops > > hw/arm/smmuv3: Migrate the accelerator feature set > > hw/arm/smmuv3-accel: Add a helper to replay all STEs > > hw/arm/smmuv3-accel: Restore accelerator state on incoming migration > > hw/arm/smmuv3-accel: Request dirty tracking on the nesting parent > > hw/arm/smmuv3: Support migration with accel=on > > > > hw/arm/smmuv3-accel.h | 2 + > > include/hw/arm/smmuv3.h | 7 +- > > hw/arm/smmuv3-accel-stubs.c | 17 ++++ > > hw/arm/smmuv3-accel.c | 192 > ++++++++++++++++++++++++++++++++++-- > > hw/arm/smmuv3.c | 6 +- > > 5 files changed, 210 insertions(+), 14 deletions(-) > >
