On Tue, May 26, 2026 at 17:17:10 +0200, Denis V. Lunev wrote:
> qemuDomainMakeCPUMigratable() strips features marked added='yes' (in
> src/cpu_map/x86_*.xml) from the migration cookie when the source CPU
> was specified as host-model. The intent was libvirt-protocol compat
> with older destinations; the cost is guest CPU compat, paid silently
> on every migration.
Sigh, yeah there's still (at least) one thing missing around vmx
migration...
> Every Intel x86 CPU model from Westmere through Sapphire Rapids
> carries 60+ added='yes' features, including
> vmx-exit-load-perf-global-ctrl and vmx-entry-load-perf-global-ctrl
> that control the LOAD_IA32_PERF_GLOBAL_CTRL allowed-1 bits of
> MSR_IA32_VMX_{EXIT,ENTRY}_CTLS. A host-model live migration on any
> of these models drops those features from the destination's qemu
> argv.
Yeah, unless those vmx features are explicitly specified in the domain
XML, libvirt adds them according to what QEMU enabled based on the -cpu
command line and then removes them during migration. The assumption is
that if QEMU automatically added them on the source host, it will also
add them on the destination host.
> Modern qemu gates the nested VMX capability MSRs on the explicit -cpu
> list, so the guest's MSR view shifts.
What exactly "modern qemu" means? Do you have an exact version? Anyway,
this sounds like a QEMU bug to me. If a combination of -cpu command line
and a machine type enabled the features on the source, the same
combination should enable them on the source as well. The machine type
does not change during migration and treating vmx features shouldn't
change either.
> Drop the strip. If a destination libvirt does not know a feature in
> the cookie, its parser rejects the migration with a precise
> unknown-feature error: operators can upgrade or narrow the source
> CPU definition. Either is visible; the status quo is not.
The problem is old libvirt was not tracking vmx features at all and thus
any domain started on new libvirt would fail to migrate to an older
libvirt. If a user explicitly required a vmx feature in domain XML, old
libvirt would correctly refuse incoming migration because of an unknown
CPU feature. But it's new libvirt suddenly recognizing features QEMU
always enabled and adding them to domain XML. We always deal with such
situation by dropping the automatically added XML elements for migration
compatibility.
> This effectively reverts 14d3517410 ("qemu: domain: Drop added
> features from migratable CPU") together with its follow-up
> aae8a5774b ("qemu: Drop vmx-* from migratable CPU model only when
> origCPU is set"), and removes the now-unused origCPU plumbing in
> qemuDomainMakeCPUMigratable() and its callers.
Unfortunately this is not the correct solution. We don't have a policy
for backward migration compatibility with old libvirt so we should keep
the compatibility as long as the domain XML does not contain anything
unknown to the old libvirt.
That said, we're missing a code that would transfer all the removed vmx
flags in a migration cookie to make sure new libvirt can see the exact
CPU definition and thus can explicitly request all the vmx features the
source QEMU added.
Jirka
