----- Original Message ----- > From: "Andrew Cooper" <andrew.coop...@citrix.com> > To: "Fonyuy-Asheri Caleb" <fonyuy-asheri.ca...@inria.fr>, "xen-devel" > <xen-devel@lists.xenproject.org> > Sent: Thursday, July 11, 2024 12:45:18 PM > Subject: Re: Help with Understanding vcpu xstate restore error during vm > migration
> On 11/07/2024 11:38 am, Fonyuy-Asheri Caleb wrote: >> Hello, >> >> I am trying to understand the causes of the vcpu xstate restore error >> during live migration. >> I get the following error during live migration: >> >> xc: error: Failed to set vcpu0's xsave info (22 = Invalid argument): >> Internal error >> >> I was able to locate the failure point to the file >> xen/arch/x86/domctl.c with the following check. >> >> if( evc->size<PV_XSAVE_HDR_SIZE|| >> evc->size>PV_XSAVE_SIZE(xfeature_mask) ) >> gotovcpuextstate_out; >> >> I know this is related to the number of xstates handled by the source >> server. Please can >> someone explain to me how these states are computed? >> >> I earlier thought it was simply the number xsave dependent features on >> the CPU but it seems >> to be more than that. >> >> Thanks in advance. > > It is certainly more complicated than that. > > What that's saying is that Xen doesn't think that the size of the blob > matches expectations. That said - I'm in the middle of rewriting this > logic because lots of it is subtly wrong. Please do you mind giving me more insight on the logic currently implemented and maybe what is wrong with it? It will be important for me since what I'm doing is research work. How do the values evc->size and xfeature_mask relate to the source and target processor xstates (or xstate management)? > > To start with, which version (or versions?) of Xen, and what hardware? Xen version 4.18.3-pre My CPU is : Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz I'm running Ubuntu 20.04. > > Are you able to capture `xen-cpuid -p` on the source and destination, as > well as `xen-cpuid -p $domid` of the VM running on the source? Yes I am able to capture them. You can find them below. > > Do you have any CPUID overrides in the VM config file? I don't have any CPUID overrides. I just let the guest (Ubuntu server) use whatever xen gives. > > Thanks, > > ~Andrew Caleb ========================================================================================================= Source server xen-cpuid -p --------------------------------------------------------------------------------------------------------- ========================================================================================================= Xen reports there are maximum 120 leaves and 2 MSRs Raw policy: 32 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000f:756e6547:6c65746e:49656e69 00000001:ffffffff -> 000306f2:00100800:7ffefbff:bfebfbff 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c10000 00000004:00000000 -> 1c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 1c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 1c004143:01c0003f:000001ff:00000000 00000004:00000003 -> 1c03c163:04c0003f:00003fff:00000006 00000005:ffffffff -> 00000040:00000040:00000003:00002120 00000006:ffffffff -> 00000077:00000002:00000009:00000000 00000007:00000000 -> 00000000:000037ab:00000000:9c000400 00000009:ffffffff -> 00000001:00000000:00000000:00000000 0000000a:ffffffff -> 07300403:00000000:00000000:00000603 0000000b:00000000 -> 00000001:00000002:00000100:00000000 0000000b:00000001 -> 00000004:00000010:00000201:00000000 0000000d:00000000 -> 00000007:00000000:00000340:00000000 0000000d:00000001 -> 00000001:00000000:00000000:00000000 0000000d:00000002 -> 00000100:00000240:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000021:2c100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 55504320:2d354520:30333632:20337620 80000004:ffffffff -> 2e322040:48473034:0000007a:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000007:ffffffff -> 00000000:00000000:00000000:00000100 80000008:ffffffff -> 0000302e:00000000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 0000000000000000 Host policy: 31 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 000306f2:00100800:77fef3ff:bfebfbff 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c10000 00000004:00000000 -> 1c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 1c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 1c004143:01c0003f:000001ff:00000000 00000004:00000003 -> 1c03c163:04c0003f:00003fff:00000006 00000007:00000000 -> 00000000:000037ab:00000000:9c000400 0000000d:00000000 -> 00000007:00000000:00000340:00000000 0000000d:00000001 -> 00000001:00000000:00000000:00000000 0000000d:00000002 -> 00000100:00000240:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000021:2c100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 55504320:2d354520:30333632:20337620 80000004:ffffffff -> 2e322040:48473034:0000007a:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000007:ffffffff -> 00000000:00000000:00000000:00000100 80000008:ffffffff -> 0000302e:00000000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 000000000c000000 PV Max policy: 33 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 000306f2:00100800:f6f83203:1fc9cbf5 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c10000 00000004:00000000 -> 1c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 1c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 1c004143:01c0003f:000001ff:00000000 00000004:00000003 -> 1c03c163:04c0003f:00003fff:00000006 00000007:00000000 -> 00000002:00000329:00000000:ac000400 0000000d:00000000 -> 00000007:00000000:00000340:00000000 0000000d:00000001 -> 00000001:00000000:00000000:00000000 0000000d:00000002 -> 00000100:00000240:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000023:28100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 55504320:2d354520:30333632:20337620 80000004:ffffffff -> 2e322040:48473034:0000007a:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000007:ffffffff -> 00000000:00000000:00000000:00000100 80000008:ffffffff -> 0000302e:00001000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 000000001c020004 HVM Max policy: 33 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 000306f2:00100800:f7fa3223:1fcbfbff 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c10000 00000004:00000000 -> 1c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 1c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 1c004143:01c0003f:000001ff:00000000 00000004:00000003 -> 1c03c163:04c0003f:00003fff:00000006 00000007:00000000 -> 00000002:000007ab:00000000:bc000400 0000000d:00000000 -> 00000007:00000000:00000340:00000000 0000000d:00000001 -> 00000001:00000000:00000000:00000000 0000000d:00000002 -> 00000100:00000240:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000023:2c100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 55504320:2d354520:30333632:20337620 80000004:ffffffff -> 2e322040:48473034:0000007a:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000007:ffffffff -> 00000000:00000000:00000000:00000100 80000008:ffffffff -> 0000302e:00101000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 000000001c020004 PV Default policy: 31 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 000306f2:00100800:f6d83203:1fc9cbf5 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c10000 00000004:00000000 -> 1c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 1c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 1c004143:01c0003f:000001ff:00000000 00000004:00000003 -> 1c03c163:04c0003f:00003fff:00000006 00000007:00000000 -> 00000000:00000329:00000000:ac000400 0000000d:00000000 -> 00000007:00000000:00000340:00000000 0000000d:00000001 -> 00000001:00000000:00000000:00000000 0000000d:00000002 -> 00000100:00000240:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000021:28100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 55504320:2d354520:30333632:20337620 80000004:ffffffff -> 2e322040:48473034:0000007a:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000008:ffffffff -> 0000302e:00001000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 000000000c000000 HVM Default policy: 31 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 000306f2:00100800:f7fa3203:1fcbfbff 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c10000 00000004:00000000 -> 1c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 1c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 1c004143:01c0003f:000001ff:00000000 00000004:00000003 -> 1c03c163:04c0003f:00003fff:00000006 00000007:00000000 -> 00000000:000007ab:00000000:bc000400 0000000d:00000000 -> 00000007:00000000:00000340:00000000 0000000d:00000001 -> 00000001:00000000:00000000:00000000 0000000d:00000002 -> 00000100:00000240:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000021:2c100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 55504320:2d354520:30333632:20337620 80000004:ffffffff -> 2e322040:48473034:0000007a:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000008:ffffffff -> 0000302e:00101000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 000000000c000000 ========================================================================================================= Target server xen-cpuid -p --------------------------------------------------------------------------------------------------------- ========================================================================================================= Xen reports there are maximum 120 leaves and 2 MSRs Raw policy: 39 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 00000016:756e6547:6c65746e:49656e69 00000001:ffffffff -> 00050654:00200800:7ffefbff:bfebfbff 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c30000 00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 3c004143:03c0003f:000003ff:00000000 00000004:00000003 -> 3c07c163:0280003f:00004fff:00000004 00000005:ffffffff -> 00000040:00000040:00000003:00002020 00000006:ffffffff -> 00000ef7:00000002:00000009:00000000 00000007:00000000 -> 00000000:d39ffffb:00000018:bc002400 0000000a:ffffffff -> 07300404:00000000:00000000:00000603 0000000b:00000000 -> 00000001:00000002:00000100:00000000 0000000b:00000001 -> 00000005:00000014:00000201:00000000 0000000d:00000000 -> 000002ff:00000000:00000a88:00000000 0000000d:00000001 -> 0000000f:00000000:00000100:00000000 0000000d:00000002 -> 00000100:00000240:00000000:00000000 0000000d:00000003 -> 00000040:000003c0:00000000:00000000 0000000d:00000004 -> 00000040:00000400:00000000:00000000 0000000d:00000005 -> 00000040:00000440:00000000:00000000 0000000d:00000006 -> 00000200:00000480:00000000:00000000 0000000d:00000007 -> 00000400:00000680:00000000:00000000 0000000d:00000008 -> 00000080:00000000:00000001:00000000 0000000d:00000009 -> 00000008:00000a80:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000121:2c100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 6c695320:20726576:34313134:55504320 80000004:ffffffff -> 32204020:4730322e:00007a48:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000007:ffffffff -> 00000000:00000000:00000000:00000100 80000008:ffffffff -> 0000302e:00000000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 0000000000000c04 Host policy: 38 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 00050654:00200800:47fee3ff:bfebfbff 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c30000 00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 3c004143:03c0003f:000003ff:00000000 00000004:00000003 -> 3c07c163:0280003f:00004fff:00000004 00000007:00000000 -> 00000000:029cffdb:00000008:bc002400 0000000d:00000000 -> 0000021b:00000000:00000a88:00000000 0000000d:00000001 -> 0000000f:00000000:00000000:00000000 0000000d:00000003 -> 00000040:000003c0:00000000:00000000 0000000d:00000004 -> 00000040:00000400:00000000:00000000 0000000d:00000009 -> 00000008:00000a80:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000121:2c100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 6c695320:20726576:34313134:55504320 80000004:ffffffff -> 32204020:4730322e:00007a48:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000007:ffffffff -> 00000000:00000000:00000000:00000100 80000008:ffffffff -> 0000302e:00000000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 0000000008000c04 PV Max policy: 32 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 00050654:00200800:c6f82203:1fc9cbf5 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c30000 00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 3c004143:03c0003f:000003ff:00000000 00000004:00000003 -> 3c07c163:0280003f:00004fff:00000004 00000007:00000000 -> 00000002:018c0b19:00000000:ac000c00 0000000d:00000000 -> 00000003:00000000:00000240:00000000 0000000d:00000001 -> 00000007:00000000:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000123:28100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 6c695320:20726576:34313134:55504320 80000004:ffffffff -> 32204020:4730322e:00007a48:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000007:ffffffff -> 00000000:00000000:00000000:00000100 80000008:ffffffff -> 0000302e:00001000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 0000000018020004 HVM Max policy: 40 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 00050654:00200800:c7fa2223:1fcbfbff 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c30000 00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 3c004143:03c0003f:000003ff:00000000 00000004:00000003 -> 3c07c163:0280003f:00004fff:00000004 00000007:00000000 -> 00000002:019c4f9b:00000008:bc000c00 0000000d:00000000 -> 0000021b:00000000:00000a88:00000000 0000000d:00000001 -> 0000000f:00000000:00000000:00000000 0000000d:00000003 -> 00000040:000003c0:00000000:00000000 0000000d:00000004 -> 00000040:00000400:00000000:00000000 0000000d:00000009 -> 00000008:00000a80:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000123:2c100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 6c695320:20726576:34313134:55504320 80000004:ffffffff -> 32204020:4730322e:00007a48:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000007:ffffffff -> 00000000:00000000:00000000:00000100 80000008:ffffffff -> 0000302e:00101000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 0000000018020004 PV Default policy: 30 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 00050654:00200800:c6d82203:1fc9cbf5 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c30000 00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 3c004143:03c0003f:000003ff:00000000 00000004:00000003 -> 3c07c163:0280003f:00004fff:00000004 00000007:00000000 -> 00000000:008c0b09:00000000:ac000400 0000000d:00000000 -> 00000003:00000000:00000240:00000000 0000000d:00000001 -> 00000007:00000000:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000121:28100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 6c695320:20726576:34313134:55504320 80000004:ffffffff -> 32204020:4730322e:00007a48:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000008:ffffffff -> 0000302e:00001000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 0000000008000004 HVM Default policy: 38 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 00050654:00200800:c7fa2203:1fcbfbff 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c30000 00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 3c004143:03c0003f:000003ff:00000000 00000004:00000003 -> 3c07c163:0280003f:00004fff:00000004 00000007:00000000 -> 00000000:009c0f8b:00000008:bc000400 0000000d:00000000 -> 00000203:00000000:00000a88:00000000 0000000d:00000001 -> 0000000f:00000000:00000000:00000000 0000000d:00000009 -> 00000008:00000a80:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000121:2c100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 6c695320:20726576:34313134:55504320 80000004:ffffffff -> 32204020:4730322e:00007a48:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000008:ffffffff -> 0000302e:00101000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 0000000008000004 ========================================================================================================= VM xen-cpuid -p $domid --------------------------------------------------------------------------------------------------------- Domain 3 policy: 31 leaves, 2 MSRs CPUID: leaf subleaf -> eax ebx ecx edx 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 00000001:ffffffff -> 000306f2:00100800:f6d83203:1fc9cbf5 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c10000 00000004:00000000 -> 1c004121:01c0003f:0000003f:00000000 00000004:00000001 -> 1c004122:01c0003f:0000003f:00000000 00000004:00000002 -> 1c004143:01c0003f:000001ff:00000000 00000004:00000003 -> 1c03c163:04c0003f:00003fff:00000006 00000007:00000000 -> 00000000:00002329:00000000:ac000400 0000000d:00000000 -> 00000007:00000000:00000340:00000000 0000000d:00000001 -> 00000001:00000000:00000000:00000000 0000000d:00000002 -> 00000100:00000240:00000000:00000000 80000000:ffffffff -> 80000008:00000000:00000000:00000000 80000001:ffffffff -> 00000000:00000000:00000021:28100800 80000002:ffffffff -> 65746e49:2952286c:6f655820:2952286e 80000003:ffffffff -> 55504320:2d354520:30333632:20337620 80000004:ffffffff -> 2e322040:48473034:0000007a:00000000 80000006:ffffffff -> 00000000:00000000:01006040:00000000 80000008:ffffffff -> 0000302e:00001000:00000000:00000000 MSRs: index -> value 000000ce -> 0000000080000000 0000010a -> 000000000c000000 =========================================================================================================
