Re: Cross vendor migration ideas
Andi Kleen wrote: But at least for Linux, if we notify the guest telling it to retune (which can include the vsyscall path) we can avoid the cost entirely. Hmm, we discussed something like this some time ago anyways (add a way to rescan CPUID features) because there are microcode updates around that change some (relatively obscure) CPUID features. But doing it fully general would be likely quite intrusive. You would need to rewrite the CPU initialization code and some other code in a callback like matter like the PCI code does for hotplug. The primary consumer would be compat mode syscall/sysenter. Page copy would be a secondary consumer. All the rest are likely in the noise. Hmm, maybe also the raid5+ algorithms. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
> Long running guests on a large farm shouldn't expect to stay on the same > host they were booted on. If they change frequently and the vendors are even matched in the farm their chance of running "fast" again will be quite good. The only worst case would be to start on one vendor, run for a short time there and then spend all the remaining time on the other. Most other cases would be better. Also again I wouldn't expect the impact to be that dramatic. While there are some workloads which are syscall latency sensitive (and not rely on the vsyscalls) a lot of others are not. > But at least for Linux, if we notify the > guest telling it to retune (which can include the vsyscall path) we can > avoid the cost entirely. Hmm, we discussed something like this some time ago anyways (add a way to rescan CPUID features) because there are microcode updates around that change some (relatively obscure) CPUID features. But doing it fully general would be likely quite intrusive. You would need to rewrite the CPU initialization code and some other code in a callback like matter like the PCI code does for hotplug. I doubt doing that would be worth the effort in real performance gains. -Andi -- [EMAIL PROTECTED] -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
- "Nitin A Kamble" <[EMAIL PROTECTED]> wrote: > Amit, Alex, > Please see my comments bellow. > > Avi, > Please have a look at the patches, and let me know the parts you > think > can be done better. > > On Fri, 2008-11-14 at 06:07 -0700, Amit Shah wrote: > > * On Thursday 13 Nov 2008 19:08:14 Alexander Graf wrote: > > > On 13.11.2008, at 05:35, Amit Shah wrote: > > > > * On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote: > > > >> On 12.11.2008, at 17:52, Amit Shah wrote: > > > >>> Hi Alex, > > > >>> > > > >>> * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote: > > > Hi, > > > > > > I was thinking a bit about cross vendor migration recently > and > > > since > > > we're doing open source development, I figured it might be a > good > > > idea > > > to talk to everyone about this. > > > > > > So why are we having a problem? > > > > > > In normal operation we don't. If we're running a 32-bit > kernel, we > > > can > > > use SYSENTER to jump from kernel<->userspace. If we're on a > 64-bit > > > kernel with 64-bit userspace, every CPU supports SYSCALL. At > least > > > Linux is being smart on this and does use exactly these two > > > capabilities in these two cases. > > > But if we're running in compat mode (64-bit kernel with > 32-bit > > > userspace), things differ. Intel supports only SYSENTER here, > while > > > AMD only supports SYSCALL. Both can still use int80. > > > > > > Operating systems detect usage of SYSCALL or SYSENTER pretty > > > early on > > > (Linux does this on vdso). So when we boot up on an Intel > machine, > > > Linux assumes that using SYSENTER in compat mode is fine. > Migrating > > > that machine to an AMD machine breaks this assumption though, > since > > > SYSENTER can't be used in compat mode. > > > On LInux, this detection is based on the CPU vendor string. > If > > > Linux > > > finds a "GenuineIntel", SYSENTER is used in compat mode, if > it's > > > "AuthenticAMD", SYSCALL is used and if none of these two is > found, > > > int80 is used. > > > > > > I tried modifying the vendor string, removed the "overwrite > the > > > vendor > > > string with the native string" hack and things look like they > work > > > just fine with Linux. > > > > > > Unfortunately right now I don't have a 64-bit Windows > installation > > > around to check if that approach works there too, but if it > does > > > and > > > no known OS breaks due to the invalid vendor string, we can > just > > > create our own virtual CPU string, no? > > > >>> > > > >>> qemu has an option for that, -cpu qemu64 IIRC. As long as we > expose > > > >>> practically correct cpuids and MSRs, this should be fine. I've > not > > > >>> tested > > > >>> qemu64 with winxp x64 though. Also, last I knew, winxp x64 > > > >>> installation > > > >>> didn't succeed with --no-kvm. qemu by default exposes an AMD > CPU > > > >>> type. > > > >> > > > >> I wasn't talking about CPUID features, but the vendor string. > Qemu64 > > > >> provides the AuthenticAMD string, so we don't run into any > issues I'm > > > >> presuming. > > > > > > > > Right -- the thing is, with the default AuthenticAMD string, > winp x64 > > > > installation fails. That has to be because of some missing > cpuids. > > > > That's one > > > > of the drawbacks of exposing a well-known CPU type. I was > suggesting > > > > we > > > > should try out the -cpu qemu64 CPU type since it exposes a non- > > > > standard CPU > > > > to see if guests and most userspace programs work fine without > any > > > > further > > > > tweaking -- see the 'cons' below for why this might be a > problem. > > > > > > I still don't really understand what you're trying to say - qemu64 > is > > > the default in KVM right now. You mean winxp64 installation > doesn't > > > > No, the default for KVM is the host CPU type. > > Amit, Aliex is correct. the default cpu for kvm is qemu64 not the > host. There are two things -- the Vendor ID and the cpuid bits. I'm talking about the vendor ID here. > I have sent the patches to add an options -cpu host. Some of the > patches > are gone in, But All the patches are not in yet. Also my patches does > not make the host option as default. I have attached the remaining > two > patches. Amit. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Andi Kleen wrote: Yes, but since we're emulating a CPU anyways we don't want vendor specific setup, since we might live migrate. I was mainly thinking of tuning. For example Linux selects the best suitable page copy function based on vendor information. Using the wrong page copy can have a large impact on your performance. We can tell Linux to retune using a custom ACPI (blech) event. Also it might break user space, unless you key the fake vendor CPUID intercept on ring 0 vs ring 3 (but even if that might not be enough because some kernel modules can call CPUID on their own) Is there userspace code that relies on GenuineIntel? Yes. Undoubtedly also the same with others. This will not work of course for userspace. I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't be that much slower than int 0x80 hopefully. Well emulating them means you're leaving the VM on every user<->kernel transition. That's a _huge_ performance hit. I don't have the numbers, but IIRC a roundtrip is ~3000 cycles. Depends on the CPU. Newer CPUs are faster. I think emulation will be slower than 3000 cycles, even on newer cpus. In addition to the #vmexit, you have to save all registers, walk the guest page tables to fetch the instruction, execute the emulator code including all the checks, go back to the guest. Then do the same again on sysexit/sysret -- the overhead doubles. Also you would select the default based on which CPU you're booting on. So initially and as long as you stay on the same vendor you'll have full performance. Only if you switch vendors later you'll also eat the emulation hit (and get it back again when you switch back to a system with the original vendor) But everything will still work correctly and that is the important part. Doesn't sound so bad to me. Long running guests on a large farm shouldn't expect to stay on the same host they were booted on. But at least for Linux, if we notify the guest telling it to retune (which can include the vsyscall path) we can avoid the cost entirely. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Glauber Costa wrote: We can possibly do it a little bit better by emulating in the guest directly. So the deal would be emulation both in the host and the guest. If the guest have it, fine. Otherwise, we use host emulation, that is slower, but works. If the guest is modifiable, this is easy. Include both syscall and sysenter paths in the vsyscall page, with a jmp to select between them. If we trap a #UD pointing at a syscall or sysenter, emulate it, and patch the jmp instruction to point at the other path. This way we have a self-adjusting syscall/sysenter entry point. We also need to tell kvm not to emulate the instruction, so that the #UD actually reaches the kernel. If the guest is not modifiable, this means patching. This is much harder than tpr patching since we don't have any space to hide our code in. Maybe we can reverse engineer the vsyscall page and hack it, but this may take a lot of work. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Cross vendor migration ideas
Comments inline. -Original Message- From: Andi Kleen <[EMAIL PROTECTED]> Sent: Saturday, November 15, 2008 07:03 To: Serebrin, Benjamin (Calendar) <[EMAIL PROTECTED]> Cc: Skywing <[EMAIL PROTECTED]>; Alexander Graf <[EMAIL PROTECTED]>; Anthony Liguori <[EMAIL PROTECTED]>; kvm@vger.kernel.org ; Amit Shah <[EMAIL PROTECTED]>; Avi Kivity <[EMAIL PROTECTED]>; Wahlig, Elsie <[EMAIL PROTECTED]>; Nakajima, Jun <[EMAIL PROTECTED]> Subject: Re: Cross vendor migration ideas > Also it might break user space, unless you key the fake vendor CPUID > intercept on ring 0 vs ring 3 (but even if that might not be enough > because some kernel modules can call CPUID on their own) > > I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't > be that much slower than int 0x80 hopefully. > > -Andi If I understand that idea correctly, I think that trying to present a differing view of the CPU vendor and/or feature set is likely to be fraught with peril and highly fragile. There are many systems where the kernel gets processor feature data in CPL=0 and makes it available to user mode. This creates an inconsistency when user mode uses cpuid directly and comes up with differing results. As a real world example, in Windows, the kernel is responsible for examining the current processors installed on the box, mapping processor feature codes to a standard encoding, and making this available to all user mode processes via a shared memory region (read only to user mode) at a well known location (SharedUserData->ProcessorFeatures). However, user mode programs are free to make their own determinations via cpuid calls if they so wish. Many programs use the standard abstracted processor feature determination mechanism (figured by the kernel), while others, for varying reasons (some good and some foolish) just call cpuid directly at CPL=3. There can even be a mix within different code modules loaded into a process. In the specific case of Windows and system services, IIRC the old-style int 2e system service interface is always initialized on 32-bit x86 kernels, even if sysenter or syscall will be preferred. As far as I know, this is just for backcompat with programs issuing system services directly as opposed to using the correct C-level abstraction layer. For Windows, it is CPL=0 inspection of processor attributes via cpuid (assuming the "disable fast system service interface" registry value is not set) that determines which is used. This occurs once, at system boot, as one would expect. That being said, I still think that it will break things in difficult to debug ways should you introduce an inconsistent view of cpuid features into the system. - S-- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
On Sat, Nov 15, 2008 at 2:39 PM, Alexander Graf <[EMAIL PROTECTED]> wrote: > > On 15.11.2008, at 14:03, Andi Kleen wrote: > >> "Serebrin, Benjamin (Calendar)" <[EMAIL PROTECTED]> >> writes: >>> >>> 64-bit OSes will use SYSCALL for 64-bit code. CPUID name string should >>> be "FakeCPUFakeCPU" >> >> One problem is that you'll disable a lot of vendor specific setup >> code if you do that. Might have all kinds of unexpected side effects. > > Yes, but since we're emulating a CPU anyways we don't want vendor specific > setup, since we might live migrate. And quirks should already be handled by > the host. We'd only nop them out usually. > >> Also it might break user space, unless you key the fake vendor CPUID >> intercept on ring 0 vs ring 3 (but even if that might not be enough >> because some kernel modules can call CPUID on their own) > > Is there userspace code that relies on GenuineIntel? What would a new vendor > do? Can't we just behave the same and have our self-invented vendor string > added to applications that direly need to know which vendor they're running > on? > >> I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't >> be that much slower than int 0x80 hopefully. > > Well emulating them means you're leaving the VM on every user<->kernel > transition. That's a _huge_ performance hit. I don't have the numbers, but > IIRC a roundtrip is ~3000 cycles. > > So yes, it definitely is safer. But it'll hurt performance a lot - which is > exactly what you'd want to avoid when migrating between vendors. We can possibly do it a little bit better by emulating in the guest directly. So the deal would be emulation both in the host and the guest. If the guest have it, fine. Otherwise, we use host emulation, that is slower, but works. > Alex > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Glauber Costa. "Free as in Freedom" http://glommer.net "The less confident you are, the more serious you have to act." -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Alexander Graf wrote: > > On 15.11.2008, at 14:03, Andi Kleen wrote: > >> "Serebrin, Benjamin (Calendar)" <[EMAIL PROTECTED]> >> writes: >>> >>> 64-bit OSes will use SYSCALL for 64-bit code. CPUID name string should >>> be "FakeCPUFakeCPU" >> >> One problem is that you'll disable a lot of vendor specific setup >> code if you do that. Might have all kinds of unexpected side effects. > > Yes, but since we're emulating a CPU anyways we don't want vendor > specific setup, since we might live migrate. I was mainly thinking of tuning. For example Linux selects the best suitable page copy function based on vendor information. Using the wrong page copy can have a large impact on your performance. > And quirks should already > be handled by the host. We'd only nop them out usually. > >> Also it might break user space, unless you key the fake vendor CPUID >> intercept on ring 0 vs ring 3 (but even if that might not be enough >> because some kernel modules can call CPUID on their own) > > Is there userspace code that relies on GenuineIntel? Yes. Undoubtedly also the same with others. >> I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't >> be that much slower than int 0x80 hopefully. > > Well emulating them means you're leaving the VM on every user<->kernel > transition. That's a _huge_ performance hit. I don't have the numbers, > but IIRC a roundtrip is ~3000 cycles. Depends on the CPU. Newer CPUs are faster. Also you would select the default based on which CPU you're booting on. So initially and as long as you stay on the same vendor you'll have full performance. Only if you switch vendors later you'll also eat the emulation hit (and get it back again when you switch back to a system with the original vendor) But everything will still work correctly and that is the important part. Doesn't sound so bad to me. -Andi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
On 15.11.2008, at 14:03, Andi Kleen wrote: "Serebrin, Benjamin (Calendar)" <[EMAIL PROTECTED]> writes: 64-bit OSes will use SYSCALL for 64-bit code. CPUID name string should be "FakeCPUFakeCPU" One problem is that you'll disable a lot of vendor specific setup code if you do that. Might have all kinds of unexpected side effects. Yes, but since we're emulating a CPU anyways we don't want vendor specific setup, since we might live migrate. And quirks should already be handled by the host. We'd only nop them out usually. Also it might break user space, unless you key the fake vendor CPUID intercept on ring 0 vs ring 3 (but even if that might not be enough because some kernel modules can call CPUID on their own) Is there userspace code that relies on GenuineIntel? What would a new vendor do? Can't we just behave the same and have our self-invented vendor string added to applications that direly need to know which vendor they're running on? I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't be that much slower than int 0x80 hopefully. Well emulating them means you're leaving the VM on every user<->kernel transition. That's a _huge_ performance hit. I don't have the numbers, but IIRC a roundtrip is ~3000 cycles. So yes, it definitely is safer. But it'll hurt performance a lot - which is exactly what you'd want to avoid when migrating between vendors. Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
"Serebrin, Benjamin (Calendar)" <[EMAIL PROTECTED]> writes: > > 64-bit OSes will use SYSCALL for 64-bit code. CPUID name string should > be "FakeCPUFakeCPU" One problem is that you'll disable a lot of vendor specific setup code if you do that. Might have all kinds of unexpected side effects. Also it might break user space, unless you key the fake vendor CPUID intercept on ring 0 vs ring 3 (but even if that might not be enough because some kernel modules can call CPUID on their own) I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't be that much slower than int 0x80 hopefully. -Andi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Amit, Alex, Please see my comments bellow. Avi, Please have a look at the patches, and let me know the parts you think can be done better. On Fri, 2008-11-14 at 06:07 -0700, Amit Shah wrote: > * On Thursday 13 Nov 2008 19:08:14 Alexander Graf wrote: > > On 13.11.2008, at 05:35, Amit Shah wrote: > > > * On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote: > > >> On 12.11.2008, at 17:52, Amit Shah wrote: > > >>> Hi Alex, > > >>> > > >>> * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote: > > Hi, > > > > I was thinking a bit about cross vendor migration recently and > > since > > we're doing open source development, I figured it might be a good > > idea > > to talk to everyone about this. > > > > So why are we having a problem? > > > > In normal operation we don't. If we're running a 32-bit kernel, we > > can > > use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit > > kernel with 64-bit userspace, every CPU supports SYSCALL. At least > > Linux is being smart on this and does use exactly these two > > capabilities in these two cases. > > But if we're running in compat mode (64-bit kernel with 32-bit > > userspace), things differ. Intel supports only SYSENTER here, while > > AMD only supports SYSCALL. Both can still use int80. > > > > Operating systems detect usage of SYSCALL or SYSENTER pretty > > early on > > (Linux does this on vdso). So when we boot up on an Intel machine, > > Linux assumes that using SYSENTER in compat mode is fine. Migrating > > that machine to an AMD machine breaks this assumption though, since > > SYSENTER can't be used in compat mode. > > On LInux, this detection is based on the CPU vendor string. If > > Linux > > finds a "GenuineIntel", SYSENTER is used in compat mode, if it's > > "AuthenticAMD", SYSCALL is used and if none of these two is found, > > int80 is used. > > > > I tried modifying the vendor string, removed the "overwrite the > > vendor > > string with the native string" hack and things look like they work > > just fine with Linux. > > > > Unfortunately right now I don't have a 64-bit Windows installation > > around to check if that approach works there too, but if it does > > and > > no known OS breaks due to the invalid vendor string, we can just > > create our own virtual CPU string, no? > > >>> > > >>> qemu has an option for that, -cpu qemu64 IIRC. As long as we expose > > >>> practically correct cpuids and MSRs, this should be fine. I've not > > >>> tested > > >>> qemu64 with winxp x64 though. Also, last I knew, winxp x64 > > >>> installation > > >>> didn't succeed with --no-kvm. qemu by default exposes an AMD CPU > > >>> type. > > >> > > >> I wasn't talking about CPUID features, but the vendor string. Qemu64 > > >> provides the AuthenticAMD string, so we don't run into any issues I'm > > >> presuming. > > > > > > Right -- the thing is, with the default AuthenticAMD string, winp x64 > > > installation fails. That has to be because of some missing cpuids. > > > That's one > > > of the drawbacks of exposing a well-known CPU type. I was suggesting > > > we > > > should try out the -cpu qemu64 CPU type since it exposes a non- > > > standard CPU > > > to see if guests and most userspace programs work fine without any > > > further > > > tweaking -- see the 'cons' below for why this might be a problem. > > > > I still don't really understand what you're trying to say - qemu64 is > > the default in KVM right now. You mean winxp64 installation doesn't > No, the default for KVM is the host CPU type. Amit, Aliex is correct. the default cpu for kvm is qemu64 not the host. I have sent the patches to add an options -cpu host. Some of the patches are gone in, But All the patches are not in yet. Also my patches does not make the host option as default. I have attached the remaining two patches. Alex, can you try these patches with "-cpu host" option and see if you can get the host vendor string in the guest for AMD box. I have already tested it on the latest Intel system. > > > work as is and we should fix it? This has nothing to do with the > > migration problems, right? > > Solutions shouldn't involve adding known regressions. If our default cpu type > changes to one that renders some of the OSes we support right now to become > nonfunctional, such changes won't be accepted. Of course, we can improve the > qemu64 cpu type to ensure the popular OS types work properly at the least. > > > >>> There are pros and cons to expose a custom vendor ID: > > >>> > > >>> pros: > > >>> - We don't need to have all the cpuid features exposed which are > > >>> expected of a > > >>> physically available CPU in the market, for example, badly-coded > > >>> applications > > >>> might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or > > >>> not,
Re: Cross vendor migration ideas
* On Thursday 13 Nov 2008 19:08:14 Alexander Graf wrote: > On 13.11.2008, at 05:35, Amit Shah wrote: > > * On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote: > >> On 12.11.2008, at 17:52, Amit Shah wrote: > >>> Hi Alex, > >>> > >>> * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote: > Hi, > > I was thinking a bit about cross vendor migration recently and > since > we're doing open source development, I figured it might be a good > idea > to talk to everyone about this. > > So why are we having a problem? > > In normal operation we don't. If we're running a 32-bit kernel, we > can > use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit > kernel with 64-bit userspace, every CPU supports SYSCALL. At least > Linux is being smart on this and does use exactly these two > capabilities in these two cases. > But if we're running in compat mode (64-bit kernel with 32-bit > userspace), things differ. Intel supports only SYSENTER here, while > AMD only supports SYSCALL. Both can still use int80. > > Operating systems detect usage of SYSCALL or SYSENTER pretty > early on > (Linux does this on vdso). So when we boot up on an Intel machine, > Linux assumes that using SYSENTER in compat mode is fine. Migrating > that machine to an AMD machine breaks this assumption though, since > SYSENTER can't be used in compat mode. > On LInux, this detection is based on the CPU vendor string. If > Linux > finds a "GenuineIntel", SYSENTER is used in compat mode, if it's > "AuthenticAMD", SYSCALL is used and if none of these two is found, > int80 is used. > > I tried modifying the vendor string, removed the "overwrite the > vendor > string with the native string" hack and things look like they work > just fine with Linux. > > Unfortunately right now I don't have a 64-bit Windows installation > around to check if that approach works there too, but if it does > and > no known OS breaks due to the invalid vendor string, we can just > create our own virtual CPU string, no? > >>> > >>> qemu has an option for that, -cpu qemu64 IIRC. As long as we expose > >>> practically correct cpuids and MSRs, this should be fine. I've not > >>> tested > >>> qemu64 with winxp x64 though. Also, last I knew, winxp x64 > >>> installation > >>> didn't succeed with --no-kvm. qemu by default exposes an AMD CPU > >>> type. > >> > >> I wasn't talking about CPUID features, but the vendor string. Qemu64 > >> provides the AuthenticAMD string, so we don't run into any issues I'm > >> presuming. > > > > Right -- the thing is, with the default AuthenticAMD string, winp x64 > > installation fails. That has to be because of some missing cpuids. > > That's one > > of the drawbacks of exposing a well-known CPU type. I was suggesting > > we > > should try out the -cpu qemu64 CPU type since it exposes a non- > > standard CPU > > to see if guests and most userspace programs work fine without any > > further > > tweaking -- see the 'cons' below for why this might be a problem. > > I still don't really understand what you're trying to say - qemu64 is > the default in KVM right now. You mean winxp64 installation doesn't No, the default for KVM is the host CPU type. > work as is and we should fix it? This has nothing to do with the > migration problems, right? Solutions shouldn't involve adding known regressions. If our default cpu type changes to one that renders some of the OSes we support right now to become nonfunctional, such changes won't be accepted. Of course, we can improve the qemu64 cpu type to ensure the popular OS types work properly at the least. > >>> There are pros and cons to expose a custom vendor ID: > >>> > >>> pros: > >>> - We don't need to have all the cpuid features exposed which are > >>> expected of a > >>> physically available CPU in the market, for example, badly-coded > >>> applications > >>> might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or > >>> not, not > >>> exposing what's actually available on every C2D out there is bad. > >>> > >>> cons: > >>> - To expose the "correct" set of feature bits for a known processor, > >>> we also > >>> need to check the family/model/stepping to support the exact same > >>> feature > >>> bits that were present in the CPU. > >>> - We might not get some optimizations that OSes might have based on > >>> CPU type, > >>> even if the host CPU qualifies for such optimizations > >>> - Standard programs like benchmarking tools, etc., might fail if > >>> they depend > >>> on the vendor string for their functionality > >>> > >>> For 32-bit guests, I think exposing a pentium4 or Athlon CPU type > >>> should be > >>> fine. For 64-bit guests, the newer the better. > >> > >> Well, we could create different CPU definitions: > >> > >> - migration safe (do what is safe for migration
Re: Cross vendor migration ideas
On 13.11.2008, at 05:35, Amit Shah wrote: * On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote: On 12.11.2008, at 17:52, Amit Shah wrote: Hi Alex, * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote: Hi, I was thinking a bit about cross vendor migration recently and since we're doing open source development, I figured it might be a good idea to talk to everyone about this. So why are we having a problem? In normal operation we don't. If we're running a 32-bit kernel, we can use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit kernel with 64-bit userspace, every CPU supports SYSCALL. At least Linux is being smart on this and does use exactly these two capabilities in these two cases. But if we're running in compat mode (64-bit kernel with 32-bit userspace), things differ. Intel supports only SYSENTER here, while AMD only supports SYSCALL. Both can still use int80. Operating systems detect usage of SYSCALL or SYSENTER pretty early on (Linux does this on vdso). So when we boot up on an Intel machine, Linux assumes that using SYSENTER in compat mode is fine. Migrating that machine to an AMD machine breaks this assumption though, since SYSENTER can't be used in compat mode. On LInux, this detection is based on the CPU vendor string. If Linux finds a "GenuineIntel", SYSENTER is used in compat mode, if it's "AuthenticAMD", SYSCALL is used and if none of these two is found, int80 is used. I tried modifying the vendor string, removed the "overwrite the vendor string with the native string" hack and things look like they work just fine with Linux. Unfortunately right now I don't have a 64-bit Windows installation around to check if that approach works there too, but if it does and no known OS breaks due to the invalid vendor string, we can just create our own virtual CPU string, no? qemu has an option for that, -cpu qemu64 IIRC. As long as we expose practically correct cpuids and MSRs, this should be fine. I've not tested qemu64 with winxp x64 though. Also, last I knew, winxp x64 installation didn't succeed with --no-kvm. qemu by default exposes an AMD CPU type. I wasn't talking about CPUID features, but the vendor string. Qemu64 provides the AuthenticAMD string, so we don't run into any issues I'm presuming. Right -- the thing is, with the default AuthenticAMD string, winp x64 installation fails. That has to be because of some missing cpuids. That's one of the drawbacks of exposing a well-known CPU type. I was suggesting we should try out the -cpu qemu64 CPU type since it exposes a non- standard CPU to see if guests and most userspace programs work fine without any further tweaking -- see the 'cons' below for why this might be a problem. I still don't really understand what you're trying to say - qemu64 is the default in KVM right now. You mean winxp64 installation doesn't work as is and we should fix it? This has nothing to do with the migration problems, right? There are pros and cons to expose a custom vendor ID: pros: - We don't need to have all the cpuid features exposed which are expected of a physically available CPU in the market, for example, badly-coded applications might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or not, not exposing what's actually available on every C2D out there is bad. cons: - To expose the "correct" set of feature bits for a known processor, we also need to check the family/model/stepping to support the exact same feature bits that were present in the CPU. - We might not get some optimizations that OSes might have based on CPU type, even if the host CPU qualifies for such optimizations - Standard programs like benchmarking tools, etc., might fail if they depend on the vendor string for their functionality For 32-bit guests, I think exposing a pentium4 or Athlon CPU type should be fine. For 64-bit guests, the newer the better. Well, we could create different CPU definitions: - migration safe (do what is safe for migration) There are multiple ways of approaching this: peg to a least-known good CPU type, all of whose instructions will work on processors from both the major vendors. However, you never know how the server pools change and you'd want to upgrade the CPU type once you know the CPUs that are installed in servers. This has to be dynamic and the management application has to take care of exposing a CPU that's of a "safe" type for the particular server pool. We have to provide ways to mask off CPUID bits as requested by the management application. (Each server sends its cpuid to the management application, which calculates the safest bits and then conveys this to each server before starting a VM.) IMHO we shouldn't really start to be smart here. There's only so much benefit in using the least common dominator between all CPUs in the datacenter vs. using the least common dominator between all possible CPUs. You'll basically end up enabling some newer
Re: Cross vendor migration ideas
Anthony Liguori wrote: I wonder if patching is an option? You would need to patch the syscall/sysenter and sysret/sysexit sites, as well as the kernel entry points, since the calling conventions are dissimilar. Finding a place to put the stubs would be difficult, at best. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Alexander Graf wrote: I wonder if patching is an option? Windows does have background daemons that check code in runtime and compares that to checksums. So binary patching might break Windows pretty easily. I'm really wondering why the CR8 patching still works - maybe even that'll break with Windows 7. kvm doesn't patch Vista since the bios isn't mapped. Vista doesn't bang heavily on the TPR like XP does so patching is not needed anyway. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
* On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote: > On 12.11.2008, at 17:52, Amit Shah wrote: > > Hi Alex, > > > > * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote: > >> Hi, > >> > >> I was thinking a bit about cross vendor migration recently and since > >> we're doing open source development, I figured it might be a good > >> idea > >> to talk to everyone about this. > >> > >> So why are we having a problem? > >> > >> In normal operation we don't. If we're running a 32-bit kernel, we > >> can > >> use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit > >> kernel with 64-bit userspace, every CPU supports SYSCALL. At least > >> Linux is being smart on this and does use exactly these two > >> capabilities in these two cases. > >> But if we're running in compat mode (64-bit kernel with 32-bit > >> userspace), things differ. Intel supports only SYSENTER here, while > >> AMD only supports SYSCALL. Both can still use int80. > >> > >> Operating systems detect usage of SYSCALL or SYSENTER pretty early on > >> (Linux does this on vdso). So when we boot up on an Intel machine, > >> Linux assumes that using SYSENTER in compat mode is fine. Migrating > >> that machine to an AMD machine breaks this assumption though, since > >> SYSENTER can't be used in compat mode. > >> On LInux, this detection is based on the CPU vendor string. If Linux > >> finds a "GenuineIntel", SYSENTER is used in compat mode, if it's > >> "AuthenticAMD", SYSCALL is used and if none of these two is found, > >> int80 is used. > >> > >> I tried modifying the vendor string, removed the "overwrite the > >> vendor > >> string with the native string" hack and things look like they work > >> just fine with Linux. > >> > >> Unfortunately right now I don't have a 64-bit Windows installation > >> around to check if that approach works there too, but if it does and > >> no known OS breaks due to the invalid vendor string, we can just > >> create our own virtual CPU string, no? > > > > qemu has an option for that, -cpu qemu64 IIRC. As long as we expose > > practically correct cpuids and MSRs, this should be fine. I've not > > tested > > qemu64 with winxp x64 though. Also, last I knew, winxp x64 > > installation > > didn't succeed with --no-kvm. qemu by default exposes an AMD CPU type. > > I wasn't talking about CPUID features, but the vendor string. Qemu64 > provides the AuthenticAMD string, so we don't run into any issues I'm > presuming. Right -- the thing is, with the default AuthenticAMD string, winp x64 installation fails. That has to be because of some missing cpuids. That's one of the drawbacks of exposing a well-known CPU type. I was suggesting we should try out the -cpu qemu64 CPU type since it exposes a non-standard CPU to see if guests and most userspace programs work fine without any further tweaking -- see the 'cons' below for why this might be a problem. > > There are pros and cons to expose a custom vendor ID: > > > > pros: > > - We don't need to have all the cpuid features exposed which are > > expected of a > > physically available CPU in the market, for example, badly-coded > > applications > > might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or > > not, not > > exposing what's actually available on every C2D out there is bad. > > > > cons: > > - To expose the "correct" set of feature bits for a known processor, > > we also > > need to check the family/model/stepping to support the exact same > > feature > > bits that were present in the CPU. > > - We might not get some optimizations that OSes might have based on > > CPU type, > > even if the host CPU qualifies for such optimizations > > - Standard programs like benchmarking tools, etc., might fail if > > they depend > > on the vendor string for their functionality > > > > For 32-bit guests, I think exposing a pentium4 or Athlon CPU type > > should be > > fine. For 64-bit guests, the newer the better. > > Well, we could create different CPU definitions: > > - migration safe (do what is safe for migration) There are multiple ways of approaching this: peg to a least-known good CPU type, all of whose instructions will work on processors from both the major vendors. However, you never know how the server pools change and you'd want to upgrade the CPU type once you know the CPUs that are installed in servers. This has to be dynamic and the management application has to take care of exposing a CPU that's of a "safe" type for the particular server pool. We have to provide ways to mask off CPUID bits as requested by the management application. (Each server sends its cpuid to the management application, which calculates the safest bits and then conveys this to each server before starting a VM.) > - CPU specific (like a Core2Duo, necessary to run Mac OS X) This doesn't need any more work -- we already have the ability to select CPU types. If the management application has knowledge of the kind of OS being installed in a VM (which thes
RE: Cross vendor migration ideas
This seems like conveniently good news. Combining all the ideas: Hide CPUID.SYSCALL to force 32-bit OSes to use 32-bit legacy SYSENTER, which is available on AMD and Intel. 64-bit OSes will use SYSCALL for 64-bit code. CPUID name string should be "FakeCPUFakeCPU" or some such so a 64-bit OS will avoid SYSCALL and SYSENTER for 32-bit code. A 64-bit Linux will use int80 for its 32-bit code, by Alex's observation. A 64-bit Windows will use a thunk that gets to long mode before using SYSCALL, by Skywing's observation. Intercept #UD and emulate any of the SYSENTER, SYSCALL, SYSRET, SYSEXIT instructions that fall through the cracks. I think this addresses Alex's observations about MacOS/Darwin's use of SYSENTER. This leaves you with a hypervisor that doesn't have to know a priori what the guest is going to do and can do nearly all its tricks through CPUID control. The only unsatisfying piece is anything high-frequency that the hypervisor ends up having to emulate. Maybe in-guest patching works on Darwin if it doesn't have any intrusion detection going on? Ben -Original Message- From: Skywing [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 12, 2008 4:02 PM To: Alexander Graf; Anthony Liguori Cc: kvm@vger.kernel.org list; Amit Shah; Avi Kivity; Wahlig, Elsie; Serebrin, Benjamin (Calendar); Nakajima, Jun Subject: RE: Cross vendor migration ideas Hi, -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alexander Graf Sent: Wednesday, November 12, 2008 10:50 AM To: Anthony Liguori Cc: kvm@vger.kernel.org list; Amit Shah; Avi Kivity; Elsie Wahlig; Serebrin, Benjamin (Calendar); Nakajima, Jun > Windows does have background daemons that check code in runtime and > compares that to checksums. So binary patching might break Windows > pretty easily. I'm really wondering why the CR8 patching still works - > maybe even that'll break with Windows 7. > Alex If you're thinking of PatchGuard, that's only there on amd64 Windows, not 32-bit x86. However, I think that patching is likely to cause other issues in general and should be avoided. BTW, 32-bit [x86] programs on amd64 Windows will all eventually get thunked through the 64-bit system call interface, which is always (hardcoded) syscall for amd64 Windows. It is only if you are running a 32-bit (x86) Windows kernel that you may optionally use one of [sysenter, syscall, int 2e]. In other words, if you are running a 64-bit Windows kernel, even the 32-bit parts will always pick syscall regardless of the reported processor feature set. The 32-bit binaries that make the system call transitions to kernel mode are different if you are running a 64-bit Windows kernel versus a 32-bit kernel; the code to differentiate which mechanism to use is removed. (In fact, to be specific, it is always 64-bit code that issues the actual syscall instruction, as the call path in the Wow64 [32-bit Windows program running under an amd64 Windows kernel] case is 32-bit program code -> 32-bit ntdll.dll -> 64-bit wow64.dll -> 64-bit ntdll.dll, where 64-bit ntdll.dll is the system-supplied binary that makes the actual system service calls. The switch from 32-bit mode to 64-bit mode in amd64 Windows is done in user mode. All system call transitions are thus done from 64-bit mode in the 64-bit Windows case. This sequence is independent of CPU vendor.) You could probably get away with not claiming support for sysenter/syscall for 32-bit guests, as Windows will degrade gracefully to int 2e in that case for guests running a 32-bit kernel. Performance implications for that still apply of course. - S -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Cross vendor migration ideas
Hi, -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alexander Graf Sent: Wednesday, November 12, 2008 10:50 AM To: Anthony Liguori Cc: kvm@vger.kernel.org list; Amit Shah; Avi Kivity; Elsie Wahlig; Serebrin, Benjamin (Calendar); Nakajima, Jun > Windows does have background daemons that check code in runtime and > compares that to checksums. So binary patching might break Windows > pretty easily. I'm really wondering why the CR8 patching still works - > maybe even that'll break with Windows 7. > Alex If you're thinking of PatchGuard, that's only there on amd64 Windows, not 32-bit x86. However, I think that patching is likely to cause other issues in general and should be avoided. BTW, 32-bit [x86] programs on amd64 Windows will all eventually get thunked through the 64-bit system call interface, which is always (hardcoded) syscall for amd64 Windows. It is only if you are running a 32-bit (x86) Windows kernel that you may optionally use one of [sysenter, syscall, int 2e]. In other words, if you are running a 64-bit Windows kernel, even the 32-bit parts will always pick syscall regardless of the reported processor feature set. The 32-bit binaries that make the system call transitions to kernel mode are different if you are running a 64-bit Windows kernel versus a 32-bit kernel; the code to differentiate which mechanism to use is removed. (In fact, to be specific, it is always 64-bit code that issues the actual syscall instruction, as the call path in the Wow64 [32-bit Windows program running under an amd64 Windows kernel] case is 32-bit program code -> 32-bit ntdll.dll -> 64-bit wow64.dll -> 64-bit ntdll.dll, where 64-bit ntdll.dll is the system-supplied binary that makes the actual system service calls. The switch from 32-bit mode to 64-bit mode in amd64 Windows is done in user mode. All system call transitions are thus done from 64-bit mode in the 64-bit Windows case. This sequence is independent of CPU vendor.) You could probably get away with not claiming support for sysenter/syscall for 32-bit guests, as Windows will degrade gracefully to int 2e in that case for guests running a 32-bit kernel. Performance implications for that still apply of course. - S -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
On 12.11.2008, at 21:06, Andi Kleen wrote:
Anthony Liguori <[EMAIL PROTECTED]> writes:
I wonder if patching is an option?
Not by patching instructions because SYSENTER and SYSCALL have
completely different semantics (e.g. regarding register use). That is
why the entry paths are different on both sides.
In theory you could switch the entry paths around, but that would
be highly kernel version specific.
One easy way is to just clear the SEP and SYSCALL32 flags
in CPUID, then int 0x80 will be always used. Of course it'll
cost you in performance significantly. I wouldn't recommend
to make it default.
SEP and SYSCALL CPUIDs are only respected on X86_32. X86_64 just
ignores them and sets X86_FEATURE_SYS{CALL,ENTER}32 in intel.c / amd.c.
That's why I'm suggesting the vendor hack, because Linux doesn't
respect the CPUID flags on x86_64.
Alex
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Anthony Liguori <[EMAIL PROTECTED]> writes: > > I wonder if patching is an option? Not by patching instructions because SYSENTER and SYSCALL have completely different semantics (e.g. regarding register use). That is why the entry paths are different on both sides. In theory you could switch the entry paths around, but that would be highly kernel version specific. One easy way is to just clear the SEP and SYSCALL32 flags in CPUID, then int 0x80 will be always used. Of course it'll cost you in performance significantly. I wouldn't recommend to make it default. -Andi -- [EMAIL PROTECTED] -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
On 12.11.2008, at 17:52, Amit Shah wrote: Hi Alex, * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote: Hi, I was thinking a bit about cross vendor migration recently and since we're doing open source development, I figured it might be a good idea to talk to everyone about this. So why are we having a problem? In normal operation we don't. If we're running a 32-bit kernel, we can use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit kernel with 64-bit userspace, every CPU supports SYSCALL. At least Linux is being smart on this and does use exactly these two capabilities in these two cases. But if we're running in compat mode (64-bit kernel with 32-bit userspace), things differ. Intel supports only SYSENTER here, while AMD only supports SYSCALL. Both can still use int80. Operating systems detect usage of SYSCALL or SYSENTER pretty early on (Linux does this on vdso). So when we boot up on an Intel machine, Linux assumes that using SYSENTER in compat mode is fine. Migrating that machine to an AMD machine breaks this assumption though, since SYSENTER can't be used in compat mode. On LInux, this detection is based on the CPU vendor string. If Linux finds a "GenuineIntel", SYSENTER is used in compat mode, if it's "AuthenticAMD", SYSCALL is used and if none of these two is found, int80 is used. I tried modifying the vendor string, removed the "overwrite the vendor string with the native string" hack and things look like they work just fine with Linux. Unfortunately right now I don't have a 64-bit Windows installation around to check if that approach works there too, but if it does and no known OS breaks due to the invalid vendor string, we can just create our own virtual CPU string, no? qemu has an option for that, -cpu qemu64 IIRC. As long as we expose practically correct cpuids and MSRs, this should be fine. I've not tested qemu64 with winxp x64 though. Also, last I knew, winxp x64 installation didn't succeed with --no-kvm. qemu by default exposes an AMD CPU type. I wasn't talking about CPUID features, but the vendor string. Qemu64 provides the AuthenticAMD string, so we don't run into any issues I'm presuming. There are pros and cons to expose a custom vendor ID: pros: - We don't need to have all the cpuid features exposed which are expected of a physically available CPU in the market, for example, badly-coded applications might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or not, not exposing what's actually available on every C2D out there is bad. cons: - To expose the "correct" set of feature bits for a known processor, we also need to check the family/model/stepping to support the exact same feature bits that were present in the CPU. - We might not get some optimizations that OSes might have based on CPU type, even if the host CPU qualifies for such optimizations - Standard programs like benchmarking tools, etc., might fail if they depend on the vendor string for their functionality For 32-bit guests, I think exposing a pentium4 or Athlon CPU type should be fine. For 64-bit guests, the newer the better. Well, we could create different CPU definitions: - migration safe (do what is safe for migration) - CPU specific (like a Core2Duo, necessary to run Mac OS X) - host (fastest possible, but no migration) I don't think we could find one definition that fits all, so the user would have to define what the usage pattern will be. I'd love to hear comments and suggestions on this and hope we'll end up in a fruitful discussion on how to improve the current situation. I have a patch ready for emulating sysenter/sysexit on AMD systems (needs testing). Patching the guest was an option that was discouraged; I had a hack ready but it was quickly shelved (again, untested). That sounds useful for misbehaving guests or cases I haven't thought of yet. Are you sure you're intercepting the SYSENTER MSRs on AMD, so you don't end up only getting 32 bits? Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Hi Alex, * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote: > Hi, > > I was thinking a bit about cross vendor migration recently and since > we're doing open source development, I figured it might be a good idea > to talk to everyone about this. > > So why are we having a problem? > > In normal operation we don't. If we're running a 32-bit kernel, we can > use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit > kernel with 64-bit userspace, every CPU supports SYSCALL. At least > Linux is being smart on this and does use exactly these two > capabilities in these two cases. > But if we're running in compat mode (64-bit kernel with 32-bit > userspace), things differ. Intel supports only SYSENTER here, while > AMD only supports SYSCALL. Both can still use int80. > > Operating systems detect usage of SYSCALL or SYSENTER pretty early on > (Linux does this on vdso). So when we boot up on an Intel machine, > Linux assumes that using SYSENTER in compat mode is fine. Migrating > that machine to an AMD machine breaks this assumption though, since > SYSENTER can't be used in compat mode. > On LInux, this detection is based on the CPU vendor string. If Linux > finds a "GenuineIntel", SYSENTER is used in compat mode, if it's > "AuthenticAMD", SYSCALL is used and if none of these two is found, > int80 is used. > > I tried modifying the vendor string, removed the "overwrite the vendor > string with the native string" hack and things look like they work > just fine with Linux. > > Unfortunately right now I don't have a 64-bit Windows installation > around to check if that approach works there too, but if it does and > no known OS breaks due to the invalid vendor string, we can just > create our own virtual CPU string, no? qemu has an option for that, -cpu qemu64 IIRC. As long as we expose practically correct cpuids and MSRs, this should be fine. I've not tested qemu64 with winxp x64 though. Also, last I knew, winxp x64 installation didn't succeed with --no-kvm. qemu by default exposes an AMD CPU type. There are pros and cons to expose a custom vendor ID: pros: - We don't need to have all the cpuid features exposed which are expected of a physically available CPU in the market, for example, badly-coded applications might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or not, not exposing what's actually available on every C2D out there is bad. cons: - To expose the "correct" set of feature bits for a known processor, we also need to check the family/model/stepping to support the exact same feature bits that were present in the CPU. - We might not get some optimizations that OSes might have based on CPU type, even if the host CPU qualifies for such optimizations - Standard programs like benchmarking tools, etc., might fail if they depend on the vendor string for their functionality For 32-bit guests, I think exposing a pentium4 or Athlon CPU type should be fine. For 64-bit guests, the newer the better. > I'd love to hear comments and suggestions on this and hope we'll end > up in a fruitful discussion on how to improve the current situation. I have a patch ready for emulating sysenter/sysexit on AMD systems (needs testing). Patching the guest was an option that was discouraged; I had a hack ready but it was quickly shelved (again, untested). Amit -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Alexander Graf wrote: On 12.11.2008, at 16:45, Anthony Liguori wrote: Alexander Graf wrote: Hi, I was thinking a bit about cross vendor migration recently and since we're doing open source development, I figured it might be a good idea to talk to everyone about this. So why are we having a problem? In normal operation we don't. If we're running a 32-bit kernel, we can use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit kernel with 64-bit userspace, every CPU supports SYSCALL. At least Linux is being smart on this and does use exactly these two capabilities in these two cases. But if we're running in compat mode (64-bit kernel with 32-bit userspace), things differ. Intel supports only SYSENTER here, while AMD only supports SYSCALL. Both can still use int80. Obviously we can trap-and-emulate but that would be slow in a relatively fast past. If we can do it without emulation, I'd greatly prefer it, as syscall/sysenter emulation in the hypervisor most probably isn't exactly fast ;-). And you don't really want to degrade performance just because you're migrating (think flashplayer here). I guess Windows 64-bit contains even more 32-bit parts. I wonder if patching is an option? Windows does have background daemons that check code in runtime and compares that to checksums. So binary patching might break Windows pretty easily. I'm really wondering why the CR8 patching still works - maybe even that'll break with Windows 7. The Hyper-V spec claims that the accesses to the TPR will always be in 5 byte instructions. Presumably this is to allow binary patching to be reliable. I'm pretty sure newer versions of Windows don't access the TPR so frequently or at least access it through CR8. With VT on modern processors, TPR patching is unnecessary so as long as they use CR8 on AMD processors, all should be okay. Although here is obviously another spot where cross vendor migration could be problematic :-) Regards, Anthony Liguori Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
On 12.11.2008, at 16:45, Anthony Liguori wrote: Alexander Graf wrote: Hi, I was thinking a bit about cross vendor migration recently and since we're doing open source development, I figured it might be a good idea to talk to everyone about this. So why are we having a problem? In normal operation we don't. If we're running a 32-bit kernel, we can use SYSENTER to jump from kernel<->userspace. If we're on a 64- bit kernel with 64-bit userspace, every CPU supports SYSCALL. At least Linux is being smart on this and does use exactly these two capabilities in these two cases. But if we're running in compat mode (64-bit kernel with 32-bit userspace), things differ. Intel supports only SYSENTER here, while AMD only supports SYSCALL. Both can still use int80. Obviously we can trap-and-emulate but that would be slow in a relatively fast past. If we can do it without emulation, I'd greatly prefer it, as syscall/ sysenter emulation in the hypervisor most probably isn't exactly fast ;-). And you don't really want to degrade performance just because you're migrating (think flashplayer here). I guess Windows 64- bit contains even more 32-bit parts. I wonder if patching is an option? Windows does have background daemons that check code in runtime and compares that to checksums. So binary patching might break Windows pretty easily. I'm really wondering why the CR8 patching still works - maybe even that'll break with Windows 7. Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Cross vendor migration ideas
Alexander Graf wrote: Hi, I was thinking a bit about cross vendor migration recently and since we're doing open source development, I figured it might be a good idea to talk to everyone about this. So why are we having a problem? In normal operation we don't. If we're running a 32-bit kernel, we can use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit kernel with 64-bit userspace, every CPU supports SYSCALL. At least Linux is being smart on this and does use exactly these two capabilities in these two cases. But if we're running in compat mode (64-bit kernel with 32-bit userspace), things differ. Intel supports only SYSENTER here, while AMD only supports SYSCALL. Both can still use int80. Obviously we can trap-and-emulate but that would be slow in a relatively fast past. I wonder if patching is an option? Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Cross vendor migration ideas
Hi, I was thinking a bit about cross vendor migration recently and since we're doing open source development, I figured it might be a good idea to talk to everyone about this. So why are we having a problem? In normal operation we don't. If we're running a 32-bit kernel, we can use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit kernel with 64-bit userspace, every CPU supports SYSCALL. At least Linux is being smart on this and does use exactly these two capabilities in these two cases. But if we're running in compat mode (64-bit kernel with 32-bit userspace), things differ. Intel supports only SYSENTER here, while AMD only supports SYSCALL. Both can still use int80. Operating systems detect usage of SYSCALL or SYSENTER pretty early on (Linux does this on vdso). So when we boot up on an Intel machine, Linux assumes that using SYSENTER in compat mode is fine. Migrating that machine to an AMD machine breaks this assumption though, since SYSENTER can't be used in compat mode. On LInux, this detection is based on the CPU vendor string. If Linux finds a "GenuineIntel", SYSENTER is used in compat mode, if it's "AuthenticAMD", SYSCALL is used and if none of these two is found, int80 is used. I tried modifying the vendor string, removed the "overwrite the vendor string with the native string" hack and things look like they work just fine with Linux. Unfortunately right now I don't have a 64-bit Windows installation around to check if that approach works there too, but if it does and no known OS breaks due to the invalid vendor string, we can just create our own virtual CPU string, no? I'd love to hear comments and suggestions on this and hope we'll end up in a fruitful discussion on how to improve the current situation. Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
