Re: About the kvm-no-adjvtime CPU property
On 6/3/2020 4:53 PM, Andrew Jones wrote: On Tue, Jun 02, 2020 at 03:47:22PM +0800, Ying Fang wrote: On 2020/6/1 20:29, Andrew Jones wrote: On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote: On 2020/6/1 16:07, Andrew Jones wrote: On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote: About the kvm-no-adjvtime CPU property Hi Andrew, To adjust virutal time, a new kvm cpu property kvm-no-adjvtime was introduced to 5.0 virt machine types. However the cpu property was enabled only for host-passthrough and max cpu model. As for other cpu model like cortex-a57, cortex-a53, cortex-a72, this kvm-adjvtime is not enabled by default, which means the virutal time can not be adjust for them. Here, for example, if VM is configured with kvm enabled: cortex-a72 We cannot adjust virtual time even if 5.0 virt machine is used. So i'd like to add it to other cpu model, do you have any suggestions here ? Hi Fang, The cpu feature only requires kvm. If a cpu model may be used with kvm, then the feature can be allowed to be used with the model. What I find interesting is that the cpu model is being used with kvm instead of 'host' or 'max'. Can you explain the reasons for that? Currently, when using yes，the cpu model is indeed used with kvm. There is a situation where the host cpu model is Cortex-A72 and a 'custom' cpu mode is used to keep insrtuction set compatible between the source and destination host machine when doing live migration. So the host physical machine cpu model is Cortex-A72 but host-passthrough model is mode used here. I mean host-passthrough model is 'not' used here. Sorry to make it confusing. I guessed as much. Are the source and destinations hosts used in the migration identical? If so, then the guest can use cpu 'host' and disable cpu features that should not be exposed (e.g. -cpu host,pmu=off). If the source and destination hosts are not identical, then I'm curious what those exact differences are. With the way AArch64 KVM works today, even using the Cortex-A72 cpu model should require identical hosts when migrating. Or, at least both hosts must be compatible with Cortex-A72 and any difference in ID registers must be somehow getting hidden from the guest. Yes, you are right. We have AAarch64 server with cpu based on Cortex-A72 and some extra instruction set added. Source host with cpu based on V1 and destination host with cpu based on V2 and they are compatible with Cortex-A72. We want to use a 'custom' cpu mode here to make it possible to do live migration between them. This is the scenario where the 'host' cpu model is not used since a 'custom' cpu model Cortex-A72 is used here . What you've described here is indeed the reason to use CPU models. I.e. enabling the migration from a host of one type to another by creating a guest that only enables the features contained in both hosts (as well as maintaining all state that describes the CPU type, e.g. MIDR). Unfortunately, unless your KVM has patches that aren't upstream, then that doesn't work on AArch64 KVM (more on that below). It may appear to be working for you, because your guest kernel and userspace don't mind the slight differences exposed to it between the hosts, or those differences are limited to explicitly disabled features. If that's the case, then I would guess that using '-cpu host' and disabling the same features would "work" as well. Yes, upstream KVM currently does not support it. We are planning to add support for the aarch64 platform since we have the situation where it is needed for our hardware. @Marc Zyngier, is there anyone who doing on this? Here's some more details on why the Cortex-A72 CPU model doesn't matter with upstream KVM. First, upstream AArch64 KVM doesn't support CPU models, and it doesn't even have a Cortex-A72 preferred target. For Cortex-A72 it will use "KVM_ARM_TARGET_GENERIC_V8", which is the same thing 'host' would do when running on a Cortex-A72. Second, if V2 of the Cortex-A72- based CPU you're using changed the revision of the MIDR, or any other state that gets passed directly through to the guest like the MIDR, then that state will change on migration. If a guest looks before migration and again after migration, then it could get confused. A guest kernel may only look once on boot and then not notice, but anything exposed to userspace is extra risky, as userspace may check more frequently. Yes, just as explained here. In short, without KVM patches that aren't upstream, then it's risky to migrate between machines with V1 and V2 of these CPUs. And, it doesn't help to use the Cortex-A72 CPU model. Thanks for your detailed introduction. Thanks, drew However the kvm-adjvtime feature is also need.
Re: About the kvm-no-adjvtime CPU property
On Tue, Jun 02, 2020 at 03:47:22PM +0800, Ying Fang wrote: > > > On 2020/6/1 20:29, Andrew Jones wrote: > > On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote: > > > > > > > > > On 2020/6/1 16:07, Andrew Jones wrote: > > > > On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote: > > > > > About the kvm-no-adjvtime CPU property > > > > > > > > > > Hi Andrew, > > > > > To adjust virutal time, a new kvm cpu property kvm-no-adjvtime > > > > > was introduced to 5.0 virt machine types. However the cpu > > > > > property was enabled only for host-passthrough and max cpu model. > > > > > As for other cpu model like cortex-a57, cortex-a53, cortex-a72, > > > > > this kvm-adjvtime is not enabled by default, which means the > > > > > virutal time can not be adjust for them. > > > > > > > > > > Here, for example, if VM is configured with kvm enabled: > > > > > > > > > > > > > > > cortex-a72 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We cannot adjust virtual time even if 5.0 virt machine is used. > > > > > So i'd like to add it to other cpu model, do you have any > > > > > suggestions here ? > > > > > > > > > > > > > > > > > > Hi Fang, > > > > > > > > The cpu feature only requires kvm. If a cpu model may be used with kvm, > > > > then the feature can be allowed to be used with the model. What I find > > > > interesting is that the cpu model is being used with kvm instead of > > > > 'host' > > > > or 'max'. Can you explain the reasons for that? Currently, when using > > > yes，the cpu model is indeed used with kvm. > > > > > > There is a situation where the host cpu model is Cortex-A72 and > > > a 'custom' cpu mode is used to keep insrtuction set compatible between > > > the source and destination host machine when doing live migration. > > > So the host physical machine cpu model is Cortex-A72 but > > > host-passthrough model is mode used here. > I mean host-passthrough model is 'not' used here. Sorry to make it > confusing. I guessed as much. > > > > Are the source and destinations hosts used in the migration identical? > > If so, then the guest can use cpu 'host' and disable cpu features that > > should not be exposed (e.g. -cpu host,pmu=off). If the source and > > destination hosts are not identical, then I'm curious what those exact > > differences are. With the way AArch64 KVM works today, even using the > > Cortex-A72 cpu model should require identical hosts when migrating. Or, > > at least both hosts must be compatible with Cortex-A72 and any difference > > in ID registers must be somehow getting hidden from the guest. > Yes, you are right. > We have AAarch64 server with cpu based on Cortex-A72 and some extra > instruction set added. Source host with cpu based on V1 and destination host > with cpu based on V2 and they are compatible with Cortex-A72. We want to use > a 'custom' cpu mode here to make it possible to do live migration between > them. This is the scenario where the 'host' cpu model is not used since a > 'custom' cpu model Cortex-A72 is used here . What you've described here is indeed the reason to use CPU models. I.e. enabling the migration from a host of one type to another by creating a guest that only enables the features contained in both hosts (as well as maintaining all state that describes the CPU type, e.g. MIDR). Unfortunately, unless your KVM has patches that aren't upstream, then that doesn't work on AArch64 KVM (more on that below). It may appear to be working for you, because your guest kernel and userspace don't mind the slight differences exposed to it between the hosts, or those differences are limited to explicitly disabled features. If that's the case, then I would guess that using '-cpu host' and disabling the same features would "work" as well. Here's some more details on why the Cortex-A72 CPU model doesn't matter with upstream KVM. First, upstream AArch64 KVM doesn't support CPU models, and it doesn't even have a Cortex-A72 preferred target. For Cortex-A72 it will use "KVM_ARM_TARGET_GENERIC_V8", which is the same thing
Re: About the kvm-no-adjvtime CPU property
On 2020/6/1 20:29, Andrew Jones wrote: On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote: On 2020/6/1 16:07, Andrew Jones wrote: On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote: About the kvm-no-adjvtime CPU property Hi Andrew, To adjust virutal time, a new kvm cpu property kvm-no-adjvtime was introduced to 5.0 virt machine types. However the cpu property was enabled only for host-passthrough and max cpu model. As for other cpu model like cortex-a57, cortex-a53, cortex-a72, this kvm-adjvtime is not enabled by default, which means the virutal time can not be adjust for them. Here, for example, if VM is configured with kvm enabled: cortex-a72 We cannot adjust virtual time even if 5.0 virt machine is used. So i'd like to add it to other cpu model, do you have any suggestions here ? Hi Fang, The cpu feature only requires kvm. If a cpu model may be used with kvm, then the feature can be allowed to be used with the model. What I find interesting is that the cpu model is being used with kvm instead of 'host' or 'max'. Can you explain the reasons for that? Currently, when using yes，the cpu model is indeed used with kvm. There is a situation where the host cpu model is Cortex-A72 and a 'custom' cpu mode is used to keep insrtuction set compatible between the source and destination host machine when doing live migration. So the host physical machine cpu model is Cortex-A72 but host-passthrough model is mode used here. I mean host-passthrough model is 'not' used here. Sorry to make it confusing. Are the source and destinations hosts used in the migration identical? If so, then the guest can use cpu 'host' and disable cpu features that should not be exposed (e.g. -cpu host,pmu=off). If the source and destination hosts are not identical, then I'm curious what those exact differences are. With the way AArch64 KVM works today, even using the Cortex-A72 cpu model should require identical hosts when migrating. Or, at least both hosts must be compatible with Cortex-A72 and any difference in ID registers must be somehow getting hidden from the guest. Yes, you are right. We have AAarch64 server with cpu based on Cortex-A72 and some extra instruction set added. Source host with cpu based on V1 and destination host with cpu based on V2 and they are compatible with Cortex-A72. We want to use a 'custom' cpu mode here to make it possible to do live migration between them. This is the scenario where the 'host' cpu model is not used since a 'custom' cpu model Cortex-A72 is used here . However the kvm-adjvtime feature is also need. So I think we should move kvm_arm_add_vcpu_properties to arm_cpu_post_init, instead of limited to 'host' and 'max' cpu model[1]. 1: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg00091.html Thanks, drew
Re: About the kvm-no-adjvtime CPU property
On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote: > > > On 2020/6/1 16:07, Andrew Jones wrote: > > On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote: > > > About the kvm-no-adjvtime CPU property > > > > > > Hi Andrew, > > > To adjust virutal time, a new kvm cpu property kvm-no-adjvtime > > > was introduced to 5.0 virt machine types. However the cpu > > > property was enabled only for host-passthrough and max cpu model. > > > As for other cpu model like cortex-a57, cortex-a53, cortex-a72, > > > this kvm-adjvtime is not enabled by default, which means the > > > virutal time can not be adjust for them. > > > > > > Here, for example, if VM is configured with kvm enabled: > > > > > > > > > cortex-a72 > > > > > > > > > > > > > > > > > > > > > > > > We cannot adjust virtual time even if 5.0 virt machine is used. > > > So i'd like to add it to other cpu model, do you have any > > > suggestions here ? > > > > > > > > > > Hi Fang, > > > > The cpu feature only requires kvm. If a cpu model may be used with kvm, > > then the feature can be allowed to be used with the model. What I find > > interesting is that the cpu model is being used with kvm instead of 'host' > > or 'max'. Can you explain the reasons for that? Currently, when using > yes，the cpu model is indeed used with kvm. > > There is a situation where the host cpu model is Cortex-A72 and > a 'custom' cpu mode is used to keep insrtuction set compatible between > the source and destination host machine when doing live migration. > So the host physical machine cpu model is Cortex-A72 but > host-passthrough model is mode used here. Are the source and destinations hosts used in the migration identical? If so, then the guest can use cpu 'host' and disable cpu features that should not be exposed (e.g. -cpu host,pmu=off). If the source and destination hosts are not identical, then I'm curious what those exact differences are. With the way AArch64 KVM works today, even using the Cortex-A72 cpu model should require identical hosts when migrating. Or, at least both hosts must be compatible with Cortex-A72 and any difference in ID registers must be somehow getting hidden from the guest. Thanks, drew
Re: About the kvm-no-adjvtime CPU property
On 2020/6/1 16:07, Andrew Jones wrote: On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote: About the kvm-no-adjvtime CPU property Hi Andrew, To adjust virutal time, a new kvm cpu property kvm-no-adjvtime was introduced to 5.0 virt machine types. However the cpu property was enabled only for host-passthrough and max cpu model. As for other cpu model like cortex-a57, cortex-a53, cortex-a72, this kvm-adjvtime is not enabled by default, which means the virutal time can not be adjust for them. Here, for example, if VM is configured with kvm enabled: cortex-a72 We cannot adjust virtual time even if 5.0 virt machine is used. So i'd like to add it to other cpu model, do you have any suggestions here ? Hi Fang, The cpu feature only requires kvm. If a cpu model may be used with kvm, then the feature can be allowed to be used with the model. What I find interesting is that the cpu model is being used with kvm instead of 'host' or 'max'. Can you explain the reasons for that? Currently, when using yes，the cpu model is indeed used with kvm. There is a situation where the host cpu model is Cortex-A72 and a 'custom' cpu mode is used to keep insrtuction set compatible between the source and destination host machine when doing live migration. So the host physical machine cpu model is Cortex-A72 but host-passthrough model is mode used here. kvm, the guest will always effectively get 'host' anyway, even when a model is provided. Indeed, for a model to work, kvm requires it to exactly match the host cpu and even then the ID registers are passed through from the host CPU, not taken from QEMU's model. In the above scenario, the cpu mode is Cortex-A72 but maybe only a subset of cpu feature is presented to the guest. Since the cpu model is not 'host' here, the kvm-adjvtime is not enabled by default. The guest virtual time will not be adjusted. Thanks, drew .
Re: About the kvm-no-adjvtime CPU property
On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote: > About the kvm-no-adjvtime CPU property > > Hi Andrew, > To adjust virutal time, a new kvm cpu property kvm-no-adjvtime > was introduced to 5.0 virt machine types. However the cpu > property was enabled only for host-passthrough and max cpu model. > As for other cpu model like cortex-a57, cortex-a53, cortex-a72, > this kvm-adjvtime is not enabled by default, which means the > virutal time can not be adjust for them. > > Here, for example, if VM is configured with kvm enabled: > > > cortex-a72 > > > > > > > > We cannot adjust virtual time even if 5.0 virt machine is used. > So i'd like to add it to other cpu model, do you have any > suggestions here ? > > Hi Fang, The cpu feature only requires kvm. If a cpu model may be used with kvm, then the feature can be allowed to be used with the model. What I find interesting is that the cpu model is being used with kvm instead of 'host' or 'max'. Can you explain the reasons for that? Currently, when using kvm, the guest will always effectively get 'host' anyway, even when a model is provided. Indeed, for a model to work, kvm requires it to exactly match the host cpu and even then the ID registers are passed through from the host CPU, not taken from QEMU's model. Thanks, drew
About the kvm-no-adjvtime CPU property
About the kvm-no-adjvtime CPU property Hi Andrew, To adjust virutal time, a new kvm cpu property kvm-no-adjvtime was introduced to 5.0 virt machine types. However the cpu property was enabled only for host-passthrough and max cpu model. As for other cpu model like cortex-a57, cortex-a53, cortex-a72, this kvm-adjvtime is not enabled by default, which means the virutal time can not be adjust for them. Here, for example, if VM is configured with kvm enabled: cortex-a72 We cannot adjust virtual time even if 5.0 virt machine is used. So i'd like to add it to other cpu model, do you have any suggestions here ?
