Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> On Wed, Jun 22, 2016 at 09:54:51 +0200, David Hildenbrand wrote: > > > On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote: > > > > I think the coffee didn't do its work already :) . I wanted to write > > > > that we can > > > > _with_ this additional query. Meaning the involved overhead would be ok > > > > - in my > > > > opinion for s390x. > > > > > > > > What we could do to avoid one compare operation would be: > > > > > > > > a) Expand the host model > > > > b) Expand the target model (because on s390x we could have migration > > > > unsafe > > > > model) > > > > c) Work with the runnability information returned via > > > > query-cpu-definitions > > > > > > > > But as we have to do b) either way on s390x, we can directly do a > > > > compare > > > > operation. (which makes implementation a lot simpler, because libvirt > > > > then > > > > doesn't have to deal with any feature/model names). > > > > > > But why do you even need to do any comparison? Isn't it possible to let > > > QEMU do it when a domain starts? The thing is we should avoid doing > > > completely different things on each architecture. > > > > > > > Sure, QEMU will of course double check when starting the guest! So trying to > > start and failing is of course an option! So no check is needed if that is > > acceptable. > > Yeah, I think it's the safest and easiest option now. > > Jirka > Alright then, this RFC already handles that properly, so that seems to be solved. The question now is if you guys see a fundamental problem in the way we want to handle CPU models. Especially a) Having flexible, not migration safe CPU models that can be expanded to migration safe models (using the expansion interface). b) Letting QEMU carry out the task of comparing and baselining to be used for e.g. for "virsh cpu-baseline" or "virsh cpu-compare". c) Indicating the host model as the expansion of "-cpu host", e.g. for "virsh capabilities" (which says "host" for now for us). Also, it will be good to know if the "expansion" interface with parameters "full" or "migratable" is really helpful to you or if I should drop that and you will come up with an own "query-host-cpu". We are also planning to implement the "query-cpu-definitions" runnability information in the future, because as you said, it might be a good way to quickly indicate runnable CPU models. But we are most likely not going to use it for e.g. comparing or baselining or detection of runnability of a more complex cpu model (having a lot of feature changes). Thanks! David -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> Question: KVM supports x2apic (and enables it by default) on any
> host CPU, because it is all emulated by KVM. Should "host-model"
> include x2apic on all hosts, or only when the host CPU has it?
> ("-cpu host" does include it).
>
> Including x2apic sounds more useful, but it doesn't match the
> "get as close to the host CPU as possible" part of your
> description.
>
>From a s390x point of view, our "host" model is about indicating the maximum
possible configuration we are able to virtualize, not strictly limiting it only
to the real host features.
We also have a feature called "sthyi", emulated by KVM, which might not be
around for the real "host". We will include that in "host", because otherwise
things seem to get really complicated with no obvious benefit (to me :) ).
Not sure how you want to handle that.
David
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 10:56:28PM +0200, Jiri Denemark wrote:
[...]
> Let me sum up what libvirt is doing (or will be doing) with guest
> CPUs... Libvirt supports three guest CPU configuration modes:
>
> - host-passthrough -- this is easy, it's just asking for "-cpu host" and
> no fancy commands are required.
>
> - host-model -- for this we need to know what CPU configuration we need
> to ask for to get as close to the host CPU as possible while being
> able to ask for the exact same CPU on a different host (after
> migration). And we need to be able to ask for this at probing time
> (when libvirtd starts rather than when starting a new domain) using
> "-machine none,accel=kvm:tcg", that is without actually creating any
> guest CPU. This is what Eduardo's query-host-cpu QMP command is useful
> for. In x86 world we could just query the guest CPU after running QEMU
> with "-cpu host", but that's unusable with "-machine none", which is
> why another way of getting this info is needed.
Question: KVM supports x2apic (and enables it by default) on any
host CPU, because it is all emulated by KVM. Should "host-model"
include x2apic on all hosts, or only when the host CPU has it?
("-cpu host" does include it).
Including x2apic sounds more useful, but it doesn't match the
"get as close to the host CPU as possible" part of your
description.
--
Eduardo
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Wed, Jun 22, 2016 at 09:54:51 +0200, David Hildenbrand wrote: > > On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote: > > > I think the coffee didn't do its work already :) . I wanted to write that > > > we can > > > _with_ this additional query. Meaning the involved overhead would be ok - > > > in my > > > opinion for s390x. > > > > > > What we could do to avoid one compare operation would be: > > > > > > a) Expand the host model > > > b) Expand the target model (because on s390x we could have migration > > > unsafe > > > model) > > > c) Work with the runnability information returned via > > > query-cpu-definitions > > > > > > But as we have to do b) either way on s390x, we can directly do a compare > > > operation. (which makes implementation a lot simpler, because libvirt then > > > doesn't have to deal with any feature/model names). > > > > But why do you even need to do any comparison? Isn't it possible to let > > QEMU do it when a domain starts? The thing is we should avoid doing > > completely different things on each architecture. > > > > Sure, QEMU will of course double check when starting the guest! So trying to > start and failing is of course an option! So no check is needed if that is > acceptable. Yeah, I think it's the safest and easiest option now. Jirka -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote: > > I think the coffee didn't do its work already :) . I wanted to write that > > we can > > _with_ this additional query. Meaning the involved overhead would be ok - > > in my > > opinion for s390x. > > > > What we could do to avoid one compare operation would be: > > > > a) Expand the host model > > b) Expand the target model (because on s390x we could have migration unsafe > > model) > > c) Work with the runnability information returned via query-cpu-definitions > > > > But as we have to do b) either way on s390x, we can directly do a compare > > operation. (which makes implementation a lot simpler, because libvirt then > > doesn't have to deal with any feature/model names). > > But why do you even need to do any comparison? Isn't it possible to let > QEMU do it when a domain starts? The thing is we should avoid doing > completely different things on each architecture. > Sure, QEMU will of course double check when starting the guest! So trying to start and failing is of course an option! So no check is needed if that is acceptable. David -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote: > I think the coffee didn't do its work already :) . I wanted to write that we > can > _with_ this additional query. Meaning the involved overhead would be ok - in > my > opinion for s390x. > > What we could do to avoid one compare operation would be: > > a) Expand the host model > b) Expand the target model (because on s390x we could have migration unsafe > model) > c) Work with the runnability information returned via query-cpu-definitions > > But as we have to do b) either way on s390x, we can directly do a compare > operation. (which makes implementation a lot simpler, because libvirt then > doesn't have to deal with any feature/model names). But why do you even need to do any comparison? Isn't it possible to let QEMU do it when a domain starts? The thing is we should avoid doing completely different things on each architecture. Jirka -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> > > > 2) Requiring a running QEMU instance to run > > > >query-cpu-model-comparison > > > > > > > > With my previous query-host-cpu proposal, the task of comparing > > > > the configuration requested by the user with host capabilities > > > > can be done directly by libvirt. This way, no extra QEMU instance > > > > needs to be run before starting a VM. > > > > > > I think we can just easily get around this by not comparing a guest CPU > > > to host (except for the explicit virConnectCompareCPU, which is not very > > > useful in its current form anyway). > > > > If there is some flexible way around that, great. But I think we (s390x) > > could > > life without this additional query. > > So if I understand correctly, you say you don't need the API to compare > guest CPU to host CPU, right? If so, that's exactly what I said too. > I think the coffee didn't do its work already :) . I wanted to write that we can _with_ this additional query. Meaning the involved overhead would be ok - in my opinion for s390x. What we could do to avoid one compare operation would be: a) Expand the host model b) Expand the target model (because on s390x we could have migration unsafe model) c) Work with the runnability information returned via query-cpu-definitions But as we have to do b) either way on s390x, we can directly do a compare operation. (which makes implementation a lot simpler, because libvirt then doesn't have to deal with any feature/model names). David -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Wed, Jun 22, 2016 at 08:51:40 +0200, David Hildenbrand wrote: > > On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote: > > > On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote: > > > I still don't think we want to set in stone that "the result the > > > guest sees when using -cpu host" is always the same as "what the > > > host supports running". > > > > > > For example: let's assume a given architecture have two features > > > (A and B) that are both supported by the host but can never be > > > enabled together. For actual "-cpu host" usage, QEMU would have > > > to choose between enabling A and B. For querying host > > > capabilities, we still want to let management software know that > > > either A or B are supported. > > > > What libvirt is really interested in is the guest CPU which would be > > used with -cpu host. This is actually what I thought query-host-cpu was > > all about. Perhaps because there's no difference for x86. > > That's also what I had in mind first. Let me explain why they are not the same > on s390x and why it is problematic (actually both types are not the same!): > > 1. Certain CPU features are only valid for LPAR guests, they can't be > virtualized for KVM guests (remember that we always have at least one level of > virtualization on s390x). So we will never be able to provide these features > to > a KVM guest, therefore we will never be able to completely mimic the real host > model. > > 2. We have a whole lot of unpublished features. We would have to include them > in > the "real host model", but we can't. For the "host" model, this is not a > problem, because we simply don't virtualize them. But the "real host model" > would then vary between say QEMZ versions, which looks really strage, because > in essance, QEMU/KVM looks like the wrong interface to query for a "real host > model". > > 3. We don't have the kernel interfaces in place to query the "real host > model". > We can only query what we are able to virtualize. And that is indeed different > compared to what I said previously. > > And as I can't see any use case for s390x, we most probably will never be able > to support the interface you are suggesting here. Which is fine, if it makes > sense for x86. Well, as I said I always thought about query-host-cpu as a way to get the CPU configuration QEMU would provide with -cpu host. Thanks to this discussion I realized its semantics is different and thus what I really need is actually the command you suggested, i.e., query-cpu-model-expansion. > > > 2) Requiring a running QEMU instance to run > > >query-cpu-model-comparison > > > > > > With my previous query-host-cpu proposal, the task of comparing > > > the configuration requested by the user with host capabilities > > > can be done directly by libvirt. This way, no extra QEMU instance > > > needs to be run before starting a VM. > > > > I think we can just easily get around this by not comparing a guest CPU > > to host (except for the explicit virConnectCompareCPU, which is not very > > useful in its current form anyway). > > If there is some flexible way around that, great. But I think we (s390x) could > life without this additional query. So if I understand correctly, you say you don't need the API to compare guest CPU to host CPU, right? If so, that's exactly what I said too. Jirka -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> On Tue, Jun 21, 2016 at 18:22:30 -0300, Eduardo Habkost wrote: > > On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote: > > [...] > > > > 1) "query-cpu-model-expansion model=host" vs "query-host-cpu": > > > > > > > > I still don't think we want to set in stone that "the result the > > > > guest sees when using -cpu host" is always the same as "what the > > > > host supports running". > > > > > > > > For example: let's assume a given architecture have two features > > > > (A and B) that are both supported by the host but can never be > > > > enabled together. For actual "-cpu host" usage, QEMU would have > > > > to choose between enabling A and B. For querying host > > > > capabilities, we still want to let management software know that > > > > either A or B are supported. > > > > > > What libvirt is really interested in is the guest CPU which would be > > > used with -cpu host. This is actually what I thought query-host-cpu was > > > all about. Perhaps because there's no difference for x86. > > > > In that case, I think it makes sense to just extend > > query-cpu-definitions or use "query-cpu-model-expansion > > model=host" instead of a query-host-cpu command. > > > > Probably query-cpu-model-expansion is better than just extending > > query-cpu-definitions, because it would allow the expansion of > > extra CPU options, like "host,migratable=off". > > Yeah, this would be even better. > > Jirka > Please be aware that we don't have anything like that on s390x, but I prepared for that requirement by being able to tell query-cpu-model-expansion how to expand (full, migratable, stable). Actually full and migratable looks the same on s390x. The plan for us is to rely on "stable" most of the time, full and migratable might be what you're looking for. David -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 18:22:30 -0300, Eduardo Habkost wrote: > On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote: > [...] > > > 1) "query-cpu-model-expansion model=host" vs "query-host-cpu": > > > > > > I still don't think we want to set in stone that "the result the > > > guest sees when using -cpu host" is always the same as "what the > > > host supports running". > > > > > > For example: let's assume a given architecture have two features > > > (A and B) that are both supported by the host but can never be > > > enabled together. For actual "-cpu host" usage, QEMU would have > > > to choose between enabling A and B. For querying host > > > capabilities, we still want to let management software know that > > > either A or B are supported. > > > > What libvirt is really interested in is the guest CPU which would be > > used with -cpu host. This is actually what I thought query-host-cpu was > > all about. Perhaps because there's no difference for x86. > > In that case, I think it makes sense to just extend > query-cpu-definitions or use "query-cpu-model-expansion > model=host" instead of a query-host-cpu command. > > Probably query-cpu-model-expansion is better than just extending > query-cpu-definitions, because it would allow the expansion of > extra CPU options, like "host,migratable=off". Yeah, this would be even better. Jirka -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote: > > On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote: > > > > (CCing libvirt people) > > > > > > > > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote: > > > > > This is our second attempt to implement CPU models for s390x. We > > > > > realized > > > > > that we also want to have features exposed via the CPU model. While > > > > > doing > > > > > that we realized that we want to have a better interface for libvirt. > > > > > > > > > > > > > Before getting into the details, I would like to clarify how the > > > > following could be accomplished using the new commands: > > > > > > > > Example: > > > > > > > > 1) User configures libvirt with: > > > > > > > >Westmere > > > > > > > > > > > > 2) libvirt will translate that to: > > > >"-cpu Westmere,+aes" or "-cpu Westmere,aes=on" > > > > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in > > > >the current host, before trying to start the VM. > > > > > > > > How exactly would this be done using the new commands? > > > > > > Hi Eduardo, > > > > > > thanks for having a look - highly appreciated that you actually map this > > > to libvirt requirements! > > > > > > That would map to a compare operation between "host" and > > > "Westmere,aes=on". > > > > > > Host could at that point already be expanded by libvirt. Doesn't matter > > > at that > > > point. > > > > > > If the result is "identica"l or "superset", it is runnable. If the result > > > is > > > "subset" or "incompatible", details about the responsible properties is > > > indicated. (I actually took that idea from your patch for indicating > > > runnability). > > > > So, I have two worries about the proposal: > > > > > > 1) "query-cpu-model-expansion model=host" vs "query-host-cpu": > > > > I still don't think we want to set in stone that "the result the > > guest sees when using -cpu host" is always the same as "what the > > host supports running". > > > > For example: let's assume a given architecture have two features > > (A and B) that are both supported by the host but can never be > > enabled together. For actual "-cpu host" usage, QEMU would have > > to choose between enabling A and B. For querying host > > capabilities, we still want to let management software know that > > either A or B are supported. > > What libvirt is really interested in is the guest CPU which would be > used with -cpu host. This is actually what I thought query-host-cpu was > all about. Perhaps because there's no difference for x86. That's also what I had in mind first. Let me explain why they are not the same on s390x and why it is problematic (actually both types are not the same!): 1. Certain CPU features are only valid for LPAR guests, they can't be virtualized for KVM guests (remember that we always have at least one level of virtualization on s390x). So we will never be able to provide these features to a KVM guest, therefore we will never be able to completely mimic the real host model. 2. We have a whole lot of unpublished features. We would have to include them in the "real host model", but we can't. For the "host" model, this is not a problem, because we simply don't virtualize them. But the "real host model" would then vary between say QEMZ versions, which looks really strage, because in essance, QEMU/KVM looks like the wrong interface to query for a "real host model". 3. We don't have the kernel interfaces in place to query the "real host model". We can only query what we are able to virtualize. And that is indeed different compared to what I said previously. And as I can't see any use case for s390x, we most probably will never be able to support the interface you are suggesting here. Which is fine, if it makes sense for x86. > > > 2) Requiring a running QEMU instance to run > >query-cpu-model-comparison > > > > With my previous query-host-cpu proposal, the task of comparing > > the configuration requested by the user with host capabilities > > can be done directly by libvirt. This way, no extra QEMU instance > > needs to be run before starting a VM. > > I think we can just easily get around this by not comparing a guest CPU > to host (except for the explicit virConnectCompareCPU, which is not very > useful in its current form anyway). If there is some flexible way around that, great. But I think we (s390x) could life without this additional query. So your first concern is valid, and if you really need this information, you should probably really create a new interface. Your second concern should not be trivial for now. We don't want to optimize for total speed at that point. David -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> On Tue, Jun 21, 2016 at 13:44:31 -0300, Eduardo Habkost wrote:
> > (CCing libvirt people)
> >
> > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> > > This is our second attempt to implement CPU models for s390x. We realized
> > > that we also want to have features exposed via the CPU model. While doing
> > > that we realized that we want to have a better interface for libvirt.
> >
> > Before getting into the details, I would like to clarify how the
> > following could be accomplished using the new commands:
> >
> > Example:
> >
> > 1) User configures libvirt with:
> >
> >Westmere
> >
> >
> > 2) libvirt will translate that to:
> >"-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
> > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
> >the current host, before trying to start the VM.
>
> While this is what libvirt currently does, I don't think it's necessary
> to keep doing that... see below.
>
> > > a) "query-cpu-model-expansion" - tell us what the "host" or a migration
> > >unsafe model looks like. Either falling back to a stable model or
> > >completely exposing all properties. We are interested in stable models.
> > > b) "query-cpu-model-comparison" - tell us how two CPU models compare,
> > > indicating which properties were responsible for the decision.
> > > c) "query-cpu-model-baseline" - create a new model out of two models,
> > > taking a requested level of stability into account.
>
> This looks like it copies current libvirt APIs, which I think is not a
> very good idea. Both CPU compare and baseline APIs in libvirt will need
> to be changed (or rather new APIs with similar functionality added) to
> become useful. I think we should first focus on making guest CPU
> configuration work the way it should work. For that I think we need some
> higher level commands.
Hi Jiri,
thanks for your reply!
Please be aware that we (s390x) have no libvirt cpu model support in place. We
are coming up with a solution that solves our problems, but instead of tagging
all interfaces with "s390x" we want interfaces that everybody might use.
I understand that x86 has some very advanced libvirt CPU model magic in place.
You can still stick to what you have right now. It doesn't work for us.
So we provide 3 basic functions (baseline, compare, expand) that can easily by
added for s390x into the CPU model. So I don't see a problem introducing
these operations in libvirt - it just works without any magic and knowledge
about features and models in libvirt.
In the end it keeps the CPU model where it belongs (QEMU) and allows us to
just fit in fine into the libvirt CPU model world that was heavily inspired
by x86 (an I can tell you, that is a very challenging task).
But - of course - we have to do some additional queries on a QEMU instance. I
don't think that this is a problem (at least not for s390x).
>
> Let me sum up what libvirt is doing (or will be doing) with guest
> CPUs... Libvirt supports three guest CPU configuration modes:
One important thing to mention here is that all of these was inspired by x86.
Some stuff doesn't even make sense on s390x ("mimic the real host"), as
it would even in practice never be possible. See my reply to Eduards patchset
I'll post later.
We want to focus on the real use cases and I think they are
- Making sure models ("Guest API") doesn't change during migration
- Allowing the user to use a certain CPU model
- Baselining/Comparing/Using the -cpu host model
>
> - host-passthrough -- this is easy, it's just asking for "-cpu host" and
> no fancy commands are required.
This should map to expanding the host model in our case.
>
> - host-model -- for this we need to know what CPU configuration we need
> to ask for to get as close to the host CPU as possible while being
> able to ask for the exact same CPU on a different host (after
> migration). And we need to be able to ask for this at probing time
> (when libvirtd starts rather than when starting a new domain) using
> "-machine none,accel=kvm:tcg", that is without actually creating any
> guest CPU. This is what Eduardo's query-host-cpu QMP command is useful
> for. In x86 world we could just query the guest CPU after running QEMU
> with "-cpu host", but that's unusable with "-machine none", which is
> why another way of getting this info is needed.
Again, we will be relying on "-cpu host" here, because the real host model
will never be possible to be fordwarded on s390x. And what's the use case
anyway? Does not make sense on s390x as far as I can tell.
We want to know what we are able to virtualize, not build up some strange
model that won't even be able to run.
>
> - custom -- the XML specifies an exact guest CPU configuration. We don't
> really need to know if that exact CPU is runnable on the current host
> in advance, we can just try to start the domain, check if the guest
> CPU matches what we asked for, and
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote: [...] > > 1) "query-cpu-model-expansion model=host" vs "query-host-cpu": > > > > I still don't think we want to set in stone that "the result the > > guest sees when using -cpu host" is always the same as "what the > > host supports running". > > > > For example: let's assume a given architecture have two features > > (A and B) that are both supported by the host but can never be > > enabled together. For actual "-cpu host" usage, QEMU would have > > to choose between enabling A and B. For querying host > > capabilities, we still want to let management software know that > > either A or B are supported. > > What libvirt is really interested in is the guest CPU which would be > used with -cpu host. This is actually what I thought query-host-cpu was > all about. Perhaps because there's no difference for x86. In that case, I think it makes sense to just extend query-cpu-definitions or use "query-cpu-model-expansion model=host" instead of a query-host-cpu command. Probably query-cpu-model-expansion is better than just extending query-cpu-definitions, because it would allow the expansion of extra CPU options, like "host,migratable=off". -- Eduardo -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote: > On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote: > > > (CCing libvirt people) > > > > > > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote: > > > > This is our second attempt to implement CPU models for s390x. We > > > > realized > > > > that we also want to have features exposed via the CPU model. While > > > > doing > > > > that we realized that we want to have a better interface for libvirt. > > > > > > Before getting into the details, I would like to clarify how the > > > following could be accomplished using the new commands: > > > > > > Example: > > > > > > 1) User configures libvirt with: > > > > > >Westmere > > > > > > > > > 2) libvirt will translate that to: > > >"-cpu Westmere,+aes" or "-cpu Westmere,aes=on" > > > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in > > >the current host, before trying to start the VM. > > > > > > How exactly would this be done using the new commands? > > > > Hi Eduardo, > > > > thanks for having a look - highly appreciated that you actually map this > > to libvirt requirements! > > > > That would map to a compare operation between "host" and "Westmere,aes=on". > > > > Host could at that point already be expanded by libvirt. Doesn't matter at > > that > > point. > > > > If the result is "identica"l or "superset", it is runnable. If the result is > > "subset" or "incompatible", details about the responsible properties is > > indicated. (I actually took that idea from your patch for indicating > > runnability). > > So, I have two worries about the proposal: > > > 1) "query-cpu-model-expansion model=host" vs "query-host-cpu": > > I still don't think we want to set in stone that "the result the > guest sees when using -cpu host" is always the same as "what the > host supports running". > > For example: let's assume a given architecture have two features > (A and B) that are both supported by the host but can never be > enabled together. For actual "-cpu host" usage, QEMU would have > to choose between enabling A and B. For querying host > capabilities, we still want to let management software know that > either A or B are supported. What libvirt is really interested in is the guest CPU which would be used with -cpu host. This is actually what I thought query-host-cpu was all about. Perhaps because there's no difference for x86. > 2) Requiring a running QEMU instance to run >query-cpu-model-comparison > > With my previous query-host-cpu proposal, the task of comparing > the configuration requested by the user with host capabilities > can be done directly by libvirt. This way, no extra QEMU instance > needs to be run before starting a VM. I think we can just easily get around this by not comparing a guest CPU to host (except for the explicit virConnectCompareCPU, which is not very useful in its current form anyway). Jirka -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 13:44:31 -0300, Eduardo Habkost wrote: > (CCing libvirt people) > > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote: > > This is our second attempt to implement CPU models for s390x. We realized > > that we also want to have features exposed via the CPU model. While doing > > that we realized that we want to have a better interface for libvirt. > > Before getting into the details, I would like to clarify how the > following could be accomplished using the new commands: > > Example: > > 1) User configures libvirt with: > >Westmere > > > 2) libvirt will translate that to: >"-cpu Westmere,+aes" or "-cpu Westmere,aes=on" > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in >the current host, before trying to start the VM. While this is what libvirt currently does, I don't think it's necessary to keep doing that... see below. > > a) "query-cpu-model-expansion" - tell us what the "host" or a migration > >unsafe model looks like. Either falling back to a stable model or > >completely exposing all properties. We are interested in stable models. > > b) "query-cpu-model-comparison" - tell us how two CPU models compare, > > indicating which properties were responsible for the decision. > > c) "query-cpu-model-baseline" - create a new model out of two models, > > taking a requested level of stability into account. This looks like it copies current libvirt APIs, which I think is not a very good idea. Both CPU compare and baseline APIs in libvirt will need to be changed (or rather new APIs with similar functionality added) to become useful. I think we should first focus on making guest CPU configuration work the way it should work. For that I think we need some higher level commands. Let me sum up what libvirt is doing (or will be doing) with guest CPUs... Libvirt supports three guest CPU configuration modes: - host-passthrough -- this is easy, it's just asking for "-cpu host" and no fancy commands are required. - host-model -- for this we need to know what CPU configuration we need to ask for to get as close to the host CPU as possible while being able to ask for the exact same CPU on a different host (after migration). And we need to be able to ask for this at probing time (when libvirtd starts rather than when starting a new domain) using "-machine none,accel=kvm:tcg", that is without actually creating any guest CPU. This is what Eduardo's query-host-cpu QMP command is useful for. In x86 world we could just query the guest CPU after running QEMU with "-cpu host", but that's unusable with "-machine none", which is why another way of getting this info is needed. - custom -- the XML specifies an exact guest CPU configuration. We don't really need to know if that exact CPU is runnable on the current host in advance, we can just try to start the domain, check if the guest CPU matches what we asked for, and we may kill QEMU if the CPU does not meet the specification. Of course, higher level management wants to know a set of host where it can run a given domain for scheduling purposes, but since they logically want to avoid tons of different CPU configs, they would just stick the CPU models predefined by QEMU. Thus giving them a way of checking what CPU models can be run on a given host with a given QEMU (using the unavailable features stuff for query-cpu-definitions usable with "-machine none") should be enough and it's even better than having to ask for every single CPU model, which is what they need to do now. There are configuration options which are somewhere between host-model and custom, but they don't bring more requirements in addition to what both of them needs. This was basically all about starting a new domain. When the domain successfully starts, we need to make sure the guest CPU does not change during save/restore or migration to another host. To achieve this, we need the same checking we need for custom mode, i.e., whether the guest CPU we got is what we asked for. In addition to this, we need a way to ask QEMU what the guest CPU looks like so that we can store it in the domain XML and ask for it during migration. I think all of this should be pretty much architecture agnostic. Of course the actual data would be quite different fro each architecture, but I believe the entry points could fit all. Or did I miss anything? Jirka -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote: > > (CCing libvirt people) > > > > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote: > > > This is our second attempt to implement CPU models for s390x. We realized > > > that we also want to have features exposed via the CPU model. While doing > > > that we realized that we want to have a better interface for libvirt. > > > > Before getting into the details, I would like to clarify how the > > following could be accomplished using the new commands: > > > > Example: > > > > 1) User configures libvirt with: > > > >Westmere > > > > > > 2) libvirt will translate that to: > >"-cpu Westmere,+aes" or "-cpu Westmere,aes=on" > > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in > >the current host, before trying to start the VM. > > > > How exactly would this be done using the new commands? > > Hi Eduardo, > > thanks for having a look - highly appreciated that you actually map this > to libvirt requirements! > > That would map to a compare operation between "host" and "Westmere,aes=on". > > Host could at that point already be expanded by libvirt. Doesn't matter at > that > point. > > If the result is "identica"l or "superset", it is runnable. If the result is > "subset" or "incompatible", details about the responsible properties is > indicated. (I actually took that idea from your patch for indicating > runnability). So, I have two worries about the proposal: 1) "query-cpu-model-expansion model=host" vs "query-host-cpu": I still don't think we want to set in stone that "the result the guest sees when using -cpu host" is always the same as "what the host supports running". For example: let's assume a given architecture have two features (A and B) that are both supported by the host but can never be enabled together. For actual "-cpu host" usage, QEMU would have to choose between enabling A and B. For querying host capabilities, we still want to let management software know that either A or B are supported. If we don't mind adding multiple new QMP commands in this series, I don't see why not having a separate "query-host-cpu" command so the semantics of each command are very clear. Then query-cpu-definitions/query-cpu-model-expansion would be about "what you really get when using -cpu ", and "query-host-cpu" would be about "what this hosts supports running". 2) Requiring a running QEMU instance to run query-cpu-model-comparison With my previous query-host-cpu proposal, the task of comparing the configuration requested by the user with host capabilities can be done directly by libvirt. This way, no extra QEMU instance needs to be run before starting a VM. In my example above, if query-cpu-model-comparison were not required, libvirt could run "query-cpu-expansion model=host" or "query-host-cpu" once at boot, and just check if "Westmere" is runnable and "aes" is present in the host data. With query-cpu-model-comparison, QEMU needs to be run one more time before starting a VM. I don't know if requiring running a new QEMU instance just to check if a CPU config is runnable would be acceptable for libvirt. I will let the libvirt people answer that. I still like the query-cpu-model-comparison proposal, because it would allow us to extend the data returned by query-cpu-model-expansion without requiring libvirt changes. It may be the best log-term option, but probably only after we add a mechanism to allow the VM to be configured after running a few QMP commands, without re-running QEMU. -- Eduardo -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> (CCing libvirt people) > > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote: > > This is our second attempt to implement CPU models for s390x. We realized > > that we also want to have features exposed via the CPU model. While doing > > that we realized that we want to have a better interface for libvirt. > > Before getting into the details, I would like to clarify how the > following could be accomplished using the new commands: > > Example: > > 1) User configures libvirt with: > >Westmere > > > 2) libvirt will translate that to: >"-cpu Westmere,+aes" or "-cpu Westmere,aes=on" > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in >the current host, before trying to start the VM. > > How exactly would this be done using the new commands? Hi Eduardo, thanks for having a look - highly appreciated that you actually map this to libvirt requirements! That would map to a compare operation between "host" and "Westmere,aes=on". Host could at that point already be expanded by libvirt. Doesn't matter at that point. If the result is "identica"l or "superset", it is runnable. If the result is "subset" or "incompatible", details about the responsible properties is indicated. (I actually took that idea from your patch for indicating runnability). Thanks! David -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> This is our second attempt to implement CPU models for s390x. We realized
> that we also want to have features exposed via the CPU model. While doing
> that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the
following could be accomplished using the new commands:
Example:
1) User configures libvirt with:
Westmere
2) libvirt will translate that to:
"-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
the current host, before trying to start the VM.
How exactly would this be done using the new commands?
>
> Unfortunately, CPU models on s390x are special and we have to take care of:
> - A CPU like z13 looks differently in various environments (under different
> LPAR versions, under different z/VM versions, under different KVM
> versions, export regulation) - we have _a lot_ of feature variability.
> - We still have certain features that are not published but might be
> implemented/introduced in the future. As they are a theoretical part
> of a CPU already, we have to find a way to model these future changes.
> - We still have certain features that are already published, but not
> implemented. Implementation might be added in the future in KVM.
> - We heavily rely on KVM to tell us which features it can actually
> virtualize - user space queries like "STFL(e)" give no guarantees.
> - Certain "subfeatures" were introduced in one run. In practice, they are
> always around, but in theory, subfeatures could be dropped in the future.
> - Just because two CPU models have the same features doesn't mean they
> are equal - some internal numbers might be different. E.g. we won't allow
> running a z13 under a zBC12 just by turning off features.
> - We cannot blindly enable all possible features for a CPU generation,
> the IBC "Instruction Blocking Control" in KVM will try to block
> instructions introduced with certain features. So a CPU generation always
> has some maximum feature set that is guaranteed to work.
>
> It all boils down to a specific released CPU to have.
> a) A fixed feature set that we expect it to be have on every hypervisor.
> b) A variable part that depends on the hypervisor and that could be
>extended in the future (adding not yet implemented features) that we
>always want to enable later on.
> c) A variable part that we want to enable only if requested - nested
>virtualization ("vsie") and assists are one example.
>
> But, the fixed feature set is not really what we want to use as a default.
> It is just like a really minimum, stable base.
>
> So we have
> a) A "stable" CPU model for each released CPU that will never change and
>maps to the minimum feature set we expect to be around on all
>hypervisors. e.g. "z13-base" or "z10EC.2-base". These are migration
>safe.
> b) A "default" CPU model for each released CPU, that can change between
>QEMU versions and that will always include the features we expect to
>be around in our currently supported environments and will contain only
>features we expect to be stable. E.g. nested virtualization will not be
>contained in these models. These models are not migration safe, e.g
>"z13" or "z10EC.2". The feature set can differ between QEMU versions.
> c) An internal "maximum" CPU model for each generation that tells us which
>features were supported as a maximum back when the hardware was
>released. This will not be exposed
>
> To not have to replicate all CPU model changes ("new default fetaures") in
> libvirt, to not duplicate the logic about compatibility and the like,
> our approach tries to keep all the QEMU logic in libvirt and provide
> standardized interfaces for libvirt to e.g. baseline, compare. This
> allows libvirt to not have to care about any model names or feature names,
> it can just pass the data from interface to interface and report it to
> the user.
>
> Also, libvirt might want to know what the "host" model looks like and
> convert a CPU model to a migration safe variant. For this reason, a QMP
> command is added that can create a migration safe variant of a variable
> CPU model, indicating only the delta changes done to a stable model.
>
> So we have:
> a) "query-cpu-model-expansion" - tell us what the "host" or a migration
>unsafe model looks like. Either falling back to a stable model or
>completely exposing all properties. We are interested in stable models.
> b) "query-cpu-model-comparison" - tell us how two CPU models compare,
> indicating which properties were responsible for the decision.
> c) "query-cpu-model-baseline" - create a new model out of two models,
> taking a requested level of stability into account.
>
> As we are aware that e.g. x86 has their own idea of a CPU
