Re: [ovs-dev] [RFC v2] [OVS/NOVA] Vhost-user backends cross-version migration support
On Wed, Aug 30, 2017 at 11:19:08PM +0300, Michael S. Tsirkin wrote: > On Wed, Aug 30, 2017 at 04:25:10PM -0300, Eduardo Habkost wrote: > > On Wed, Aug 30, 2017 at 05:23:39PM +0300, Michael S. Tsirkin wrote: > > > On Wed, Aug 30, 2017 at 10:17:27AM -0300, Eduardo Habkost wrote: > > > > I'm CCing libvir-list and qemu-devel because I would like to get > > > > feedback from libvirt and QEMU developers too. > > > > > > > > On Tue, Aug 08, 2017 at 10:49:21PM +0300, Michael S. Tsirkin wrote: > > > > > On Tue, Jul 18, 2017 at 03:42:08PM +0200, Maxime Coquelin wrote: > > > > > > This is an revival from a thread I initiated earlier this year [0], > > > > > > that > > > > > > I had to postpone due to other priorities. > > > > > > > > > > > > First, I'd like to thanks reviewers of my first proposal, this new > > > > > > version tries to address the comments made: > > > > > > 1.This is Nova's role and not Libvirt's to query hosts supported > > > > > > compatibility modes and to select one, since Nova adds the > > > > > > vhost-user > > > > > > ports and has visibility on other hosts. Hence I remove libvirt ML > > > > > > and > > > > > > add Openstack one in the recipient list. > > > > > > 2. By default, the compatibility version selected is the most > > > > > > recent > > > > > > one, except if the admin selects on older compat version. > > > > > > > > > > > > The goal of this thread is to draft a solution based on the outcomes > > > > > > of discussions with contributors of the different parties (DPDK/OVS > > > > > > /Nova/...). > > > > > > > > > > > > I'm really interested on feedback from OVS & Nova contributors, > > > > > > as my experience with these projects is rather limited. > > > > > > > > > > > > Problem statement: > > > > > > == > > > > > > > > > > > > When migrating a VM from one host to another, the interfaces > > > > > > exposed by > > > > > > QEMU must stay unchanged in order to guarantee a successful > > > > > > migration. > > > > > > In the case of vhost-user interface, parameters like supported > > > > > > Virtio > > > > > > feature set, max number of queues, max vring sizes,... must remain > > > > > > compatible. Indeed, the frontend not being re-initialized, no > > > > > > re-negotiation happens at migration time. > > > > > > > > > > > > For example, we have a VM that runs on host A, which has its > > > > > > vhost-user > > > > > > backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the > > > > > > Guest > > > > > > also support this feature, it is successfully negotiated, and guest > > > > > > transmit packets using indirect descriptor tables, that the backend > > > > > > knows to handle. > > > > > > > > > > > > At some point, the VM is being migrated to host B, which runs an > > > > > > older > > > > > > version of the backend not supporting this > > > > > > VIRTIO_F_RING_INDIRECT_DESC > > > > > > feature. The migration would break, because the Guest still have the > > > > > > VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains > > > > > > some > > > > > > decriptors pointing to indirect tables, that backend B doesn't know > > > > > > to > > > > > > handle. > > > > > > This is just an example about Virtio features compatibility, but > > > > > > other > > > > > > backend implementation details could cause other failures. (e.g. > > > > > > configurable queues sizes) > > > > > > > > > > > > What we need is to be able to query the destination host's backend > > > > > > to > > > > > > ensure migration is possible before it is initiated. > > > > > > > > > > This remided me strongly of the issues around the virtual CPU modeling > > > > > in KVM, see > > > > > http
Re: [ovs-dev] [RFC v2] [OVS/NOVA] Vhost-user backends cross-version migration support
On Wed, Aug 30, 2017 at 05:23:39PM +0300, Michael S. Tsirkin wrote: > On Wed, Aug 30, 2017 at 10:17:27AM -0300, Eduardo Habkost wrote: > > I'm CCing libvir-list and qemu-devel because I would like to get > > feedback from libvirt and QEMU developers too. > > > > On Tue, Aug 08, 2017 at 10:49:21PM +0300, Michael S. Tsirkin wrote: > > > On Tue, Jul 18, 2017 at 03:42:08PM +0200, Maxime Coquelin wrote: > > > > This is an revival from a thread I initiated earlier this year [0], that > > > > I had to postpone due to other priorities. > > > > > > > > First, I'd like to thanks reviewers of my first proposal, this new > > > > version tries to address the comments made: > > > > 1.This is Nova's role and not Libvirt's to query hosts supported > > > > compatibility modes and to select one, since Nova adds the vhost-user > > > > ports and has visibility on other hosts. Hence I remove libvirt ML and > > > > add Openstack one in the recipient list. > > > > 2. By default, the compatibility version selected is the most recent > > > > one, except if the admin selects on older compat version. > > > > > > > > The goal of this thread is to draft a solution based on the outcomes > > > > of discussions with contributors of the different parties (DPDK/OVS > > > > /Nova/...). > > > > > > > > I'm really interested on feedback from OVS & Nova contributors, > > > > as my experience with these projects is rather limited. > > > > > > > > Problem statement: > > > > == > > > > > > > > When migrating a VM from one host to another, the interfaces exposed by > > > > QEMU must stay unchanged in order to guarantee a successful migration. > > > > In the case of vhost-user interface, parameters like supported Virtio > > > > feature set, max number of queues, max vring sizes,... must remain > > > > compatible. Indeed, the frontend not being re-initialized, no > > > > re-negotiation happens at migration time. > > > > > > > > For example, we have a VM that runs on host A, which has its vhost-user > > > > backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest > > > > also support this feature, it is successfully negotiated, and guest > > > > transmit packets using indirect descriptor tables, that the backend > > > > knows to handle. > > > > > > > > At some point, the VM is being migrated to host B, which runs an older > > > > version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC > > > > feature. The migration would break, because the Guest still have the > > > > VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some > > > > decriptors pointing to indirect tables, that backend B doesn't know to > > > > handle. > > > > This is just an example about Virtio features compatibility, but other > > > > backend implementation details could cause other failures. (e.g. > > > > configurable queues sizes) > > > > > > > > What we need is to be able to query the destination host's backend to > > > > ensure migration is possible before it is initiated. > > > > > > This remided me strongly of the issues around the virtual CPU modeling > > > in KVM, see > > > https://wiki.qemu.org/index.php/Features/CPUModels#Querying_host_capabilities > > > > > > QEMU recently gained query-cpu-model-expansion to allow capability > > > queries. > > > > > > Cc Eduardo accordingly. Eduardo, could you please take a look - > > > how is the problem solved on the KVM/VCPU side? Do the above > > > problem and solution for vhost look similar? > > > > (Sorry for taking so long to reply) > > > > CPU configuration in QEMU has the additional problem of features > > depending on host hardware and kernel capabilities (not just QEMU > > software capabilities). Do you have vhost-user features that > > depend on the host kernel or hardware too, or all of them just > > depend on the vhost-user backend software? > > vhost-net features depend on host kernel. > > > If it depends only on software, a solution similar to how > > machine-types work in QEMU sound enough. If features depend on > > host kernel or host hardware too, it is a bit more complex: it > > means you need an interface to find out if each configu
Re: [ovs-dev] [RFC v2] [OVS/NOVA] Vhost-user backends cross-version migration support
I'm CCing libvir-list and qemu-devel because I would like to get feedback from libvirt and QEMU developers too. On Tue, Aug 08, 2017 at 10:49:21PM +0300, Michael S. Tsirkin wrote: > On Tue, Jul 18, 2017 at 03:42:08PM +0200, Maxime Coquelin wrote: > > This is an revival from a thread I initiated earlier this year [0], that > > I had to postpone due to other priorities. > > > > First, I'd like to thanks reviewers of my first proposal, this new > > version tries to address the comments made: > > 1.This is Nova's role and not Libvirt's to query hosts supported > > compatibility modes and to select one, since Nova adds the vhost-user > > ports and has visibility on other hosts. Hence I remove libvirt ML and > > add Openstack one in the recipient list. > > 2. By default, the compatibility version selected is the most recent > > one, except if the admin selects on older compat version. > > > > The goal of this thread is to draft a solution based on the outcomes > > of discussions with contributors of the different parties (DPDK/OVS > > /Nova/...). > > > > I'm really interested on feedback from OVS & Nova contributors, > > as my experience with these projects is rather limited. > > > > Problem statement: > > == > > > > When migrating a VM from one host to another, the interfaces exposed by > > QEMU must stay unchanged in order to guarantee a successful migration. > > In the case of vhost-user interface, parameters like supported Virtio > > feature set, max number of queues, max vring sizes,... must remain > > compatible. Indeed, the frontend not being re-initialized, no > > re-negotiation happens at migration time. > > > > For example, we have a VM that runs on host A, which has its vhost-user > > backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest > > also support this feature, it is successfully negotiated, and guest > > transmit packets using indirect descriptor tables, that the backend > > knows to handle. > > > > At some point, the VM is being migrated to host B, which runs an older > > version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC > > feature. The migration would break, because the Guest still have the > > VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some > > decriptors pointing to indirect tables, that backend B doesn't know to > > handle. > > This is just an example about Virtio features compatibility, but other > > backend implementation details could cause other failures. (e.g. > > configurable queues sizes) > > > > What we need is to be able to query the destination host's backend to > > ensure migration is possible before it is initiated. > > This remided me strongly of the issues around the virtual CPU modeling > in KVM, see > https://wiki.qemu.org/index.php/Features/CPUModels#Querying_host_capabilities > > QEMU recently gained query-cpu-model-expansion to allow capability queries. > > Cc Eduardo accordingly. Eduardo, could you please take a look - > how is the problem solved on the KVM/VCPU side? Do the above > problem and solution for vhost look similar? (Sorry for taking so long to reply) CPU configuration in QEMU has the additional problem of features depending on host hardware and kernel capabilities (not just QEMU software capabilities). Do you have vhost-user features that depend on the host kernel or hardware too, or all of them just depend on the vhost-user backend software? If it depends only on software, a solution similar to how machine-types work in QEMU sound enough. If features depend on host kernel or host hardware too, it is a bit more complex: it means you need an interface to find out if each configurable feature/version is really available on the host. (In the case of CPU models, we started with an interface that reported which CPU models were runnable on the host. But as libvirt allows enabling/disabling individual CPU features, the interface had to be extended to report which CPU features were available/unavailable on the host.) * * * Now, there's one thing that seems very different here: the guest-visible interface is not defined only by QEMU, but also by the vhost-user backend. Is that correct? This means QEMU won't fully control the resulting guest ABI anymore. I would really prefer if we could keep libvirt+QEMU in control of the guest ABI as usual, making QEMU configure all the guest-visible vhost-user features. But I understand this would require additional interfaces between QEMU and libvirt, and extending the libvirt APIs. So, if QEMU is really not going to control the resulting guest ABI completely, can we at least provide a mechanism which QEMU can use to ask vhost-user for guest ABI details on migration, and block migration if vhost-user was misconfigured on the destination host when migrating? > > > The below proposal has been drafted based on how Qemu manages machine types: > > > > Proposal > > > > > > The idea is to hav
