Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators
On 2018年01月05日 18:25, Liang, Cunming wrote: -Original Message- From: Jason Wang [mailto:jasow...@redhat.com] Sent: Friday, January 5, 2018 4:39 PM To: Liang, Cunming ; Bie, Tiwei Cc: Tan, Jianfeng ; virtio-...@lists.oasis-open.org; m...@redhat.com; qemu-devel@nongnu.org; alex.william...@redhat.com; Wang, Xiao W ; stefa...@redhat.com; Wang, Zhihong ; pbonz...@redhat.com; Daly, Dan Subject: Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators On 2018年01月05日 14:58, Liang, Cunming wrote: Thanks for the pointer. Looks rather interesting. We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details). This is exactly what I mean. Form my point of view, there's no need for any extension for vhost protocol, we just need to reuse qemu iothread to implement a userspace vhost dataplane and do the mdev inside that thread. On functional perspective, it makes sense to have qemu native support of those certain usage. However, qemu doesn't have to take responsibility for dataplane. There're already huge amounts of codes for different devices emulation, leveraging external dataplane library is an effective way to introduce more. This does not mean to drop external dataplane library. Actually, you can link dpdk to qemu directly. It's not a bad idea, then the interface comes to be new API/ABI definition of external dataplane library instead of existing vhost protocol. These API/ABI should be qemu internal which should be much flexible than vhost-user. dpdk as a library is not a big deal to link with, customized application is. In addition, it will ask for qemu to provide flexible process model then. Lots of application level features (e.g. hot upgrade/fix) becomes burden. Don't quite get this, I think we can solve this by migration. Even if a dpdk userspace backend can do this, it can only do upgrade and fix for network datapath. This is not a complete solution obviously. It's nice to discuss this but it was a little bit out of the topic. I'm open to that option, keep eyes on any proposal there. The beauty of vhost_user is to open a door for variable userland workloads(e.g. vswitch). The dataplane connected with VM usually need to be close integrated with those userland workloads, a control place interface(vhost-user) is better than a datapath interface(e.g. provided by dataplace in qemu iothread). Do we really need vswitch for vDPA? Accelerators come into the picture of vswitch, which usually provides in-chip EMC for early classification. It gives a fast path for those throughput sensitive(SLA) VNF to bypass the further table lookup. It co-exists other VNF whose SLA level is best effort but requires more functions(e.g. stateful conntrack, security check, even higher layer WAF support) support, DPDK based datapath still boost the throughput there. It's not used to be a single choice of dedicated or shared datapath, usually they're co-exist. So if I understand this correctly, the "vswtich" here is a hardware function (something like smart NICs or OVS offloaded). So the question still, is vhost-user a must in this case? On workloads point of view, it's not excited to be part of qemu process. Don't see why, qemu have dataplane for virtio-blk/scsi. Qemu has vhost-user for scsi too. I'm not saying which one is bad, just point out sometime it's very workloads driven. Network is different with blk/scsi/crypto. What's the main difference from your point of view which makes vhost-user a must in this case? That comes up with the idea of vhost-user extension. Userland workloads decides to enable accelerators or not, qemu provides the common control plane infrastructure. It brings extra complexity: endless new types of messages and a huge brunch of bugs. And what's more important, the split model tends to be less efficient in some cases, e.g guest IOMMU integration. I'm pretty sure we will meet more in the future. vIOMMU relevant message has been supported by vhost protocol. It's independent effort there. The point is vIOMMU integration is very inefficient in vhost-user for some cases. If you have lots of dynamic mappings, it can have only 5%-10% performance compared to vIOMMU disabled. A huge amount of translation request will be generated in this case. The main issue here is you can not offload datapath completely to vhost-user backends completely, IOMMU translations were still done in qemu. This is one of the defect of vhost-user when datapath need to access the device state. I don't see this patch introduce endless new types. Not this patch but we can imagine vhost-user protocol will become complex in the future. My taking of your fundamental concern is about continues adding new features on vhost-user. Feel f
Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators
On 2017年12月22日 14:41, Tiwei Bie wrote: This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU. When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU. The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to): - consistent device interface from guest OS; - max flexibility on control path and hardware design; - leveraging the existing virtio live-migration framework; But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support: 1) guest driver notifies the device directly; 2) device interrupts the guest directly; So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through. A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch. There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated! Tiwei Bie (3): vhost-user: support receiving file descriptors in slave_read vhost-user: introduce shared vhost-user state vhost-user: add VFIO based accelerators support docs/interop/vhost-user.txt | 57 ++ hw/scsi/vhost-user-scsi.c | 6 +- hw/vfio/common.c| 2 +- hw/virtio/vhost-user.c | 430 +++- hw/virtio/vhost.c | 3 +- hw/virtio/virtio-pci.c | 8 - hw/virtio/virtio-pci.h | 8 + include/hw/vfio/vfio.h | 2 + include/hw/virtio/vhost-user.h | 43 include/hw/virtio/virtio-scsi.h | 6 +- net/vhost-user.c| 30 +-- 11 files changed, 561 insertions(+), 34 deletions(-) create mode 100644 include/hw/virtio/vhost-user.h I may miss something, but may I ask why you must implement them through vhost-use/dpdk. It looks to me you could put all of them in qemu which could simplify a lots of things (just like userspace NVME driver wrote by Fam). Thanks
Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators
On Wed, Jan 03, 2018 at 10:34:36PM +0800, Jason Wang wrote: > On 2017年12月22日 14:41, Tiwei Bie wrote: > > This RFC patch set does some small extensions to vhost-user protocol > > to support VFIO based accelerators, and makes it possible to get the > > similar performance of VFIO passthru while keeping the virtio device > > emulation in QEMU. > > > > When we have virtio ring compatible devices, it's possible to setup > > the device (DMA mapping, PCI config, etc) based on the existing info > > (memory-table, features, vring info, etc) which is available on the > > vhost-backend (e.g. DPDK vhost library). Then, we will be able to > > use such devices to accelerate the emulated device for the VM. And > > we call it vDPA: vhost DataPath Acceleration. The key difference > > between VFIO passthru and vDPA is that, in vDPA only the data path > > (e.g. ring, notify and queue interrupt) is pass-throughed, the device > > control path (e.g. PCI configuration space and MMIO regions) is still > > defined and emulated by QEMU. > > > > The benefits of keeping virtio device emulation in QEMU compared > > with virtio device VFIO passthru include (but not limit to): > > > > - consistent device interface from guest OS; > > - max flexibility on control path and hardware design; > > - leveraging the existing virtio live-migration framework; > > > > But the critical issue in vDPA is that the data path performance is > > relatively low and some host threads are needed for the data path, > > because some necessary mechanisms are missing to support: > > > > 1) guest driver notifies the device directly; > > 2) device interrupts the guest directly; > > > > So this patch set does some small extensions to vhost-user protocol > > to make both of them possible. It leverages the same mechanisms (e.g. > > EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to > > achieve the data path pass through. > > > > A new protocol feature bit is added to negotiate the accelerator feature > > support. Two new slave message types are added to enable the notify and > > interrupt passthru for each queue. From the view of vhost-user protocol > > design, it's very flexible. The passthru can be enabled/disabled for > > each queue individually, and it's possible to accelerate each queue by > > different devices. More design and implementation details can be found > > from the last patch. > > > > There are some rough edges in this patch set (so this is a RFC patch > > set for now), but it's never too early to hear the thoughts from the > > community! So any comments and suggestions would be really appreciated! > > > > Tiwei Bie (3): > >vhost-user: support receiving file descriptors in slave_read > >vhost-user: introduce shared vhost-user state > >vhost-user: add VFIO based accelerators support > > > > docs/interop/vhost-user.txt | 57 ++ > > hw/scsi/vhost-user-scsi.c | 6 +- > > hw/vfio/common.c| 2 +- > > hw/virtio/vhost-user.c | 430 > > +++- > > hw/virtio/vhost.c | 3 +- > > hw/virtio/virtio-pci.c | 8 - > > hw/virtio/virtio-pci.h | 8 + > > include/hw/vfio/vfio.h | 2 + > > include/hw/virtio/vhost-user.h | 43 > > include/hw/virtio/virtio-scsi.h | 6 +- > > net/vhost-user.c| 30 +-- > > 11 files changed, 561 insertions(+), 34 deletions(-) > > create mode 100644 include/hw/virtio/vhost-user.h > > > > I may miss something, but may I ask why you must implement them through > vhost-use/dpdk. It looks to me you could put all of them in qemu which could > simplify a lots of things (just like userspace NVME driver wrote by Fam). > Thanks for your comments! :-) Yeah, you're right. We can also implement everything in QEMU like the userspace NVME driver by Fam. It was also described by Cunming on the KVM Forum 2017. Below is the link to the slides: https://events.static.linuxfound.org/sites/events/files/slides/KVM17%27-vDPA.pdf We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details). And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands to QEMU: - the switching among different accelerators and software version can be done at runtime in vhost process; - use different accelerators to accelerate different queue pairs or just accelerate some (instead of all) queue pairs; Best regards, Tiwei Bie
Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators
On 2018年01月04日 14:18, Tiwei Bie wrote: On Wed, Jan 03, 2018 at 10:34:36PM +0800, Jason Wang wrote: On 2017年12月22日 14:41, Tiwei Bie wrote: This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU. When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU. The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to): - consistent device interface from guest OS; - max flexibility on control path and hardware design; - leveraging the existing virtio live-migration framework; But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support: 1) guest driver notifies the device directly; 2) device interrupts the guest directly; So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through. A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch. There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated! Tiwei Bie (3): vhost-user: support receiving file descriptors in slave_read vhost-user: introduce shared vhost-user state vhost-user: add VFIO based accelerators support docs/interop/vhost-user.txt | 57 ++ hw/scsi/vhost-user-scsi.c | 6 +- hw/vfio/common.c| 2 +- hw/virtio/vhost-user.c | 430 +++- hw/virtio/vhost.c | 3 +- hw/virtio/virtio-pci.c | 8 - hw/virtio/virtio-pci.h | 8 + include/hw/vfio/vfio.h | 2 + include/hw/virtio/vhost-user.h | 43 include/hw/virtio/virtio-scsi.h | 6 +- net/vhost-user.c| 30 +-- 11 files changed, 561 insertions(+), 34 deletions(-) create mode 100644 include/hw/virtio/vhost-user.h I may miss something, but may I ask why you must implement them through vhost-use/dpdk. It looks to me you could put all of them in qemu which could simplify a lots of things (just like userspace NVME driver wrote by Fam). Thanks for your comments! :-) Yeah, you're right. We can also implement everything in QEMU like the userspace NVME driver by Fam. It was also described by Cunming on the KVM Forum 2017. Below is the link to the slides: https://events.static.linuxfound.org/sites/events/files/slides/KVM17%27-vDPA.pdf Thanks for the pointer. Looks rather interesting. We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details). This is exactly what I mean. Form my point of view, there's no need for any extension for vhost protocol, we just need to reuse qemu iothread to implement a userspace vhost dataplane and do the mdev inside that thread. And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands to QEMU: Maybe I was wrong but I don't think we care about the complexity of command line or monitor command in this case. - the switching among different accelerators and software version can be done at runtime in vhost process; - use different accelerators to accelerate different queue pairs or just accelerate some (instead of all) queue pairs; Well, technically, if we want, these could be implemented in qemu too. And here's some more advantages if you
Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators
> -Original Message- > From: Jason Wang [mailto:jasow...@redhat.com] > Sent: Thursday, January 4, 2018 3:22 PM > To: Bie, Tiwei > Cc: Tan, Jianfeng ; virtio-...@lists.oasis-open.org; > m...@redhat.com; Liang, Cunming ; qemu- > de...@nongnu.org; alex.william...@redhat.com; Wang, Xiao W > ; stefa...@redhat.com; Wang, Zhihong > ; pbonz...@redhat.com; Daly, Dan > > Subject: Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support > VFIO based accelerators > > > > On 2018年01月04日 14:18, Tiwei Bie wrote: > > On Wed, Jan 03, 2018 at 10:34:36PM +0800, Jason Wang wrote: > >> On 2017年12月22日 14:41, Tiwei Bie wrote: > >>> This RFC patch set does some small extensions to vhost-user protocol > >>> to support VFIO based accelerators, and makes it possible to get the > >>> similar performance of VFIO passthru while keeping the virtio device > >>> emulation in QEMU. > >>> > >>> When we have virtio ring compatible devices, it's possible to setup > >>> the device (DMA mapping, PCI config, etc) based on the existing info > >>> (memory-table, features, vring info, etc) which is available on the > >>> vhost-backend (e.g. DPDK vhost library). Then, we will be able to > >>> use such devices to accelerate the emulated device for the VM. And > >>> we call it vDPA: vhost DataPath Acceleration. The key difference > >>> between VFIO passthru and vDPA is that, in vDPA only the data path > >>> (e.g. ring, notify and queue interrupt) is pass-throughed, the > >>> device control path (e.g. PCI configuration space and MMIO regions) > >>> is still defined and emulated by QEMU. > >>> > >>> The benefits of keeping virtio device emulation in QEMU compared > >>> with virtio device VFIO passthru include (but not limit to): > >>> > >>> - consistent device interface from guest OS; > >>> - max flexibility on control path and hardware design; > >>> - leveraging the existing virtio live-migration framework; > >>> > >>> But the critical issue in vDPA is that the data path performance is > >>> relatively low and some host threads are needed for the data path, > >>> because some necessary mechanisms are missing to support: > >>> > >>> 1) guest driver notifies the device directly; > >>> 2) device interrupts the guest directly; > >>> > >>> So this patch set does some small extensions to vhost-user protocol > >>> to make both of them possible. It leverages the same mechanisms (e.g. > >>> EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to > >>> achieve the data path pass through. > >>> > >>> A new protocol feature bit is added to negotiate the accelerator > >>> feature support. Two new slave message types are added to enable the > >>> notify and interrupt passthru for each queue. From the view of > >>> vhost-user protocol design, it's very flexible. The passthru can be > >>> enabled/disabled for each queue individually, and it's possible to > >>> accelerate each queue by different devices. More design and > >>> implementation details can be found from the last patch. > >>> > >>> There are some rough edges in this patch set (so this is a RFC patch > >>> set for now), but it's never too early to hear the thoughts from the > >>> community! So any comments and suggestions would be really > appreciated! > >>> > >>> Tiwei Bie (3): > >>> vhost-user: support receiving file descriptors in slave_read > >>> vhost-user: introduce shared vhost-user state > >>> vhost-user: add VFIO based accelerators support > >>> > >>>docs/interop/vhost-user.txt | 57 ++ > >>>hw/scsi/vhost-user-scsi.c | 6 +- > >>>hw/vfio/common.c| 2 +- > >>>hw/virtio/vhost-user.c | 430 > +++- > >>>hw/virtio/vhost.c | 3 +- > >>>hw/virtio/virtio-pci.c | 8 - > >>>hw/virtio/virtio-pci.h | 8 + > >>>include/hw/vfio/vfio.h | 2 + > >>>include/hw/virtio/vhost-user.h | 43 > >>>include/hw/virtio/virtio-scsi.h | 6 +- > >>>net/vhost-user.c| 30 +-- > >>>11 files changed, 561 insertions(+), 34 deletions(-) > >>>
Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators
On 2018年01月05日 14:58, Liang, Cunming wrote: Thanks for the pointer. Looks rather interesting. We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details). This is exactly what I mean. Form my point of view, there's no need for any extension for vhost protocol, we just need to reuse qemu iothread to implement a userspace vhost dataplane and do the mdev inside that thread. On functional perspective, it makes sense to have qemu native support of those certain usage. However, qemu doesn't have to take responsibility for dataplane. There're already huge amounts of codes for different devices emulation, leveraging external dataplane library is an effective way to introduce more. This does not mean to drop external dataplane library. Actually, you can link dpdk to qemu directly. The beauty of vhost_user is to open a door for variable userland workloads(e.g. vswitch). The dataplane connected with VM usually need to be close integrated with those userland workloads, a control place interface(vhost-user) is better than a datapath interface(e.g. provided by dataplace in qemu iothread). Do we really need vswitch for vDPA? On workloads point of view, it's not excited to be part of qemu process. Don't see why, qemu have dataplane for virtio-blk/scsi. That comes up with the idea of vhost-user extension. Userland workloads decides to enable accelerators or not, qemu provides the common control plane infrastructure. It brings extra complexity: endless new types of messages and a huge brunch of bugs. And what's more important, the split model tends to be less efficient in some cases, e.g guest IOMMU integration. I'm pretty sure we will meet more in the future. And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands to QEMU: Maybe I was wrong but I don't think we care about the complexity of command line or monitor command in this case. - the switching among different accelerators and software version can be done at runtime in vhost process; - use different accelerators to accelerate different queue pairs or just accelerate some (instead of all) queue pairs; Well, technically, if we want, these could be implemented in qemu too. You're right if just considering I/O. The ways to consume those I/O is another perspective. Simply 1:1 associating guest virtio-net and accelerator w/ SW datapath fallback is not the whole picture. Pay attention: 1) What I mean is not a fallback here. You can still do a lot of tricks e.g offloading datapath to hardware or doorbell map. 2) Qemu supports (very old and inefficient) a split model of device emulation and network backend. This means we can switch between backends (though not implemented). It's variable usages on workload side to abstract the device (e.g. port re-presenter for vswitch) and etc. I don't think qemu is interested for all bunch of things there. Again, you can link any dataplane to qemu directly instead of using vhost-user if vhost-user tends to be less useful in some cases (vDPA is one of the case I think). Thanks
Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators
> -Original Message- > From: Jason Wang [mailto:jasow...@redhat.com] > Sent: Friday, January 5, 2018 4:39 PM > To: Liang, Cunming ; Bie, Tiwei > > Cc: Tan, Jianfeng ; virtio-...@lists.oasis-open.org; > m...@redhat.com; qemu-devel@nongnu.org; alex.william...@redhat.com; > Wang, Xiao W ; stefa...@redhat.com; Wang, > Zhihong ; pbonz...@redhat.com; Daly, Dan > > Subject: Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support > VFIO based accelerators > > > > On 2018年01月05日 14:58, Liang, Cunming wrote: > >> Thanks for the pointer. Looks rather interesting. > >> > >>> We're also working on it (including defining a standard device for > >>> vhost data path acceleration based on mdev to hide vendor specific > >>> details). > >> This is exactly what I mean. Form my point of view, there's no need > >> for any extension for vhost protocol, we just need to reuse qemu > >> iothread to implement a userspace vhost dataplane and do the mdev > inside that thread. > > On functional perspective, it makes sense to have qemu native support of > those certain usage. However, qemu doesn't have to take responsibility for > dataplane. There're already huge amounts of codes for different devices > emulation, leveraging external dataplane library is an effective way to > introduce more. > > This does not mean to drop external dataplane library. Actually, you can link > dpdk to qemu directly. It's not a bad idea, then the interface comes to be new API/ABI definition of external dataplane library instead of existing vhost protocol. dpdk as a library is not a big deal to link with, customized application is. In addition, it will ask for qemu to provide flexible process model then. Lots of application level features (e.g. hot upgrade/fix) becomes burden. I'm open to that option, keep eyes on any proposal there. > > > The beauty of vhost_user is to open a door for variable userland > workloads(e.g. vswitch). The dataplane connected with VM usually need to > be close integrated with those userland workloads, a control place > interface(vhost-user) is better than a datapath interface(e.g. provided by > dataplace in qemu iothread). > > Do we really need vswitch for vDPA? Accelerators come into the picture of vswitch, which usually provides in-chip EMC for early classification. It gives a fast path for those throughput sensitive(SLA) VNF to bypass the further table lookup. It co-exists other VNF whose SLA level is best effort but requires more functions(e.g. stateful conntrack, security check, even higher layer WAF support) support, DPDK based datapath still boost the throughput there. It's not used to be a single choice of dedicated or shared datapath, usually they're co-exist. > > > On workloads point of view, it's not excited to be part of qemu process. > > Don't see why, qemu have dataplane for virtio-blk/scsi. Qemu has vhost-user for scsi too. I'm not saying which one is bad, just point out sometime it's very workloads driven. Network is different with blk/scsi/crypto. > > > That comes up with the idea of vhost-user extension. Userland workloads > decides to enable accelerators or not, qemu provides the common control > plane infrastructure. > > It brings extra complexity: endless new types of messages and a huge brunch > of bugs. And what's more important, the split model tends to be less efficient > in some cases, e.g guest IOMMU integration. I'm pretty sure we will meet > more in the future. vIOMMU relevant message has been supported by vhost protocol. It's independent effort there. I don't see this patch introduce endless new types. My taking of your fundamental concern is about continues adding new features on vhost-user. Feel free to correct me if I misunderstood your point. > > >>> And IMO it's also not a bad idea to extend vhost-user protocol to > >>> support the accelerators if possible. And it could be more flexible > >>> because it could support (for example) below things easily without > >>> introducing any complex command line options or monitor commands to > >>> QEMU: > >> Maybe I was wrong but I don't think we care about the complexity of > >> command line or monitor command in this case. > >> > >>> - the switching among different accelerators and software version > >>> can be done at runtime in vhost process; > >>> - use different accelerators to accelerate different queue pairs > >>> or just accelerate some (instead of all) queue pairs; > >> Well, technically, if we want, these could be impl
