On 9/8/2015 12:45 PM, Tetsuya Mukawa wrote: > On 2015/09/07 14:54, Xie, Huawei wrote: >> On 8/26/2015 5:23 PM, Tetsuya Mukawa wrote: >>> On 2015/08/25 18:56, Xie, Huawei wrote: >>>> On 8/25/2015 10:59 AM, Tetsuya Mukawa wrote: >>>>> Hi Xie and Yanping, >>>>> >>>>> >>>>> May I ask you some questions? >>>>> It seems we are also developing an almost same one. >>>> Good to know that we are tackling the same problem and have the similar >>>> idea. >>>> What is your status now? We had the POC running, and compliant with >>>> dpdkvhost. >>>> Interrupt like notification isn't supported. >>> We implemented vhost PMD first, so we just start implementing it. >>> >>>>> On 2015/08/20 19:14, Xie, Huawei wrote: >>>>>> Added dev at dpdk.org >>>>>> >>>>>> On 8/20/2015 6:04 PM, Xie, Huawei wrote: >>>>>>> Yanping: >>>>>>> I read your mail, seems what we did are quite similar. Here i wrote a >>>>>>> quick mail to describe our design. Let me know if it is the same thing. >>>>>>> >>>>>>> Problem Statement: >>>>>>> We don't have a high performance networking interface in container for >>>>>>> NFV. Current veth pair based interface couldn't be easily accelerated. >>>>>>> >>>>>>> The key components involved: >>>>>>> 1. DPDK based virtio PMD driver in container. >>>>>>> 2. device simulation framework in container. >>>>>>> 3. dpdk(or kernel) vhost running in host. >>>>>>> >>>>>>> How virtio is created? >>>>>>> A: There is no "real" virtio-pci device in container environment. >>>>>>> 1). Host maintains pools of memories, and shares memory to container. >>>>>>> This could be accomplished through host share a huge page file to >>>>>>> container. >>>>>>> 2). Containers creates virtio rings based on the shared memory. >>>>>>> 3). Container creates mbuf memory pools on the shared memory. >>>>>>> 4) Container send the memory and vring information to vhost through >>>>>>> vhost message. This could be done either through ioctl call or vhost >>>>>>> user message. >>>>>>> >>>>>>> How vhost message is sent? >>>>>>> A: There are two alternative ways to do this. >>>>>>> 1) The customized virtio PMD is responsible for all the vring creation, >>>>>>> and vhost message sending. >>>>> Above is our approach so far. >>>>> It seems Yanping also takes this kind of approach. >>>>> We are using vhost-user functionality instead of using the vhost-net >>>>> kernel module. >>>>> Probably this is the difference between Yanping and us. >>>> In my current implementation, the device simulation layer talks to "user >>>> space" vhost through cuse interface. It could also be done through vhost >>>> user socket. This isn't the key point. >>>> Here vhost-user is kind of confusing, maybe user space vhost is more >>>> accurate, either cuse or unix domain socket. :). >>>> >>>> As for yanping, they are now connecting to vhost-net kernel module, but >>>> they are also trying to connect to "user space" vhost. Correct me if >>>> wrong. >>>> Yes, there is some difference between these two. Vhost-net kernel module >>>> could directly access other process's memory, while using >>>> vhost-user(cuse/user), we need do the memory mapping. >>>>> BTW, we are going to submit a vhost PMD for DPDK-2.2. >>>>> This PMD is implemented on librte_vhost. >>>>> It allows DPDK application to handle a vhost-user(cuse) backend as a >>>>> normal NIC port. >>>>> This PMD should work with both Xie and Yanping approach. >>>>> (In the case of Yanping approach, we may need vhost-cuse) >>>>> >>>>>>> 2) We could do this through a lightweight device simulation framework. >>>>>>> The device simulation creates simple PCI bus. On the PCI bus, >>>>>>> virtio-net PCI devices are created. The device simulations provides >>>>>>> IOAPI for MMIO/IO access. >>>>> Does it mean you implemented a kernel module? >>>>> If so, do you still need vhost-cuse functionality to handle vhost >>>>> messages n userspace? >>>> The device simulation is a library running in user space in container. >>>> It is linked with DPDK app. It creates pseudo buses and virtio-net PCI >>>> devices. >>>> The virtio-container-PMD configures the virtio-net pseudo devices >>>> through IOAPI provided by the device simulation rather than IO >>>> instructions as in KVM. >>>> Why we use device simulation? >>>> We could create other virtio devices in container, and provide an common >>>> way to talk to vhost-xx module. >>> Thanks for explanation. >>> At first reading, I thought the difference between approach1 and >>> approach2 is whether we need to implement a new kernel module, or not. >>> But I understand how you implemented. >>> >>> Please let me explain our design more. >>> We might use a kind of similar approach to handle a pseudo virtio-net >>> device in DPDK. >>> (Anyway, we haven't finished implementing yet, this overview might have >>> some technical problems) >>> >>> Step1. Separate virtio-net and vhost-user socket related code from QEMU, >>> then implement it as a separated program. >>> The program also has below features. >>> - Create a directory that contains almost same files like >>> /sys/bus/pci/device/<pci address>/* >>> (To scan these file located on outside sysfs, we need to fix EAL) >>> - This dummy device is driven by dummy-virtio-net-driver. This name is >>> specified by '<pci addr>/driver' file. >>> - Create a shared file that represents pci configuration space, then >>> mmap it, also specify the path in '<pci addr>/resource_path' >>> >>> The program will be GPL, but it will be like a bridge on the shared >>> memory between virtio-net PMD and DPDK vhost backend. >>> Actually, It will work under virtio-net PMD, but we don't need to link it. >>> So I guess we don't have GPL license issue. >>> >>> Step2. Fix pci scan code of EAL to scan dummy devices. >>> - To scan above files, extend pci_scan() of EAL. >>> >>> Step3. Add a new kdrv type to EAL. >>> - To handle the 'dummy-virtio-net-driver', add a new kdrv type to EAL. >>> >>> Step4. Implement pci_dummy_virtio_net_map/unmap(). >>> - It will have almost same functionality like pci_uio_map(), but for >>> dummy virtio-net device. >>> - The dummy device will be mmaped using a path specified in '<pci >>> addr>/resource_path'. >>> >>> Step5. Add a new compile option for virtio-net device to replace IO >>> functions. >>> - The IO functions of virtio-net PMD will be replaced by read() and >>> write() to access to the shared memory. >>> - Add notification mechanism to IO functions. This will be used when >>> write() to the shared memory is done. >>> (Not sure exactly, but probably we need it) >>> >>> Does it make sense? >>> I guess Step1&2 is different from your approach, but the rest might be >>> similar. >>> >>> Actually, we just need sysfs entries for a virtio-net dummy device, but >>> so far, I don't have a fine way to register them from user space without >>> loading a kernel module. >> Tetsuya: >> I don't quite get the details. Who will create those sysfs entries? A >> kernel module right? > Hi Xie, > > I don't create sysfs entries. Just create a directory that contains > files looks like sysfs entries. > And initialize EAL with not only sysfs but also the above directory. > > In quoted last sentence, I wanted to say we just needed files looks like > sysfs entries. > But I don't know a good way to create files under sysfs without loading > kernel module. > This is because I try to create the additional directory. > >> The virtio-net is configured through read/write to sharing >> memory(between host and guest), right? > Yes, I agree. > >> Where is shared vring created and shared memory created, on shared huge >> page between host and guest? > The vritqueues(vrings) are on guest hugepage. > > Let me explain. > Guest container should have read/write access to a part of hugepage > directory on host. > (For example, /mnt/huge/conainer1/ is shared between host and guest.) > Also host and guest needs to communicate through a unix domain socket. > (For example, host and guest can communicate with using > "/tmp/container1/sock") > > If we can do like above, a virtio-net PMD on guest can creates > virtqueues(vrings) on it's hugepage, and writes these information to a > pseudo virtio-net device that is a process created in guest container. > Then the pseudo virtio-net device sends it to vhost-user backend(host > DPDK application) through a unix domain socket. > > So with my plan, there are 3 processes. > DPDK applications on host and guest, also a process that works like > virtio-net device. > >> Who will talk to dpdkvhost? > If we need to talk to a cuse device or the vhost-net kernel module, an > above pseudo virtio-net device could talk to. > (But, so far, my target is only vhost-user.) > >>> This is because I need to change pci_scan() also. >>> >>> It seems you have implemented a virtio-net pseudo device as BSD license. >>> If so, this kind of PMD would be nice to use it. >> Currently it is based on native linux kvm tool. > Great, I hadn't noticed this option. > >>> In the case that it takes much time to implement some lost >>> functionalities like interrupt mode, using QEMU code might be an one of >>> options. >> For interrupt mode, i plan to use eventfd for sleep/wake, have not tried >> yet. >>> Anyway, we just need a fine virtual NIC between containers and host. >>> So we don't hold to our approach and implementation. >> Do you have comments to my implementation? >> We could publish the version without the device framework first for >> reference. > No I don't have. Could you please share it? > I am looking forward to seeing it. OK, we are removing the device framework. Hope to publish it in one month's time.
> > Tetsuya >
