[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On 2016/06/06 19:50, Tan, Jianfeng wrote: >> Please let me make sure how we can invoke many DPDK applications in >> hundreds containers. >> (Do we have a way to do? Or, will we have it in the future?) > > Just to add some option here, we cannot say no to that kind of use case. > To have many instances, we can: > > (1) add a restriction of "cpu share" on each instance, relying on kernel > to schedule. > (2) enable interrupt mode, so that one instance can go to sleep when it > has no pkts to receive and awoke by vhost backend when pkts come. > > Option 2 is my choice. Hi Yuanhan and Jianfeng, Thanks for your descriptions about how you will invoke many DPDK applications in containers. I guess we have almost talked everything we need to consider to choose one of container implementations. We may have one conclusion about this choice. If we can easily maintain virtio device implementation, also if we have an use-case to invoke hundreds of DPDK applications in containers, I guess Jianfeng's implementation will be nice. Anyway, we just follow virtio maintainers choice. Thanks, Tetsuya > > Thanks, > Jianfeng > >> >> Thanks, >> Tetsuya > >
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On Tue, Jun 07, 2016 at 04:12:28PM +0900, Tetsuya Mukawa wrote: > On 2016/06/06 19:50, Tan, Jianfeng wrote: > >> Please let me make sure how we can invoke many DPDK applications in > >> hundreds containers. > >> (Do we have a way to do? Or, will we have it in the future?) > > > > Just to add some option here, we cannot say no to that kind of use case. > > To have many instances, we can: > > > > (1) add a restriction of "cpu share" on each instance, relying on kernel > > to schedule. > > (2) enable interrupt mode, so that one instance can go to sleep when it > > has no pkts to receive and awoke by vhost backend when pkts come. > > > > Option 2 is my choice. > > Hi Yuanhan and Jianfeng, > > Thanks for your descriptions about how you will invoke many DPDK > applications in containers. > I guess we have almost talked everything we need to consider to choose > one of container implementations. > > We may have one conclusion about this choice. > If we can easily maintain virtio device implementation, AFAIK, yes. > also if we have > an use-case to invoke hundreds of DPDK applications in containers, I Don't know yet, but it seems easier to achieve that with Jianfeng's solution. > guess Jianfeng's implementation will be nice. I'm afraid that's what I'm seeing. > Anyway, we just follow virtio maintainers choice. Thanks, and of course, contribution is huge welcome so that we could have a better container solution! --yliu
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
Hi, On 6/6/2016 5:30 PM, Tetsuya Mukawa wrote: > On 2016/06/06 17:49, Yuanhan Liu wrote: >> On Mon, Jun 06, 2016 at 05:33:31PM +0900, Tetsuya Mukawa wrote: > [My solution] > - Pros > Basic principle of my implementation is not to reinvent the wheel. Yes, that's a good point. However, it's not that hard as we would have thought in the first time: the tough part that dequeue/enqueue packets from/to vring is actually offloaded to DPDK vhost-user. That means we only need re-implement the control path of virtio-net device, plus the vhost-user frontend. If you have a detailed look of your patchset as well Jianfeng's, you might find that the two patchset are actually with same code size. >>> Yes, I know this. >>> So far, the amount of code is almost same, but in the future we may need >>> to implement more, if virtio-net specification is revised. >> It didn't take too much effort to implement from scratch, I doubt it >> will for future revise. And, virtio-net spec is unlikely revised, or >> to be precisely, unlikely revised quite often. Therefore, I don't see >> big issues here. >> > We can use a virtio-net device of QEMU implementation, it means we don't > need to maintain virtio-net device by ourselves, and we can use all of > functions supported by QEMU virtio-net device. > - Cons > Need to invoke QEMU process. Another thing is that it makes the usage a bit harder: look at the long qemu cli options of your example usage. It also has some traps, say, "--enable-kvm" is not allowed, which is a default option used with QEMU. >>> Probably a kind of shell script will help the users. >> Yeah, that would help. But if we have a choice to make it simpler in the >> beginning, why not then? :-) >> And judging that we actually don't take too much effort to implement a virtio device emulation, I'd prefer it slightly. I guess something light weight and easier for use is more important here. >>> This is very important point. >>> If so, we don't need much effort when virtio-spec is changed. >> I'd assume so. >> Actually, I have foreseen another benefit of adding virtio-user device emulation: we now might be able to add a rte_vhost_dequeue/enqueue_burst() unit test case. We simply can't do it before, since we depend on QEMU for testing, which is not acceptable for a unit test case. Making it be a unit test case would help us spotting any bad changes that would introduce bugs easily and automatically. >>> As you mentioned above, QEMU process is not related with >>> dequeuing/enqueuing. >>> So I guess we may have a testing for rte_vhost_dequeue/enqueue_burst() >>> regardless of choice. >> Yes, we don't need the dequeue/enqueue part, but we need the vhost-user >> initialization part from QEMU vhost-user. Now that we have vhost-user >> frontend from virtio-user, we have no dependency on QEMU any more. >> > Anyway, we can choose one of belows. > 1. Take advantage of invoking less processes. > 2. Take advantage of maintainability of virtio-net device. >>> If container usage that DPDK assumes is to invoke hundreds containers in >>> one host, >> I barely know about container, but I would assume that's not rare. > Hi Yuanhan, > > It's great to hear it's not so hard to maintain Jiangfeng's virtio-net > device features. > > Please let me make sure how we can invoke many DPDK applications in > hundreds containers. > (Do we have a way to do? Or, will we have it in the future?) Just to add some option here, we cannot say no to that kind of use case. To have many instances, we can: (1) add a restriction of "cpu share" on each instance, relying on kernel to schedule. (2) enable interrupt mode, so that one instance can go to sleep when it has no pkts to receive and awoke by vhost backend when pkts come. Option 2 is my choice. Thanks, Jianfeng > > Thanks, > Tetsuya
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
Hi, On 6/6/2016 5:28 PM, Tetsuya Mukawa wrote: > On 2016/06/06 17:03, Tan, Jianfeng wrote: >> Hi, >> >> >> On 6/6/2016 1:10 PM, Tetsuya Mukawa wrote: >>> Hi Yuanhan, >>> >>> Sorry for late replying. >>> >>> On 2016/06/03 13:17, Yuanhan Liu wrote: On Thu, Jun 02, 2016 at 06:30:18PM +0900, Tetsuya Mukawa wrote: > Hi Yuanhan, > > On 2016/06/02 16:31, Yuanhan Liu wrote: >> But still, I'd ask do we really need 2 virtio for container solutions? > I appreciate your comments. No, I appreciate your effort for contributing to DPDK! vhost-pmd stuff is just brilliant! > Let me have time to discuss it with our team. I'm wondering could we have one solution only. IMO, the drawback of having two (quite different) solutions might outweighs the benefit it takes. Say, it might just confuse user. >>> I agree with this. >>> If we have 2 solutions, it would confuse the DPDK users. >>> OTOH, I'm wondering could you adapt to Jianfeng's solution? If not, what's the missing parts, and could we fix it? I'm thinking having one unified solution will keep ours energy/focus on one thing, making it better and better! Having two just splits the energy; it also introduces extra burden for maintaining. >>> Of course, I adopt Jiangeng's solution basically. >>> Actually, his solution is almost similar I tried to implement at first. >>> >>> I guess here is pros/cons of 2 solutions. >>> >>> [Jianfeng's solution] >>> - Pros >>> Don't need to invoke QEMU process. >>> - Cons >>> If virtio-net specification is changed, we need to implement it by >>> ourselves. >> It will barely introduce any change when virtio-net specification is >> changed as far as I can see. The only part we care is the how desc, >> avail, used distribute on memory, which is a very small part. > It's a good news, because we don't pay much effort to follow latest > virtio-net specification. > >> It's true that my solution now seriously depend on vhost-user protocol, >> which is defined in QEMU. I cannot see a big problem there so far. >> >>>Also, LSC interrupt and control queue functions are not >>> supported yet. >>> I agree both functions may not be so important, and if we need it >>> we can implement them, but we need to pay energy to implement them. >> LSC is really less important than rxq interrupt (IMO). We don't know how >> long will rxq interrupt of virtio be available for QEMU, but we can >> accelerate it if we avoid using QEMU. >> >> Actually, if the vhost backend is vhost-user (the main use case), >> current qemu have limited control queue support, because it needs the >> support from the vhost user backend. >> >> Add one more con of my solution: >> - Need to write another logic to support other virtio device (say >> virtio-scsi), if it's easier of Tetsuya's solution to do that? >> > Probably, my solution will be easier to do that. > My solution has enough facility to access to io port and PCI > configuration space of virtio-scsi device of QEMU. > So, if you invoke with QEMU with virtio-scsi, only you need to do is > changing PCI interface of current virtio-scsi PMD. > (I just assume currently we have virtio-scsi PMD.) > If the virtio-scsi PMD works on QEMU, same code should work with only > changing PCI interface. > >>> [My solution] >>> - Pros >>> Basic principle of my implementation is not to reinvent the wheel. >>> We can use a virtio-net device of QEMU implementation, it means we don't >>> need to maintain virtio-net device by ourselves, and we can use all of >>> functions supported by QEMU virtio-net device. >>> - Cons >>> Need to invoke QEMU process. >> Two more possible cons: >> a) This solution also needs to maintain qtest utility, right? > But the spec of qtest will be more stable than virtio-net. > >> b) There's still address arrange restriction, right? Although we can use >> "--base-virtaddr=0x4" to relieve this question, but how about if >> there are 2 or more devices? (By the way, is there still address arrange >> requirement for 32 bit system) > Our solutions are a virtio-net driver, and a vhost-user backend driver > needs to access to memory allocated by virtio-net driver. > If an application has 2 devices, it means 2 vhost-user backend PMD needs > to access to the same application memory, right? > Also, currently each virtio-net device has an one QEMU process. > So, I am not sure what will be problem if we have 2 devices. OK, my bad. Multiple devices should have just one "--base-virtaddr=0x4". > > BTW, 44bits limitations comes from current QEMU implementation itself. > (Actually, if modern virtio device is used, we should be able to remove > the restriction.) Good to know. > >> c) Actually, IMO this solution is sensitive to any virtio spec change >> (io port, pci configuration space). > In this case, virtio-net PMD itself will need to be fixed. > Then, my implementation will be also fixed with the same way. > Current implementation has only
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On 2016/06/06 17:49, Yuanhan Liu wrote: > On Mon, Jun 06, 2016 at 05:33:31PM +0900, Tetsuya Mukawa wrote: [My solution] - Pros Basic principle of my implementation is not to reinvent the wheel. >>> >>> Yes, that's a good point. However, it's not that hard as we would have >>> thought in the first time: the tough part that dequeue/enqueue packets >>> from/to vring is actually offloaded to DPDK vhost-user. That means we >>> only need re-implement the control path of virtio-net device, plus the >>> vhost-user frontend. If you have a detailed look of your patchset as >>> well Jianfeng's, you might find that the two patchset are actually with >>> same code size. >> >> Yes, I know this. >> So far, the amount of code is almost same, but in the future we may need >> to implement more, if virtio-net specification is revised. > > It didn't take too much effort to implement from scratch, I doubt it > will for future revise. And, virtio-net spec is unlikely revised, or > to be precisely, unlikely revised quite often. Therefore, I don't see > big issues here. > We can use a virtio-net device of QEMU implementation, it means we don't need to maintain virtio-net device by ourselves, and we can use all of functions supported by QEMU virtio-net device. - Cons Need to invoke QEMU process. >>> >>> Another thing is that it makes the usage a bit harder: look at the >>> long qemu cli options of your example usage. It also has some traps, >>> say, "--enable-kvm" is not allowed, which is a default option used >>> with QEMU. >> >> Probably a kind of shell script will help the users. > > Yeah, that would help. But if we have a choice to make it simpler in the > beginning, why not then? :-) > >>> >>> And judging that we actually don't take too much effort to implement >>> a virtio device emulation, I'd prefer it slightly. I guess something >>> light weight and easier for use is more important here. >> >> This is very important point. >> If so, we don't need much effort when virtio-spec is changed. > > I'd assume so. > >>> Actually, I have foreseen another benefit of adding virtio-user device >>> emulation: we now might be able to add a rte_vhost_dequeue/enqueue_burst() >>> unit test case. We simply can't do it before, since we depend on QEMU >>> for testing, which is not acceptable for a unit test case. Making it >>> be a unit test case would help us spotting any bad changes that would >>> introduce bugs easily and automatically. >> >> As you mentioned above, QEMU process is not related with >> dequeuing/enqueuing. >> So I guess we may have a testing for rte_vhost_dequeue/enqueue_burst() >> regardless of choice. > > Yes, we don't need the dequeue/enqueue part, but we need the vhost-user > initialization part from QEMU vhost-user. Now that we have vhost-user > frontend from virtio-user, we have no dependency on QEMU any more. > Anyway, we can choose one of belows. 1. Take advantage of invoking less processes. 2. Take advantage of maintainability of virtio-net device. >> >> If container usage that DPDK assumes is to invoke hundreds containers in >> one host, > > I barely know about container, but I would assume that's not rare. Hi Yuanhan, It's great to hear it's not so hard to maintain Jiangfeng's virtio-net device features. Please let me make sure how we can invoke many DPDK applications in hundreds containers. (Do we have a way to do? Or, will we have it in the future?) Thanks, Tetsuya > >> we should take Jiangfeng's solution. >> >> Also, if implementing a new feature and maintaining Jiangfeng's >> virtio-net device are not so hard, > > As stated, I would assume so. > > --yliu > >> we should take his solution. >> >> I guess this is the point we need to consider. >> What do you think? >> >> Thanks, >> Tetsuya >> Honestly, I'm OK if my solution is not merged. Thus, it should be decided to let DPDK better. What do you think? Which is better for DPDK? Thanks, Tetsuya > > --yliu >
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On 2016/06/06 17:03, Tan, Jianfeng wrote: > Hi, > > > On 6/6/2016 1:10 PM, Tetsuya Mukawa wrote: >> Hi Yuanhan, >> >> Sorry for late replying. >> >> On 2016/06/03 13:17, Yuanhan Liu wrote: >>> On Thu, Jun 02, 2016 at 06:30:18PM +0900, Tetsuya Mukawa wrote: Hi Yuanhan, On 2016/06/02 16:31, Yuanhan Liu wrote: > But still, I'd ask do we really need 2 virtio for container solutions? I appreciate your comments. >>> No, I appreciate your effort for contributing to DPDK! vhost-pmd stuff >>> is just brilliant! >>> Let me have time to discuss it with our team. >>> I'm wondering could we have one solution only. IMO, the drawback of >>> having two (quite different) solutions might outweighs the benefit >>> it takes. Say, it might just confuse user. >> I agree with this. >> If we have 2 solutions, it would confuse the DPDK users. >> >>> OTOH, I'm wondering could you adapt to Jianfeng's solution? If not, >>> what's the missing parts, and could we fix it? I'm thinking having >>> one unified solution will keep ours energy/focus on one thing, making >>> it better and better! Having two just splits the energy; it also >>> introduces extra burden for maintaining. >> Of course, I adopt Jiangeng's solution basically. >> Actually, his solution is almost similar I tried to implement at first. >> >> I guess here is pros/cons of 2 solutions. >> >> [Jianfeng's solution] >> - Pros >> Don't need to invoke QEMU process. >> - Cons >> If virtio-net specification is changed, we need to implement it by >> ourselves. > > It will barely introduce any change when virtio-net specification is > changed as far as I can see. The only part we care is the how desc, > avail, used distribute on memory, which is a very small part. It's a good news, because we don't pay much effort to follow latest virtio-net specification. > > It's true that my solution now seriously depend on vhost-user protocol, > which is defined in QEMU. I cannot see a big problem there so far. > >> Also, LSC interrupt and control queue functions are not >> supported yet. >> I agree both functions may not be so important, and if we need it >> we can implement them, but we need to pay energy to implement them. > > LSC is really less important than rxq interrupt (IMO). We don't know how > long will rxq interrupt of virtio be available for QEMU, but we can > accelerate it if we avoid using QEMU. > > Actually, if the vhost backend is vhost-user (the main use case), > current qemu have limited control queue support, because it needs the > support from the vhost user backend. > > Add one more con of my solution: > - Need to write another logic to support other virtio device (say > virtio-scsi), if it's easier of Tetsuya's solution to do that? > Probably, my solution will be easier to do that. My solution has enough facility to access to io port and PCI configuration space of virtio-scsi device of QEMU. So, if you invoke with QEMU with virtio-scsi, only you need to do is changing PCI interface of current virtio-scsi PMD. (I just assume currently we have virtio-scsi PMD.) If the virtio-scsi PMD works on QEMU, same code should work with only changing PCI interface. >> >> [My solution] >> - Pros >> Basic principle of my implementation is not to reinvent the wheel. >> We can use a virtio-net device of QEMU implementation, it means we don't >> need to maintain virtio-net device by ourselves, and we can use all of >> functions supported by QEMU virtio-net device. >> - Cons >> Need to invoke QEMU process. > > Two more possible cons: > a) This solution also needs to maintain qtest utility, right? But the spec of qtest will be more stable than virtio-net. > b) There's still address arrange restriction, right? Although we can use > "--base-virtaddr=0x4" to relieve this question, but how about if > there are 2 or more devices? (By the way, is there still address arrange > requirement for 32 bit system) Our solutions are a virtio-net driver, and a vhost-user backend driver needs to access to memory allocated by virtio-net driver. If an application has 2 devices, it means 2 vhost-user backend PMD needs to access to the same application memory, right? Also, currently each virtio-net device has an one QEMU process. So, I am not sure what will be problem if we have 2 devices. BTW, 44bits limitations comes from current QEMU implementation itself. (Actually, if modern virtio device is used, we should be able to remove the restriction.) > c) Actually, IMO this solution is sensitive to any virtio spec change > (io port, pci configuration space). In this case, virtio-net PMD itself will need to be fixed. Then, my implementation will be also fixed with the same way. Current implementation has only PCI abstraction that Yuanhan introduced, so you may think my solution depends on above things, but actually, my implementation depends on only how to access to io port and PCI configuration space. This is what "qtest.h" provides. Thanks, Tetsuya > >>
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On 2016/06/06 16:21, Yuanhan Liu wrote: > On Mon, Jun 06, 2016 at 02:10:46PM +0900, Tetsuya Mukawa wrote: >> Hi Yuanhan, >> >> Sorry for late replying. > > Never mind. > >> >> On 2016/06/03 13:17, Yuanhan Liu wrote: >>> On Thu, Jun 02, 2016 at 06:30:18PM +0900, Tetsuya Mukawa wrote: Hi Yuanhan, On 2016/06/02 16:31, Yuanhan Liu wrote: > But still, I'd ask do we really need 2 virtio for container solutions? I appreciate your comments. >>> >>> No, I appreciate your effort for contributing to DPDK! vhost-pmd stuff >>> is just brilliant! >>> Let me have time to discuss it with our team. >>> >>> I'm wondering could we have one solution only. IMO, the drawback of >>> having two (quite different) solutions might outweighs the benefit >>> it takes. Say, it might just confuse user. >> >> I agree with this. >> If we have 2 solutions, it would confuse the DPDK users. >> >>> >>> OTOH, I'm wondering could you adapt to Jianfeng's solution? If not, >>> what's the missing parts, and could we fix it? I'm thinking having >>> one unified solution will keep ours energy/focus on one thing, making >>> it better and better! Having two just splits the energy; it also >>> introduces extra burden for maintaining. >> >> Of course, I adopt Jiangeng's solution basically. >> Actually, his solution is almost similar I tried to implement at first. >> >> I guess here is pros/cons of 2 solutions. >> >> [Jianfeng's solution] >> - Pros >> Don't need to invoke QEMU process. >> - Cons >> If virtio-net specification is changed, we need to implement it by >> ourselves. Also, LSC interrupt and control queue functions are not >> supported yet. > > Jianfeng have made and sent out the patch to enable ctrl queue and > multiple queue support. Sorry, I haven't noticed that ctrl queue has been already enabled. > > For the LSC part, no much idea yet so far. But I'm assuming it will > not take too much effort, either. > >> I agree both functions may not be so important, and if we need it >> we can implement them, but we need to pay energy to implement them. >> >> [My solution] >> - Pros >> Basic principle of my implementation is not to reinvent the wheel. > > Yes, that's a good point. However, it's not that hard as we would have > thought in the first time: the tough part that dequeue/enqueue packets > from/to vring is actually offloaded to DPDK vhost-user. That means we > only need re-implement the control path of virtio-net device, plus the > vhost-user frontend. If you have a detailed look of your patchset as > well Jianfeng's, you might find that the two patchset are actually with > same code size. Yes, I know this. So far, the amount of code is almost same, but in the future we may need to implement more, if virtio-net specification is revised. > >> We can use a virtio-net device of QEMU implementation, it means we don't >> need to maintain virtio-net device by ourselves, and we can use all of >> functions supported by QEMU virtio-net device. >> - Cons >> Need to invoke QEMU process. > > Another thing is that it makes the usage a bit harder: look at the > long qemu cli options of your example usage. It also has some traps, > say, "--enable-kvm" is not allowed, which is a default option used > with QEMU. Probably a kind of shell script will help the users. > > And judging that we actually don't take too much effort to implement > a virtio device emulation, I'd prefer it slightly. I guess something > light weight and easier for use is more important here. This is very important point. If so, we don't need much effort when virtio-spec is changed. > > Actually, I have foreseen another benefit of adding virtio-user device > emulation: we now might be able to add a rte_vhost_dequeue/enqueue_burst() > unit test case. We simply can't do it before, since we depend on QEMU > for testing, which is not acceptable for a unit test case. Making it > be a unit test case would help us spotting any bad changes that would > introduce bugs easily and automatically. As you mentioned above, QEMU process is not related with dequeuing/enqueuing. So I guess we may have a testing for rte_vhost_dequeue/enqueue_burst() regardless of choice. >> Anyway, we can choose one of belows. >> 1. Take advantage of invoking less processes. >> 2. Take advantage of maintainability of virtio-net device. If container usage that DPDK assumes is to invoke hundreds containers in one host, we should take Jiangfeng's solution. Also, if implementing a new feature and maintaining Jiangfeng's virtio-net device are not so hard, we should take his solution. I guess this is the point we need to consider. What do you think? Thanks, Tetsuya >> >> Honestly, I'm OK if my solution is not merged. >> Thus, it should be decided to let DPDK better. >> >> What do you think? >> Which is better for DPDK? >> >> Thanks, >> Tetsuya >> >>> >>> --yliu >>>
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On Mon, Jun 06, 2016 at 05:33:31PM +0900, Tetsuya Mukawa wrote: > >> [My solution] > >> - Pros > >> Basic principle of my implementation is not to reinvent the wheel. > > > > Yes, that's a good point. However, it's not that hard as we would have > > thought in the first time: the tough part that dequeue/enqueue packets > > from/to vring is actually offloaded to DPDK vhost-user. That means we > > only need re-implement the control path of virtio-net device, plus the > > vhost-user frontend. If you have a detailed look of your patchset as > > well Jianfeng's, you might find that the two patchset are actually with > > same code size. > > Yes, I know this. > So far, the amount of code is almost same, but in the future we may need > to implement more, if virtio-net specification is revised. It didn't take too much effort to implement from scratch, I doubt it will for future revise. And, virtio-net spec is unlikely revised, or to be precisely, unlikely revised quite often. Therefore, I don't see big issues here. > >> We can use a virtio-net device of QEMU implementation, it means we don't > >> need to maintain virtio-net device by ourselves, and we can use all of > >> functions supported by QEMU virtio-net device. > >> - Cons > >> Need to invoke QEMU process. > > > > Another thing is that it makes the usage a bit harder: look at the > > long qemu cli options of your example usage. It also has some traps, > > say, "--enable-kvm" is not allowed, which is a default option used > > with QEMU. > > Probably a kind of shell script will help the users. Yeah, that would help. But if we have a choice to make it simpler in the beginning, why not then? :-) > > > > And judging that we actually don't take too much effort to implement > > a virtio device emulation, I'd prefer it slightly. I guess something > > light weight and easier for use is more important here. > > This is very important point. > If so, we don't need much effort when virtio-spec is changed. I'd assume so. > > Actually, I have foreseen another benefit of adding virtio-user device > > emulation: we now might be able to add a rte_vhost_dequeue/enqueue_burst() > > unit test case. We simply can't do it before, since we depend on QEMU > > for testing, which is not acceptable for a unit test case. Making it > > be a unit test case would help us spotting any bad changes that would > > introduce bugs easily and automatically. > > As you mentioned above, QEMU process is not related with > dequeuing/enqueuing. > So I guess we may have a testing for rte_vhost_dequeue/enqueue_burst() > regardless of choice. Yes, we don't need the dequeue/enqueue part, but we need the vhost-user initialization part from QEMU vhost-user. Now that we have vhost-user frontend from virtio-user, we have no dependency on QEMU any more. > >> Anyway, we can choose one of belows. > >> 1. Take advantage of invoking less processes. > >> 2. Take advantage of maintainability of virtio-net device. > > If container usage that DPDK assumes is to invoke hundreds containers in > one host, I barely know about container, but I would assume that's not rare. > we should take Jiangfeng's solution. > > Also, if implementing a new feature and maintaining Jiangfeng's > virtio-net device are not so hard, As stated, I would assume so. --yliu > we should take his solution. > > I guess this is the point we need to consider. > What do you think? > > Thanks, > Tetsuya > > >> > >> Honestly, I'm OK if my solution is not merged. > >> Thus, it should be decided to let DPDK better. > >> > >> What do you think? > >> Which is better for DPDK? > >> > >> Thanks, > >> Tetsuya > >> > >>> > >>> --yliu > >>>
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
Hi, On 6/6/2016 1:10 PM, Tetsuya Mukawa wrote: > Hi Yuanhan, > > Sorry for late replying. > > On 2016/06/03 13:17, Yuanhan Liu wrote: >> On Thu, Jun 02, 2016 at 06:30:18PM +0900, Tetsuya Mukawa wrote: >>> Hi Yuanhan, >>> >>> On 2016/06/02 16:31, Yuanhan Liu wrote: But still, I'd ask do we really need 2 virtio for container solutions? >>> I appreciate your comments. >> No, I appreciate your effort for contributing to DPDK! vhost-pmd stuff >> is just brilliant! >> >>> Let me have time to discuss it with our team. >> I'm wondering could we have one solution only. IMO, the drawback of >> having two (quite different) solutions might outweighs the benefit >> it takes. Say, it might just confuse user. > I agree with this. > If we have 2 solutions, it would confuse the DPDK users. > >> OTOH, I'm wondering could you adapt to Jianfeng's solution? If not, >> what's the missing parts, and could we fix it? I'm thinking having >> one unified solution will keep ours energy/focus on one thing, making >> it better and better! Having two just splits the energy; it also >> introduces extra burden for maintaining. > Of course, I adopt Jiangeng's solution basically. > Actually, his solution is almost similar I tried to implement at first. > > I guess here is pros/cons of 2 solutions. > > [Jianfeng's solution] > - Pros > Don't need to invoke QEMU process. > - Cons > If virtio-net specification is changed, we need to implement it by > ourselves. It will barely introduce any change when virtio-net specification is changed as far as I can see. The only part we care is the how desc, avail, used distribute on memory, which is a very small part. It's true that my solution now seriously depend on vhost-user protocol, which is defined in QEMU. I cannot see a big problem there so far. > Also, LSC interrupt and control queue functions are not > supported yet. > I agree both functions may not be so important, and if we need it > we can implement them, but we need to pay energy to implement them. LSC is really less important than rxq interrupt (IMO). We don't know how long will rxq interrupt of virtio be available for QEMU, but we can accelerate it if we avoid using QEMU. Actually, if the vhost backend is vhost-user (the main use case), current qemu have limited control queue support, because it needs the support from the vhost user backend. Add one more con of my solution: - Need to write another logic to support other virtio device (say virtio-scsi), if it's easier of Tetsuya's solution to do that? > > [My solution] > - Pros > Basic principle of my implementation is not to reinvent the wheel. > We can use a virtio-net device of QEMU implementation, it means we don't > need to maintain virtio-net device by ourselves, and we can use all of > functions supported by QEMU virtio-net device. > - Cons > Need to invoke QEMU process. Two more possible cons: a) This solution also needs to maintain qtest utility, right? b) There's still address arrange restriction, right? Although we can use "--base-virtaddr=0x4" to relieve this question, but how about if there are 2 or more devices? (By the way, is there still address arrange requirement for 32 bit system) c) Actually, IMO this solution is sensitive to any virtio spec change (io port, pci configuration space). > > > Anyway, we can choose one of belows. > 1. Take advantage of invoking less processes. > 2. Take advantage of maintainability of virtio-net device. > > Honestly, I'm OK if my solution is not merged. > Thus, it should be decided to let DPDK better. Yes, agreed. Thanks, Jianfeng > > What do you think? > Which is better for DPDK? > > Thanks, > Tetsuya > >> --yliu >>
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On Mon, Jun 06, 2016 at 02:10:46PM +0900, Tetsuya Mukawa wrote: > Hi Yuanhan, > > Sorry for late replying. Never mind. > > On 2016/06/03 13:17, Yuanhan Liu wrote: > > On Thu, Jun 02, 2016 at 06:30:18PM +0900, Tetsuya Mukawa wrote: > >> Hi Yuanhan, > >> > >> On 2016/06/02 16:31, Yuanhan Liu wrote: > >>> But still, I'd ask do we really need 2 virtio for container solutions? > >> > >> I appreciate your comments. > > > > No, I appreciate your effort for contributing to DPDK! vhost-pmd stuff > > is just brilliant! > > > >> Let me have time to discuss it with our team. > > > > I'm wondering could we have one solution only. IMO, the drawback of > > having two (quite different) solutions might outweighs the benefit > > it takes. Say, it might just confuse user. > > I agree with this. > If we have 2 solutions, it would confuse the DPDK users. > > > > > OTOH, I'm wondering could you adapt to Jianfeng's solution? If not, > > what's the missing parts, and could we fix it? I'm thinking having > > one unified solution will keep ours energy/focus on one thing, making > > it better and better! Having two just splits the energy; it also > > introduces extra burden for maintaining. > > Of course, I adopt Jiangeng's solution basically. > Actually, his solution is almost similar I tried to implement at first. > > I guess here is pros/cons of 2 solutions. > > [Jianfeng's solution] > - Pros > Don't need to invoke QEMU process. > - Cons > If virtio-net specification is changed, we need to implement it by > ourselves. Also, LSC interrupt and control queue functions are not > supported yet. Jianfeng have made and sent out the patch to enable ctrl queue and multiple queue support. For the LSC part, no much idea yet so far. But I'm assuming it will not take too much effort, either. > I agree both functions may not be so important, and if we need it > we can implement them, but we need to pay energy to implement them. > > [My solution] > - Pros > Basic principle of my implementation is not to reinvent the wheel. Yes, that's a good point. However, it's not that hard as we would have thought in the first time: the tough part that dequeue/enqueue packets from/to vring is actually offloaded to DPDK vhost-user. That means we only need re-implement the control path of virtio-net device, plus the vhost-user frontend. If you have a detailed look of your patchset as well Jianfeng's, you might find that the two patchset are actually with same code size. > We can use a virtio-net device of QEMU implementation, it means we don't > need to maintain virtio-net device by ourselves, and we can use all of > functions supported by QEMU virtio-net device. > - Cons > Need to invoke QEMU process. Another thing is that it makes the usage a bit harder: look at the long qemu cli options of your example usage. It also has some traps, say, "--enable-kvm" is not allowed, which is a default option used with QEMU. And judging that we actually don't take too much effort to implement a virtio device emulation, I'd prefer it slightly. I guess something light weight and easier for use is more important here. Actually, I have foreseen another benefit of adding virtio-user device emulation: we now might be able to add a rte_vhost_dequeue/enqueue_burst() unit test case. We simply can't do it before, since we depend on QEMU for testing, which is not acceptable for a unit test case. Making it be a unit test case would help us spotting any bad changes that would introduce bugs easily and automatically. --yliu > Anyway, we can choose one of belows. > 1. Take advantage of invoking less processes. > 2. Take advantage of maintainability of virtio-net device. > > Honestly, I'm OK if my solution is not merged. > Thus, it should be decided to let DPDK better. > > What do you think? > Which is better for DPDK? > > Thanks, > Tetsuya > > > > > --yliu > >
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
Hi Yuanhan, Sorry for late replying. On 2016/06/03 13:17, Yuanhan Liu wrote: > On Thu, Jun 02, 2016 at 06:30:18PM +0900, Tetsuya Mukawa wrote: >> Hi Yuanhan, >> >> On 2016/06/02 16:31, Yuanhan Liu wrote: >>> But still, I'd ask do we really need 2 virtio for container solutions? >> >> I appreciate your comments. > > No, I appreciate your effort for contributing to DPDK! vhost-pmd stuff > is just brilliant! > >> Let me have time to discuss it with our team. > > I'm wondering could we have one solution only. IMO, the drawback of > having two (quite different) solutions might outweighs the benefit > it takes. Say, it might just confuse user. I agree with this. If we have 2 solutions, it would confuse the DPDK users. > > OTOH, I'm wondering could you adapt to Jianfeng's solution? If not, > what's the missing parts, and could we fix it? I'm thinking having > one unified solution will keep ours energy/focus on one thing, making > it better and better! Having two just splits the energy; it also > introduces extra burden for maintaining. Of course, I adopt Jiangeng's solution basically. Actually, his solution is almost similar I tried to implement at first. I guess here is pros/cons of 2 solutions. [Jianfeng's solution] - Pros Don't need to invoke QEMU process. - Cons If virtio-net specification is changed, we need to implement it by ourselves. Also, LSC interrupt and control queue functions are not supported yet. I agree both functions may not be so important, and if we need it we can implement them, but we need to pay energy to implement them. [My solution] - Pros Basic principle of my implementation is not to reinvent the wheel. We can use a virtio-net device of QEMU implementation, it means we don't need to maintain virtio-net device by ourselves, and we can use all of functions supported by QEMU virtio-net device. - Cons Need to invoke QEMU process. Anyway, we can choose one of belows. 1. Take advantage of invoking less processes. 2. Take advantage of maintainability of virtio-net device. Honestly, I'm OK if my solution is not merged. Thus, it should be decided to let DPDK better. What do you think? Which is better for DPDK? Thanks, Tetsuya > > --yliu >
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
2016-06-03 12:17, Yuanhan Liu: > On Thu, Jun 02, 2016 at 06:30:18PM +0900, Tetsuya Mukawa wrote: > > Hi Yuanhan, > > > > On 2016/06/02 16:31, Yuanhan Liu wrote: > > > But still, I'd ask do we really need 2 virtio for container solutions? > > > > I appreciate your comments. > > No, I appreciate your effort for contributing to DPDK! vhost-pmd stuff > is just brilliant! > > > Let me have time to discuss it with our team. > > I'm wondering could we have one solution only. IMO, the drawback of > having two (quite different) solutions might outweighs the benefit > it takes. Say, it might just confuse user. +1 > OTOH, I'm wondering could you adapt to Jianfeng's solution? If not, > what's the missing parts, and could we fix it? I'm thinking having > one unified solution will keep ours energy/focus on one thing, making > it better and better! Having two just splits the energy; it also > introduces extra burden for maintaining. +1
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On Thu, Jun 02, 2016 at 06:30:18PM +0900, Tetsuya Mukawa wrote: > Hi Yuanhan, > > On 2016/06/02 16:31, Yuanhan Liu wrote: > > But still, I'd ask do we really need 2 virtio for container solutions? > > I appreciate your comments. No, I appreciate your effort for contributing to DPDK! vhost-pmd stuff is just brilliant! > Let me have time to discuss it with our team. I'm wondering could we have one solution only. IMO, the drawback of having two (quite different) solutions might outweighs the benefit it takes. Say, it might just confuse user. OTOH, I'm wondering could you adapt to Jianfeng's solution? If not, what's the missing parts, and could we fix it? I'm thinking having one unified solution will keep ours energy/focus on one thing, making it better and better! Having two just splits the energy; it also introduces extra burden for maintaining. --yliu
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
Hi Yuanhan, On 2016/06/02 16:31, Yuanhan Liu wrote: > But still, I'd ask do we really need 2 virtio for container solutions? I appreciate your comments. Let me have time to discuss it with our team. Thanks, Tetsuya
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
On Thu, Jun 02, 2016 at 12:29:39PM +0900, Tetsuya Mukawa wrote: > The patches will work on below patch series. > - [PATCH v5 0/8] virtio support for container > > It seems his implementation will be changed a bit. > So, this patch series are also going to be changed to follow his > implementation. Hi Tetsuya, TBH, I was considering to reject your v4: the code was quite messy. But this v5 changed my mind a bit: it's much cleaner. But still, I'd ask do we really need 2 virtio for container solutions? That results to the same question that I'm sure you have already answered before: in which way your solution outweighs Jianfeng's? The reason I want to ask again is: 1), I wasn't actively participating the discussion in last release, besides some common comments on virtio, 2), maybe it's time to make a decision that should we take one solution only, if so, which one, or should we take both? Thomas is Cc'ed, hope he can help on the decision making. --yliu
[dpdk-dev] [PATCH v5 0/6] Virtio-net PMD: QEMU QTest extension for container
The patches will work on below patch series. - [PATCH v5 0/8] virtio support for container It seems his implementation will be changed a bit. So, this patch series are also going to be changed to follow his implementation. [Changes] v5 changes: - Rebase on latest dpdk-next-virtio. - Follow Jianfeng's implementation to support virtual virtio-net device. - Split the patch series like followings. - This patch series. Only support basic functions. The functions to handle LSC interrupt and '--range-virtaddr' was removed from this patch series. This patch needs EAL memory mapped between (1<<31) to (1<<44). To allocate such a memory, just assume the user will use '--base-virtaddr'. If appropriate memory cannot be allocated, this PMD will exit as error. Then the users can try other values. - Supplement patches to support link status interrupt. - Supplement patches to support '--range-virtaddr'. This EAL option will help to allocate memory mapped between (1<<31) to (1<<44). v4 changes: - Rebase on latest master. - Split patches. - To abstract qtest code more, change interface between current virtio code and qtest code. - Rename qtest.c to qtest_utils.c - Change implementation like below. - Set pci device information out of qtest abstraction, then pass it to qtest to initialize devices. - Remove redundant condition checking from qtest_raw_send/recv(). - Fix return value of qtest_raw_send(). v3 changes: - Rebase on latest master. - remove "-qtest-virtio" option, then add "--range-virtaddr" and "--align-memsize" options. - Fix typos in qtest.c v2 changes: - Rebase on above patch seiries. - Rebase on master - Add "--qtest-virtio" EAL option. - Fixes in qtest.c - Fix error handling for the case qtest connection is closed. - Use eventfd for interrupt messaging. - Use linux header for PCI register definitions. - Fix qtest_raw_send/recv to handle error correctly. - Fix bit mask of PCI_CONFIG_ADDR. - Describe memory and ioport usage of qtest guest in qtest.c - Remove loop that is for finding PCI devices. [Abstraction] Normally, virtio-net PMD only works on VM, because there is no virtio-net device on host. This patches extend virtio-net PMD to be able to work on host as virtual PMD. But we didn't implement virtio-net device as a part of virtio-net PMD. To prepare virtio-net device for the PMD, start QEMU process with special QTest mode, then connect it from virtio-net PMD through unix domain socket. The PMD can connect to anywhere QEMU virtio-net device can. For example, the PMD can connects to vhost-net kernel module and vhost-user backend application. Similar to virtio-net PMD on QEMU, application memory that uses virtio-net PMD will be shared between vhost backend application. But vhost backend application memory will not be shared. Main target of this PMD is container like docker, rkt, lxc and etc. We can isolate related processes(virtio-net PMD process, QEMU and vhost-user backend process) by container. But, to communicate through unix domain socket, shared directory will be needed. [How to use] Please use QEMU-2.5.1, or above. (So far, QEMU-2.5.1 hasn't been released yet, so please checkout master from QEMU repository) - Compile Set "CONFIG_RTE_VIRTIO_VDEV_QTEST=y" in config/common_linux. Then compile it. - Start QEMU like below. $ qemu-system-x86_64 \ -machine pc-i440fx-1.4,accel=qtest \ -display none -qtest-log /dev/null \ -qtest unix:/tmp/socket,server \ -netdev type=tap,script=/etc/qemu-ifup,id=net0,queues=1 \ -device virtio-net-pci,netdev=net0,mq=on,disable-modern=false,addr=3 \ -chardev socket,id=chr1,path=/tmp/ivshmem,server \ -device ivshmem,size=1G,chardev=chr1,vectors=1,addr=4 - Start DPDK application like below $ testpmd -c f -n 1 -m 1024 --no-pci --base-virtaddr=0x4 \ --vdev="eth_virtio_qtest0,qtest=/tmp/socket,ivshmem=/tmp/ivshmem"\ -- --disable-hw-vlan --txqflags=0xf00 -i (*1) Please Specify same memory size in QEMU and DPDK command line. (*2) Should use qemu-2.5.1, or above. (*3) QEMU process is needed per port. (*4) virtio-1.0 device are only supported. (*5) The vhost backends like vhost-net and vhost-user can be specified. (*6) In most cases, just using above command is enough, but you can also specify other QEMU virtio-net options. (*7) Only checked "pc-i440fx-1.4" machine, but may work with other machines. It depends on a machine has piix3 south bridge. If the machine doesn't have, virtio-net PMD cannot receive status changed interrupts. (*8) Should not add "--enable-kvm" to QEMU command line. [Detailed Description] - virtio-net device implementation The PMD uses QEMU virtio-net device. To do that, QEMU QTest functionality is used. QTest is a test framework of QEMU devices. It allows us to implement a device driver
