[dpdk-dev] [PATCH v5 0/7] virtio ring layout optimization and simple rx/tx processing
On Tue, Oct 27, 2015 at 10:15:01AM +0800, Yuanhan Liu wrote: > On Tue, Oct 27, 2015 at 01:44:09AM +, Tan, Jianfeng wrote: > > > > > > > -Original Message- > > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Huawei Xie > > > Sent: Sunday, October 25, 2015 11:35 PM > > > To: dev at dpdk.org > > > Subject: [dpdk-dev] [PATCH v5 0/7] virtio ring layout optimization and > > > simple > > > rx/tx processing > > > > > > Changes in v5: > > > - Call __rte_pktmbuf_prefree_seg to check refcnt when free mbufs > > > > > > Changes in v4: > > > - Fix the error in virtio tx ring layout ascii chart in the commit message > > > - Move virtio_xmit_cleanup ahead to free descriptors earlier > > > - Test merge-able feature when select simple rx/tx functions > > [...] > > > > > Acked-by Jianfeng Tan > > Jianfeng, > > I often see a reply like this, just dropping an ACK at the end of > long email, and no more, which takes me (as well as others) some > time to scroll it many times to the bottom till see that. > > TBH, it's always a bit frustrating that, after the scroll effort, > I just see such a reply that could have been put on the top of > the email so that I can see it with a glimpse only. > > So, top reply would be good for this case, or you could reply like > what I did, removing other context to make your reply fit in one > screen. > > --yliu +1 When ack'ing patches, please place the ack on the line underneath the signoff and delete the rest of the email below, as it's unneeded. /Bruce
[dpdk-dev] [PATCH v5 0/7] virtio ring layout optimization and simple rx/tx processing
On Tue, Oct 27, 2015 at 01:44:09AM +, Tan, Jianfeng wrote: > > > > -Original Message- > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Huawei Xie > > Sent: Sunday, October 25, 2015 11:35 PM > > To: dev at dpdk.org > > Subject: [dpdk-dev] [PATCH v5 0/7] virtio ring layout optimization and > > simple > > rx/tx processing > > > > Changes in v5: > > - Call __rte_pktmbuf_prefree_seg to check refcnt when free mbufs > > > > Changes in v4: > > - Fix the error in virtio tx ring layout ascii chart in the commit message > > - Move virtio_xmit_cleanup ahead to free descriptors earlier > > - Test merge-able feature when select simple rx/tx functions [...] > > Acked-by Jianfeng Tan Jianfeng, I often see a reply like this, just dropping an ACK at the end of long email, and no more, which takes me (as well as others) some time to scroll it many times to the bottom till see that. TBH, it's always a bit frustrating that, after the scroll effort, I just see such a reply that could have been put on the top of the email so that I can see it with a glimpse only. So, top reply would be good for this case, or you could reply like what I did, removing other context to make your reply fit in one screen. --yliu
[dpdk-dev] [PATCH v5 0/7] virtio ring layout optimization and simple rx/tx processing
> -Original Message- > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Huawei Xie > Sent: Sunday, October 25, 2015 11:35 PM > To: dev at dpdk.org > Subject: [dpdk-dev] [PATCH v5 0/7] virtio ring layout optimization and simple > rx/tx processing > > Changes in v5: > - Call __rte_pktmbuf_prefree_seg to check refcnt when free mbufs > > Changes in v4: > - Fix the error in virtio tx ring layout ascii chart in the commit message > - Move virtio_xmit_cleanup ahead to free descriptors earlier > - Test merge-able feature when select simple rx/tx functions > > Changes in v3: > - Remove unnecessary NULL test for rte_free > - Remove unnecessary assign of local var after free > - Remove return at the end of void function > - Remove always_inline attribute for virtio_xmit_cleanup > - Reword some commit messages > - Add TODO in the commit message of simple tx patch > > Changes in v2: > - Remove the configure macro > - Enable simple R/TX processing when user specifies simple txq flags > - Reword some comments and commit messages > > In DPDK based switching enviroment, mostly vhost runs on a dedicated core > while virtio processing in guest VMs runs on other different cores. > Take RX for example, with generic implementation, for each guest buffer, > a) virtio driver allocates a descriptor from free descriptor list > b) modify the entry of avail ring to point to allocated descriptor > c) after packet is received, free the descriptor > > When vhost fetches the avail ring, it need to fetch the modified L1 cache > from virtio core, which is a heavy cost in current CPU implementation. > > This idea of this optimization is: > allocate the fixed descriptor for each entry of avail ring, so avail ring > will > always be the same during the run. > This removes L1M cache transfer from virtio core to vhost core for avail ring. > (Note we couldn't avoid the cache transfer for descriptors). > Besides, descriptor allocation and free operation is eliminated. > This also makes vector procesing possible to further accelerate the > processing. > > This is the layout for the avail ring(take 256 ring entries for example), with > each entry pointing to the descriptor with the same index. > avail > idx > + > | > +++---+-+--+ > | 0 | 1 | 2 | ... | 254 | 255 | avail ring > +-+--+-+--+-+-+-+---+--+---+ > ||| | | | > ||| | | | > vvv | v v > +-+--+-+--+-+-+-+---+--+---+ > | 0 | 1 | 2 | ... | 254 | 255 | desc ring > +++---+-+--+ > | > | > +++---+-+--+ > | 0 | 1 | 2 | | 254 | 255 | used ring > +++---+-+--+ > | > + > > This is the ring layout for TX. > As we need one virtio header for each xmit packet, we have 128 slots > available. > > ++ > || > || > +-+-+-+--+--+--+--+ > | 0 | 1 | ... | 127 || 128 | 129 | ... | 255 | avail ring > +--+--+--+--+-+---+--+---+--+---+--+--+---+ >| || || | | | >v vv || v v v > +--+--+--+--+-+---+--+---+--+---+--+--+---+ > | 127 | 128 | ... | 255 || 127 | 128 | ... | 255 | desc ring for > virtio_net_hdr > +--+--+--+--+-+---+--+---+--+---+--+--+---+ >| || || | | | >v vv || v v v > +--+--+--+--+-+---+--+---+--+---+--+--+---+ > | 0 | 1 | ... | 127 || 0 | 1 | ... | 127 | desc ring for tx dat > +-+-+-+--+--+--+--+ > || > || > ++ > > > Performance boost could be observed only if the virtio backend isn't the > bottleneck or in VM2VM case. > There are also several vhost optimization patches to be submitted later. > > > Huawei Xie (7): > virtio: add virtio_rxtx.h header file > virtio: add software rx ring, fake_buf into virtqueue > virtio: rx/tx ring layout optimization > virtio: fill RX avail ring with blank mbufs > virtio: virtio vec rx > virtio: simple tx routine > virtio: pick simple rx/tx func > > drivers/net/virtio/Makefile | 2 +- > drivers/net/virtio/virtio_ethdev.c | 12 +- > drivers/net/virtio/vir
[dpdk-dev] [PATCH v5 0/7] virtio ring layout optimization and simple rx/tx processing
Changes in v5: - Call __rte_pktmbuf_prefree_seg to check refcnt when free mbufs Changes in v4: - Fix the error in virtio tx ring layout ascii chart in the commit message - Move virtio_xmit_cleanup ahead to free descriptors earlier - Test merge-able feature when select simple rx/tx functions Changes in v3: - Remove unnecessary NULL test for rte_free - Remove unnecessary assign of local var after free - Remove return at the end of void function - Remove always_inline attribute for virtio_xmit_cleanup - Reword some commit messages - Add TODO in the commit message of simple tx patch Changes in v2: - Remove the configure macro - Enable simple R/TX processing when user specifies simple txq flags - Reword some comments and commit messages In DPDK based switching enviroment, mostly vhost runs on a dedicated core while virtio processing in guest VMs runs on other different cores. Take RX for example, with generic implementation, for each guest buffer, a) virtio driver allocates a descriptor from free descriptor list b) modify the entry of avail ring to point to allocated descriptor c) after packet is received, free the descriptor When vhost fetches the avail ring, it need to fetch the modified L1 cache from virtio core, which is a heavy cost in current CPU implementation. This idea of this optimization is: allocate the fixed descriptor for each entry of avail ring, so avail ring will always be the same during the run. This removes L1M cache transfer from virtio core to vhost core for avail ring. (Note we couldn't avoid the cache transfer for descriptors). Besides, descriptor allocation and free operation is eliminated. This also makes vector procesing possible to further accelerate the processing. This is the layout for the avail ring(take 256 ring entries for example), with each entry pointing to the descriptor with the same index. avail idx + | +++---+-+--+ | 0 | 1 | 2 | ... | 254 | 255 | avail ring +-+--+-+--+-+-+-+---+--+---+ ||| | | | ||| | | | vvv | v v +-+--+-+--+-+-+-+---+--+---+ | 0 | 1 | 2 | ... | 254 | 255 | desc ring +++---+-+--+ | | +++---+-+--+ | 0 | 1 | 2 | | 254 | 255 | used ring +++---+-+--+ | + This is the ring layout for TX. As we need one virtio header for each xmit packet, we have 128 slots available. ++ || || +-+-+-+--+--+--+--+ | 0 | 1 | ... | 127 || 128 | 129 | ... | 255 | avail ring +--+--+--+--+-+---+--+---+--+---+--+--+---+ | || || | | | v vv || v v v +--+--+--+--+-+---+--+---+--+---+--+--+---+ | 127 | 128 | ... | 255 || 127 | 128 | ... | 255 | desc ring for virtio_net_hdr +--+--+--+--+-+---+--+---+--+---+--+--+---+ | || || | | | v vv || v v v +--+--+--+--+-+---+--+---+--+---+--+--+---+ | 0 | 1 | ... | 127 || 0 | 1 | ... | 127 | desc ring for tx dat +-+-+-+--+--+--+--+ || || ++ Performance boost could be observed only if the virtio backend isn't the bottleneck or in VM2VM case. There are also several vhost optimization patches to be submitted later. Huawei Xie (7): virtio: add virtio_rxtx.h header file virtio: add software rx ring, fake_buf into virtqueue virtio: rx/tx ring layout optimization virtio: fill RX avail ring with blank mbufs virtio: virtio vec rx virtio: simple tx routine virtio: pick simple rx/tx func drivers/net/virtio/Makefile | 2 +- drivers/net/virtio/virtio_ethdev.c | 12 +- drivers/net/virtio/virtio_ethdev.h | 5 + drivers/net/virtio/virtio_rxtx.c| 56 - drivers/net/virtio/virtio_rxtx.h| 39 +++ drivers/net/virtio/virtio_rxtx_simple.c | 414 drivers/net/virtio/virtqueue.h | 5 + 7 files changed, 529 insertions(+), 4 deletions(-) create mode 100644 drivers/net/virtio/virtio_rxtx.h create mode 100644 drivers/net/virtio/virtio_rxtx_simple.c -- 1.8.1.4
