[dpdk-dev] [PATCH v7 3/8] vhost: vring queue setup for multiple queue support
On Thu, Oct 22, 2015 at 09:49:58AM +, Xie, Huawei wrote:
> On 10/21/2015 11:48 AM, Yuanhan Liu wrote:
> > All queue pairs, including the default (the first) queue pair,
> > are allocated dynamically, when a vring_call message is received
> > first time for a specific queue pair.
> >
> > This is a refactor work for enabling vhost-user multiple queue;
> > it should not break anything as it does no functional changes:
> > we don't support mq set, so there is only one mq at max.
> >
> > This patch is based on Changchun's patch.
> >
> [...]
> >
> > void
> > @@ -290,13 +298,9 @@ user_get_vring_base(struct vhost_device_ctx ctx,
> > * sent and only sent in vhost_vring_stop.
> > * TODO: cleanup the vring, it isn't usable since here.
> > */
> > - if ((dev->virtqueue[VIRTIO_RXQ]->kickfd) >= 0) {
> > - close(dev->virtqueue[VIRTIO_RXQ]->kickfd);
> > - dev->virtqueue[VIRTIO_RXQ]->kickfd = -1;
> > - }
> > - if ((dev->virtqueue[VIRTIO_TXQ]->kickfd) >= 0) {
> > - close(dev->virtqueue[VIRTIO_TXQ]->kickfd);
> > - dev->virtqueue[VIRTIO_TXQ]->kickfd = -1;
> > + if ((dev->virtqueue[state->index]->kickfd) >= 0) {
> > + close(dev->virtqueue[state->index]->kickfd);
> > + dev->virtqueue[state->index]->kickfd = -1;
> > }
> Since we change the behavior here, better list in the commit message as
> well.
I checked the code again, and found I should not change that:
GET_VRING_BASE is sent per virt queue pair.
BTW, it's wrong to do this kind of stuff here, we need fix
it in future.
>
> >
> >
> > @@ -680,13 +704,21 @@ set_vring_call(struct vhost_device_ctx ctx, struct
> > vhost_vring_file *file)
> > {
> > struct virtio_net *dev;
> > struct vhost_virtqueue *vq;
> > + uint32_t cur_qp_idx = file->index / VIRTIO_QNUM;
> >
> > dev = get_device(ctx);
> > if (dev == NULL)
> > return -1;
> >
> > + /* alloc vring queue pair if it is a new queue pair */
> > + if (cur_qp_idx + 1 > dev->virt_qp_nb) {
> > + if (alloc_vring_queue_pair(dev, cur_qp_idx) < 0)
> > + return -1;
> > + }
> > +
> Here we rely on the fact that this set_vring_call message is sent in the
> continuous ascending order of queue idx 0, 1, 2, ...
That's true.
>
> > /* file->index refers to the queue index. The txq is 1, rxq is 0. */
> > vq = dev->virtqueue[file->index];
> > + assert(vq != NULL);
> >
> If we allocate the queue until the we receive the first vring message,
> better add comment that we rely on this fact.
Will do that.
> Could we add the vhost-user message to tell us the queue number QEMU
> allocates before vring message?
We may need do that. But it's too late to make it in v2.2
--yliu
> > if (vq->callfd >= 0)
> > close(vq->callfd);
>
[dpdk-dev] [PATCH v7 3/8] vhost: vring queue setup for multiple queue support
On 10/21/2015 11:48 AM, Yuanhan Liu wrote:
> All queue pairs, including the default (the first) queue pair,
> are allocated dynamically, when a vring_call message is received
> first time for a specific queue pair.
>
> This is a refactor work for enabling vhost-user multiple queue;
> it should not break anything as it does no functional changes:
> we don't support mq set, so there is only one mq at max.
>
> This patch is based on Changchun's patch.
>
[...]
>
> void
> @@ -290,13 +298,9 @@ user_get_vring_base(struct vhost_device_ctx ctx,
>* sent and only sent in vhost_vring_stop.
>* TODO: cleanup the vring, it isn't usable since here.
>*/
> - if ((dev->virtqueue[VIRTIO_RXQ]->kickfd) >= 0) {
> - close(dev->virtqueue[VIRTIO_RXQ]->kickfd);
> - dev->virtqueue[VIRTIO_RXQ]->kickfd = -1;
> - }
> - if ((dev->virtqueue[VIRTIO_TXQ]->kickfd) >= 0) {
> - close(dev->virtqueue[VIRTIO_TXQ]->kickfd);
> - dev->virtqueue[VIRTIO_TXQ]->kickfd = -1;
> + if ((dev->virtqueue[state->index]->kickfd) >= 0) {
> + close(dev->virtqueue[state->index]->kickfd);
> + dev->virtqueue[state->index]->kickfd = -1;
> }
Since we change the behavior here, better list in the commit message as
well.
>
>
> @@ -680,13 +704,21 @@ set_vring_call(struct vhost_device_ctx ctx, struct
> vhost_vring_file *file)
> {
> struct virtio_net *dev;
> struct vhost_virtqueue *vq;
> + uint32_t cur_qp_idx = file->index / VIRTIO_QNUM;
>
> dev = get_device(ctx);
> if (dev == NULL)
> return -1;
>
> + /* alloc vring queue pair if it is a new queue pair */
> + if (cur_qp_idx + 1 > dev->virt_qp_nb) {
> + if (alloc_vring_queue_pair(dev, cur_qp_idx) < 0)
> + return -1;
> + }
> +
Here we rely on the fact that this set_vring_call message is sent in the
continuous ascending order of queue idx 0, 1, 2, ...
> /* file->index refers to the queue index. The txq is 1, rxq is 0. */
> vq = dev->virtqueue[file->index];
> + assert(vq != NULL);
>
If we allocate the queue until the we receive the first vring message,
better add comment that we rely on this fact.
Could we add the vhost-user message to tell us the queue number QEMU
allocates before vring message?
> if (vq->callfd >= 0)
> close(vq->callfd);
[dpdk-dev] [PATCH v7 3/8] vhost: vring queue setup for multiple queue support
On Tue, Oct 20, 2015 at 09:45:12PM -0700, Stephen Hemminger wrote:
> On Wed, 21 Oct 2015 11:48:09 +0800
> Yuanhan Liu wrote:
>
> > struct virtio_net {
> > - struct vhost_virtqueue *virtqueue[VIRTIO_QNUM];/**< Contains
> > all virtqueue information. */
> > + struct vhost_virtqueue *virtqueue[VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MAX];
> > /**< Contains all virtqueue information. */
> > struct virtio_memory*mem; /**< QEMU memory and memory
> > region information. */
>
> Since vhost_virtqueue takes up space, why not put it at end of array,
> that way offsets are smaller and all the early fields will be in
> adjacent cache lines.
Makes sense, and here it is:
-- >8 --
>From 382d0abb744c577f60dbc8e1cdd766615d20b930 Mon Sep 17 00:00:00 2001
From: Yuanhan Liu
Date: Fri, 18 Sep 2015 16:01:10 +0800
Subject: [PATCH] vhost: vring queue setup for multiple queue support
All queue pairs, including the default (the first) queue pair,
are allocated dynamically, when a vring_call message is received
first time for a specific queue pair.
This is a refactor work for enabling vhost-user multiple queue;
it should not break anything as it does no functional changes:
we don't support mq set, so there is only one mq at max.
This patch is based on Changchun's patch.
Signed-off-by: Ouyang Changchun
Signed-off-by: Yuanhan Liu
Acked-by: Flavio Leitner
---
v8: move virtuque field to the end of `virtio_net' struct.
---
lib/librte_vhost/rte_virtio_net.h | 3 +-
lib/librte_vhost/vhost_user/virtio-net-user.c | 44
lib/librte_vhost/virtio-net.c | 153 +++---
3 files changed, 117 insertions(+), 83 deletions(-)
diff --git a/lib/librte_vhost/rte_virtio_net.h
b/lib/librte_vhost/rte_virtio_net.h
index e3a21e5..9a32a95 100644
--- a/lib/librte_vhost/rte_virtio_net.h
+++ b/lib/librte_vhost/rte_virtio_net.h
@@ -96,7 +96,6 @@ struct vhost_virtqueue {
* Device structure contains all configuration information relating to the
device.
*/
struct virtio_net {
- struct vhost_virtqueue *virtqueue[VIRTIO_QNUM];/**< Contains
all virtqueue information. */
struct virtio_memory*mem; /**< QEMU memory and memory
region information. */
uint64_tfeatures; /**< Negotiated feature set. */
uint64_tprotocol_features; /**< Negotiated
protocol feature set. */
@@ -104,7 +103,9 @@ struct virtio_net {
uint32_tflags; /**< Device flags. Only used to
check if device is running on data core. */
#define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ)
charifname[IF_NAME_SZ]; /**< Name of the tap
device or socket path. */
+ uint32_tvirt_qp_nb; /**< number of queue pair we
have allocated */
void*priv; /**< private context */
+ struct vhost_virtqueue *virtqueue[VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MAX];
/**< Contains all virtqueue information. */
} __rte_cache_aligned;
/**
diff --git a/lib/librte_vhost/vhost_user/virtio-net-user.c
b/lib/librte_vhost/vhost_user/virtio-net-user.c
index 360254e..e83d279 100644
--- a/lib/librte_vhost/vhost_user/virtio-net-user.c
+++ b/lib/librte_vhost/vhost_user/virtio-net-user.c
@@ -206,25 +206,33 @@ err_mmap:
}
static int
+vq_is_ready(struct vhost_virtqueue *vq)
+{
+ return vq && vq->desc &&
+ vq->kickfd != -1 &&
+ vq->callfd != -1;
+}
+
+static int
virtio_is_ready(struct virtio_net *dev)
{
struct vhost_virtqueue *rvq, *tvq;
+ uint32_t i;
- /* mq support in future.*/
- rvq = dev->virtqueue[VIRTIO_RXQ];
- tvq = dev->virtqueue[VIRTIO_TXQ];
- if (rvq && tvq && rvq->desc && tvq->desc &&
- (rvq->kickfd != -1) &&
- (rvq->callfd != -1) &&
- (tvq->kickfd != -1) &&
- (tvq->callfd != -1)) {
- RTE_LOG(INFO, VHOST_CONFIG,
- "virtio is now ready for processing.\n");
- return 1;
+ for (i = 0; i < dev->virt_qp_nb; i++) {
+ rvq = dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_RXQ];
+ tvq = dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_TXQ];
+
+ if (!vq_is_ready(rvq) || !vq_is_ready(tvq)) {
+ RTE_LOG(INFO, VHOST_CONFIG,
+ "virtio is not ready for processing.\n");
+ return 0;
+ }
}
+
RTE_LOG(INFO, VHOST_CONFIG,
- "virtio isn't ready for processing.\n");
- return 0;
+ "virtio is now ready for processing.\n");
+ return 1;
}
void
@@ -290,13 +298,9 @@ user_get_vring_base(struct vhost_device_ctx ctx,
* sent and only sent in vhost_vring_stop.
* TODO: cleanup the vring, it isn't usable since here.
*/
- if
[dpdk-dev] [PATCH v7 3/8] vhost: vring queue setup for multiple queue support
All queue pairs, including the default (the first) queue pair,
are allocated dynamically, when a vring_call message is received
first time for a specific queue pair.
This is a refactor work for enabling vhost-user multiple queue;
it should not break anything as it does no functional changes:
we don't support mq set, so there is only one mq at max.
This patch is based on Changchun's patch.
Signed-off-by: Ouyang Changchun
Signed-off-by: Yuanhan Liu
Acked-by: Flavio Leitner
---
lib/librte_vhost/rte_virtio_net.h | 3 +-
lib/librte_vhost/vhost_user/virtio-net-user.c | 44
lib/librte_vhost/virtio-net.c | 144 --
3 files changed, 114 insertions(+), 77 deletions(-)
diff --git a/lib/librte_vhost/rte_virtio_net.h
b/lib/librte_vhost/rte_virtio_net.h
index e3a21e5..5dd6493 100644
--- a/lib/librte_vhost/rte_virtio_net.h
+++ b/lib/librte_vhost/rte_virtio_net.h
@@ -96,7 +96,7 @@ struct vhost_virtqueue {
* Device structure contains all configuration information relating to the
device.
*/
struct virtio_net {
- struct vhost_virtqueue *virtqueue[VIRTIO_QNUM];/**< Contains
all virtqueue information. */
+ struct vhost_virtqueue *virtqueue[VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MAX];
/**< Contains all virtqueue information. */
struct virtio_memory*mem; /**< QEMU memory and memory
region information. */
uint64_tfeatures; /**< Negotiated feature set. */
uint64_tprotocol_features; /**< Negotiated
protocol feature set. */
@@ -104,6 +104,7 @@ struct virtio_net {
uint32_tflags; /**< Device flags. Only used to
check if device is running on data core. */
#define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ)
charifname[IF_NAME_SZ]; /**< Name of the tap
device or socket path. */
+ uint32_tvirt_qp_nb; /**< number of queue pair we
have allocated */
void*priv; /**< private context */
} __rte_cache_aligned;
diff --git a/lib/librte_vhost/vhost_user/virtio-net-user.c
b/lib/librte_vhost/vhost_user/virtio-net-user.c
index 360254e..e83d279 100644
--- a/lib/librte_vhost/vhost_user/virtio-net-user.c
+++ b/lib/librte_vhost/vhost_user/virtio-net-user.c
@@ -206,25 +206,33 @@ err_mmap:
}
static int
+vq_is_ready(struct vhost_virtqueue *vq)
+{
+ return vq && vq->desc &&
+ vq->kickfd != -1 &&
+ vq->callfd != -1;
+}
+
+static int
virtio_is_ready(struct virtio_net *dev)
{
struct vhost_virtqueue *rvq, *tvq;
+ uint32_t i;
- /* mq support in future.*/
- rvq = dev->virtqueue[VIRTIO_RXQ];
- tvq = dev->virtqueue[VIRTIO_TXQ];
- if (rvq && tvq && rvq->desc && tvq->desc &&
- (rvq->kickfd != -1) &&
- (rvq->callfd != -1) &&
- (tvq->kickfd != -1) &&
- (tvq->callfd != -1)) {
- RTE_LOG(INFO, VHOST_CONFIG,
- "virtio is now ready for processing.\n");
- return 1;
+ for (i = 0; i < dev->virt_qp_nb; i++) {
+ rvq = dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_RXQ];
+ tvq = dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_TXQ];
+
+ if (!vq_is_ready(rvq) || !vq_is_ready(tvq)) {
+ RTE_LOG(INFO, VHOST_CONFIG,
+ "virtio is not ready for processing.\n");
+ return 0;
+ }
}
+
RTE_LOG(INFO, VHOST_CONFIG,
- "virtio isn't ready for processing.\n");
- return 0;
+ "virtio is now ready for processing.\n");
+ return 1;
}
void
@@ -290,13 +298,9 @@ user_get_vring_base(struct vhost_device_ctx ctx,
* sent and only sent in vhost_vring_stop.
* TODO: cleanup the vring, it isn't usable since here.
*/
- if ((dev->virtqueue[VIRTIO_RXQ]->kickfd) >= 0) {
- close(dev->virtqueue[VIRTIO_RXQ]->kickfd);
- dev->virtqueue[VIRTIO_RXQ]->kickfd = -1;
- }
- if ((dev->virtqueue[VIRTIO_TXQ]->kickfd) >= 0) {
- close(dev->virtqueue[VIRTIO_TXQ]->kickfd);
- dev->virtqueue[VIRTIO_TXQ]->kickfd = -1;
+ if ((dev->virtqueue[state->index]->kickfd) >= 0) {
+ close(dev->virtqueue[state->index]->kickfd);
+ dev->virtqueue[state->index]->kickfd = -1;
}
return 0;
diff --git a/lib/librte_vhost/virtio-net.c b/lib/librte_vhost/virtio-net.c
index deac6b9..57fb7b1 100644
--- a/lib/librte_vhost/virtio-net.c
+++ b/lib/librte_vhost/virtio-net.c
@@ -36,6 +36,7 @@
#include
#include
#include
+#include
#include
#include
#ifdef RTE_LIBRTE_VHOST_NUMA
@@ -178,6 +179,15 @@ add_config_ll_entry(struct virtio_net_config_ll
*new_ll_dev)
}
+static void
+cleanup_vq(struct vhost_virtqueue *vq)
+{
+ if
[dpdk-dev] [PATCH v7 3/8] vhost: vring queue setup for multiple queue support
On Wed, 21 Oct 2015 11:48:09 +0800
Yuanhan Liu wrote:
> struct virtio_net {
> - struct vhost_virtqueue *virtqueue[VIRTIO_QNUM];/**< Contains
> all virtqueue information. */
> + struct vhost_virtqueue *virtqueue[VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MAX];
> /**< Contains all virtqueue information. */
> struct virtio_memory*mem; /**< QEMU memory and memory
> region information. */
Since vhost_virtqueue takes up space, why not put it at end of array,
that way offsets are smaller and all the early fields will be in
adjacent cache lines.
