Re: [ovs-dev] [PATCH v5 4/4] dpif-netdev: Introduce pmd-rxq-affinity.

[Daniele Di Proietto]

ofputil_parse_key_value() to parse the affinity seems like a good idea, thanks!


I got a compiler warning on an unused variable.  I fixed that and applied the 
series to master.

Thanks,

Daniele



On 27/07/2016 07:44, "Ilya Maximets"  wrote:

>New 'other_config:pmd-rxq-affinity' field for Interface table to
>perform manual pinning of RX queues to desired cores.
>
>This functionality is required to achieve maximum performance because
>all kinds of ports have different cost of rx/tx operations and
>only user can know about expected workload on different ports.
>
>Example:
>   # ./bin/ovs-vsctl set interface dpdk0 options:n_rxq=4 \
> other_config:pmd-rxq-affinity="0:3,1:7,3:8"
>   Queue #0 pinned to core 3;
>   Queue #1 pinned to core 7;
>   Queue #2 not pinned.
>   Queue #3 pinned to core 8;
>
>It's decided to automatically isolate cores that have rxq explicitly
>assigned to them because it's useful to keep constant polling rate on
>some performance critical ports while adding/deleting other ports
>without explicit pinning of all ports.
>
>Signed-off-by: Ilya Maximets 
>---
> INSTALL.DPDK.md  |  49 +++-
> NEWS |   2 +
> lib/dpif-netdev.c| 216 +--
> tests/pmd.at |   6 ++
> vswitchd/vswitch.xml |  23 ++
> 5 files changed, 254 insertions(+), 42 deletions(-)
>
>diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md
>index 5407794..7609aa7 100644
>--- a/INSTALL.DPDK.md
>+++ b/INSTALL.DPDK.md
>@@ -289,14 +289,57 @@ advanced install guide [INSTALL.DPDK-ADVANCED.md]
>  # Check current stats
>ovs-appctl dpif-netdev/pmd-stats-show
> 
>+ # Clear previous stats
>+   ovs-appctl dpif-netdev/pmd-stats-clear
>+ ```
>+
>+  7. Port/rxq assigment to PMD threads
>+
>+ ```
>  # Show port/rxq assignment
>ovs-appctl dpif-netdev/pmd-rxq-show
>+ ```
> 
>- # Clear previous stats
>-   ovs-appctl dpif-netdev/pmd-stats-clear
>+ To change default rxq assignment to pmd threads rxqs may be manually
>+ pinned to desired cores using:
>+
>+ ```
>+ ovs-vsctl set Interface  \
>+   other_config:pmd-rxq-affinity=
>  ```
>+ where:
>+
>+ ```
>+  ::= NULL | 
>+  ::=  |
>+   , 
>+  ::=  : 
>+ ```
>+
>+ Example:
>+
>+ ```
>+ ovs-vsctl set interface dpdk0 options:n_rxq=4 \
>+   other_config:pmd-rxq-affinity="0:3,1:7,3:8"
>+
>+ Queue #0 pinned to core 3;
>+ Queue #1 pinned to core 7;
>+ Queue #2 not pinned.
>+ Queue #3 pinned to core 8;
>+ ```
>+
>+ After that PMD threads on cores where RX queues was pinned will become
>+ `isolated`. This means that this thread will poll only pinned RX queues.
>+
>+ WARNING: If there are no `non-isolated` PMD threads, `non-pinned` RX 
>queues
>+ will not be polled. Also, if provided `core_id` is not available (ex. 
>this
>+ `core_id` not in `pmd-cpu-mask`), RX queue will not be polled by any
>+ PMD thread.
>+
>+ Isolation of PMD threads also can be checked using
>+ `ovs-appctl dpif-netdev/pmd-rxq-show` command.
> 
>-  7. Stop vswitchd & Delete bridge
>+  8. Stop vswitchd & Delete bridge
> 
>  ```
>  ovs-appctl -t ovs-vswitchd exit
>diff --git a/NEWS b/NEWS
>index 73d3fcf..1a34f75 100644
>--- a/NEWS
>+++ b/NEWS
>@@ -45,6 +45,8 @@ Post-v2.5.0
>Old 'other_config:n-dpdk-rxqs' is no longer supported.
>Not supported by vHost interfaces. For them number of rx and tx queues
>is applied from connected virtio device.
>+ * New 'other_config:pmd-rxq-affinity' field for PMD interfaces, that
>+   allows to pin port's rx queues to desired cores.
>  * New appctl command 'dpif-netdev/pmd-rxq-show' to check the port/rxq
>assignment.
>  * Type of log messages from PMD threads changed from INFO to DBG.
>diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
>index 1ef0cd7..33f1216 100644
>--- a/lib/dpif-netdev.c
>+++ b/lib/dpif-netdev.c
>@@ -53,7 +53,9 @@
> #include "openvswitch/list.h"
> #include "openvswitch/match.h"
> #include "openvswitch/ofp-print.h"
>+#include "openvswitch/ofp-util.h"
> #include "openvswitch/ofpbuf.h"
>+#include "openvswitch/shash.h"
> #include "openvswitch/vlog.h"
> #include "ovs-numa.h"
> #include "ovs-rcu.h"
>@@ -62,7 +64,7 @@
> #include "pvector.h"
> #include "random.h"
> #include "seq.h"
>-#include "openvswitch/shash.h"
>+#include "smap.h"
> #include "sset.h"
> #include "timeval.h"
> #include "tnl-neigh-cache.h"
>@@ -252,6 +254,12 @@ enum pmd_cycles_counter_type {
> 
> #define XPS_TIMEOUT_MS 500LL
> 
>+/* Contained by struct dp_netdev_port's 'rxqs' member.  */
>+struct dp_netdev_rxq {
>+struct netdev_rxq *rxq;
>+unsigned core_id;   /* Сore to which this queue is pinned. */
>+};
>+
> /* A port in a netdev-based datapath. */
> struct dp_netdev_port {
> odp_port_t 

[ovs-dev] [PATCH v5 4/4] dpif-netdev: Introduce pmd-rxq-affinity.

[Ilya Maximets]

New 'other_config:pmd-rxq-affinity' field for Interface table to
perform manual pinning of RX queues to desired cores.

This functionality is required to achieve maximum performance because
all kinds of ports have different cost of rx/tx operations and
only user can know about expected workload on different ports.

Example:
# ./bin/ovs-vsctl set interface dpdk0 options:n_rxq=4 \
  other_config:pmd-rxq-affinity="0:3,1:7,3:8"
Queue #0 pinned to core 3;
Queue #1 pinned to core 7;
Queue #2 not pinned.
Queue #3 pinned to core 8;

It's decided to automatically isolate cores that have rxq explicitly
assigned to them because it's useful to keep constant polling rate on
some performance critical ports while adding/deleting other ports
without explicit pinning of all ports.

Signed-off-by: Ilya Maximets 
---
 INSTALL.DPDK.md  |  49 +++-
 NEWS |   2 +
 lib/dpif-netdev.c| 216 +--
 tests/pmd.at |   6 ++
 vswitchd/vswitch.xml |  23 ++
 5 files changed, 254 insertions(+), 42 deletions(-)

diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md
index 5407794..7609aa7 100644
--- a/INSTALL.DPDK.md
+++ b/INSTALL.DPDK.md
@@ -289,14 +289,57 @@ advanced install guide [INSTALL.DPDK-ADVANCED.md]
  # Check current stats
ovs-appctl dpif-netdev/pmd-stats-show
 
+ # Clear previous stats
+   ovs-appctl dpif-netdev/pmd-stats-clear
+ ```
+
+  7. Port/rxq assigment to PMD threads
+
+ ```
  # Show port/rxq assignment
ovs-appctl dpif-netdev/pmd-rxq-show
+ ```
 
- # Clear previous stats
-   ovs-appctl dpif-netdev/pmd-stats-clear
+ To change default rxq assignment to pmd threads rxqs may be manually
+ pinned to desired cores using:
+
+ ```
+ ovs-vsctl set Interface  \
+   other_config:pmd-rxq-affinity=
  ```
+ where:
+
+ ```
+  ::= NULL | 
+  ::=  |
+   , 
+  ::=  : 
+ ```
+
+ Example:
+
+ ```
+ ovs-vsctl set interface dpdk0 options:n_rxq=4 \
+   other_config:pmd-rxq-affinity="0:3,1:7,3:8"
+
+ Queue #0 pinned to core 3;
+ Queue #1 pinned to core 7;
+ Queue #2 not pinned.
+ Queue #3 pinned to core 8;
+ ```
+
+ After that PMD threads on cores where RX queues was pinned will become
+ `isolated`. This means that this thread will poll only pinned RX queues.
+
+ WARNING: If there are no `non-isolated` PMD threads, `non-pinned` RX 
queues
+ will not be polled. Also, if provided `core_id` is not available (ex. this
+ `core_id` not in `pmd-cpu-mask`), RX queue will not be polled by any
+ PMD thread.
+
+ Isolation of PMD threads also can be checked using
+ `ovs-appctl dpif-netdev/pmd-rxq-show` command.
 
-  7. Stop vswitchd & Delete bridge
+  8. Stop vswitchd & Delete bridge
 
  ```
  ovs-appctl -t ovs-vswitchd exit
diff --git a/NEWS b/NEWS
index 73d3fcf..1a34f75 100644
--- a/NEWS
+++ b/NEWS
@@ -45,6 +45,8 @@ Post-v2.5.0
Old 'other_config:n-dpdk-rxqs' is no longer supported.
Not supported by vHost interfaces. For them number of rx and tx queues
is applied from connected virtio device.
+ * New 'other_config:pmd-rxq-affinity' field for PMD interfaces, that
+   allows to pin port's rx queues to desired cores.
  * New appctl command 'dpif-netdev/pmd-rxq-show' to check the port/rxq
assignment.
  * Type of log messages from PMD threads changed from INFO to DBG.
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 1ef0cd7..33f1216 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -53,7 +53,9 @@
 #include "openvswitch/list.h"
 #include "openvswitch/match.h"
 #include "openvswitch/ofp-print.h"
+#include "openvswitch/ofp-util.h"
 #include "openvswitch/ofpbuf.h"
+#include "openvswitch/shash.h"
 #include "openvswitch/vlog.h"
 #include "ovs-numa.h"
 #include "ovs-rcu.h"
@@ -62,7 +64,7 @@
 #include "pvector.h"
 #include "random.h"
 #include "seq.h"
-#include "openvswitch/shash.h"
+#include "smap.h"
 #include "sset.h"
 #include "timeval.h"
 #include "tnl-neigh-cache.h"
@@ -252,6 +254,12 @@ enum pmd_cycles_counter_type {
 
 #define XPS_TIMEOUT_MS 500LL
 
+/* Contained by struct dp_netdev_port's 'rxqs' member.  */
+struct dp_netdev_rxq {
+struct netdev_rxq *rxq;
+unsigned core_id;   /* Сore to which this queue is pinned. */
+};
+
 /* A port in a netdev-based datapath. */
 struct dp_netdev_port {
 odp_port_t port_no;
@@ -259,11 +267,12 @@ struct dp_netdev_port {
 struct hmap_node node;  /* Node in dp_netdev's 'ports'. */
 struct netdev_saved_flags *sf;
 unsigned n_rxq; /* Number of elements in 'rxq' */
-struct netdev_rxq **rxq;
+struct dp_netdev_rxq *rxqs;
 atomic_bool dynamic_txqs;   /* If true XPS will be used. */
 unsigned *txq_used; /* Number of threads