[ovs-dev] [PATCH v13] netdev-dpdk: Add custom rx-steering configuration.
Some control protocols are used to maintain link status between forwarding engines (e.g. LACP). When the system is not sized properly, the PMD threads may not be able to process all incoming traffic from the configured Rx queues. When a signaling packet of such protocols is dropped, it can cause link flapping, worsening the situation. Use the RTE flow API to redirect these protocols into a dedicated Rx queue. The assumption is made that the ratio between control protocol traffic and user data traffic is very low and thus this dedicated Rx queue will never get full. Re-program the RSS redirection table to only use the other Rx queues. The additional Rx queue will be assigned a PMD core like any other Rx queue. Polling that extra queue may introduce increased latency and a slight performance penalty at the benefit of preventing link flapping. This feature must be enabled per port on specific protocols via the rx-steering option. This option takes "rss" followed by a "+" separated list of protocol names. It is only supported on ethernet ports. This feature is experimental. If the user has already configured multiple Rx queues on the port, an additional one will be allocated for control packets. If the hardware cannot satisfy the number of requested Rx queues, the last Rx queue will be assigned for control plane. If only one Rx queue is available, the rx-steering feature will be disabled. If the hardware does not support the RTE flow matchers/actions, the rx-steering feature will be completely disabled on the port. It cannot be enabled when other_config:hw-offload=true as it may conflict with the offloaded RTE flows. Similarly, if hw-offload is enabled, custom rx-steering will be forcibly disabled on all ports. Example use: ovs-vsctl add-bond br-phy bond0 phy0 phy1 -- \ set interface phy0 type=dpdk options:dpdk-devargs=:ca:00.0 -- \ set interface phy0 options:rx-steering=rss+lacp -- \ set interface phy1 type=dpdk options:dpdk-devargs=:ca:00.1 -- \ set interface phy1 options:rx-steering=rss+lacp As a starting point, only one protocol is supported: LACP. Other protocols can be added in the future. NIC compatibility should be checked. To validate that this works as intended, I used a traffic generator to generate random traffic slightly above the machine capacity at line rate on a two ports bond interface. OVS is configured to receive traffic on two VLANs and pop/push them in a br-int bridge based on tags set on patch ports. +--+ | DUT | |++| || br-int || in_port=patch10,actions=mod_dl_src:$patch11,mod_dl_dst:$tgen1,output:patch11 |||| in_port=patch11,actions=mod_dl_src:$patch10,mod_dl_dst:$tgen0,output:patch10 || patch10patch11 || |+---|---|+| || | | |+---|---|+| || patch00patch01 || || tag:10tag:20 || |||| || br-phy || default flow, action=NORMAL |||| || bond0|| balance-slb, lacp=passive, lacp-time=fast ||phy0 phy1 || |+--|-|---+| +---|-|+ | | +---|-|+ | port0 port1 | balance L3/L4, lacp=active, lacp-time=fast | lag | mode trunk VLANs 10, 20 | | |switch| | | | vlan 10vlan 20 | mode access | port2 port3 | +-|--|-+ | | +-|--|-+ | tgen0 tgen1 | Random traffic that is properly balanced | | across the bond ports in both directions. | traffic generator | +--+ Without rx-steering, the bond0 links are randomly switching to "defaulted" when one of the LACP packets sent by the switch is dropped because the RX queues are full and the PMD threads did not process them fast enough. When that happens, all traffic must go through a single link which causes above line rate traffic to be dropped. ~# ovs-appctl lacp/show-stats bond0 bond0 statistics member: phy0: TX PDUs: 347246 RX PDUs: 14865 RX Bad PDUs: 0 RX Marker Request PDUs: 0 Link Expired: 168 Link Defaulted: 0 Carrier Status Changed: 0 member: phy1: TX PDUs: 347245 RX PDUs: 14919 RX Bad PDUs: 0 RX Marker Request PDUs: 0 Link Expired: 147 Link Defaulted: 1 Carrier Status Changed: 0 When rx-steering is enabled, no LACP packet is dropped and the bond links remain enabled at all times, maximizing the throughput. Neither the "Link Expired" nor the "Link Defaulted" counters are incremented anymore. This feature may be considered as "QoS". However, it does not work by limiting the rate of traffic explicitly. It only guarantees that some protocols have a lower chance of being dropped because the PMD cores cannot keep
Re: [ovs-dev] [PATCH v6 3/4] netdev-dpdk: Add support for egress packet-per-second policing.
Bleep bloop. Greetings miter, I am a robot and I have tried out your patch. Thanks for your contribution. I encountered some error that I wasn't expecting. See the details below. checkpatch: WARNING: Line is 80 characters long (recommended limit is 79) #308 FILE: lib/netdev-dpdk.c:5063: pps_drop = pkts_policer_run_single_packet(&policer->egress_tb, pkt_cnt); Lines checked: 600, Warnings: 1, Errors: 0 Please check this out. If you feel there has been an error, please email acon...@redhat.com Thanks, 0-day Robot ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [PATCH v6 4/4] netdev-dpdk: Add support for ingress packet-per-second policing.
From: Lin Huang
OvS has supported packet-per-second policer which can be set at ingress
and egress side in kernel datapath. But the userspace datapath dosen't
support for ingress and egress packet-per-second policing now.
So, this patch add support for userspace ingress pps policing by using
native ovs token bucket library. Token bucket is accumulated by 'rate'
tokens per millisecond and store maxiumim tokens at 'burst' bucket size.
One token in the bucket means one packet (1 kpkts * millisecond) which
will drop or pass by policer.
This patch reuse 'ingress_policing_kpkts_rate' and
'ingress_policing_kpkts_burst' options at interface table. Now userspace
ingress policer supports setting packet-per-second limits in addition to
the previously configurable byte rate settings.
Examples:
$ ovs-vsctl set interface dpdk0 ingress_policing_rate=12300
$ ovs-vsctl set interface dpdk0 ingress_policing_burst=12300
$ ovs-vsctl set interface dpdk0 ingress_policing_kpkts_rate=123
$ ovs-vsctl set interface dpdk0 ingress_policing_kpkts_burst=123
Add some unit tests for ingress packet-per-second policing.
Signed-off-by: Lin Huang
---
Documentation/topics/dpdk/qos.rst | 14 +-
NEWS | 2 +
lib/netdev-dpdk.c | 71 --
tests/system-dpdk.at | 218 ++
4 files changed, 288 insertions(+), 17 deletions(-)
diff --git a/Documentation/topics/dpdk/qos.rst
b/Documentation/topics/dpdk/qos.rst
index 5f0b1469a..4812ed37c 100644
--- a/Documentation/topics/dpdk/qos.rst
+++ b/Documentation/topics/dpdk/qos.rst
@@ -106,13 +106,19 @@ Refer to ``vswitch.xml`` for more details on egress
policer.
Rate Limiting (Ingress Policing)
-Assuming you have a :doc:`vhost-user port ` receiving traffic
-consisting of packets of size 64 bytes, the following command would limit the
-reception rate of the port to ~1,000,000 packets per second::
+Assuming you have a :doc:`vhost-user port ` receiving traffic,
+the following command would limit the reception rate of the port to
+~1,000,000 bits per second::
-$ ovs-vsctl set interface vhost-user0 ingress_policing_rate=368000 \
+$ ovs-vsctl set interface vhost-user0 ingress_policing_rate=1000 \
ingress_policing_burst=1000`
+or, the following command would limit the reception rate of the port to
+~1,000,000 packets per second::
+
+$ ovs-vsctl set interface dpdk0 ingress_policing_kpkts_rate=1000 \
+ingress_policing_kpkts_burst=1000`
+
To examine the ingress policer configuration of the port::
$ ovs-vsctl list interface vhost-user0
diff --git a/NEWS b/NEWS
index d88f490b1..91f62a0e4 100644
--- a/NEWS
+++ b/NEWS
@@ -46,6 +46,8 @@ Post-v3.1.0
table to check the status.
* Added new configuration options 'kpkts_rate' and 'kpkts_burst' for
'egress-policer' to support kilo packet-per-second policing.
+ * Added support for ingress kilo packet-per-second policing configured by
+ ingress_policing_kpkts_rate/burst options.
v3.1.0 - 16 Feb 2023
diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
index f97153665..b6efa70a4 100644
--- a/lib/netdev-dpdk.c
+++ b/lib/netdev-dpdk.c
@@ -413,6 +413,8 @@ struct ingress_policer {
struct rte_meter_srtcm_params app_srtcm_params;
struct rte_meter_srtcm in_policer;
struct rte_meter_srtcm_profile in_prof;
+struct token_bucket tb;
+enum policer_type type;
rte_spinlock_t policer_lock;
};
@@ -499,6 +501,9 @@ struct netdev_dpdk {
uint32_t policer_rate;
uint32_t policer_burst;
+uint32_t policer_kpkts_rate;
+uint32_t policer_kpkts_burst;
+
/* Array of vhost rxq states, see vring_state_changed. */
bool *vhost_rxq_enabled;
);
@@ -1377,6 +1382,8 @@ common_construct(struct netdev *netdev, dpdk_port_t
port_no,
ovsrcu_init(&dev->ingress_policer, NULL);
dev->policer_rate = 0;
dev->policer_burst = 0;
+dev->policer_kpkts_rate = 0;
+dev->policer_kpkts_burst = 0;
netdev->n_rxq = 0;
netdev->n_txq = 0;
@@ -2471,14 +2478,22 @@ static int
ingress_policer_run(struct ingress_policer *policer, struct rte_mbuf **pkts,
int pkt_cnt, bool should_steal)
{
+int bps_drop = 0, pps_drop = 0;
int cnt = 0;
rte_spinlock_lock(&policer->policer_lock);
-cnt = srtcm_policer_run_single_packet(&policer->in_policer,
- &policer->in_prof,
- pkts, pkt_cnt);
+if (policer->type & POLICER_BPS) {
+bps_drop = srtcm_policer_run_single_packet(&policer->in_policer,
+ &policer->in_prof,
+ pkts, pkt_cnt);
+}
+
+if (policer->type & POLICER_PKTPS) {
+pps_drop = pkts_policer_run_single_packet(&policer->tb, pkt_cnt);
+}
rte_spinlock_unlock(&policer->policer_lo
[ovs-dev] [PATCH v6 3/4] netdev-dpdk: Add support for egress packet-per-second policing.
From: Lin Huang
OvS has supported packet-per-second policer which can be set at ingress
and egress side in kernel datapath. But the userspace datapath doesn't
support for ingress and egress packet-per-second policing now.
So, this patch add support for userspace egress pps policing by using
native ovs token bucket library. Token bucket is accumulated by 'rate'
tokens per millisecond and store maximum tokens at 'burst' bucket size.
One token in the bucket means one packet (1 kpkts * millisecond) which
will drop or pass by policer.
This patch add new configuration option 'kpkts_rate' and 'kpkts_burst'
for egress-policer QoS type which now supports setting packet-per-second
limits in addition to the previously configurable byte rate settings.
Examples:
$ovs-vsctl set port vhost-user0 qos=@newqos --
--id=@newqos create qos type=egress-policer \
other-config:cir=123000 other-config:cbs=123000
other-config:kpkts_rate=123 other-config:kpkts_burst=123
Add some unit tests for egress packet-per-second policing.
Signed-off-by: Lin Huang
---
Documentation/topics/dpdk/qos.rst | 15 ++-
NEWS | 2 +
lib/netdev-dpdk.c | 132
tests/system-dpdk.at | 194 +-
vswitchd/vswitch.xml | 10 ++
5 files changed, 327 insertions(+), 26 deletions(-)
diff --git a/Documentation/topics/dpdk/qos.rst
b/Documentation/topics/dpdk/qos.rst
index a98ec672f..5f0b1469a 100644
--- a/Documentation/topics/dpdk/qos.rst
+++ b/Documentation/topics/dpdk/qos.rst
@@ -36,14 +36,21 @@ QoS (Egress Policing)
Single Queue Policer
-Assuming you have a :doc:`vhost-user port ` transmitting traffic
-consisting of packets of size 64 bytes, the following command would limit the
-egress transmission rate of the port to ~1,000,000 packets per second::
+Assuming you have a :doc:`vhost-user port ` transmitting traffic,
+the following command would limit the egress transmission rate of the port to
+~1,000,000 bytes per second:
$ ovs-vsctl set port vhost-user0 qos=@newqos -- \
---id=@newqos create qos type=egress-policer other-config:cir=4600 \
+--id=@newqos create qos type=egress-policer other-config:cir=100 \
other-config:cbs=2048`
+or, the following command would limit the egress transmission rate of the port
+to ~1,000,000 packets per second:
+
+ ovs-vsctl set port vhost-user0 qos=@newqos -- \
+ --id=@newqos create qos type=egress-policer \
+ other-config:kpkts_rate=1000 other-config:kpkts_burst=1000
+
To examine the QoS configuration of the port, run::
$ ovs-appctl -t ovs-vswitchd qos/show vhost-user0
diff --git a/NEWS b/NEWS
index 0b5dc3db1..d88f490b1 100644
--- a/NEWS
+++ b/NEWS
@@ -44,6 +44,8 @@ Post-v3.1.0
* IP and L4 checksum offload support is now enabled by default for
interfaces that support it. See the 'status' column in the 'interface'
table to check the status.
+ * Added new configuration options 'kpkts_rate' and 'kpkts_burst' for
+ 'egress-policer' to support kilo packet-per-second policing.
v3.1.0 - 16 Feb 2023
diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
index 2f7f74395..f97153665 100644
--- a/lib/netdev-dpdk.c
+++ b/lib/netdev-dpdk.c
@@ -19,6 +19,7 @@
#include
#include
+#include
#include
#include
#include
@@ -59,6 +60,7 @@
#include "openvswitch/ofp-parse.h"
#include "openvswitch/ofp-print.h"
#include "openvswitch/shash.h"
+#include "openvswitch/token-bucket.h"
#include "openvswitch/vlog.h"
#include "ovs-numa.h"
#include "ovs-rcu.h"
@@ -91,6 +93,8 @@ static bool per_port_memory = false; /* Status of per port
memory support */
#define OVS_CACHE_LINE_SIZE CACHE_LINE_SIZE
#define OVS_VPORT_DPDK "ovs_dpdk"
+#define MAX_KPKTS_PARAMETER 4294967U /* UINT32_MAX / 1000 */
+
/*
* need to reserve tons of extra space in the mbufs so we can align the
* DMA addresses to 4KB.
@@ -400,6 +404,11 @@ struct dpdk_tx_queue {
);
};
+enum policer_type {
+POLICER_BPS = 1 << 0, /* Rate value in bytes/sec. */
+POLICER_PKTPS = 1 << 1, /* Rate value in packet/sec. */
+};
+
struct ingress_policer {
struct rte_meter_srtcm_params app_srtcm_params;
struct rte_meter_srtcm in_policer;
@@ -2418,6 +2427,46 @@ srtcm_policer_run_single_packet(struct rte_meter_srtcm
*meter,
return cnt;
}
+static int
+pkts_policer_run_single_packet(struct token_bucket *tb, int pkt_cnt)
+{
+int cnt = 0;
+long long int now = time_msec();
+
+for (int i = 0; i < pkt_cnt; i++) {
+/* Handle current packet. */
+if (!token_bucket_withdraw(tb, 1, now)) {
+/* Count dropped packets. */
+cnt++;
+}
+}
+
+return cnt;
+}
+
+static int
+pkts_policer_profile_config(struct token_bucket *tb,
+uint32_t kpkts_rate, uint32_t kpkts_burst)
+{
+if (kpkts_rate >
[ovs-dev] [PATCH v6 2/4] netdev-dpdk: Make srtcm_policer not to free pkts.
From: Lin Huang
Now srtcm_policer will free pkts, if packets are exceed rate limit.
This patch change srtcm_policer not to free pkts, just count dropped packets.
Signed-off-by: Lin Huang
---
lib/netdev-dpdk.c | 30 +++---
1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
index 63dac689e..2f7f74395 100644
--- a/lib/netdev-dpdk.c
+++ b/lib/netdev-dpdk.c
@@ -2399,8 +2399,7 @@ netdev_dpdk_srtcm_policer_pkt_handle(struct
rte_meter_srtcm *meter,
static int
srtcm_policer_run_single_packet(struct rte_meter_srtcm *meter,
struct rte_meter_srtcm_profile *profile,
-struct rte_mbuf **pkts, int pkt_cnt,
-bool should_steal)
+struct rte_mbuf **pkts, int pkt_cnt)
{
int i = 0;
int cnt = 0;
@@ -2410,16 +2409,9 @@ srtcm_policer_run_single_packet(struct rte_meter_srtcm
*meter,
for (i = 0; i < pkt_cnt; i++) {
pkt = pkts[i];
/* Handle current packet */
-if (netdev_dpdk_srtcm_policer_pkt_handle(meter, profile,
- pkt, current_time)) {
-if (cnt != i) {
-pkts[cnt] = pkt;
-}
+if (!netdev_dpdk_srtcm_policer_pkt_handle(meter, profile,
+ pkt, current_time)) {
cnt++;
-} else {
-if (should_steal) {
-rte_pktmbuf_free(pkt);
-}
}
}
@@ -2435,10 +2427,14 @@ ingress_policer_run(struct ingress_policer *policer,
struct rte_mbuf **pkts,
rte_spinlock_lock(&policer->policer_lock);
cnt = srtcm_policer_run_single_packet(&policer->in_policer,
&policer->in_prof,
- pkts, pkt_cnt, should_steal);
+ pkts, pkt_cnt);
rte_spinlock_unlock(&policer->policer_lock);
-return cnt;
+if (should_steal && cnt) {
+rte_pktmbuf_free_bulk(&pkts[pkt_cnt - cnt], cnt);
+}
+
+return pkt_cnt - cnt;
}
static bool
@@ -4976,9 +4972,13 @@ egress_policer_run(struct qos_conf *conf, struct
rte_mbuf **pkts, int pkt_cnt,
cnt = srtcm_policer_run_single_packet(&policer->egress_meter,
&policer->egress_prof, pkts,
- pkt_cnt, should_steal);
+ pkt_cnt);
-return cnt;
+if (should_steal && cnt) {
+rte_pktmbuf_free_bulk(&pkts[pkt_cnt - cnt], cnt);
+}
+
+return pkt_cnt - cnt;
}
static const struct dpdk_qos_ops egress_policer_ops = {
--
2.39.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [PATCH v6 1/4] token-bucket: Make token-bucket timestamp updated by caller.
From: Lin Huang
Now, token-bucket 'last_fill' is updated by token_bucket_withdraw() itself.
Add a new function parameter 'now' to update timestamp by caller.
Signed-off-by: Lin Huang
---
include/openvswitch/token-bucket.h | 3 ++-
lib/token-bucket.c | 4 ++--
lib/vlog.c | 3 ++-
ofproto/pinsched.c | 2 +-
4 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/include/openvswitch/token-bucket.h
b/include/openvswitch/token-bucket.h
index 580747f61..d1191e956 100644
--- a/include/openvswitch/token-bucket.h
+++ b/include/openvswitch/token-bucket.h
@@ -40,7 +40,8 @@ void token_bucket_init(struct token_bucket *,
unsigned int rate, unsigned int burst);
void token_bucket_set(struct token_bucket *,
unsigned int rate, unsigned int burst);
-bool token_bucket_withdraw(struct token_bucket *, unsigned int n);
+bool token_bucket_withdraw(struct token_bucket *tb, unsigned int n,
+ long long int now);
void token_bucket_wait_at(struct token_bucket *, unsigned int n,
const char *where);
#define token_bucket_wait(bucket, n)\
diff --git a/lib/token-bucket.c b/lib/token-bucket.c
index 0badeb46b..60eb26e53 100644
--- a/lib/token-bucket.c
+++ b/lib/token-bucket.c
@@ -59,10 +59,10 @@ token_bucket_set(struct token_bucket *tb,
* if 'tb' contained fewer than 'n' tokens (and thus 'n' tokens could not be
* removed) . */
bool
-token_bucket_withdraw(struct token_bucket *tb, unsigned int n)
+token_bucket_withdraw(struct token_bucket *tb, unsigned int n,
+ long long int now)
{
if (tb->tokens < n) {
-long long int now = time_msec();
if (now > tb->last_fill) {
unsigned long long int elapsed_ull
= (unsigned long long int) now - tb->last_fill;
diff --git a/lib/vlog.c b/lib/vlog.c
index b2653142f..380dcd479 100644
--- a/lib/vlog.c
+++ b/lib/vlog.c
@@ -1321,7 +1321,8 @@ vlog_should_drop(const struct vlog_module *module, enum
vlog_level level,
}
ovs_mutex_lock(&rl->mutex);
-if (!token_bucket_withdraw(&rl->token_bucket, VLOG_MSG_TOKENS)) {
+if (!token_bucket_withdraw(&rl->token_bucket, VLOG_MSG_TOKENS,
+ time_msec())) {
time_t now = time_now();
if (!rl->n_dropped) {
rl->first_dropped = now;
diff --git a/ofproto/pinsched.c b/ofproto/pinsched.c
index 59561f076..a39e4d2ee 100644
--- a/ofproto/pinsched.c
+++ b/ofproto/pinsched.c
@@ -184,7 +184,7 @@ get_tx_packet(struct pinsched *ps)
static bool
get_token(struct pinsched *ps)
{
-return token_bucket_withdraw(&ps->token_bucket, 1000);
+return token_bucket_withdraw(&ps->token_bucket, 1000, time_msec());
}
void
--
2.39.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [PATCH v6 0/4] netdev-dpdk: Add support for userspace port-based packet-per-second policing.
From: Lin Huang v5->v6: fix code by reviewing. v5->v4: police pkts bps and pps at the same time. Get the maximum from the number of dropped packets, and drop that. v4->v3: rewrite egress_policer_details_to_param func. add a new pkts_policer_profile_config func. v2->v3: police and free pkts by bulk. fix the appropriate error code. v1->v2: delete duplicated code. Lin Huang (4): token-bucket: Make token-bucket timestamp updated by caller. netdev-dpdk: Make srtcm_policer not to free pkts. netdev-dpdk: Add support for egress packet-per-second policing. netdev-dpdk: Add support for ingress packet-per-second policing. Documentation/topics/dpdk/qos.rst | 29 +- NEWS | 4 + include/openvswitch/token-bucket.h | 3 +- lib/netdev-dpdk.c | 229 lib/token-bucket.c | 4 +- lib/vlog.c | 3 +- ofproto/pinsched.c | 2 +- tests/system-dpdk.at | 412 - vswitchd/vswitch.xml | 10 + 9 files changed, 635 insertions(+), 61 deletions(-) -- 2.39.1 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
