Hi Billy,
>Hi All,
>
>This patch set provides a method to request ingress scheduling on interfaces.
>It also provides an implemtation of same for DPDK physical ports.
>
>This allows specific packet types to be:
>* forwarded to their destination port ahead of other packets.
>and/or
>* be less likely to be dropped in an overloaded situation.
>
>It was previously discussed
>https://mail.openvswitch.org/pipermail/ovs-discuss/2017-May/044395.html
>and RFC'd
>https://mail.openvswitch.org/pipermail/ovs-dev/2017-July/335237.html
>
>Limitations of this patch:
>* The patch uses the Flow Director filter API in DPDK and has only been tested
>on Fortville (XL710) NIC.
>* Prioritization is limited to:
>** eth_type
>** Fully specified 5-tuple src & dst ip and port numbers for UDP & TCP packets
>* ovs-appctl dpif-netdev/pmd-*-show o/p should indicate rxq prioritization.
>* any requirements for a more granular prioritization mechanism
>
>Initial results:
>* even when userspace OVS is very much overloaded and
> dropping significant numbers of packets the drop rate for prioritized traffic
> is running at 1/1000th of the drop rate for non-prioritized traffic.
>
>* the latency profile of prioritized traffic through userspace OVS is also much
> improved
>
>1e0 |*
> |*
>1e-1 |* | Non-prioritized pkt latency
> |* * Prioritized pkt latency
>1e-2 |*
> |*
>1e-3 |* |
> |* |
>1e-4 |* | | |
> |* |* | |
>1e-5 |* |* | | |
> |* |* |* | | |
>1e-6 |* |* |* |* | |
> |* |* |* |* |* |
>1e-7 |* |* |* |* |* |*
> |* |* |* |* |* |*
>1e-8 |* |* |* |* |* |*
> 0-1 1-20 20-40 40-50 50-60 60-70 ... 120-400
> Latency (us)
>
> Proportion of packets per latency bin @ 80% Max Throughput
> (Log scale)
>
Thanks for working on this feature. I started reviewing the code initially but
later decided to test it first as it uses XL710 NIC Flow director features and
wanted to
Know the implications if any. I had few observations here and would like to
know if you have seen this during your unit tests.
1) With this patch series, Rx Burst Bulk Allocation call back function is
invoked instead of vector rx function.
Meaning i40e_recv_pkts_bulk_alloc() gets invoked instead of
i40e_recv_pkts_vec(). Please check i40e_set_rx_function() of i40e DPDK
drivers.
I am speculating this may be due to the enabling flow director and rules.
I don't know the implications of using bulk_alloc() function, maybe we should
check with DPDK guys on this.
2) When I tried to prioritize the udp pkts for specific IPs and Ports, I see a
massive performance drop. I am using XL710 NIC with stable firmware version.
Below are my steps.
- Start OvS and make sure the the n_rxq for DPDK0, DPDK1 ports is set
to 2.
- Do simple P2P test with single
stream(ip_src=8.18.8.1,ip_dst=101.10.10.1,udp_src=10001,udp_dst=5001) and
check the throughput.
- Prioritize the active stream.
ovs-vsctl set interface dpdk0
other_config:ingress_sched=udp,ip_src=8.18.8.1,ip_dst=101.10.10.1,udp_src=10001,udp_dst=5001
- Throughput drop is observed now. (~1.7Mpps)
A bit of debugging in to case 2, I found that "miniflow_hash_5tuple()" is
getting invoked and consuming 10% of the total cycles.
one of the commits had below lines.
----------------------------dpdk_eth_dev_queue_setup------------------------------
/* Ingress scheduling requires ETH_MQ_RX_NONE so limit it to when exactly
* two rxqs are defined. Otherwise MQ will not work as expected. */
if (dev->ingress_sched_str && n_rxq == 2) {
conf.rxmode.mq_mode = ETH_MQ_RX_NONE;
}
else {
conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
}
---------------------------------------------------------------------------------------------
Does ingress scheduling turn off RSS? This will be big drawback as calculating
hash in SW consumes significant cycles.
3) This is another corner case.
- Here n_rxq set to 4 for my DPDK ports. start OvS and traffic is started
and throughput is as expected.
- Now prioritize the stream
ovs-vsctl set interface dpdk0
other_config:ingress_sched=udp,ip_src=8.18.8.1,ip_dst=101.10.10.1,udp_src=10001,udp_dst=5001
- The above command shouldn't take in to affect as n_rxq is set to 4 and
not 2 and the same is logged appropriately.
"2017-07-28T11:11:57.792Z|00104|netdev_dpdk|ERR|Interface dpdk0:
Ingress scheduling config ignored; Requires n_rxq==2.
2017-07-28T11:11:57.809Z|00105|dpdk|INFO|PMD: i40e_pf_config_rss():
Max of contiguous 4 PF queues are configured"
- However the throughput drop is observed in this case too.
Let me know if the above is reproducible on your setup. Meanwhile I will review
the code.
Regards,
Bhanuprakash.
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
