Good discussion. Some thoughts: 1. Prioritizing queues by assigning them to dedicated PMDs is a simple and effective but very crude method, considering that you have to reserve an entire (logical) core for that. So I am all for a more economic and perhaps slightly less deterministic option!
2. Offering the option to prioritize certain queues in OVS-DPDK is a highly desirable feature. We have at least one important use case in OpenStack (prioritizing "in-band" infrastructure control plane traffic over tenant data, in case both are carried on the same physical network). In our case the traffic separation would be done per VLAN. Can we add this to the list of supported filters? 3. It would be nice to be able to combine priority queues with filters with a number of RSS queues without filter. Is this a XL710 HW limitation or only a limitation of the drivers and DPDK APIs? BR, Jan > From: ovs-dev-boun...@openvswitch.org [mailto:ovs-dev- > boun...@openvswitch.org] On Behalf Of O Mahony, Billy > Sent: Thursday, 17 August, 2017 18:07 > To: Kevin Traynor <ktray...@redhat.com>; d...@openvswitch.org > Subject: Re: [ovs-dev] [PATCH 0/4] prioritizing latency sensitive traffic > > Hi Kevin, > > Thanks for the comments - more inline. > > Billy. > > > -----Original Message----- > > From: Kevin Traynor [mailto:ktray...@redhat.com] > > Sent: Thursday, August 17, 2017 3:37 PM > > To: O Mahony, Billy <billy.o.mah...@intel.com>; d...@openvswitch.org > > Subject: Re: [ovs-dev] [PATCH 0/4] prioritizing latency sensitive > > traffic > > > > Hi Billy, > > > > I just happened to be about to send a reply to the previous patchset, > > so adding comments here instead. > > > > On 08/17/2017 03:24 PM, Billy O'Mahony wrote: > > > Hi All, > > > > > > v2: Addresses various review comments; Applies cleanly on 0bedb3d6. > > > > > > This patch set provides a method to request ingress scheduling on > > interfaces. > > > It also provides an implemtation of same for DPDK physical ports. > > > > > > This allows specific packet types to be: > > > * forwarded to their destination port ahead of other packets. > > > and/or > > > * be less likely to be dropped in an overloaded situation. > > > > > > It was previously discussed > > > https://mail.openvswitch.org/pipermail/ovs-discuss/2017- > > May/044395.htm > > > l > > > and RFC'd > > > https://mail.openvswitch.org/pipermail/ovs-dev/2017-July/335237.html > > > > > > Limitations of this patch: > > > * The patch uses the Flow Director filter API in DPDK and has only > > > been tested on Fortville (XL710) NIC. > > > * Prioritization is limited to: > > > ** eth_type > > > ** Fully specified 5-tuple src & dst ip and port numbers for UDP & > > > TCP packets > > > * ovs-appctl dpif-netdev/pmd-*-show o/p should indicate rxq > > prioritization. > > > * any requirements for a more granular prioritization mechanism > > > > > > > In general I like the idea of splitting priority traffic to a specific > > queue but I have concerns about the implementation. I shared most of > > these when we met already but adding here too. Not a detailed review. > [[BO'M]] No worries. If we get the high-level sorted out first the details > will > fall into place :) > > > > - It is using deprecated DPDK filter API. > > http://dpdk.org/doc/guides/rel_notes/deprecation.html > [[BO'M]] Yes it looks like a move to the shiny new Flow API is in order. > > > > - It is an invasive change that seems to be for only one Intel NIC in > > the DPDK datapath. Even then it is very limited as it only works when > > that Intel NIC is using exactly 2 rx queues. > [[BO'M]] That's the current case but is really a limitation of > FlowDirectorAPI/DPDK/XL710 combination. Maybe Flow API will allow to > RSS over many queues and place the prioritized traffic on another queue. > > > > - It's a hardcoded opaque QoS which will have a negative impact on > > whichever queues happen to land on the same pmd so it's unpredictable > > which queues will be affected. It could effect other latency sensitive > > traffic that cannot by prioritized because of the limitations above. > > > > - I guess multiple priority queues could land on the same pmd and > > starve each other? > [[BO'M]] Interaction with pmd assignment is definitely an issue that needs > to be addressed. I know there is work in-flight in that regard so it will be > easier to address that when the in-flight work lands. > > > > I think a more general, less restricted scheme using DPDK rte_flow API > > with controls on the effects to other traffic is needed. Perhaps if a > > user is very concerned with latency on traffic from a port, they would > > be ok with dedicating a pmd to it. > [[BO'M]] You are proposing to prioritize queues by allocating a single pmd > to them rather than by changing the pmds read algorithm to favor > prioritized queues? For sure that could be another implementation of the > solution. > > If we look at the patch set as containing two distinct things as per the cover > letter "the patch set provides a method to request ingress scheduling on > interfaces. It also provides an implementation of same for DPDK physical > ports." Then this would change the second part put the first would be still > valid. Each port type in any case would have to come up with it's own > implementation - it's just for non-physical ports than cannot offload the > prioritization decision it not worth the effort - as was noted in an earlier > RFC. > > > > > thanks, > > Kevin. > > > > > Initial results: > > > * even when userspace OVS is very much overloaded and > > > dropping significant numbers of packets the drop rate for prioritized > traffic > > > is running at 1/1000th of the drop rate for non-prioritized traffic. > > > > > > * the latency profile of prioritized traffic through userspace OVS > > > is also > > much > > > improved > > > > > > 1e0 |* > > > |* > > > 1e-1 |* | Non-prioritized pkt latency > > > |* * Prioritized pkt latency > > > 1e-2 |* > > > |* > > > 1e-3 |* | > > > |* | > > > 1e-4 |* | | | > > > |* |* | | > > > 1e-5 |* |* | | | > > > |* |* |* | | | > > > 1e-6 |* |* |* |* | | > > > |* |* |* |* |* | > > > 1e-7 |* |* |* |* |* |* > > > |* |* |* |* |* |* > > > 1e-8 |* |* |* |* |* |* > > > 0-1 1-20 20-40 40-50 50-60 60-70 ... 120-400 > > > Latency (us) > > > > > > Proportion of packets per latency bin @ 80% Max Throughput > > > (Log scale) > > > > > > > > > Regards, > > > Billy. > > > > > > billy O'Mahony (4): > > > netdev: Add set_ingress_sched to netdev api > > > netdev-dpdk: Apply ingress_sched config to dpdk phy ports > > > dpif-netdev: Add rxq prioritization > > > docs: Document ingress scheduling feature > > > > > > Documentation/howto/dpdk.rst | 31 +++++++ > > > include/openvswitch/ofp-parse.h | 3 + > > > lib/dpif-netdev.c | 25 ++++-- > > > lib/netdev-bsd.c | 1 + > > > lib/netdev-dpdk.c | 192 > > +++++++++++++++++++++++++++++++++++++++- > > > lib/netdev-dummy.c | 1 + > > > lib/netdev-linux.c | 1 + > > > lib/netdev-provider.h | 10 +++ > > > lib/netdev-vport.c | 1 + > > > lib/netdev.c | 22 +++++ > > > lib/netdev.h | 1 + > > > vswitchd/bridge.c | 4 + > > > vswitchd/vswitch.xml | 31 +++++++ > > > 13 files changed, 315 insertions(+), 8 deletions(-) > > > > > _______________________________________________ > dev mailing list > d...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
