Re: [ovs-discuss] prioritizing latency-sensitive traffic
This does sound like a reasonable approach to me. On Mon, May 08, 2017 at 10:42:07AM +, O Mahony, Billy wrote: > Hi All, > > Running with the other_config suggestion from Ben & Darrell far below, here > is some more details on what an implementation of this could look like. I'll > start coding an RFC patch shortly. > > Comments, criticisms & improvements welcome. > > Regards, > Billy. > > RFC > > Proposed Feature: Interface ingress scheduling > > Purpose: > To allow certain packets ingressing an Interface to be prioritized. Which > means such packets are: > a) forwarded to their destination port before non-prioritized packets > and/or > b) are less likely to be dropped in an overloaded situation than prioritized > packets. > > Details: > Ingress scheduling is supported with the best effort of the Interface. It may > be dependant on the interface type and it's supporting implementation > devices. So different interface types may have different levels of support > for the feature and the same interface type attached to different devices > (e.g physical NICs or vhost ports, the device driver used, the NIC model) may > also have different levels of support. > > It's unlikely that any implementation will allow prioritization on an > arbitrary combinarion of ovs-fields (7) - check the Interface type's > documentation for details. > > If for any reason the prioritizion request cannot be fully honored a warning > is logged. > > The general form of control of the feature is > >ovs-vsctl set Interface other_config:prioritize=<ovs-fields(7) > match specifier> > > Example: > > To give priority to packets with an IP DSCP field value of 1: > > ovs-vsctl set Interface dpdk0 other_config:prioritize=ip_dscp=1 > > > > * Intially only support one ingress priority match. Multiple priorities could > be specified later. > * The prioritization algorithm (e.g strict priority queueing and so on) is > currently unspecified and implementation dependent. > * This proposal's interaction with bonding is to be defined. > > * The subset of match fields supported is dependent on the interface type and > it's underlying implementation devices (NICs, vhost ports etc). > * If for any reason the prioritize request cannot be honored a warning is > logged > * the configuration stays in the Interface table it is not rolled back > (this is normal once schema constraints are met the entry stays in the db and > db watchers (such as ovs-vswitchd are triggered)) > * possible failure reaons (for netdev-dpdk Interfaces) > * the prioritize fields are not supported > * NIC rx multi-queue is not enabled (ie n_rxq < 2) - implementation on > dpdk ports will be via h/w filtering (supported by dpdk interface) but which > requires multiqueue to be enabled. It would be too surprising to change the > n_rxq setting implicitly. > * pmd-rxq-affinity is set. A simple & sensible interaction between these > two features is not simple to define. For now using the existing > pmd-rqx-affinity feature precludes other_config:prioritize. > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Monday, April 24, 2017 4:14 PM > > To: O Mahony, Billy <billy.o.mah...@intel.com> > > Cc: ovs-discuss@openvswitch.org; Darrell Ball <db...@vmware.com>; Kevin > > Traynor <ktray...@redhat.com> > > Subject: Re: [ovs-discuss] prioritizing latency-sensitive traffic > > > > If that's useful, then it makes sense to me and I'd have no objection. > > > > On Mon, Apr 24, 2017 at 10:06:17AM +, O Mahony, Billy wrote: > > > Hi Ben, Darrell, > > > > > > I've done some PoC work on this kind of traffic prioritization. However > > > with > > OVS-DPDK's run-to-completion model the issue I find the same issue as you > > outlined - by the time the priority of the packet has been determined most > > of the effort to process the packet has already been spent. > > > > > > So, I relied on using hardware offload i.e. FlowDirector on the NIC to > > analyse and allocate packets to high and low priority queues and then > > modifying the PMD (dpif_netdev_run ) to read preferentially from the high > > priority queue. The results were good for overload situations - packets on > > the priority queue not dropped. In terms of latency there was an > > improvement but there was still a long tail to the latency profile. i..e The > > latency profile moved down but not left. > > > > > > As Darrell pointed out for egress scheduling, perhaps this kind of in
Re: [ovs-discuss] prioritizing latency-sensitive traffic
If that's useful, then it makes sense to me and I'd have no objection. On Mon, Apr 24, 2017 at 10:06:17AM +, O Mahony, Billy wrote: > Hi Ben, Darrell, > > I've done some PoC work on this kind of traffic prioritization. However with > OVS-DPDK's run-to-completion model the issue I find the same issue as you > outlined - by the time the priority of the packet has been determined most of > the effort to process the packet has already been spent. > > So, I relied on using hardware offload i.e. FlowDirector on the NIC to > analyse and allocate packets to high and low priority queues and then > modifying the PMD (dpif_netdev_run ) to read preferentially from the high > priority queue. The results were good for overload situations - packets on > the priority queue not dropped. In terms of latency there was an improvement > but there was still a long tail to the latency profile. i..e The latency > profile moved down but not left. > > As Darrell pointed out for egress scheduling, perhaps this kind of ingress > prioritization can be encapsulated as an optional property of a port? > > If the port can implement the prioritization (either by offloading to > hardware or in software) it can accept the property being set; If not it can > return ENOTSUPP? > > There is currently HWOL use-cases being gathered for OVS-DPDK: > https://docs.google.com/document/d/1A8adu1xzg53bzcFKINhffYyC0nCQjqwqnI1jAFx2sck/edit?usp=sharing > + Kevin who is co-ordinating that. > > Thanks, > Billy. > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Friday, April 21, 2017 10:39 PM > > To: O Mahony, Billy <billy.o.mah...@intel.com> > > Cc: ovs-discuss@openvswitch.org; Darrell Ball <db...@vmware.com> > > Subject: Re: [ovs-discuss] prioritizing latency-sensitive traffic > > > > Thanks for letting us know. I'm happy to continue the conversation if there > > are interesting ideas; it's a frustrating situation, frankly, and I'd love > > to hear > > creative approaches. > > > > On Tue, Apr 18, 2017 at 10:01:28AM +, O Mahony, Billy wrote: > > > Hi Ben, Darrell, > > > > > > It sounds like the general feeling is that any kind of tc pre-processing > > > is not > > worth it and the existing egress queing/QoS facilities should suffice. > > > > > > Thanks for your comments. > > > > > > /Billy > > > > > > > > > > > > > -Original Message- > > > > From: Ben Pfaff [mailto:b...@ovn.org] > > > > Sent: Thursday, April 13, 2017 7:47 PM > > > > To: O Mahony, Billy <billy.o.mah...@intel.com> > > > > Cc: ovs-discuss@openvswitch.org > > > > Subject: Re: [ovs-discuss] prioritizing latency-sensitive traffic > > > > > > > > I don't know how much more OVS can contribute to this than it already > > does. > > > > By the time that OVS has classified a packet to the extent that is > > > > necessary to determine that it should be handled with a high > > > > priority, OVS has already done most of the work that it does on a > > > > packet. > > > [[BO'M]] I'm investigating how I could go about classifying packets > > > before the main The work to transmit the > > > > packet is not part of OVS's job, it is the job of the driver, and at > > > > most OVS can mark the packet with a priority or a queue. OVS can > > > > already do that. So the usual answer is that it's a matter of > > > > configuring QoS in the driver to do what the user wants. > > > > > > > > On Mon, Apr 10, 2017 at 09:30:12AM +, O Mahony, Billy wrote: > > > > > Hi Everybody, > > > > > > > > > > I just wanted to reflag this discussion below about possible > > > > > methods of > > > > how to prioritize certain types of traffic handled by OVS. > > > > > > > > > > By prioritize I mean either or both of: > > > > > a) 'priority' packets are sent to their destination port faster > > > > > than other packets > > > > > b) in an overloaded situation the switch drops non-prioritized > > > > > packets > > > > rather than prioritized packets. > > > > > > > > > > Also just to be clear I am discussing kernel ovs here. Also I'm > > > > > looking at > > > > doing this without writing new code - ie is it possible and if so > > > > how is it configured using existing OVS. > > >
Re: [ovs-discuss] prioritizing latency-sensitive traffic
Hi Ben, Darrell, I've done some PoC work on this kind of traffic prioritization. However with OVS-DPDK's run-to-completion model the issue I find the same issue as you outlined - by the time the priority of the packet has been determined most of the effort to process the packet has already been spent. So, I relied on using hardware offload i.e. FlowDirector on the NIC to analyse and allocate packets to high and low priority queues and then modifying the PMD (dpif_netdev_run ) to read preferentially from the high priority queue. The results were good for overload situations - packets on the priority queue not dropped. In terms of latency there was an improvement but there was still a long tail to the latency profile. i..e The latency profile moved down but not left. As Darrell pointed out for egress scheduling, perhaps this kind of ingress prioritization can be encapsulated as an optional property of a port? If the port can implement the prioritization (either by offloading to hardware or in software) it can accept the property being set; If not it can return ENOTSUPP? There is currently HWOL use-cases being gathered for OVS-DPDK: https://docs.google.com/document/d/1A8adu1xzg53bzcFKINhffYyC0nCQjqwqnI1jAFx2sck/edit?usp=sharing + Kevin who is co-ordinating that. Thanks, Billy. > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Friday, April 21, 2017 10:39 PM > To: O Mahony, Billy <billy.o.mah...@intel.com> > Cc: ovs-discuss@openvswitch.org; Darrell Ball <db...@vmware.com> > Subject: Re: [ovs-discuss] prioritizing latency-sensitive traffic > > Thanks for letting us know. I'm happy to continue the conversation if there > are interesting ideas; it's a frustrating situation, frankly, and I'd love to > hear > creative approaches. > > On Tue, Apr 18, 2017 at 10:01:28AM +, O Mahony, Billy wrote: > > Hi Ben, Darrell, > > > > It sounds like the general feeling is that any kind of tc pre-processing is > > not > worth it and the existing egress queing/QoS facilities should suffice. > > > > Thanks for your comments. > > > > /Billy > > > > > > > > > -Original Message- > > > From: Ben Pfaff [mailto:b...@ovn.org] > > > Sent: Thursday, April 13, 2017 7:47 PM > > > To: O Mahony, Billy <billy.o.mah...@intel.com> > > > Cc: ovs-discuss@openvswitch.org > > > Subject: Re: [ovs-discuss] prioritizing latency-sensitive traffic > > > > > > I don't know how much more OVS can contribute to this than it already > does. > > > By the time that OVS has classified a packet to the extent that is > > > necessary to determine that it should be handled with a high > > > priority, OVS has already done most of the work that it does on a packet. > > [[BO'M]] I'm investigating how I could go about classifying packets > > before the main The work to transmit the > > > packet is not part of OVS's job, it is the job of the driver, and at > > > most OVS can mark the packet with a priority or a queue. OVS can > > > already do that. So the usual answer is that it's a matter of > > > configuring QoS in the driver to do what the user wants. > > > > > > On Mon, Apr 10, 2017 at 09:30:12AM +, O Mahony, Billy wrote: > > > > Hi Everybody, > > > > > > > > I just wanted to reflag this discussion below about possible > > > > methods of > > > how to prioritize certain types of traffic handled by OVS. > > > > > > > > By prioritize I mean either or both of: > > > > a) 'priority' packets are sent to their destination port faster > > > > than other packets > > > > b) in an overloaded situation the switch drops non-prioritized > > > > packets > > > rather than prioritized packets. > > > > > > > > Also just to be clear I am discussing kernel ovs here. Also I'm > > > > looking at > > > doing this without writing new code - ie is it possible and if so > > > how is it configured using existing OVS. > > > > > > > > Thanks again, > > > > Billy. > > > > > > > > > -Original Message- > > > > > From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > > > > > boun...@openvswitch.org] On Behalf Of O Mahony, Billy > > > > > Sent: Friday, November 25, 2016 5:04 PM > > > > > To: ovs-discuss@openvswitch.org > > > > > Subject: [ovs-discuss] prioritizing latency-sensitive traffic > > > > > > > > > > Hi, > > > > > > > > >
Re: [ovs-discuss] prioritizing latency-sensitive traffic
Thanks for letting us know. I'm happy to continue the conversation if there are interesting ideas; it's a frustrating situation, frankly, and I'd love to hear creative approaches. On Tue, Apr 18, 2017 at 10:01:28AM +, O Mahony, Billy wrote: > Hi Ben, Darrell, > > It sounds like the general feeling is that any kind of tc pre-processing is > not worth it and the existing egress queing/QoS facilities should suffice. > > Thanks for your comments. > > /Billy > > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Thursday, April 13, 2017 7:47 PM > > To: O Mahony, Billy <billy.o.mah...@intel.com> > > Cc: ovs-discuss@openvswitch.org > > Subject: Re: [ovs-discuss] prioritizing latency-sensitive traffic > > > > I don't know how much more OVS can contribute to this than it already does. > > By the time that OVS has classified a packet to the extent that is > > necessary to > > determine that it should be handled with a high priority, OVS has already > > done most of the work that it does on a packet. > [[BO'M]] I'm investigating how I could go about classifying packets before > the main > The work to transmit the > > packet is not part of OVS's job, it is the job of the driver, and at most > > OVS can > > mark the packet with a priority or a queue. OVS can already do that. So > > the > > usual answer is that it's a matter of configuring QoS in the driver to do > > what > > the user wants. > > > > On Mon, Apr 10, 2017 at 09:30:12AM +, O Mahony, Billy wrote: > > > Hi Everybody, > > > > > > I just wanted to reflag this discussion below about possible methods of > > how to prioritize certain types of traffic handled by OVS. > > > > > > By prioritize I mean either or both of: > > > a) 'priority' packets are sent to their destination port faster than > > > other packets > > > b) in an overloaded situation the switch drops non-prioritized packets > > rather than prioritized packets. > > > > > > Also just to be clear I am discussing kernel ovs here. Also I'm looking at > > doing this without writing new code - ie is it possible and if so how is it > > configured using existing OVS. > > > > > > Thanks again, > > > Billy. > > > > > > > -Original Message- > > > > From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > > > > boun...@openvswitch.org] On Behalf Of O Mahony, Billy > > > > Sent: Friday, November 25, 2016 5:04 PM > > > > To: ovs-discuss@openvswitch.org > > > > Subject: [ovs-discuss] prioritizing latency-sensitive traffic > > > > > > > > Hi, > > > > > > > > I have been performing tests investigating latency profiles of > > > > low-bandwidth time-sensitive traffic when the system is busy with > > 'normal' traffic. > > > > Unsurprisingly the latency-sensitive traffic is affected by the > > > > normal traffic and has basically the same latency profile as the normal > > traffic. > > > > > > > > I would like to be able to perform prioritization of traffic as some > > > > protocols such as PTP would benefit greatly from having it's packets > > > > 'jump > > the queue'. > > > > > > > > From skimming the documentation it looks that ingress QoS offers > > > > only policing (rate-limiting). Is this actually the case or maybe > > > > I'm not looking in the right place? > > > > > > > > But if so, I am looking at some alternatives: > > > > > > > > a) create two separate egress ports and have PTP listen on one port, > > > > everything else listen on the other port and use normal forwarding > > > > rules to send PTP traffic incoming from eth0 to it's own port. Something > > like: > > > > > > > > other apps ptp_daemon > > > > + + > > > > + + > > > >if_norm if_ptp > > > >++ > > > >|| > > > >|| > > > > ++++ > > > > | | > > > > |ovs | > > > > | | > > > > +-++ > > > > | > > > > + > > > > eth0 > > > > > > > > b) create prioritized queu
Re: [ovs-discuss] prioritizing latency-sensitive traffic
Hi Ben, Darrell, It sounds like the general feeling is that any kind of tc pre-processing is not worth it and the existing egress queing/QoS facilities should suffice. Thanks for your comments. /Billy > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Thursday, April 13, 2017 7:47 PM > To: O Mahony, Billy <billy.o.mah...@intel.com> > Cc: ovs-discuss@openvswitch.org > Subject: Re: [ovs-discuss] prioritizing latency-sensitive traffic > > I don't know how much more OVS can contribute to this than it already does. > By the time that OVS has classified a packet to the extent that is necessary > to > determine that it should be handled with a high priority, OVS has already > done most of the work that it does on a packet. [[BO'M]] I'm investigating how I could go about classifying packets before the main The work to transmit the > packet is not part of OVS's job, it is the job of the driver, and at most OVS > can > mark the packet with a priority or a queue. OVS can already do that. So the > usual answer is that it's a matter of configuring QoS in the driver to do what > the user wants. > > On Mon, Apr 10, 2017 at 09:30:12AM +, O Mahony, Billy wrote: > > Hi Everybody, > > > > I just wanted to reflag this discussion below about possible methods of > how to prioritize certain types of traffic handled by OVS. > > > > By prioritize I mean either or both of: > > a) 'priority' packets are sent to their destination port faster than > > other packets > > b) in an overloaded situation the switch drops non-prioritized packets > rather than prioritized packets. > > > > Also just to be clear I am discussing kernel ovs here. Also I'm looking at > doing this without writing new code - ie is it possible and if so how is it > configured using existing OVS. > > > > Thanks again, > > Billy. > > > > > -Original Message- > > > From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > > > boun...@openvswitch.org] On Behalf Of O Mahony, Billy > > > Sent: Friday, November 25, 2016 5:04 PM > > > To: ovs-discuss@openvswitch.org > > > Subject: [ovs-discuss] prioritizing latency-sensitive traffic > > > > > > Hi, > > > > > > I have been performing tests investigating latency profiles of > > > low-bandwidth time-sensitive traffic when the system is busy with > 'normal' traffic. > > > Unsurprisingly the latency-sensitive traffic is affected by the > > > normal traffic and has basically the same latency profile as the normal > traffic. > > > > > > I would like to be able to perform prioritization of traffic as some > > > protocols such as PTP would benefit greatly from having it's packets 'jump > the queue'. > > > > > > From skimming the documentation it looks that ingress QoS offers > > > only policing (rate-limiting). Is this actually the case or maybe > > > I'm not looking in the right place? > > > > > > But if so, I am looking at some alternatives: > > > > > > a) create two separate egress ports and have PTP listen on one port, > > > everything else listen on the other port and use normal forwarding > > > rules to send PTP traffic incoming from eth0 to it's own port. Something > like: > > > > > > other apps ptp_daemon > > > + + > > > + + > > >if_norm if_ptp > > >++ > > >|| > > >|| > > > ++++ > > > | | > > > |ovs | > > > | | > > > +-++ > > > | > > > + > > > eth0 > > > > > > b) create prioritized queues on a port and use match and actions > > > such as > > > set_queue(queue) and enqueue(port, queue) on ingress traffic to > > > forward the PTP traffic to the higher priority queue. However I > > > think queue priority for this case only relates to which queue get > > > to consume the bandwidth of the port first and not about changing > > > the order in which the packets egress the port. > > > > > > c) Or perhaps I can re-use tc PRIO or CBQ qdiscs by passing all > > > traffic to tc first before ovs? > > > > > > other apps > > > | > > > | > > >if_norm > > >+ > > >| > > >|
Re: [ovs-discuss] prioritizing latency-sensitive traffic
Hi Everybody, I just wanted to reflag this discussion below about possible methods of how to prioritize certain types of traffic handled by OVS. By prioritize I mean either or both of: a) 'priority' packets are sent to their destination port faster than other packets b) in an overloaded situation the switch drops non-prioritized packets rather than prioritized packets. Also just to be clear I am discussing kernel ovs here. Also I'm looking at doing this without writing new code - ie is it possible and if so how is it configured using existing OVS. Thanks again, Billy. > -Original Message- > From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > boun...@openvswitch.org] On Behalf Of O Mahony, Billy > Sent: Friday, November 25, 2016 5:04 PM > To: ovs-discuss@openvswitch.org > Subject: [ovs-discuss] prioritizing latency-sensitive traffic > > Hi, > > I have been performing tests investigating latency profiles of low-bandwidth > time-sensitive traffic when the system is busy with 'normal' traffic. > Unsurprisingly the latency-sensitive traffic is affected by the normal traffic > and has basically the same latency profile as the normal traffic. > > I would like to be able to perform prioritization of traffic as some protocols > such as PTP would benefit greatly from having it's packets 'jump the queue'. > > From skimming the documentation it looks that ingress QoS offers only > policing (rate-limiting). Is this actually the case or maybe I'm not looking > in the > right place? > > But if so, I am looking at some alternatives: > > a) create two separate egress ports and have PTP listen on one port, > everything else listen on the other port and use normal forwarding rules to > send PTP traffic incoming from eth0 to it's own port. Something like: > > other apps ptp_daemon > + + > + + >if_norm if_ptp >++ >|| >|| > ++++ > | | > |ovs | > | | > +-++ > | > + > eth0 > > b) create prioritized queues on a port and use match and actions such as > set_queue(queue) and enqueue(port, queue) on ingress traffic to forward > the PTP traffic to the higher priority queue. However I think queue priority > for this case only relates to which queue get to consume the bandwidth of > the port first and not about changing the order in which the packets egress > the port. > > c) Or perhaps I can re-use tc PRIO or CBQ qdiscs by passing all traffic to tc > first > before ovs? > > other apps > | > | >if_norm >+ >| >| > +--+ > | | > |ovs | > | | > +-++ > | > | > tc - if_ptp ptp_daemon > + > eth0 > > Any thoughts, ideas or clarifications most welcome. > > Thanks, > Billy. > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
