On Thu, Oct 18, 2018 at 09:43:11PM +0530, Sriharsha Basavapatna via dev wrote: > With the current OVS offload design, when an offload-device fails to add a > flow rule and returns an error, OVS adds the rule to the kernel datapath. > The flow gets processed by the kernel datapath for the entire life of that > flow. This is fine when an error is returned by the device due to lack of > support for certain keys or actions. > > But when an error is returned due to temporary conditions such as lack of > resources to add a flow rule, the flow continues to be processed by kernel > even when resources become available later. That is, those flows never get > offloaded again. This problem becomes more pronounced when a flow that has > been initially offloaded may have a smaller packet rate than a later flow > that could not be offloaded due to lack of resources. This leads to > inefficient use of HW resources and wastage of host CPU cycles. > > This patch-set addresses this issue by providing a way to detect temporary > offload resource constraints (Out-Of-Resource or OOR condition) and to > selectively and dynamically offload flows with a higher packets-per-second > (pps) rate. This dynamic rebalancing is done periodically on netdevs that > are in OOR state until resources become available to offload all pending > flows. > > The patch-set involves the following changes at a high level: > > 1. Detection of Out-Of-Resources (OOR) condition on an offload-capable > netdev. > 2. Gathering flow offload selection criteria for all flows on an OOR netdev; > i.e, packets-per-second (pps) rate of flows for offloaded and > non-offloaded (pending) flows. > 3. Dynamically replacing offloaded flows with a lower pps-rate, with > non-offloaded flows with a higher pps-rate, on an OOR netdev. A new > OpenvSwitch configuration option - "offload-rebalance" to enable > this policy. > > Cost/benefits data points: > > 1. Rough cost of the new rebalancing, in terms of CPU time: > > Ran a test that replaced 256 low pps-rate flows(pings) with 256 high > pps-rate flows(iperf), in a system with 4 cpus (Intel Xeon E5 @ 2.40GHz; > 2 cores with hw threads enabled, rest disabled). The data showed that cpu > utilization increased by about ~20%. This increase occurs during the > specific second in which rebalancing is done. And subsequently (from the > next second), cpu utilization decreases significantly due to offloading > of higher pps-rate flows. So effectively there's a bump in cpu utilization > at the time of rebalancing, that is more than compensated by reduced cpu > utilization once the right flows get offloaded. > > 2. Rough benefits to the user in terms of offload performance: > > The benefits to the user is reduced cpu utilization in the host, since > higher pps-rate flows get offloaded, replacing lower pps-rate flows. > Replacing a single offloaded flood ping flow with an iperf flow (multiple > connections), shows that the cpu %used that was originally 100% on a > single cpu (rebalancing disabled) goes down to 35% (rebalancing enabled). > That is, cpu utilization decreased 65% after rebalancing. > > 3. Circumstances under which the benefits would show up: > > The rebalancing benefits would show up once offload resources are > exhausted and new flows with higher pps-rate are initiated, that would > otherwise be handled by kernel datapath costing host cpu cycles. > > This can be observed using 'ovs appctl dpctl/ dump-flows' command. Prior > to rebalancing, any high pps-rate flows that couldn't be offloaded due to > resource crunch would show up in the output of 'dump-flows type=ovs' and > after rebalancing such flows would appear in the output of > 'dump-flows type=offloaded'.
Thanks, applied to master. _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
