On Thu, Aug 25, 2016 at 1:32 PM, Rick Jones <rick.jon...@hpe.com> wrote: > On 08/25/2016 12:49 PM, Eric Dumazet wrote: >> >> On Thu, 2016-08-25 at 12:23 -0700, Alexander Duyck wrote: >>> >>> A simpler approach is provided with this patch. With it we disable XPS >>> any >>> time a socket is not present for a given flow. By doing this we can >>> avoid >>> using XPS for any routing or bridging situations in which XPS is likely >>> more of a hinderance than a help. >> >> >> Yes, but this will destroy isolation for people properly doing VM cpu >> pining. > > > Why not simply stop enabling XPS by default. Treat it like RPS and RFS > (unless I've missed a patch...). The people who are already doing the extra > steps to pin VMs can enable XPS in that case. It isn't clear that one > should always pin VMs - for example if a (public) cloud needed to > oversubscribe the cores. > > happy benchmarking, > > rick jones
The big motivation for most of the drivers to have it is because XPS can provide very good savings when you end up aligning the entire transmit path by also controlling the memory allocations and IRQ affinity. What most of these devices try to do by default is isolate all of a given flow to a single CPU so you can get as close to linear scaling as possible. The problem is with XPS disabled by default you take a pretty serious performance hit for all of the non-virtualization cases since you can easily end up crossing CPUs or worse yet NUMA nodes when processing Tx traffic. Really if anything I still think we need to do something to enforce ordering of flows in regards to things that make use of netif_rx instead of using a NAPI processing path. From what I can tell we will always end up running into reordering issues as long as you can have a single thread bouncing between CPUs and spraying packets on to the various backlogs. It is one of the reasons why I had mentioned possibly enabling RPS for these type of interfaces as without it you will still get some reordering, it just varies in the degree as to how much. Essentially all that is happening when you disable XPS is that you shorten the queues, but there are still multiple queues in play. You still can end up with packets getting reordered between the various per_cpu backlogs. - Alex
