On 02/04/2016 11:38 AM, Tom Herbert wrote:
On Thu, Feb 4, 2016 at 11:13 AM, Rick Jones <rick.jon...@hpe.com> wrote:
The Intel folks suggested something about the process scheduler moving the
sender around and ultimately causing some packet re-ordering. That could I
suppose explain the TCP_STREAM difference, but not the TCP_RR since that has
just a single segment in flight at one time.
XPS has OOO avoidance for TCP, that should not be a problem.
What/how much should I read into:
With XPS TCPOFOQueue: 78206
Without XPS TCPOFOQueue: 967
out of the netstat statistics on the receiving VM?
I can try to get perf/whatnot installed on the systems - suggestions as to
what metrics to look at are welcome.
I'd start with verifying the XPS configuration is sane and then trying
to reproduce the issue outside of using VMs, if both of those are okay
then maybe look at some sort of bad interaction with OpenStack
configuration.
Fair enough - what is the definition of "sane" for an XPS configuration?
Here is what it looks like before I disabled it:
$ for i in `find
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0 -name
xps_cpus`; do echo $i `cat $i`; done
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-0/xps_cpus
0000,00000001
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-1/xps_cpus
0000,00000002
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-2/xps_cpus
0000,00000004
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-3/xps_cpus
0000,00000008
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-4/xps_cpus
0000,00000010
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-5/xps_cpus
0000,00000020
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-6/xps_cpus
0000,00000040
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-7/xps_cpus
0000,00000080
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-8/xps_cpus
0000,00000100
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-9/xps_cpus
0000,00000200
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-10/xps_cpus
0000,00000400
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-11/xps_cpus
0000,00000800
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-12/xps_cpus
0000,00001000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-13/xps_cpus
0000,00002000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-14/xps_cpus
0000,00004000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-15/xps_cpus
0000,00008000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-16/xps_cpus
0000,00010000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-17/xps_cpus
0000,00020000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-18/xps_cpus
0000,00040000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-19/xps_cpus
0000,00080000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-20/xps_cpus
0000,00100000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-21/xps_cpus
0000,00200000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-22/xps_cpus
0000,00400000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-23/xps_cpus
0000,00800000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-24/xps_cpus
0000,01000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-25/xps_cpus
0000,02000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-26/xps_cpus
0000,04000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-27/xps_cpus
0000,08000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-28/xps_cpus
0000,10000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-29/xps_cpus
0000,20000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-30/xps_cpus
0000,40000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-31/xps_cpus
0000,80000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-32/xps_cpus
0001,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-33/xps_cpus
0002,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-34/xps_cpus
0004,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-35/xps_cpus
0008,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-36/xps_cpus
0010,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-37/xps_cpus
0020,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-38/xps_cpus
0040,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-39/xps_cpus
0080,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-40/xps_cpus
0100,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-41/xps_cpus
0200,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-42/xps_cpus
0400,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-43/xps_cpus
0800,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-44/xps_cpus
1000,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-45/xps_cpus
2000,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-46/xps_cpus
4000,00000000
/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.0/net/eth0/queues/tx-47/xps_cpus
8000,00000000
Which looks like it simply got spread across all the "CPUs" (HTs
included) in the system.
happy benchmarking,
rick jones
