I went ahead and put the patched kernel on both systems. I was getting
mixed results - in one direction, results in the 8Gbit/s range, in the
other in the 7 Gbit/s. I noticed that interrupts were going to
different CPUs so I started playing with IRQ assignments, and bound all
interrupts of the 82599ES to CPU0 to remove that variable. At that
point I started getting 8.X Gbit/s consistently in either direction.
root@qu-stbaz1-perf0000:~# HDR="-P 1"; for i in `seq 1 5`; do netperf -H
192.168.0.22 -l 30 $HDR -c -C -- -O
throughput,local_cpu_util,local_sd,local_cpu_peak_util,local_cpu_peak_id,remote_cpu_util,remote_sd,remote_cpu_peak_util,remote_cpu_peak_id;
HDR="-P 0"; done
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.0.22 () port 0 AF_INET : demo
Throughput Local Local Local Local Remote Remote Remote Remote
CPU Service Peak Peak CPU Service Peak Peak
Util Demand Per CPU Per CPU Util Demand Per CPU Per CPU
% Util % ID % Util % ID
8768.48 1.95 0.582 62.22 0 4.36 1.304 99.97 0
8757.99 1.95 0.583 62.27 0 4.37 1.307 100.00 0
8793.86 2.01 0.600 64.32 0 4.23 1.262 100.00 0
8720.98 1.93 0.580 61.67 0 4.45 1.337 99.97 0
8380.49 1.84 0.575 58.74 0 4.39 1.374 100.00 0
root@qu-stbaz1-perf0001:~# HDR="-P 1"; for i in `seq 1 5`; do netperf -H
192.168.0.21 -l 30 $HDR -c -C -- -O
throughput,local_cpu_util,local_sd,local_cpu_peak_util,local_cpu_peak_id,remote_cpu_util,remote_sd,remote_cpu_peak_util,remote_cpu_peak_id;
HDR="-P 0"; done
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.0.21 () port 0 AF_INET : demo
Throughput Local Local Local Local Remote Remote Remote Remote
CPU Service Peak Peak CPU Service Peak Peak
Util Demand Per CPU Per CPU Util Demand Per CPU Per CPU
% Util % ID % Util % ID
8365.16 1.93 0.604 61.64 0 4.57 1.431 99.97 0
8724.08 2.01 0.604 64.31 0 4.66 1.401 100.00 0
8653.70 1.98 0.600 63.37 0 4.67 1.414 99.90 0
8748.05 1.99 0.596 63.62 0 4.62 1.383 99.97 0
8756.66 1.99 0.595 63.55 0 4.52 1.354 99.97 0
If I switch the interrupts to a core on the other socket, throughput
drops to 7.5 Gbit/s or so either way.
I'm still trying to get onto the consoles to check power management
settings. Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz are the processors in
use.
happy benchmarking,
rick
PS FWIW, if I shift from using just the linux native vxlan to a "mostly
full" set of OpenStack compute node plumbing - two OVS bridges and a
linux bridge and associated plumbing with a vxlan tunnel defined in OVS,
but nothing above the Linux bridge (and no VMs) I see more like 4.9
Gbit/s. The veth pair connecting the linux bridge to the top ovs bridge
show rx checksum and gro enabled. the linux bridge itself shows GRO but
rx checksum off (fixed). I'm not sure how to go about checking the OVS
constructs.
root@qu-stbaz1-perf0000:/home/raj# cat raj_full_stack.sh
ovs-vsctl add-br br-tun
ovs-vsctl add-port br-tun vxlan0 -- set interface vxlan0 type=vxlan
options:remote_ip=$1 options:key=99 options:dst_port=4789
ovs-vsctl add-port br-tun patch-tun -- set interface patch-tun
type=patch options:peer=patch-int
ovs-vsctl add-br br-int
ovs-vsctl add-port br-int patch-int -- set interface patch-int
type=patch options:peer=patch-tun
brctl addbr qbr
ip link add dev qvb type veth peer name qvo
brctl addif qbr qvb
ovs-vsctl add-port br-int qvo
ifconfig qbr $2
ifconfig qbr mtu 1450
ifconfig qvb up
ifconfig qvo up
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
