On Thu, May 8, 2014 at 5:51 AM, Arno Toell <[email protected]> wrote: > Hello, > > as there have been a lot of discussions involving the OVS performance > compared to Linux bridges. I'm about to deploy OVS at large scale in our > cloud so that I decided to make some benchmarks on my own to argue about > OVS performance compared to what Linux delivers as is. > > I thought you could be possibly interested in these numbers, hence I'D > like to share them with you. > > > Lab setup: > > I've connected two Dell PowerEdge R320 (Intel Xeon E5-2407 0 @ 2.20GHz, > 32 GiB RAM) together. There is no switch in between, servers are > directly cross connected. The network server adapter is a two port Intel > X520-T2 card (2 x 10G SFP+). I'm using Debian Wheezy (kernel > 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3) as host system. For OVS I'm using > the Debian package as well (1.4.2+git20120612-9.1~deb7u1). I did not > bother with different kernel and OVS versions, as the tested version is > what is considerable for production, and the achieved performance seems > not substantially worse than vanilla bridges. > > > Network configuration: > > I did all tests on server 1 (tengig1). The peer was configured with > Linux LACP bonding (hashing mode layer3+4). This setup did not change > across experiments. I did not do bidirectional connection tests. Thus, > this server is mostly just for passive opening of incoming connections > for benchmarks. > > server 2: tengig2 (passive side) > > 10.10.10.2/24 @ bond0 > 10.10.100.2/24 @ bond0.100 (VLAN 100) > > server 1: tengig2 (active side) > > untagged traffic: 10.10.10.1/24 > tagged traffic: 10.10.100.1/24 > interfaces vary depending on the setup > > I did enable jumbo frames (MTU 9000) on all interfaces and verified, > packets are not fragmented. > > Benchmarking Methodology: > > I'm using nuttcp to generate constant bitrate UDP traffic. There is > nothing else going on on the link. > > I've increased the sending and receive window on both sides (as I did > tests with TCP as well): > > root@tengig1:~# echo 4194304 > /proc/sys/net/core/wmem_max > root@tengig1:~# echo 4194304 > /proc/sys/net/core/rmem_max > > > --------------------------------------------------------------------------------------- > BASELINE > --------------------------------------------------------------------------------------- > > Linux bonding driver, no bridge at all > > root@tengig1:~# cat /proc/net/bonding/bond0 > Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) > > Bonding Mode: IEEE 802.3ad Dynamic link aggregation > Transmit Hash Policy: layer3+4 (1) > MII Status: up > MII Polling Interval (ms): 0 > Up Delay (ms): 0 > Down Delay (ms): 0 > > 802.3ad info > LACP rate: slow > Min links: 0 > Aggregator selection policy (ad_select): stable > Active Aggregator Info: > Aggregator ID: 3 > Number of ports: 2 > Actor Key: 33 > Partner Key: 33 > Partner Mac Address: 90:e2:ba:69:bd:3c > > Slave Interface: eth2 > MII Status: up > Speed: 10000 Mbps > Duplex: full > Link Failure Count: 0 > Permanent HW addr: 90:e2:ba:69:b6:c8 > Aggregator ID: 3 > Slave queue ID: 0 > > Slave Interface: eth3 > MII Status: up > Speed: 10000 Mbps > Duplex: full > Link Failure Count: 0 > Permanent HW addr: 90:e2:ba:69:b6:c9 > Aggregator ID: 3 > Slave queue ID: 0 > > > root@tengig1:~# nuttcp -l8000 -u -w4m -R20G -i1 -N4 10.10.10.2 > 1476.5778 MB / 1.00 sec = 12386.2834 Mbps 0 / 193538 ~drop/pkt > 0.00 ~%loss > 1476.5625 MB / 1.00 sec = 12386.1678 Mbps 0 / 193536 ~drop/pkt > 0.00 ~%loss > 1476.3794 MB / 1.00 sec = 12384.9662 Mbps 0 / 193512 ~drop/pkt > 0.00 ~%loss > 1476.2115 MB / 1.00 sec = 12383.1743 Mbps 0 / 193490 ~drop/pkt > 0.00 ~%loss > 1476.4252 MB / 1.00 sec = 12385.3378 Mbps 0 / 193518 ~drop/pkt > 0.00 ~%loss > 1476.2344 MB / 1.00 sec = 12383.3662 Mbps 0 / 193493 ~drop/pkt > 0.00 ~%loss > 1476.3336 MB / 1.00 sec = 12384.5822 Mbps 0 / 193506 ~drop/pkt > 0.00 ~%loss > 1476.2497 MB / 1.00 sec = 12383.4942 Mbps 0 / 193495 ~drop/pkt > 0.00 ~%loss > 1476.1734 MB / 1.00 sec = 12383.2134 Mbps 0 / 193485 ~drop/pkt > 0.00 ~%loss > 1476.1581 MB / 1.00 sec = 12382.7263 Mbps 0 / 193483 ~drop/pkt > 0.00 ~%loss > 7.5760 MB / 1.00 sec = 63.5517 Mbps 0 / 993 ~drop/pkt > 0.00 ~%loss > > 14770.8817 MB / 10.00 sec = 12390.3865 Mbps 73 %TX 63 %RX 0 / 1936049 > drop/pkt 0.00 %loss > > > n.b.: The traffic is nicely split across both trunk links. Yet I do not > approach the wire speed. I did not investigate why, it might be either > related to driver settings or concurrency issues. However, this > experiment serves as baseline comparison for remaining experiments > > > --------------------------------------------------------------------------------------- > LINUX VLAN > --------------------------------------------------------------------------------------- > > Linux bonding driver, no bridge at all. However, use tagged frames. > > root@tengig1:~# ip link add link bond0 name bond0.100 type vlan id 100 > root@tengig1:~# ip addr add 10.10.100.1/24 dev bond0.100 > root@tengig1:~# ip link set bond0.100 up > > > root@tengig1:~# ping -M do -s 8000 10.10.100.2 -c1 > PING 10.10.100.2 (10.10.100.2) 8000(8028) bytes of data. > 8008 bytes from 10.10.100.2: icmp_req=1 ttl=64 time=0.378 ms > > --- 10.10.100.2 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.378/0.378/0.378/0.000 ms > > root@tengig1:~# nuttcp -l8000 -u -w4m -R20G -i1 -N4 10.10.100.2 > 1424.0494 MB / 1.00 sec = 11945.1470 Mbps 9604 / 196257 ~drop/pkt > 4.89 ~%loss > 1424.0417 MB / 1.00 sec = 11946.2297 Mbps 7833 / 194485 ~drop/pkt > 4.03 ~%loss > 1424.1028 MB / 1.00 sec = 11945.7980 Mbps 12335 / 198995 ~drop/pkt > 6.20 ~%loss > 1424.0723 MB / 1.00 sec = 11946.4380 Mbps 7681 / 194337 ~drop/pkt > 3.95 ~%loss > 1423.7061 MB / 1.00 sec = 11942.4582 Mbps 12371 / 198979 ~drop/pkt > 6.22 ~%loss > 1425.3082 MB / 1.00 sec = 11956.7944 Mbps 6061 / 192879 ~drop/pkt > 3.14 ~%loss > 1423.9655 MB / 1.00 sec = 11944.6580 Mbps 13831 / 200473 ~drop/pkt > 6.90 ~%loss > 1425.0336 MB / 1.00 sec = 11954.4664 Mbps 6065 / 192847 ~drop/pkt > 3.14 ~%loss > 1425.0031 MB / 1.00 sec = 11953.3975 Mbps 13667 / 200445 ~drop/pkt > 6.82 ~%loss > 1425.6897 MB / 1.00 sec = 11959.9467 Mbps 7569 / 194437 ~drop/pkt > 3.89 ~%loss > 10.3607 MB / 1.00 sec = 86.9096 Mbps -1348 / 10 ~drop/pkt > -13480.00000 ~%loss > > 14255.3406 MB / 10.00 sec = 11957.9606 Mbps 99 %TX 74 %RX 99368 / > 1967844 drop/pkt 5.05 %loss > > > --------------------------------------------------------------------------------------- > LINUX BRIDGE > --------------------------------------------------------------------------------------- > > Linux bonding driver, use the on-board Linux bridge. Changes to the > previous experiment: > > > ip link add veth0 type veth peer name veth1 > brctl addbr br0 > brctl addif br0 veth0 > brctl addif br0 bond0 > ip link set veth0 up > ip link set mtu 9000 dev veth0 > ip link set mtu 9000 dev veth1 > ip link set veth1 up > ip link set veth1 up > ip addr del 10.10.10.1/24 dev bond0 > ip addr add 10.10.10.1/24 dev veth1 > ip link set br0 up > > root@tengig1:~# ping -M do -s 8000 10.10.10.2 -c1 > PING 10.10.10.2 (10.10.10.2) 8000(8028) bytes of data. > 8008 bytes from 10.10.10.2: icmp_req=1 ttl=64 time=0.376 ms > > --- 10.10.10.2 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.376/0.376/0.376/0.000 ms > > root@tengig1:~# nuttcp -l8000 -u -w4m -R20G -i1 -N4 10.10.10.2 > 1250.5188 MB / 1.00 sec = 10489.9756 Mbps 0 / 163908 ~drop/pkt > 0.00 ~%loss > 1250.7706 MB / 1.00 sec = 10492.2870 Mbps 0 / 163941 ~drop/pkt > 0.00 ~%loss > 1253.0441 MB / 1.00 sec = 10511.2960 Mbps 0 / 164239 ~drop/pkt > 0.00 ~%loss > 1251.8082 MB / 1.00 sec = 10500.9385 Mbps 0 / 164077 ~drop/pkt > 0.00 ~%loss > 1250.5417 MB / 1.00 sec = 10490.3040 Mbps 0 / 163911 ~drop/pkt > 0.00 ~%loss > 1251.2970 MB / 1.00 sec = 10496.6610 Mbps 0 / 164010 ~drop/pkt > 0.00 ~%loss > 1252.7924 MB / 1.00 sec = 10509.1630 Mbps 0 / 164206 ~drop/pkt > 0.00 ~%loss > 1251.7242 MB / 1.00 sec = 10500.2240 Mbps 0 / 164066 ~drop/pkt > 0.00 ~%loss > 1251.9379 MB / 1.00 sec = 10502.0160 Mbps 0 / 164094 ~drop/pkt > 0.00 ~%loss > 1251.7548 MB / 1.00 sec = 10500.4905 Mbps 0 / 164070 ~drop/pkt > 0.00 ~%loss > 0.0381 MB / 1.00 sec = 0.3200 Mbps 0 / 5 ~drop/pkt > 0.00 ~%loss > > 12516.2277 MB / 10.00 sec = 10499.1135 Mbps 99 %TX 53 %RX 0 / 1640527 > drop/pkt 0.00 %loss > > > > > > > --------------------------------------------------------------------------------------- > LINUX BRIDGE + Linux VLAN > --------------------------------------------------------------------------------------- > > Linux bonding driver, use the on-board Linux bridge. Moreover, send > tagged frames outgoing. Changes to the previous experiment: > > > root@tengig1:~# brctl delif br0 bond0 > root@tengig1:~# brctl addif br0 bond0.100 > root@tengig1:~# ip addr del 10.10.100.1/24 dev bond0.100 > root@tengig1:~# ip addr add 10.10.100.1/24 dev veth1 > root@tengig1:~# ping -M do -s 8000 10.10.100.2 -c1 > PING 10.10.100.2 (10.10.100.2) 8000(8028) bytes of data. > 8008 bytes from 10.10.100.2: icmp_req=1 ttl=64 time=0.406 ms > > --- 10.10.100.2 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.406/0.406/0.406/0.000 ms > > > root@tengig1:~# nuttcp -l8000 -u -w4m -R20G -i1 -N4 10.10.100.2 > 1234.6954 MB / 1.00 sec = 10357.2517 Mbps 0 / 161834 ~drop/pkt > 0.00 ~%loss > 1232.8644 MB / 1.00 sec = 10341.8092 Mbps 0 / 161594 ~drop/pkt > 0.00 ~%loss > 1233.3221 MB / 1.00 sec = 10346.1353 Mbps 0 / 161654 ~drop/pkt > 0.00 ~%loss > 1233.8028 MB / 1.00 sec = 10349.6189 Mbps 0 / 161717 ~drop/pkt > 0.00 ~%loss > 1233.2764 MB / 1.00 sec = 10345.7410 Mbps 0 / 161648 ~drop/pkt > 0.00 ~%loss > 1231.2317 MB / 1.00 sec = 10328.0515 Mbps 0 / 161380 ~drop/pkt > 0.00 ~%loss > 1230.9265 MB / 1.00 sec = 10326.0182 Mbps 0 / 161340 ~drop/pkt > 0.00 ~%loss > 1230.6824 MB / 1.00 sec = 10323.4539 Mbps 0 / 161308 ~drop/pkt > 0.00 ~%loss > 1233.2764 MB / 1.00 sec = 10345.7410 Mbps 0 / 161648 ~drop/pkt > 0.00 ~%loss > 1233.0704 MB / 1.00 sec = 10343.5888 Mbps 0 / 161621 ~drop/pkt > 0.00 ~%loss > 0.0381 MB / 1.00 sec = 0.3200 Mbps 0 / 5 ~drop/pkt > 0.00 ~%loss > > 12327.1866 MB / 10.00 sec = 10340.5713 Mbps 99 %TX 53 %RX 0 / 1615749 > drop/pkt 0.00 %loss > > > > --------------------------------------------------------------------------------------- > OVS BRIDGE + LINUX BOND > --------------------------------------------------------------------------------------- > > Use OVS as bridge (without brcompat module), however use the Linux > bonding driver. No VLAN tagging. > > root@tengig1:~# ip link set br0 down > root@tengig1:~# brctl delbr br0 > root@tengig1:~# ovs-vsctl add-br br0 > root@tengig1:~# ovs-vsctl add-port br0 bond0 > root@tengig1:~# ovs-vsctl add-port br0 veth0 > root@tengig1:~# ping -M do -s 8000 10.10.10.2 -c1 > PING 10.10.10.2 (10.10.10.2) 8000(8028) bytes of data. > 8008 bytes from 10.10.10.2: icmp_req=1 ttl=64 time=0.547 ms > > --- 10.10.10.2 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.547/0.547/0.547/0.000 ms > > > root@tengig1:~# nuttcp -l8000 -u -w4m -R20G -i1 -N4 10.10.10.2 > 1464.8056 MB / 1.00 sec = 12287.0779 Mbps 0 / 191995 ~drop/pkt > 0.00 ~%loss > 1462.2574 MB / 1.00 sec = 12266.8437 Mbps 0 / 191661 ~drop/pkt > 0.00 ~%loss > 1462.3032 MB / 1.00 sec = 12266.2219 Mbps 0 / 191667 ~drop/pkt > 0.00 ~%loss > 1462.8525 MB / 1.00 sec = 12271.7501 Mbps 0 / 191739 ~drop/pkt > 0.00 ~%loss > 1461.2808 MB / 1.00 sec = 12257.6585 Mbps 0 / 191533 ~drop/pkt > 0.00 ~%loss > 1455.8945 MB / 1.00 sec = 12213.3799 Mbps 0 / 190827 ~drop/pkt > 0.00 ~%loss > 1455.6351 MB / 1.00 sec = 12210.2758 Mbps 0 / 190793 ~drop/pkt > 0.00 ~%loss > 1456.1310 MB / 1.00 sec = 12215.4128 Mbps 0 / 190858 ~drop/pkt > 0.00 ~%loss > 1456.5887 MB / 1.00 sec = 12218.2755 Mbps 0 / 190918 ~drop/pkt > 0.00 ~%loss > 1455.5359 MB / 1.00 sec = 12210.3840 Mbps 0 / 190780 ~drop/pkt > 0.00 ~%loss > 0.0763 MB / 1.00 sec = 0.6400 Mbps 0 / 10 ~drop/pkt > 0.00 ~%loss > > 14593.3609 MB / 10.00 sec = 12241.4789 Mbps 99 %TX 62 %RX 0 / 1912781 > drop/pkt 0.00 %loss > > > > --------------------------------------------------------------------------------------- > OVS BRIDGE + OVS BOND > --------------------------------------------------------------------------------------- > > Same setup as before, however do not use the Linux bonding driver but > use OVS' on board LACP support. Note that only one link is used. The > other one is purely passive. I am not sure if/how I can convince OVS to > do L4 hashing and use both links simultaneously - I thought I did though. > > > > root@tengig1:~# ovs-vsctl del-port bond0 > root@tengig1:~# ifdown bond0 > root@tengig1:~# ovs-vsctl add-bond br0 bond0 eth2 eth3 lacp=active > root@tengig1:~# ovs-vsctl set Port bond0 bond_mode=balance-tcp > root@tengig1:~# ip link set eth2 up > root@tengig1:~# ip link set eth3 up > root@tengig1:~# ip link set mtu 9000 dev eth2 > root@tengig1:~# ip link set mtu 9000 dev eth3 > root@tengig1:~# ping -M do -s 8000 10.10.10.2 -c1 > PING 10.10.10.2 (10.10.10.2) 8000(8028) bytes of data. > 8008 bytes from 10.10.10.2: icmp_req=1 ttl=64 time=0.616 ms > > --- 10.10.10.2 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.616/0.616/0.616/0.000 ms > > root@tengig1:~# ovs-appctl bond/show bond0 > bond_mode: balance-tcp > bond-hash-algorithm: balance-tcp > bond-hash-basis: 0 > updelay: 0 ms > downdelay: 0 ms > next rebalance: 7321 ms > lacp_negotiated: true > > slave eth3: enabled > may_enable: true > > slave eth2: enabled > active slave > may_enable: true > > > > root@tengig1:~# nuttcp -l8000 -u -w4m -R20G -i1 -N4 10.10.10.2 > 1180.9921 MB / 1.00 sec = 9906.5729 Mbps 7912 / 162707 ~drop/pkt > 4.86 ~%loss > 1181.9611 MB / 1.00 sec = 9915.2460 Mbps 8912 / 163834 ~drop/pkt > 5.44 ~%loss > 1182.7850 MB / 1.00 sec = 9921.7613 Mbps 10108 / 165138 ~drop/pkt > 6.12 ~%loss > 1182.7469 MB / 1.00 sec = 9921.7587 Mbps 9580 / 164605 ~drop/pkt > 5.82 ~%loss > 1181.3583 MB / 1.00 sec = 9909.7736 Mbps 9608 / 164451 ~drop/pkt > 5.84 ~%loss > 1182.5333 MB / 1.00 sec = 9919.9866 Mbps 9612 / 164609 ~drop/pkt > 5.84 ~%loss > 1183.2047 MB / 1.00 sec = 9925.3010 Mbps 9684 / 164769 ~drop/pkt > 5.88 ~%loss > 1181.8161 MB / 1.00 sec = 9913.9407 Mbps 8960 / 163863 ~drop/pkt > 5.47 ~%loss > 1180.7327 MB / 1.00 sec = 9904.5158 Mbps 9668 / 164429 ~drop/pkt > 5.88 ~%loss > 1181.2363 MB / 1.00 sec = 9909.0965 Mbps 8928 / 163755 ~drop/pkt > 5.45 ~%loss > 7.9956 MB / 1.00 sec = 67.0703 Mbps 80 / 1128 ~drop/pkt > 7.09 ~%loss > > 11827.3621 MB / 10.00 sec = 9921.2634 Mbps 99 %TX 53 %RX 94036 / > 1644272 drop/pkt 5.72 %loss > > > > --------------------------------------------------------------------------------------- > OVS BRIDGE + OVS BOND + OVS VLAN > --------------------------------------------------------------------------------------- > > All of it. Use the OVS bridge, OVS bonding driver and the OVS VLAN tag > support. I tag the veth port on access. > > root@tengig1:~# ovs-vsctl del-port veth0 > root@tengig1:~# ovs-vsctl add-port br0 veth0 tag=100 > root@tengig1:~# ping -M do -s 8000 10.10.100.2 -c1 > PING 10.10.100.2 (10.10.100.2) 8000(8028) bytes of data. > 8008 bytes from 10.10.100.2: icmp_req=1 ttl=64 time=0.550 ms > > --- 10.10.100.2 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.550/0.550/0.550/0.000 ms > root@tengig1:~# nuttcp -l8000 -u -w4m -R20G -i1 -N4 10.10.10.2 > 1180.7480 MB / 1.00 sec = 9904.7429 Mbps 8897 / 163660 ~drop/pkt > 5.44 ~%loss > 1181.1752 MB / 1.00 sec = 9908.4457 Mbps 9427 / 164246 ~drop/pkt > 5.74 ~%loss > 1181.5643 MB / 1.00 sec = 9911.6701 Mbps 8821 / 163691 ~drop/pkt > 5.39 ~%loss > 1181.7474 MB / 1.00 sec = 9913.2160 Mbps 9775 / 164669 ~drop/pkt > 5.94 ~%loss > 1180.4886 MB / 1.00 sec = 9902.6461 Mbps 8517 / 163246 ~drop/pkt > 5.22 ~%loss > 1180.1910 MB / 1.00 sec = 9899.7937 Mbps 8835 / 163525 ~drop/pkt > 5.40 ~%loss > 1181.8924 MB / 1.00 sec = 9914.7988 Mbps 9709 / 164622 ~drop/pkt > 5.90 ~%loss > 1181.2973 MB / 1.00 sec = 9909.0635 Mbps 8371 / 163206 ~drop/pkt > 5.13 ~%loss > 1181.0074 MB / 1.00 sec = 9907.3746 Mbps 9672 / 164469 ~drop/pkt > 5.88 ~%loss > 1182.1518 MB / 1.00 sec = 9916.2808 Mbps 7652 / 162599 ~drop/pkt > 4.71 ~%loss > 7.9651 MB / 1.00 sec = 66.8165 Mbps 60 / 1104 ~drop/pkt > 5.43 ~%loss > 0.0000 MB / 1.00 sec = 0.0000 Mbps 0 / 0 ~drop/pkt > 0.00 ~%loss > > 11820.2286 MB / 10.00 sec = 9915.1357 Mbps 99 %TX 49 %RX 90806 / > 1640107 drop/pkt 5.54 %loss > > > > Let me know in a timely manner if you want me to verify other > experiments, or different setups. The lab setup is still up and running, > but I'm likely going to shut it down soon. > > Overall I'm very happy with the delivered OVS performance. According to > other threads on this list I feared it could be much worse, but it > doesn't seem this would be the case - at least for OVS 1.4. >
Thanks for detailed results. Can you try current LTS 1.9 or most recent release OVS 2.1.2. There have been lot of changes since 1.4 release. _______________________________________________ discuss mailing list [email protected] http://openvswitch.org/mailman/listinfo/discuss
