Well, to add some datapoints, I just did a quick test om my private homeserver, testing with iperf3
-s / iperf3 -c <server-ip>
I have not much knowledge of iperf3 network test and it's validity, but there is definitely some
differences to be seen.
Summary:
FBSD <-> FBSD 3,5Gbit/sec
FBSD -> Centos: 3GBit/Sec
Centos -> FBSD: 12 Gbit/sec
Centos <-> Centos: 12 Gbit/sec
OpenSuse <-> OpenSuse 13 GBit/sec
So it ssems there is some fishy when Freebsd is used as client
xen running on opensuse, 11 vm's, freebsd , openSuse and Centos. No vm has been optimized in any
way, everything is out of the box.
===========================
Test 1, Freebsd 12 -> freebsd 12
===========================
Server Freebsd 12:
ccepted connection from 192.168.1.100, port 39518
[ 5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 374 MBytes 3.14 Gbits/sec
[ 5] 1.00-2.00 sec 444 MBytes 3.72 Gbits/sec
[ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec
[ 5] 3.00-4.00 sec 438 MBytes 3.67 Gbits/sec
[ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec
[ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec
[ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec
[ 5] 7.00-8.00 sec 432 MBytes 3.63 Gbits/sec
[ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec
[ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec
[ 5] 10.00-10.00 sec 332 KBytes 2.17 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver
Client, freebsd 12:
Connecting to host 10.192.168.99, port 5201
[ 5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 239 178 KBytes
[ 5] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec 834 74.1 KBytes
[ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec 794 38.5 KBytes
[ 5] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 808 198 KBytes
[ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 777 67.0 KBytes
[ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec 795 120 KBytes
[ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec 763 72.7 KBytes
[ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 810 329 KBytes
[ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec 777 219 KBytes
[ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 688 304 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 7285 sender
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver
==========================
test 2, centos 7.6 -> Freebsd 12
==========================
Server Freebsd 12:
Accepted connection from 192.168.1.162, port 49180
[ 5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.17 GBytes 10.0 Gbits/sec
[ 5] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec
[ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec
[ 5] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec
[ 5] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec
[ 5] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec
[ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec
[ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec
[ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec
[ 5] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver
Client, centos 7.6:
Connecting to host 192.168.1.99, port 5201
[ 4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.17 GBytes 10.1 Gbits/sec 0 822 KBytes
[ 4] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.53 MBytes
[ 4] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.57 MBytes
[ 4] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.64 MBytes
[ 4] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec 0 1.70 MBytes
[ 4] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.70 MBytes
[ 4] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes
[ 4] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.70 MBytes
[ 4] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes
[ 4] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.70 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver
==========================
test 3, Freebsd 12 -> Centos 7.6
==========================
Server, Centos 7.6:
Accepted connection from 192.168.1.100, port 20309
[ 5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec
[ 5] 1.00-2.00 sec 379 MBytes 3.18 Gbits/sec
[ 5] 2.00-3.00 sec 368 MBytes 3.08 Gbits/sec
[ 5] 3.00-4.00 sec 371 MBytes 3.11 Gbits/sec
[ 5] 4.00-5.00 sec 374 MBytes 3.14 Gbits/sec
[ 5] 5.00-6.00 sec 362 MBytes 3.04 Gbits/sec
[ 5] 6.00-7.00 sec 381 MBytes 3.19 Gbits/sec
[ 5] 7.00-8.00 sec 384 MBytes 3.22 Gbits/sec
[ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec
[ 5] 9.00-10.00 sec 366 MBytes 3.07 Gbits/sec
[ 5] 10.00-10.13 sec 56.4 MBytes 3.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.13 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.13 sec 3.66 GBytes 3.10 Gbits/sec receiver
Client, Freebsd 12:
5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 387 MBytes 3.25 Gbits/sec 83 309 KBytes
[ 5] 1.00-2.00 sec 380 MBytes 3.19 Gbits/sec 88 11.4 KBytes
[ 5] 2.00-3.00 sec 365 MBytes 3.06 Gbits/sec 83 54.2 KBytes
[ 5] 3.00-4.00 sec 372 MBytes 3.12 Gbits/sec 86 322 KBytes
[ 5] 4.00-5.00 sec 370 MBytes 3.11 Gbits/sec 83 2.85 KBytes
[ 5] 5.00-6.00 sec 366 MBytes 3.07 Gbits/sec 84 298 KBytes
[ 5] 6.00-7.00 sec 382 MBytes 3.20 Gbits/sec 87 356 KBytes
[ 5] 7.00-8.00 sec 380 MBytes 3.19 Gbits/sec 87 369 KBytes
[ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec 87 29.9 KBytes
[ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 86 419 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec 854 sender
[ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec receiver
==========================
test 4, Centos 7.6 -> Centos 7.6
==========================
Server:
Accepted connection from 192.168.1.162, port 47606
[ 5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 1.60 GBytes 13.7 Gbits/sec
[ 5] 1.00-2.00 sec 1.59 GBytes 13.7 Gbits/sec
[ 5] 2.00-3.00 sec 1.60 GBytes 13.8 Gbits/sec
[ 5] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec
[ 5] 4.00-5.00 sec 1.49 GBytes 12.8 Gbits/sec
[ 5] 5.00-6.00 sec 1.38 GBytes 11.9 Gbits/sec
[ 5] 6.00-7.00 sec 1.52 GBytes 13.1 Gbits/sec
[ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec
[ 5] 8.00-9.00 sec 1.51 GBytes 13.0 Gbits/sec
[ 5] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec
[ 5] 10.00-10.03 sec 48.1 MBytes 11.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.03 sec 15.5 GBytes 13.2 Gbits/sec receiver
Client, Centos 7.6
Connecting to host 192.168.1.155, port 5201
[ 4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.41 MBytes
[ 4] 1.00-2.00 sec 1.58 GBytes 13.5 Gbits/sec 0 1.46 MBytes
[ 4] 2.00-3.00 sec 1.61 GBytes 13.9 Gbits/sec 0 1.91 MBytes
[ 4] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec 0 2.13 MBytes
[ 4] 4.00-5.00 sec 1.47 GBytes 12.6 Gbits/sec 0 2.49 MBytes
[ 4] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec 0 2.56 MBytes
[ 4] 6.00-7.00 sec 1.53 GBytes 13.1 Gbits/sec 0 2.71 MBytes
[ 4] 7.00-8.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.73 MBytes
[ 4] 8.00-9.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.81 MBytes
[ 4] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec 0 2.84 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec receiver
===============================
test4, Opensuse 15.1 -> OpenSuse 15.1
================================
Server, Opensuse 15.1:
[ 5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.51 GBytes 13.0 Gbits/sec
[ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec
[ 5] 2.00-3.00 sec 1.58 GBytes 13.6 Gbits/sec
[ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec
[ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec
[ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec
[ 5] 6.00-7.00 sec 1.58 GBytes 13.5 Gbits/sec
[ 5] 7.00-8.00 sec 1.60 GBytes 13.8 Gbits/sec
[ 5] 8.00-9.00 sec 1.60 GBytes 13.7 Gbits/sec
[ 5] 9.00-10.00 sec 1.61 GBytes 13.8 Gbits/sec
[ 5] 10.00-10.04 sec 68.0 MBytes 14.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec receiver
Client, OpenSuse 15.1:
[ 5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.58 GBytes 13.6 Gbits/sec 0 723 KBytes
[ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec 0 805 KBytes
[ 5] 2.00-3.00 sec 1.57 GBytes 13.5 Gbits/sec 0 926 KBytes
[ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec 0 926 KBytes
[ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec 0 970 KBytes
[ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.05 MBytes
[ 5] 6.00-7.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.05 MBytes
[ 5] 7.00-8.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.05 MBytes
[ 5] 8.00-9.00 sec 1.59 GBytes 13.7 Gbits/sec 0 1.05 MBytes
[ 5] 9.00-10.00 sec 1.62 GBytes 13.9 Gbits/sec 0 1.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 15.9 GBytes 13.7 Gbits/sec 0 sender
[ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec receiver
I've noticed very slow networking speed between VM's with FreeBSD on the
same host (XCP-ng 7.6.0) for more recent FreeBSD-versions.
I've made some tests that show me that something happened from 10.4-RELEASE
to 11.0-RELEASE that had a huge impact on network performance, and
12.0-RELEASE is even slower.
My test setup:
Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source.
Network: Internal Private Network on the host (not connected to a PIF).
Each VM has only one VIF connected to this network.
VM's:
2 x 12.0-RELEASE
2 x 11.0-RELEASE
2 x 10.4-RELEASE
All clean identical installs from XenOrchestra, only installed iperf on
each VM for testing. (xe-guest-utilities makes no difference in my tests,
I've tried with and without).
iperf -s on first server listed below, and iperf -c <ip> -r on the second
to test speed back and forth:
12.0 <-> 12.0: 50Mbit as client and server
12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server)
12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server).
11.0 <-> 11.0: 219Mbit as client, 99Mbit as server
10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server
As I side note, not sure if related, but I've noticed that I can't run
iperf with -r flag on 10.4-RELEASE. I get this error message:
iperf -c 172.31.16.122 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
write failed: Broken pipe
------------------------------------------------------------
Client connecting to 172.31.16.122, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec
I can run iperf -s fine, and iperf -c <ip> from the other 10.4 VM though:
iperf -c 172.31.16.122
------------------------------------------------------------
Client connecting to 172.31.16.122, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec
What have I tried to solve this?
I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via
XCP-ng. Disabled basically everything without any difference in iperf
results: other-config (MRW): ethtool-sg: off; ethtool-tso: off;
ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off
Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum
-rxcsum -tso -lro and no difference here either.
Any ideas of how to proceed now to find a solution for this?
Regards,
Christian
--
mvh,
Bjarne
_______________________________________________
[email protected] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[email protected]"
