On 11/21/12 7:41 AM, Marc Peters wrote:
Hi list,
we are experiencing low throughput on interncontinental connections with
our FreeBSD Servers. We made several tests and are wondering, why this
would be. The first tests were on an IPSEC VPN between our datacenter in
DE and Santa Clara, CA. We are connected with two gigabit uplinks in
each DC. Pushing data by scp between our FreeBSD servers takes ages.
Starting with several MB/s it drops to 60-70KB/s:
[root@freebsd ~]# ls -alh test.tgz
-rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz
[root@freebsd ~]# scp test.tgz 172.16.3.10:.
Password:
test.tgz 28% 17MB 75.3KB/s 09:32 ETA
For comparision, we did a similiar test with Linux, which didn't show
this behaviour:
root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50:
root@172.16.4.50's password:
jdk-6u33-linux-x64.bin 100%
69MB 3.4MB/s 00:20
root@linux:~#
Otherwise, the servers are really fast, when copying data to a machine
nearby:
[root@freebsd ~]# ls -alh test
-rw-r--r-- 1 root wheel 1G Nov 21 13:43 test
[root@freebsd ~]# scp test 172.16.3.11:
Password:
test 100% 1000MB 38.5MB/s 00:26
Intercontinental ftp downloads are the same:
[root@freebsd ~]# fetch
ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso
FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps
[root@freebsd ~]# fetch
ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso
FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s
[root@freebsd ~]# fetch
ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso
FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C
Linux:
root@linux:~# wget
ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso
--2012-11-21 15:07:57--
ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso
=> `FreeBSD-9.1-RC3-amd64-disc1.iso'
Resolving ftp1.de.freebsd.org... 137.226.34.42
Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1)
/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done.
==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896
==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done.
Length: 718800896 (686M) (unauthoritative)
100%[=====================================================================>]
718,800,896 19.1M/s in 61s
2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso'
saved [718800896]
Doing some googling brought up a lot of tuning hints, but nothing worked
for us. We tweaked some sysctls:
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.hostcache.expire=1
but to no good. Disabling MSI and TSO4 for the card didn't change
anything, too.
The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and
pciconf -lvc at the end of this mail).
Can someone hit me with a cluestick to get the BSDs on speed?
you really do need to get a tcpdump of the transfer under slow
conditions and a SIFTR output to match.
What is the ping time between the hosts. that will allow you to work
out how large a window you should have.
marc
PS: The version is FreeBSD-RC2 amd64, because we need the patch for
process migration on the CPUs which didn't make it 9.0 or an errata, as
we were the only ones, hitting this bug (so kib@ said).
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
