Re: ssh over WAN: TCP window too small
On 8/25/15 3:47 PM, Chris Stankevitz wrote: Can anyone explain my abysmally small TCP window? So I believe this is the story: 1. openssh limits the size of some outgoing buffer to 65KB 2. openssh/HPN tries to improve on this by increasing the size of the outgoing buffer to match getsockopt(SO_SNDBUF) 3. When asked for the current SO_SNDBUF, FreeBSD 10.1 reports the high watermark of the outgoing buffer, not its capacity. 4. (2) is essentially a no-op because of (3). 5. openssh/HPN can be tricked into increasing its outgoing buffer by increasing sendspace/recvspace My comments: - (3) is not what I would expect -- perhaps the ssh/HPN folks would agree with me. Shouldn't getsockopt(SO_SNDBUF) return the same value set by setsockopt(SO_SNDBUF)? - I do not understand the mechanism by which (5) works. Chris ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
On 8/27/15 6:14 AM, Kurt Lidl wrote: # tcp options for long-haul speedups kern.ipc.maxsockbuf=4194304 # (2 * default 2097152) net.inet.tcp.mssdflt=1448 # (default 576) net.inet.tcp.sendbuf_max=4194304# (2 * default 2097152) net.inet.tcp.recvbuf_max=4194304# (2 * default 2097152) net.inet.tcp.syncache.rexmtlimit=1 # (default 3) net.inet.tcp.recvspace=262144 # (4 * default 65,536) net.inet.tcp.sendspace=262144 # (4 * default 65,536) net.inet.tcp.sendbuf_inc=65536 # (8 * default 8192) net.inet.tcp.recvbuf_inc=131072 # (8 * default 16384) Kurt, Thank you. FYI the default for sendspace is 32768 (not 65536). With these parameters my S-BCNT increases from ~60K to ~200K when doing `ssh < /dev/zero`. Amusingly, something somewhere limits bytes-in-flight to 172,888 bytes (I believe the ssh sending client is limiting -- although I'm not sure why). This effectively caps my bandwidth to 20 Mbps. With iperf this limitation does not exist -- nor does the need to tune these values (except for buf_inc which can get the ball rolling faster as you pointed out). Chris ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
On 8/26/15 5:46 PM, David DeSimone wrote: On 8/26/15 1:24 AM, John-Mark Gurney wrote: 94146 ssh 6.686140 CALL read(0x4,0x7fff6c70,0x4000) 94146 ssh 6.686154 GIO fd 4 read 4096 bytes [ read of stdin (/dev/zero) snipped) It would be interesting to know how long from the read of stdin (and is it really reading stdin in 4k blocks? If so, that should be fixed) The read is making a call with 0x4000 = 16k buffer size, but it only receives 4k, probably because that is the max size of the pipe buffer. In that example I used `ssh < /dev/null`. I would have used 'dd if=/dev/zero bs=1m | ssh` but at the time I did not know about `foo | ktrace bar`. Although if there is such a thing named "pipe buffer" I'm not sure it would have made a difference... Chris ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
John-Mark, I'm going to rearrange the conversation a bit in the quotes below: On 8/26/15 3:32 PM, John-Mark Gurney wrote: So, I looked at what getsockopt SO_RCVBUF returns, and it returns: case SO_RCVBUF: optval = so->so_rcv.sb_hiwat; Which is NOT S-BMAX, so it looks like OpenSSH only ever gets 66052 bytes for the buffer... I believe that explains everything we are seeing. HPN developers thought SO_RCVBUF would return S-BMAX but it instead returns S-HIWA. If it's decided to base it's waiting for ack on SO_RCVBUF, then this is probably where the issue is... Try setting HPNBufferSize to something resonably large, like 1MB, or above your bandwidth-delay product to see if this will make a difference.. Per: Conditions: HPNBufferSize SET, TCPRcvBufPoll enabled, TCPRcvBuf NOT Set Result: HPN Buffer Size = grows to HPNBufferSize The buffer will grow up to the maximum size specified here. Setting HPNBufferSize to 1 MB does not change S-BCNT which remains stuck at ~60KB. Although I might not necessarily expect a change. From README.hpn: HPNBufferSize: This is the default buffer size the HPN functionality uses when interacting with non-HPN SSH installations. It would be interesting to know how long from the read of stdin (and is it really reading stdin in 4k blocks? If so, that should be fixed) to the write out the socket... Basicly, how long does it take to encrypt the data... Note in the above the blocking call to select at time 6.592065 that takes ~0.1 second. This is the reason my connection is slow. The content appears to be encrypted... I presume it's an application-level ssh ack. I posted the in-context kdump at: https://www.stankevitz.com/ssh-ktrace.txt stdin is read in 4k blocks... but each "cycle" reads 3 of these blocks. I define as "cycle" as the period beginning and ending at a 0.1 second select(2) sleep. Looks like encryption of a 4KB block takes 10 microseconds, which means: for every 0.1 seconds in select(2), .30 seconds are encrypting. net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 65536 Try increasing these and reporting back... the buf_auto should override those, but this may be limiting it... Okay. Now we're getting somewhere. Increase sendspace on "sending client": no change Increase recvspace on "sending client": no change Increase sendspace on "receiving server": no change Increase recvspace on "receiving server": sending client's S-BCNT increases proportionally! I'm going to try Kurts parameters now... Chris ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
On Tue, Aug 25, 2015 at 2:12 PM, Chris Stankevitz wrote: > On 8/25/15 4:11 PM, Bjoern A. Zeeb wrote: > >> >> On 25 Aug 2015, at 22:47 , Chris Stankevitz wrote: >>> >>> Can anyone recommend some tools/tricks to figure out what in FreeBSD >>> and/or >>> >> >> base SSH is limiting the send/recv buffer and/or TCP window? > >> >> if you have the memory, try these sysctls: >> >> kern.ipc.maxsockbuf=146800640 >> net.inet.tcp.recvbuf_max=67108864 >> net.inet.tcp.sendbuf_max=67108864 >> > > Bjoern, > > Thank you for the reply. Before your suggestion my sysctls are: > > kern.ipc.maxsockbuf=2097152 > net.inet.tcp.recvbuf_max=2097152 > net.inet.tcp.sendbuf_max=2097152 > > Each of these is much larger than the limit I am experiencing (~64,000). > So I [naively] expect changing these values will have no effect. Let me > try... > > ... okay... sure enough the sysctrl changes you suggest did not change the > 64,000 bytes-in-flight limit I am experiencing. > > Thanks for the idea (and keep 'em coming!), > > Chris > My former employer, ESnet, was heavily involved in moving very large amounts of data (petabytes) over very long (intercontinental), very fat (100G) pipes. In an effort to improve customer satisfaction they have done extensive research into the issues involved and have published much of it at http://fasterdata.es.net. In particular, they have documented the issues with ssh over long latency links at http://fasterdata.es.net/data-transfer-tools/say-no-to-scp/. It is VERY hard to get good performance on high latency links in the bast of cases and, unfortunately, ssh/scp makes it not the best of cases. -- Kevin Oberman, Goat herder and Retired Network Engineer ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
Chris Stankevitz wrote: Hi, # cat /dev/urandom | ssh root at host 'cat > /dev/null' I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). iperf with default options gets the window opened to 500 KBytes (yielding 35 Mbps). Both sides of the connection: FreeBSD 10.1 w/default sshd options (except I permit root login). In particular, HPN is not disabled. Can anyone explain my abysmally small TCP window? Can anyone recommend some tools/tricks to figure out what in FreeBSD and/or base SSH is limiting the send/recv buffer and/or TCP window? I know this response is a little late to the party, but... I spent a bit of time last year tuning my FreeBSD 10.1 host to be able to transfer a bunch of data between the east coast of the US and the west cost. My WAN link was more like 70ms @ 75 Mbps, so not too different than yours. The other end of the connection was also a FreeBSD 10.1 host. I have the following in my /etc/sysctl.conf - and I get pretty much all 75Mbps when I scp or rsync a file: # tcp options for long-haul speedups kern.ipc.maxsockbuf=4194304 # (2 * default 2097152) net.inet.tcp.mssdflt=1448 # (default 576) net.inet.tcp.sendbuf_max=4194304# (2 * default 2097152) net.inet.tcp.recvbuf_max=4194304# (2 * default 2097152) net.inet.tcp.syncache.rexmtlimit=1 # (default 3) net.inet.tcp.recvspace=262144 # (4 * default 65,536) net.inet.tcp.sendspace=262144 # (4 * default 65,536) net.inet.tcp.sendbuf_inc=65536 # (8 * default 8192) net.inet.tcp.recvbuf_inc=131072 # (8 * default 16384) One thing that was noticed - it can take a really, really, really long time for the TCP window to open up the whole way with the default net.inet.tcp.sendbuf_inc setting! -Kurt ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: ssh over WAN: TCP window too small
On 8/26/15 1:24 AM, John-Mark Gurney wrote: > > 94146 ssh 6.686140 CALL read(0x4,0x7fff6c70,0x4000) > > 94146 ssh 6.686154 GIO fd 4 read 4096 bytes > >[ read of stdin (/dev/zero) snipped) > > It would be interesting to know how long from the read of stdin (and is > it really reading stdin in 4k blocks? If so, that should be fixed) The read is making a call with 0x4000 = 16k buffer size, but it only receives 4k, probably because that is the max size of the pipe buffer. This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. makes no warranty that this email is error or virus free. Thank you. This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. NTT America makes no warranty that this email is error or virus free. Thank you. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
Hi, > On 26 Aug 2015, at 08:47, Chris Stankevitz wrote: > > Hi, > > # cat /dev/urandom | ssh root@host 'cat > /dev/null' > > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). > tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). iperf > with default options gets the window opened to 500 KBytes (yielding 35 Mbps). Given that you are TCP window limited, do you have something in the middle preventing the windows size negotiation from working? A stateful firewall somewhere, perhaps? > Both sides of the connection: FreeBSD 10.1 w/default sshd options (except I > permit root login). In particular, HPN is not disabled. > > Can anyone explain my abysmally small TCP window? > > Can anyone recommend some tools/tricks to figure out what in FreeBSD and/or > base SSH is limiting the send/recv buffer and/or TCP window? Regards, Jan. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
Chris Stankevitz wrote this message on Wed, Aug 26, 2015 at 14:30 -0700:
> Thanks again. I appreciate you teaching me "how to fish". I basically
> spent all morning reading kdump output.
No worries, glad you're learning...
> On 8/26/15 1:24 AM, John-Mark Gurney wrote:
> > It really looks like we should set TCPRcvBufPoll by default on
> > FreeBSD...
>
> According to /etc/ssh/sshd_config, TCPRcvBufPoll defaults to true.
> ktrace (thank you for showing me this) shows the ssh process
> continuously polling SO_RCVBUF.
>
> In my case with TCPRcvBufPoll on and HPNBufferSize and TCPRcvBuf unset,
> ssh never sets SO_RCVBUF or SO_SNDBUF so presumably FreeBSD has complete
> control over those values (you saw the same thing).
>
> But perhaps all this is moot in my case because netstat shows that the
> sender and receiver have sufficiently large SND/RCV buffers: (can you
> confirm my interpretation is correct?)
>
> Sender S-BMAX 662816
> Receiver R-BMAX 528416
Yes, this is correct... So definately not an issue w/ window size of
socket buffer size...
> >> I will experiment with TCPRcvBuf.
> >
> > It does look like the values are in KB
>
> Yes, README.hpn says they are in KB. However, even though it is
> described in README.hpn, I cannot set TCPRcvBuf in sshd_config:
>
> /etc/ssh/sshd_config: line 141: Bad configuration option: TCPRcvBuf
> /etc/ssh/sshd_config: line 141: Bad configuration option: TcpRcvBuf
Looks like:
https://www.psc.edu/index.php/networking/695
Says that TCPRcvBuf is client side only.. Which is odd, but not an
issue w/ FreeBSD...
> However, as I described above, I believe the buffer size is a red herring.
Agreed
> ktrace shows pretty clearly what is happening. ssh isn't even bothering
> to read from /dev/zero until select(2) gives the green light. And in my
> case select(2) blocks ssh for 0.1 second for every ~30KB of data:
>
>
> 94146 ssh 6.592042 CALL getsockopt(0x3,SOL_SOCKET,SO_RCVBUF
> 94146 ssh 6.592054 RET getsockopt 0
> 94146 ssh 6.592065 CALL select(0x7,0x803817168,0x803817178,0,0)
> 94146 ssh 6.685873 RET select 1
> 94146 ssh 6.685907 CALL read(0x3,0x7fffae40,0x2000)
> 94146 ssh 6.685921 GIO fd 3 read 36 bytes
>[ read of fd 3 snipped]
> 94146 ssh 6.685931 RET read 36/0x24
> 94146 ssh 6.685950 CALL getpid
> 94146 ssh 6.685962 RET getpid 94146/0x16fc2
> 94146 ssh 6.685974 CALL getpid
> 94146 ssh 6.685984 RET getpid 94146/0x16fc2
> 94146 ssh 6.685996 CALL getpid
> 94146 ssh 6.686006 RET getpid 94146/0x16fc2
> 94146 ssh 6.686017 CALL getpid
> 94146 ssh 6.686027 RET getpid 94146/0x16fc2
> 94146 ssh 6.686091 CALL getsockopt(0x3,SOL_SOCKET,SO_RCVBUF
> 94146 ssh 6.686103 RET getsockopt 0
> 94146 ssh 6.686116 CALL select(0x7,0x803817168,0x803817178,0,0)
> 94146 ssh 6.686128 RET select 2
> 94146 ssh 6.686140 CALL read(0x4,0x7fff6c70,0x4000)
> 94146 ssh 6.686154 GIO fd 4 read 4096 bytes
>[ read of stdin (/dev/zero) snipped)
It would be interesting to know how long from the read of stdin (and is
it really reading stdin in 4k blocks? If so, that should be fixed)
to the write out the socket... Basicly, how long does it take to encrypt
the data...
> Note in the above the blocking call to select at time 6.592065 that
> takes ~0.1 second. This is the reason my connection is slow. The
> content appears to be encrypted... I presume it's an application-level
> ssh ack.
>
> BTW when I disable HPN (HPNdisabled=yes) that same select happens, but
> it blocks for less time (~0.05 second) and roughly doubles my throughput.
>
> > Also, don't forget that if you set this in .ssh/config, you only set
> > the client size recive buffer, not the server side, so you'd probably
> > need to add this to the server's sshd_config to enable it for server
> > receive side...
>
> I understand.
>
> > ktrace -i ssh ...
>
> Thank you, this is awesome. Is there a way for me to ktrace 'b' in this
> example: `a | b | c`? I tried `ktrace a | b | c` and `ktrace test.sh`
When you do this, make sure you ktrace -i to inherit the ktrace flag
to the children... In the case of a | b | c, just doing:
a | ktrace b | c
Should work too...
> (which included a|b|c) but neither seemed to work. I'm using stream
> redirection now but it doesn't afford me the bs control of dd. Perhaps
> named pipes is the solution.
>
> > Oh, I forgot to ask to make sure that net.inet.tcp.{send,recv}buf_auto
> > is enabled:
>
> Unfortunately it is enabled :-/
>
> > Maybe a dump of your net.inet.tcp might also be helpful...
>
> This should all be standard out-of-the-box:
[...]
> net.inet.tcp.sendspace: 32768
> net.inet.tcp.recvspace: 65536
Try increasing these and reporting back... the buf_auto should override
those, but this may be limiting it...
> net.inet.tcp.cc.algorithm: newreno
> net.inet.tcp.cc.availa
Re: ssh over WAN: TCP window too small
On Wed, Aug 26, 2015 at 02:30:07PM -0700, Chris Stankevitz wrote: > > ktrace -i ssh ... > > Thank you, this is awesome. Is there a way for me to ktrace 'b' in this > example: `a | b | c`? I tried `ktrace a | b | c` and `ktrace test.sh` > (which included a|b|c) but neither seemed to work. I'm using stream > redirection now but it doesn't afford me the bs control of dd. Perhaps > named pipes is the solution. a | ktrace b | c or ktrace -di test.sh (I suspect only -i is needed, but I've gotten so used to using both flags) You can also start ktrace on an existing process if you know its PID ktrace -p To stop all ongoing tracing: ktrace -C Regards, Gary ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
John-Mark,
Thanks again. I appreciate you teaching me "how to fish". I basically
spent all morning reading kdump output.
On 8/26/15 1:24 AM, John-Mark Gurney wrote:
> It really looks like we should set TCPRcvBufPoll by default on
> FreeBSD...
According to /etc/ssh/sshd_config, TCPRcvBufPoll defaults to true.
ktrace (thank you for showing me this) shows the ssh process
continuously polling SO_RCVBUF.
In my case with TCPRcvBufPoll on and HPNBufferSize and TCPRcvBuf unset,
ssh never sets SO_RCVBUF or SO_SNDBUF so presumably FreeBSD has complete
control over those values (you saw the same thing).
But perhaps all this is moot in my case because netstat shows that the
sender and receiver have sufficiently large SND/RCV buffers: (can you
confirm my interpretation is correct?)
Sender S-BMAX 662816
Receiver R-BMAX 528416
I will experiment with TCPRcvBuf.
It does look like the values are in KB
Yes, README.hpn says they are in KB. However, even though it is
described in README.hpn, I cannot set TCPRcvBuf in sshd_config:
/etc/ssh/sshd_config: line 141: Bad configuration option: TCPRcvBuf
/etc/ssh/sshd_config: line 141: Bad configuration option: TcpRcvBuf
However, as I described above, I believe the buffer size is a red herring.
ktrace shows pretty clearly what is happening. ssh isn't even bothering
to read from /dev/zero until select(2) gives the green light. And in my
case select(2) blocks ssh for 0.1 second for every ~30KB of data:
94146 ssh 6.592042 CALL getsockopt(0x3,SOL_SOCKET,SO_RCVBUF
94146 ssh 6.592054 RET getsockopt 0
94146 ssh 6.592065 CALL select(0x7,0x803817168,0x803817178,0,0)
94146 ssh 6.685873 RET select 1
94146 ssh 6.685907 CALL read(0x3,0x7fffae40,0x2000)
94146 ssh 6.685921 GIO fd 3 read 36 bytes
[ read of fd 3 snipped]
94146 ssh 6.685931 RET read 36/0x24
94146 ssh 6.685950 CALL getpid
94146 ssh 6.685962 RET getpid 94146/0x16fc2
94146 ssh 6.685974 CALL getpid
94146 ssh 6.685984 RET getpid 94146/0x16fc2
94146 ssh 6.685996 CALL getpid
94146 ssh 6.686006 RET getpid 94146/0x16fc2
94146 ssh 6.686017 CALL getpid
94146 ssh 6.686027 RET getpid 94146/0x16fc2
94146 ssh 6.686091 CALL getsockopt(0x3,SOL_SOCKET,SO_RCVBUF
94146 ssh 6.686103 RET getsockopt 0
94146 ssh 6.686116 CALL select(0x7,0x803817168,0x803817178,0,0)
94146 ssh 6.686128 RET select 2
94146 ssh 6.686140 CALL read(0x4,0x7fff6c70,0x4000)
94146 ssh 6.686154 GIO fd 4 read 4096 bytes
[ read of stdin (/dev/zero) snipped)
Note in the above the blocking call to select at time 6.592065 that
takes ~0.1 second. This is the reason my connection is slow. The
content appears to be encrypted... I presume it's an application-level
ssh ack.
BTW when I disable HPN (HPNdisabled=yes) that same select happens, but
it blocks for less time (~0.05 second) and roughly doubles my throughput.
Also, don't forget that if you set this in .ssh/config, you only set
the client size recive buffer, not the server side, so you'd probably
need to add this to the server's sshd_config to enable it for server
receive side...
I understand.
ktrace -i ssh ...
Thank you, this is awesome. Is there a way for me to ktrace 'b' in this
example: `a | b | c`? I tried `ktrace a | b | c` and `ktrace test.sh`
(which included a|b|c) but neither seemed to work. I'm using stream
redirection now but it doesn't afford me the bs control of dd. Perhaps
named pipes is the solution.
Oh, I forgot to ask to make sure that net.inet.tcp.{send,recv}buf_auto
is enabled:
Unfortunately it is enabled :-/
Maybe a dump of your net.inet.tcp might also be helpful...
This should all be standard out-of-the-box:
net.inet.tcp.rfc1323: 1
net.inet.tcp.mssdflt: 536
net.inet.tcp.keepidle: 720
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 32768
net.inet.tcp.recvspace: 65536
net.inet.tcp.keepinit: 75000
net.inet.tcp.delacktime: 100
net.inet.tcp.v6mssdflt: 1220
net.inet.tcp.cc.algorithm: newreno
net.inet.tcp.cc.available: newreno
net.inet.tcp.hostcache.cachelimit: 15360
net.inet.tcp.hostcache.hashsize: 512
net.inet.tcp.hostcache.bucketlimit: 30
net.inet.tcp.hostcache.count: 6
net.inet.tcp.hostcache.expire: 3600
net.inet.tcp.hostcache.prune: 300
net.inet.tcp.hostcache.purge: 0
net.inet.tcp.log_in_vain: 0
net.inet.tcp.blackhole: 0
net.inet.tcp.delayed_ack: 1
net.inet.tcp.drop_synfin: 0
net.inet.tcp.rfc3042: 1
net.inet.tcp.rfc3390: 1
net.inet.tcp.experimental.initcwnd10: 1
net.inet.tcp.rfc3465: 1
net.inet.tcp.abc_l_var: 2
net.inet.tcp.ecn.enable: 0
net.inet.tcp.ecn.maxretries: 1
net.inet.tcp.insecure_rst: 0
net.inet.tcp.recvbuf_auto: 1
net.inet.tcp.recvbuf_inc: 16384
net.inet.tcp.recvbuf_max: 2097152
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.tso: 1
net.inet.tcp.sendbuf_auto: 1
net.inet.tcp.sendbuf_inc: 8192
net.inet.tcp.sendbuf_max: 2097152
net.inet.tcp.reass.maxsegments
Re: ssh over WAN: TCP window too small
Chris Stankevitz wrote: # cat /dev/urandom | ssh root@host 'cat > /dev/null' I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). iperf with default options gets the window opened to 500 KBytes (yielding 35 Mbps). Both sides of the connection: FreeBSD 10.1 w/default sshd options (except I permit root login). In particular, HPN is not disabled. Can anyone explain my abysmally small TCP window? Can anyone recommend some tools/tricks to figure out what in FreeBSD and/or base SSH is limiting the send/recv buffer and/or TCP window? As an alternative to ssh for copying large files across high-BDP WAN links consider sysutils/bbcp, optionally coupled with security/hpenc for encryption. Mark ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 19:55 -0700: > John-Mark, > > Thank you for your reply. > > On 8/25/15 6:03 PM, John-Mark Gurney wrote: > > Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700: > >> # cat /dev/urandom | ssh root@host 'cat > /dev/null' > > > > Don't use this for testing... use /dev/zero or some other device > > that can produce data faster than this... > > Okay. As I'm sure you can imagine, I used urandom to avoid compression > artifacts. My urandom produces data at ~300 Mbps... but I will use > /dev/zero from now on. Yeh, unless you enable compression, ssh doesn't use it, so it won't be an issue... Also, if you want to free up even more cpu, you can test w/ "-c none" which disables encryption... > > So, our SSH does have the HPN patches: > > https://www.psc.edu/index.php/hpn-ssh > > > > and the README says: > > BUFFER SIZES: > > - if HPN is disabled the receive buffer size will be set to the OpenSSH > > default > >of 64K. > > Yes... I spent some time reading that document and fretting over whether > or not HPN was really incorporated in my setup. I "confirmed" that it > was available and enabled by setting "HPNDisabled no" and restarting > sshd (on both sides) without complaint. I'm half-tempted to build from > ports to be certain. > > > Looks like there are undocumented options like TCPRcvBuf that you can > > use to adjust the recv buffer window... > > According to the HPN README the default (which I am using) is the > "system wide TCP receive buffer size". I don't know what value that is > or where it comes from (net.inet.tcp.???). I will experiment with > TCPRcvBuf. It does look like the values are in KB, as I tried to set it to 3, and I got this error message: Couldn't set socket receive buffer to 3072: No buffer space available Also, don't forget that if you set this in .ssh/config, you only set the client size recive buffer, not the server side, so you'd probably need to add this to the server's sshd_config to enable it for server receive side... > > We have code that will auto grow > > buffer sizes properly so that slow connections won't use up too much > > buffer space... > > That is what I expected, although I believe openssh tries thwart/limit > this by requesting particular buffer sizes (I'm really unqualified to > talk about this). And it is my understanding that HPN undoes these > limitations although I'm not sure if it opens the door to FreeBSD having > full control or uses its own voodoo. You can verify this w/ ktrace -i ssh ... Then after that, you do a kdump | grep SO_RCVBUF | grep setsockopt to see if the program set any... If you see something like: $ kdump | grep SO_RCVBUF | grep setsockopt 6641 ssh CALL setsockopt(0x3,SOL_SOCKET,SO_RCVBUF,0x62dd8c,0x4) 6641 ssh CALL setsockopt(0x7,SOL_SOCKET,SO_RCVBUF,0x7fffcaa4,0x4) Then the buffer size is being set... I don't see this w/o tcprcvbuf in my config file, but I do when I add it... > > In a quick test of mine, I'm seeing a buffer size of ~520k from my > > MacOSX box, and ~776k from my 9.2-R box... Server in both cases is > > a June -CURRENT > > Thank you for those numbers. Since my system is basically stock, I > wonder if my bad behavior is an artifact of something on my network. > Did you invoke ssh more or less as "cat /dev/zero | ssh root@host 'cat > > /dev/null'"? Are you quoting S-BCNT numbers? The exact command is: dd if=/dev/zero bs=1m | ssh carbon dd of=/dev/null bs=1m And the numbers I was quoting was the R-BMAX numbers... As that is: R-BMAXMaximum bytes that can be used in the receive buffer. S-BCNT is just the number of bytes waiting to be sent, not the largest possible number that could be buffered... You really can't depend upon that number as only on high latency links will that have an appreciable number, otherwise you'll likely catch it between the ack draining it, and the program running to refill it... > > netstat -xAanfinet is helpful on this... > > That is brilliant! I was using pcap and wireshark to deduce some of > that info. Yep, there are lots of great debugging ways.. You could have also used dtrace for some of this too.. :) > I include my sender and receiver netstat's below for the ssh-ing > /dev/zero. It differs from iperf (which works well), most notably in > S-BCNT (~1MB for iperf, ~64kB for ssh). I think in my case the question is: > > - who is keeping S-BCNT so low (openssh, HPN, or FreeBSD)? As it's easier to change the client's recv buffer, you might want to try the command: ssh dd if=/dev/zero bs=1m > /dev/null And then you can play around w/ tcprcvbuf, though you should verify that SO_RCVBUF is being set before this though... > - Is the limitation introduced by the sending or receiving system? > > - what is the mechanism by which S-BCNT grows when using ssh over > long/fat pipe
Re: ssh over WAN: TCP window too small
John-Mark, Thank you for your reply. On 8/25/15 6:03 PM, John-Mark Gurney wrote: Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700: # cat /dev/urandom | ssh root@host 'cat > /dev/null' Don't use this for testing... use /dev/zero or some other device that can produce data faster than this... Okay. As I'm sure you can imagine, I used urandom to avoid compression artifacts. My urandom produces data at ~300 Mbps... but I will use /dev/zero from now on. So, our SSH does have the HPN patches: https://www.psc.edu/index.php/hpn-ssh and the README says: BUFFER SIZES: - if HPN is disabled the receive buffer size will be set to the OpenSSH default of 64K. Yes... I spent some time reading that document and fretting over whether or not HPN was really incorporated in my setup. I "confirmed" that it was available and enabled by setting "HPNDisabled no" and restarting sshd (on both sides) without complaint. I'm half-tempted to build from ports to be certain. Looks like there are undocumented options like TCPRcvBuf that you can use to adjust the recv buffer window... According to the HPN README the default (which I am using) is the "system wide TCP receive buffer size". I don't know what value that is or where it comes from (net.inet.tcp.???). I will experiment with TCPRcvBuf. We have code that will auto grow buffer sizes properly so that slow connections won't use up too much buffer space... That is what I expected, although I believe openssh tries thwart/limit this by requesting particular buffer sizes (I'm really unqualified to talk about this). And it is my understanding that HPN undoes these limitations although I'm not sure if it opens the door to FreeBSD having full control or uses its own voodoo. In a quick test of mine, I'm seeing a buffer size of ~520k from my MacOSX box, and ~776k from my 9.2-R box... Server in both cases is a June -CURRENT Thank you for those numbers. Since my system is basically stock, I wonder if my bad behavior is an artifact of something on my network. Did you invoke ssh more or less as "cat /dev/zero | ssh root@host 'cat > /dev/null'"? Are you quoting S-BCNT numbers? netstat -xAanfinet is helpful on this... That is brilliant! I was using pcap and wireshark to deduce some of that info. I include my sender and receiver netstat's below for the ssh-ing /dev/zero. It differs from iperf (which works well), most notably in S-BCNT (~1MB for iperf, ~64kB for ssh). I think in my case the question is: - who is keeping S-BCNT so low (openssh, HPN, or FreeBSD)? - Is the limitation introduced by the sending or receiving system? - what is the mechanism by which S-BCNT grows when using ssh over long/fat pipes? Thank you again, Chris SSH Sender Recv-Q 0 Send-Q 50132 R-MBUF 0 S-MBUF 16 R-CLUS 0 S-CLUS 14 R-HIWA 66052 S-HIWA 82852 R-LOWA 1 S-LOWA 2048 R-BCNT 0 S-BCNT 57344 R-BMAX 528416 S-BMAX 662816 rexmt 0.29 persist 0 keep7199.98 2msl0 delack 0 rcvtime 0.01 SSH Receiver Recv-Q 0 Send-Q 36 R-MBUF 0 S-MBUF 1 R-CLUS 0 S-CLUS 0 R-HIWA 66052 S-HIWA 33700 R-LOWA 1 S-LOWA 2048 R-BCNT 0 S-BCNT 256 R-BMAX 528416 S-BMAX 269600 rexmt 0.24 persist 0 keep7199.96 2msl0 delack 0.06 rcvtime 0.03 ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700: > # cat /dev/urandom | ssh root@host 'cat > /dev/null' Don't use this for testing... use /dev/zero or some other device that can produce data faster than this... > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). > tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). > iperf with default options gets the window opened to 500 KBytes > (yielding 35 Mbps). > > Both sides of the connection: FreeBSD 10.1 w/default sshd options > (except I permit root login). In particular, HPN is not disabled. > > Can anyone explain my abysmally small TCP window? Looks like ssh is propbably hard setting the send/recv buffers to values that are too small... So, our SSH does have the HPN patches: https://www.psc.edu/index.php/hpn-ssh and the README says: BUFFER SIZES: - if HPN is disabled the receive buffer size will be set to the OpenSSH default of 64K. You can read more at: https://svnweb.freebsd.org/base/stable/10/crypto/openssh/README.hpn?annotate=256281 Looks like there are undocumented options like TCPRcvBuf that you can use to adjust the recv buffer window... It looks like OpenSSH hard sets the buffer sizes for some reason... On FreeBSD, these should never be set unless the option is provided and you know what you are doing.. We have code that will auto grow buffer sizes properly so that slow connections won't use up too much buffer space... > Can anyone recommend some tools/tricks to figure out what in FreeBSD > and/or base SSH is limiting the send/recv buffer and/or TCP window? Seems like from looking at the code, things should "just work", so not sure why you are seeing the smaller window size... In a quick test of mine, I'm seeing a buffer size of ~520k from my MacOSX box, and ~776k from my 9.2-R box... Server in both cases is a June -CURRENT... netstat -xAanfinet is helpful on this... Hope this helps! -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
On 8/25/15 4:11 PM, Bjoern A. Zeeb wrote: On 25 Aug 2015, at 22:47 , Chris Stankevitz wrote: Can anyone recommend some tools/tricks to figure out what in FreeBSD and/or >> base SSH is limiting the send/recv buffer and/or TCP window? if you have the memory, try these sysctls: kern.ipc.maxsockbuf=146800640 net.inet.tcp.recvbuf_max=67108864 net.inet.tcp.sendbuf_max=67108864 Bjoern, Thank you for the reply. Before your suggestion my sysctls are: kern.ipc.maxsockbuf=2097152 net.inet.tcp.recvbuf_max=2097152 net.inet.tcp.sendbuf_max=2097152 Each of these is much larger than the limit I am experiencing (~64,000). So I [naively] expect changing these values will have no effect. Let me try... ... okay... sure enough the sysctrl changes you suggest did not change the 64,000 bytes-in-flight limit I am experiencing. Thanks for the idea (and keep 'em coming!), Chris ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ssh over WAN: TCP window too small
> On 25 Aug 2015, at 22:47 , Chris Stankevitz wrote: > > Hi, > > # cat /dev/urandom | ssh root@host 'cat > /dev/null' > > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). > tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). iperf > with default options gets the window opened to 500 KBytes (yielding 35 Mbps). > > Both sides of the connection: FreeBSD 10.1 w/default sshd options (except I > permit root login). In particular, HPN is not disabled. > > Can anyone explain my abysmally small TCP window? > > Can anyone recommend some tools/tricks to figure out what in FreeBSD and/or > base SSH is limiting the send/recv buffer and/or TCP window? if you have the memory, try these sysctls: kern.ipc.maxsockbuf=146800640 net.inet.tcp.recvbuf_max=67108864 net.inet.tcp.sendbuf_max=67108864 ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
ssh over WAN: TCP window too small
Hi, # cat /dev/urandom | ssh root@host 'cat > /dev/null' I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). iperf with default options gets the window opened to 500 KBytes (yielding 35 Mbps). Both sides of the connection: FreeBSD 10.1 w/default sshd options (except I permit root login). In particular, HPN is not disabled. Can anyone explain my abysmally small TCP window? Can anyone recommend some tools/tricks to figure out what in FreeBSD and/or base SSH is limiting the send/recv buffer and/or TCP window? Thank you, Chris ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
