On Fri, Dec 03, 2010 at 05:59:54PM +0100, Claudio Jeker wrote:
> On Fri, Dec 03, 2010 at 04:14:37PM +0100, Alexander Bluhm wrote:
> > On Fri, Dec 03, 2010 at 01:12:57PM +0100, Claudio Jeker wrote:
> > > Window size scaling is disabled when an application is issuing a
> > > setsockopt() changing SO_SNDBUF or SO_RCVBUF.
> >
> > tcp_update_sndspace() still rounds up to tp->t_maxseg even if
> > SO_SNDBUF has been set.
> >
> > I was always wondering why the code in tcp_update_sndspace()
> > is different from tcp_update_rcvspace().
> >
> > tcp_update_sndspace()
> > ...
> > /* round to MSS boundary */
> > nmax = roundup(nmax, tp->t_maxseg);
> >
> > if (nmax != so->so_snd.sb_hiwat)
> > sbreserve(&so->so_snd, nmax);
> > }
> >
> > tcp_update_rcvspace()
> > ...
> > if (nmax == so->so_rcv.sb_hiwat)
> > return;
> >
> > /* round to MSS boundary */
> > nmax = roundup(nmax, tp->t_maxseg);
> > sbreserve(&so->so_rcv, nmax);
> > }
> >
>
> IIRC I changed the rcvspace version because else it would alternate between
> sb_hiwat and roundup(sb_hiwat, tp->t_maxseg). The problem was that
> roundup(sb_max, tp->t_maxseg) was bigger then sb_max and so the
> sbreserve() would be called everytime. I guess tcp_update_sndspace() may
> have a similar issue but it needs to be fixed differently because the two
> algorithms are too different.
>
> Btw. the scaling functions are probably not perfect and if people can come
> up with a better algorithm I'm happy to test it but I'm currently to busy
> with other stuff to do it myself.
So I've been doing some playing. And I've found something that seems to work
on my home DSL link, and home Gigabit LAN a bit better than what's currently
in there.
I discovered that the other end (running Linux) didn't have tcp timestamps
enabled. Testing from a Linux machine with timestamps enabled I can retreive
at around 53 megs with timestamps disabled on the server. With timestamps
enabled on server speed drops to 50 megs/sec with Linux, and 14.6megs/sec with
OpenBSD-current default.
With my changes I'm getting 50 megs/sec with OpenBSD under Virtualbox, with
timestamps on server, within a few percent of Linux. So that makes me
comfortable about LAN performance. If I disable timestamps on server it drops
to 46.5megs/sec. Which to my mind is acceptable.
The thing of more concern is performance over the internet though, which from
home I'm now getting on 10 meg http request around 1000k/sec compared to around
500k/sec without changes. Where Linux client is averaging around 800k/sec. (the
web server is a VPS running Linux and international)
The thing is, I'm still resorting to increasing the initial window size which I
seem to recall being seen as a negative direction to go in because of memory
pressure etc. I'm raising it to 64k currently. And then I've adjusted the way
recv scaling works as so:
/* automatic buffer scaling */
if (tp->rfbuf_cnt > so->so_rcv.sb_hiwat / 8 * 7)
nmax = MIN(sb_max, so->so_rcv.sb_hiwat +
so->so_rcv.sb_hiwat * 9/8);
Because of the rounding up below that I'm assuming it shouldn't matter if it
ends up with weird values. I've also bumped the maximum window size up to 512k
The weird thing is, that was just my first stab-in-the-dark and with smaller
window sizes it should actually be less aggressive than the current system.
So I'm wondering if it has someting to do with how the other end is ramping
up or such.
I also discovered when I was doing tests last night that the reason one
connection wasn't scaling up seemed to be because SACK was being used so
I'm keeping in mind that slowly raising it may make retransmissions less
likely.
On my mind currently is that 64k is a better default than 16k. But that if
there is memory pressure a reduction could be made. And it seems that most
operating systems are defaulting more around those numbers now days. So it
may make sense to instead look at some kind of slow connection sysctl with 0
for 8k window size, 64k max, 1 for 32k window size, 256k max, 2 for 64k window
size, 512k max. On a 56k modem even 8k window size can lead to massive
congestion though.
Ben.
