On Wed, Jun 22, 2016 at 10:54:27PM +1000, David Gwynne wrote:
> secondly, allocating more than 4g at a time to socket buffers is
> generally a waste of memory. in practice you should scale the amount
> of memory available to sockets according to the size of the tcp
> windows you need to saturate the bandwidth available to the box.
Currently OpenBSD limits the socket buffer size to 256k.
#define SB_MAX (256*1024) /* default for max chars in sockbuf */
For downloading large files from the internet this is not sufficinet
anymore. After customer complaints we have increased the limit to
1MB. This still does not give maximum throughput, but granting
more could easily result in running out of mbufs. 16MB would be
sufficent.
Besides from single connections with high throughput we also have
a lot of long running connections, say some 10000. Each connection
over a relay needs two sockets and four socket buffers. With 1MB
limit and 10000 connections the theoretical maximum is 40GB.
It is hard to figure out which connections need socket buffer space
in advance. tcp_update_{snd,rcv}space() adjusts it dynamically,
there sbchecklowmem() has a first come first serve policy. Another
challenge is, that the peers on both sides of the relay can decide
wether they fill our buffers.
Besides from finding a smarter algorithm to distribute the socket
buffer space, increasing the number of mbufs could be a solution.
Our server machines mostly relay connection data, there I seems
seductive to use much more mbuf memory to speed up TCP connetions.
Without 64 bit DMA most memory of the machine is unused.
Also modern BIOS maps only 2GB in low region. All DMA devices must
share these. Putting mbufs high should reduce pressure.
Of course there are problems with network adaptors that support
less DMA space and with hotplug configurations. For a general
solution we can implement bounce buffers, disable the feature on
such machines or have a knob.
bluhm
