On 12 September 2016 at 20:25, Mike Belopuhov <m...@belopuhov.com> wrote: > On Fri, Sep 02, 2016 at 13:43 +0200, Alexander Bluhm wrote: >> On Fri, Sep 02, 2016 at 10:33:19AM +0200, Andreas Bartelt wrote: >> > On 09/02/16 10:24, Alexander Bluhm wrote: >> > > I see a performance drop to 10 Mbit/sec on some old i386 machines >> > > with em(4). Can you try this kernel diff to see wether it is the >> > > same problem? >> > > >> > >> > yes, >50 MB/s now. Thanks! >> >> So I think this happens: >> >> sosend() uses m_getuio() now to allocate a MAXMCLBYTES mbuf cluster, >> that is 65536 bytes. sbreserve() calculates the upper cluster limit >> min(cc * 2, sb_max + (sb_max / MCLBYTES) * MSIZE). In our case it >> is min(1024*16 * 2, (256*1024) + ((256*1024) / (1 << 11)) * 256) >> == min(32768, 294912) == 32768. >> >> So after allocating a single mbuf cluster the sending socket buffer >> has no space anymore. As tcp_output() keeps the mbuf cluster for >> retransmits, it will be freed only after all ACKs have been received. >> That kills performance totally. >> >> To allow cycling through the mbufs periodically, I think we need >> space for at least 3 of them. Note that this diff also affects the >> mbuf size on the receiving side, but I think it does not matter >> much as the data size is also limited. >> >> Andreas, can you revert the diff I sent previously and try this one >> instead? >> >> ok? >> >> bluhm >> > > I see the same problem on large MTU (9000) networks now. > s/MAXMCLBYTES/PAGE_SIZE/ in m_getuio fixes it again for me, > while bumping 3 to 6 in sbreserve doesn't. > > Can somebody confirm this to rule out some local issues? >
Nevermind. I've found and fixed a subtle bug in my own math.
