Though it would not cause your problem, the reason for this is: In 3.10.18: https://elixir.free-electrons.com/linux/v3.10.18/source/net/ipv4/tcp.c
void tcp_init_mem <https://elixir.free-electrons.com/linux/v3.10.18/ident/tcp_init_mem>(struct net *net){ unsigned long limit <https://elixir.free-electrons.com/linux/v3.10.18/ident/limit> = nr_free_buffer_pages <https://elixir.free-electrons.com/linux/v3.10.18/ident/nr_free_buffer_pages>() / 8; limit <https://elixir.free-electrons.com/linux/v3.10.18/ident/limit> = max <https://elixir.free-electrons.com/linux/v3.10.18/ident/max>(limit <https://elixir.free-electrons.com/linux/v3.10.18/ident/limit>, 128UL); net->ipv4.sysctl_tcp_mem <https://elixir.free-electrons.com/linux/v3.10.18/ident/sysctl_tcp_mem>[0] = limit <https://elixir.free-electrons.com/linux/v3.10.18/ident/limit> / 4 * 3; net->ipv4.sysctl_tcp_mem <https://elixir.free-electrons.com/linux/v3.10.18/ident/sysctl_tcp_mem>[1] = limit <https://elixir.free-electrons.com/linux/v3.10.18/ident/limit>; net->ipv4.sysctl_tcp_mem <https://elixir.free-electrons.com/linux/v3.10.18/ident/sysctl_tcp_mem>[2] = net->ipv4.sysctl_tcp_mem <https://elixir.free-electrons.com/linux/v3.10.18/ident/sysctl_tcp_mem>[0] * 2;} In 4.4.49: https://elixir.free-electrons.com/linux/v4.4.49/source/net/ipv4/tcp.c static void __init <https://elixir.free-electrons.com/linux/v4.4.49/ident/__init> tcp_init_mem <https://elixir.free-electrons.com/linux/v4.4.49/ident/tcp_init_mem>(void){ unsigned long limit <https://elixir.free-electrons.com/linux/v4.4.49/ident/limit> = nr_free_buffer_pages <https://elixir.free-electrons.com/linux/v4.4.49/ident/nr_free_buffer_pages>() / 16; limit <https://elixir.free-electrons.com/linux/v4.4.49/ident/limit> = max <https://elixir.free-electrons.com/linux/v4.4.49/ident/max>(limit <https://elixir.free-electrons.com/linux/v4.4.49/ident/limit>, 128UL); sysctl_tcp_mem <https://elixir.free-electrons.com/linux/v4.4.49/ident/sysctl_tcp_mem>[0] = limit <https://elixir.free-electrons.com/linux/v4.4.49/ident/limit> / 4 * 3; /* 4.68 % */ sysctl_tcp_mem <https://elixir.free-electrons.com/linux/v4.4.49/ident/sysctl_tcp_mem>[1] = limit <https://elixir.free-electrons.com/linux/v4.4.49/ident/limit>; /* 6.25 % */ sysctl_tcp_mem <https://elixir.free-electrons.com/linux/v4.4.49/ident/sysctl_tcp_mem>[2] = sysctl_tcp_mem <https://elixir.free-electrons.com/linux/v4.4.49/ident/sysctl_tcp_mem>[0] * 2; /* 9.37 % */} As you can see, the limit is halved in 4.4.49. The relevant commit: commit b66e91ccbc34ebd5a2f90f9e1bc1597e2924a500 Author: Eric Dumazet <eduma...@google.com> Date: Fri May 15 12:39:30 2015 -0700 tcp: halves tcp_mem[] limits Allowing tcp to use ~19% of physical memory is way too much, and allowed bugs to be hidden. Add to this that some drivers use a full page per incoming frame, so real cost can be twice the advertized one. Reduce tcp_mem by 50 % as a first step to sanity. tcp_mem[0,1,2] defaults are now 4.68%, 6.25%, 9.37% of physical memory. \ Signed-off-by: Eric Dumazet <eduma...@google.com> Signed-off-by: David S. Miller <da...@davemloft.net> Regards, - Krishna On Fri, Nov 3, 2017 at 6:19 PM, Mark Brookes <m...@loadbalancer.org> wrote: > Hi All, > > We have been investigating an issue with reduced throughput. (its quite > possible that its nothing to do with HAProxy.) > I thought I would just check here to see if this rings a bell with anyone. > > We are currently looking to update our kernel from 3.10.18 to 4.4.49. It > appears that in the move from 3.x.x to 4.x.x at some point the kernel devs > change the tcp_mem calculation which results in halving the values based on > the same amount of RAM. Although that isnt the problem it just highlighted > it. > > Our test setup is > > Mutiple Clients --> Haproxy --> Real Server. > > If I run a fairly heavy load using iperf through haproxy using the 3.10.18 > kernel and I check - > > > cat proc/net/sockstat > sockets: used 193 > TCP: inuse 116 orphan 0 tw 17 alloc 118 mem 25591 > UDP: inuse 12 mem 3 > UDPLITE: inuse 0 > RAW: inuse 1 > FRAG: inuse 0 memory 0 > > cat /proc/sys/net/ipv4/tcp_mem > 89544 119392 179088 > > When I reboot into the 4.4.49 kernel and run the same test I get - > > cat proc/net/sockstat > sockets: used 198 > TCP: inuse 115 orphan 0 tw 18 alloc 117 mem 43957 > UDP: inuse 12 mem 2 > UDPLITE: inuse 0 > RAW: inuse 1 > FRAG: inuse 0 memory 0 > > cat /proc/sys/net/ipv4/tcp_mem > 44721 59631 89442 > > Haproxy ------ > Build options : > TARGET = linux2628 > CPU = generic > CC = gcc > CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing > -Wdeclaration-after-statement -fwrapv > OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > Encrypted password support via crypt(3): yes > Built with zlib version : 1.2.3 > Running on zlib version : 1.2.3 > Compression algorithms supported : identity("identity"), > deflate("deflate"), raw-deflate("deflate"), gzip("gzip") > Built with OpenSSL version : OpenSSL 1.0.2j-fips 26 Sep 2016 > Running on OpenSSL version : OpenSSL 1.0.2j-fips 26 Sep 2016 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports prefer-server-ciphers : yes > Built with PCRE version : 7.8 2008-09-05 > Running on PCRE version : 7.8 2008-09-05 > PCRE library supports JIT : no (USE_PCRE_JIT not set) > Built without Lua support > Built with transparent proxy support using: IP_TRANSPARENT > IPV6_TRANSPARENT IP_FREEBIND > > Available polling systems : > epoll : pref=300, test result OK > poll : pref=200, test result OK > select : pref=150, test result OK > Total: 3 (3 usable), will use epoll. > > Available filters : > [SPOE] spoe > [TRACE] trace > [COMP] compression > > Ive tried 4.4.95 kernel and get the same result Ive also tried the 4.9.59. > > Ive tried the latest 1.7.9 HAProxy too. > > Does anyone have any ideas? > > Thanks > -- > > Mark Brookes > Loadbalancer.org Ltd. > www.loadbalancer.org > > > +44 (0)330 380 1064 > m...@loadbalancer.org >
