Re: Recommendations for a new haproxy installation
On Thu, May 28, 2015 at 10:43:37AM -0600, Shawn Heisey wrote: > On 4/30/2015 11:50 PM, Willy Tarreau wrote: > > If you're working on preparing the OS, please *do* verify that > > conntrack is properly tuned (large hash table with at least 1/4 of the > > total number of sessions). Otherwise under load it will become > > extremely slow. > > When I asked about recommendations earlier, I was not using the > firewall, but now circumstances (FTP load balancing) will force me into > turning the firewall on. At that point, I expect will need to pay > attention to netfilter tuning. > > I found another message on the Internet where you advised someone to > look at nf_conntrack_max and nf_conntrack_htable_size. On a recent > (3.13) kernel, the htable size parameter doesn't seem to exist. I found > netfilter/nf_conntrack_max which is set to 65536. There is also > /netfilter/nf_conntrack_expect_max which is set to 256. > > Is a value of 65536 for nf_conntrack_max high enough? I'm definitely no > expert, but that certainly seems like a pretty high number, although I > did see one recommendation of 262144, and another where they used 10485760. No, 64k is pretty low. It means that you can have one connection per source port to a single destination port between a single source and a single destination. Add to that the fact that on a proxy you'll have connections on both sides, it means that you'll saturate with 32k forwarded connections. And if you add to that that a TIME_WAIT stays for 60s, your 32k forwarded connections turn into an absolute maximum of 500 connections per second. It can be fine for most uses. My hope DSL proxy doesn't need this for example. But a server facing the internet definitely needs a bit more. To give you an idea, you would just have to meet a bug in a script running on one of your servers that retries in loops to fetch an object via haproxy, and possibly after a few seconds all your entries are in use and you need to wait for them to expire. At the very least, ensure that you can sustain a communication on *all* source ports from a single client to a server, which means 64k per side thus 128k total. That will protect you against such type of accidents. > I would expect normal peak traffic to be below a few hundred requests > per second. If we ever saw 1000-2000 per second, I'm not sure our > current backend hardware could keep up. The calculus is easy : take the highest number you'd dream of. Let's say 2000/s. Multiply it by the session's life time, which is the sum of all states from SYN_RECV to TIME_WAIT and even CLOSE for a firewall. Typically with an ideal server responding immediately, you'd have 60s TIME_WAIT plus 10s CLOSE, which is 70s. You then get 14 concurrent connections per side. That's 280k total. It's about the worst case scenario based on your input. In practice, haproxy will ensure that no TIME_WAITs are kept on their way to the server. It's very important to target the highest possible peak. Some admins consider that if they break at certain rates it's not a problem, but it's the worst thing that can happen : your site falling in front of the most witnesses. Very bad publicity. So take some margins and consider your site's growth and the fact that you will forget about this tunable in 6 months and you don't want to have a bad surprize next year. Regards, Willy
Re: Recommendations for a new haproxy installation
On 4/30/2015 11:50 PM, Willy Tarreau wrote: > If you're working on preparing the OS, please *do* verify that > conntrack is properly tuned (large hash table with at least 1/4 of the > total number of sessions). Otherwise under load it will become > extremely slow. When I asked about recommendations earlier, I was not using the firewall, but now circumstances (FTP load balancing) will force me into turning the firewall on. At that point, I expect will need to pay attention to netfilter tuning. I found another message on the Internet where you advised someone to look at nf_conntrack_max and nf_conntrack_htable_size. On a recent (3.13) kernel, the htable size parameter doesn't seem to exist. I found netfilter/nf_conntrack_max which is set to 65536. There is also /netfilter/nf_conntrack_expect_max which is set to 256. Is a value of 65536 for nf_conntrack_max high enough? I'm definitely no expert, but that certainly seems like a pretty high number, although I did see one recommendation of 262144, and another where they used 10485760. I would expect normal peak traffic to be below a few hundred requests per second. If we ever saw 1000-2000 per second, I'm not sure our current backend hardware could keep up. Thanks, Shawn
Re: Recommendations for a new haproxy installation
On Wed, Apr 29, 2015 at 03:00:58PM -0600, Shawn Heisey wrote: > I have an existing load balancer installation that I have been slowly > migrating from IPVS to haproxy. It's CentOS 6, so many components are > out of date, such as TLS support. > > Once that migration is done, I would like to entirely replace the > hardware and load an ideal software environment for haproxy. > > The new machines have Ubuntu 14, so the openssl version is fairly new, > but not the newest available. The CPU is an Intel Xeon E5-2430, which > has built-in TLS acceleration. It has 16GB of memory. The machine is > dedicated for load balancing. > > How can I be sure that openssl is compiled with support for TLS > acceleration in the CPU? I don't understand what type of acceleration your talking about. There are multiple features improving TLS performance such as AES-NI, MULX, AVX2, etc. Anyway you can use "openssl speed" to compare between the original lib and your newly built one. You can force some algorithms to verify specific improvements, such as "openssl speed rsa2048" or "openssl speed ecdhp256" etc. > I am compiling haproxy from source. Would you > recommend that I install a separate and newer openssl from source for > explicit use with haproxy, and tweak its config for the specific > hardware it's on? As I mentionned in the other mail, I still prefer a dynamic build to make it easier to update it, but I know it's not necessarily easy because the soname doesn't change between versions... > The CPU has 6 hyperthreaded CPU cores. I know that haproxy can be run > in multiprocess mode to take advantage of multiple CPU cores, but is > that a recommended and stable config? Yes, but there are a few things that are not shared such as stick-tables and stats. Also you won't be able to use "peers" to replicate stick-tables between nodes. And health-checks will be done for each process. > If it is, then I will do it just > so I'm taking full advantage of the hardware. If it's just for SSL performance, you can also have one TLS-only instance running in multi-process to decipher the traffic and a single process one for the rest. You just use abstract sockets between the processes, they're cheaper than TCP and that's all. Example : global nbproc 6 listen ssl-offload bind :443 process 1 crt ... bind :443 process 2 crt ... bind :443 process 3 crt ... bind :443 process 4 crt ... bind :443 process 5 crt ... bind :443 process 6 crt ... server clear abns@haproxy-clear-listener send-proxy-v2 And the other one : global nbproc 1 frontend clear-text bind :80 ... bind abns@haproxy-clear-listener accept-proxy ... You can put the two in the same config if you want, but then don't forget to specify "bind-process 1" in all instances except the first one. Peers will fail as well (though I have an idea how to fix that). > I know from the list > history that stats don't aggregate across processes, but as long as I > can figure out how to look at all the stats, that shouldn't be a problem. To access all the stats, simply have different ports bound to different processes and visit all of them. For the CLI, you can define as many as you want and specify "process" on the stats line as well to limit the scope of the socket. > Is there anything else I should be aware of or think about as I work on > the OS and software for this replacement hardware? If you're working on preparing the OS, please *do* verify that conntrack is properly tuned (large hash table with at least 1/4 of the total number of sessions). Otherwise under load it will become extremely slow. Regards, Willy
Re: Recommendations for a new haproxy installation
Hi, On Wed, Apr 29, 2015 at 10:58:36PM -0600, Shawn Heisey wrote: > On 4/29/2015 3:00 PM, Shawn Heisey wrote: > > How can I be sure that openssl is compiled with support for TLS > > acceleration in the CPU? I am compiling haproxy from source. Would you > > recommend that I install a separate and newer openssl from source for > > explicit use with haproxy, and tweak its config for the specific > > hardware it's on? > > Followup on the openssl part of my email. > > I built and installed openssl 1.0.2a from source, with this config line: > > ./config no-shared enable-ec_nistp_64_gcc_128 threads This looks correct to me. > Then I built haproxy using this command: > > make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 CPU=native > SSL_INC=/usr/local/ssl/include SSL_LIB=/usr/local/ssl/lib ADDLIB=-ldl > > Here's the 'haproxy -vv' and 'uname -a' output: > > --- > HA-Proxy version 1.5.11 2015/01/31 > Copyright 2000-2015 Willy Tarreau > > Build options : > TARGET = linux2628 > CPU = native > CC = gcc > CFLAGS = -O2 -march=native -g -fno-strict-aliasing > OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 > > Encrypted password support via crypt(3): yes > Built with zlib version : 1.2.8 > Compression algorithms supported : identity, deflate, gzip > Built with OpenSSL version : OpenSSL 1.0.2a 19 Mar 2015 > Running on OpenSSL version : OpenSSL 1.0.2a 19 Mar 2015 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports prefer-server-ciphers : yes > Built with PCRE version : 8.31 2012-07-06 > PCRE library supports JIT : no (USE_PCRE_JIT not set) > Built with transparent proxy support using: IP_TRANSPARENT > IPV6_TRANSPARENT IP_FREEBIND > > Available polling systems : > epoll : pref=300, test result OK >poll : pref=200, test result OK > select : pref=150, test result OK > Total: 3 (3 usable), will use epoll. Looks good as well. > --- > Linux lb1 3.13.0-49-generic #83-Ubuntu SMP Fri Apr 10 20:11:33 UTC 2015 > x86_64 x86_64 x86_64 GNU/Linux > --- OK. > Can anyone who's knowledgeable about this look over what I've done and > tell me if they'd do something different? That's fine for me. > I also still need some assistance with the rest of my original email. I do not have much advice to add beyond this. > Side issue, mentioning in case it's important, though I suspect it > isn't: When I built openssl with the indicated config, 'make test' > failed, but it passed on an earlier build with 'shared' instead of > 'no-shared'. I rebuilt with no-shared because haproxy was dynamically > linking the older openssl library installed from ubuntu packages, > instead of the newer library used for compile. You'd still better run with the shared mode in my opinion, because openssl provides frequent updates and 1.0.2 is fairly new, so you'll certainly have to rebuild often. On the other hand, it might be an opportunity to think about upgrading haproxy as well. But you need to be reactive since openssl updates are generally for security issues (eg heartbleed and such), so you don't want to wait. Regards, Willy
Re: Recommendations for a new haproxy installation
On 4/29/2015 3:00 PM, Shawn Heisey wrote: > How can I be sure that openssl is compiled with support for TLS > acceleration in the CPU? I am compiling haproxy from source. Would you > recommend that I install a separate and newer openssl from source for > explicit use with haproxy, and tweak its config for the specific > hardware it's on? Followup on the openssl part of my email. I built and installed openssl 1.0.2a from source, with this config line: ./config no-shared enable-ec_nistp_64_gcc_128 threads Then I built haproxy using this command: make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 CPU=native SSL_INC=/usr/local/ssl/include SSL_LIB=/usr/local/ssl/lib ADDLIB=-ldl Here's the 'haproxy -vv' and 'uname -a' output: --- HA-Proxy version 1.5.11 2015/01/31 Copyright 2000-2015 Willy Tarreau Build options : TARGET = linux2628 CPU = native CC = gcc CFLAGS = -O2 -march=native -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.8 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.2a 19 Mar 2015 Running on OpenSSL version : OpenSSL 1.0.2a 19 Mar 2015 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.31 2012-07-06 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. --- Linux lb1 3.13.0-49-generic #83-Ubuntu SMP Fri Apr 10 20:11:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux --- Can anyone who's knowledgeable about this look over what I've done and tell me if they'd do something different? I also still need some assistance with the rest of my original email. Side issue, mentioning in case it's important, though I suspect it isn't: When I built openssl with the indicated config, 'make test' failed, but it passed on an earlier build with 'shared' instead of 'no-shared'. I rebuilt with no-shared because haproxy was dynamically linking the older openssl library installed from ubuntu packages, instead of the newer library used for compile. Thanks, Shawn
Recommendations for a new haproxy installation
I have an existing load balancer installation that I have been slowly migrating from IPVS to haproxy. It's CentOS 6, so many components are out of date, such as TLS support. Once that migration is done, I would like to entirely replace the hardware and load an ideal software environment for haproxy. The new machines have Ubuntu 14, so the openssl version is fairly new, but not the newest available. The CPU is an Intel Xeon E5-2430, which has built-in TLS acceleration. It has 16GB of memory. The machine is dedicated for load balancing. How can I be sure that openssl is compiled with support for TLS acceleration in the CPU? I am compiling haproxy from source. Would you recommend that I install a separate and newer openssl from source for explicit use with haproxy, and tweak its config for the specific hardware it's on? The CPU has 6 hyperthreaded CPU cores. I know that haproxy can be run in multiprocess mode to take advantage of multiple CPU cores, but is that a recommended and stable config? If it is, then I will do it just so I'm taking full advantage of the hardware. I know from the list history that stats don't aggregate across processes, but as long as I can figure out how to look at all the stats, that shouldn't be a problem. Is there anything else I should be aware of or think about as I work on the OS and software for this replacement hardware? Thanks, Shawn
