Hi Valentine, hi Willy,
after spending some time testing I agree tuning maxconn/fd-limits is hard ... With 8GB RAM we can still OOM with 1M FDs / 500k maxconn (no TLS), but it appears to be around the sweetspot. It thought it would require more memory considering that we suggest 1GB of memory for 20k non-TLS connections or 8k TLS connections, but my test was indeed synthetic with zero features used, and it's not only about haproxy userspace but the system as well. lukas@dev:~/haproxy$ git grep -B3 -A1 "GB of RAM" doc/configuration.txt- global maxconn. Also, keep in mind that a connection contains two buffers doc/configuration.txt- of tune.bufsize (16kB by default) each, as well as some other data resulting doc/configuration.txt- in about 33 kB of RAM being consumed per established connection. That means doc/configuration.txt: that a medium system equipped with 1GB of RAM can withstand around doc/configuration.txt- 20000-25000 concurrent connections if properly tuned. -- doc/intro.txt- - 1300 HTTPS connections per second using TLS connections renegotiated with doc/intro.txt- RSA2048; doc/intro.txt- doc/intro.txt: - 20000 concurrent saturated connections per GB of RAM, including the memory doc/intro.txt- required for system buffers; it is possible to do better with careful tuning doc/intro.txt- but this result it easy to achieve. doc/intro.txt- doc/intro.txt: - about 8000 concurrent TLS connections (client-side only) per GB of RAM, doc/intro.txt- including the memory required for system buffers; lukas@dev:~/haproxy$ On Thu, 11 Jul 2024 at 08:05, Willy Tarreau <w...@1wt.eu> wrote: > > What I would really like is to no longer see any maxconn in a regular > configuration because there's no good value and we've seen them copied > over and over. By setting a global maxconn you force yourself to at least think about it. I'm assuming by "regular configuration" you mean small scale/size? In this case I agree. > I'm confused now, I don't see how, given that the change only *lowers* > an existing limit, it never raises it. It's precisely because of the > risk of OOM with OSes switching the default from one million FDs to one > billion that we're proposing to keep the previous limit of 1 million as > a sane upper bound. The only risk I'm seeing would be users discovering > that they cannot accept more than ~500k concurrent connections on a large > system. But I claim that those dealing with such loads *do* careful size > and configure their systems and services (RAM, fd, conntrack, monitoring > tools etc). Thus I'm not sure which scenario you have in mind that this > change could result in such a report as above. True, I confused memory required for initialization with memory allocated when actually used. On Thu, 11 Jul 2024 at 14:44, Willy Tarreau <w...@1wt.eu> wrote: > > My take on this limit is that most users should not care. Those dealing > with high loads have to do their homework and are used to doing this, > and those deploying in extremely small environments are also used to > adjusting limits (even sometimes rebuilding with specific options), and > I'm fine with leaving a bit of work for both extremities. Considering how non-trivial tuning maxconn/fd-hard-limit/ulimit for a specific memory size and configuration is, I (now) have to agree. On Tue, 16 Jul 2024 at 16:22, Valentine Krasnobaeva <vkrasnoba...@haproxy.com> wrote: > > Our issue in GITHUB: https://github.com/haproxy/haproxy/issues/2621 > >> You agree that this is the environment systemd sets us up with, right? > > Yes, as it was investigated by Apollon systemd/256~rc3-3 now sets the > file descriptor hard limit to kernel max on boot. Yes, I will comment regarding some additional systemd context on the github issue (but nothing changes for haproxy). On Tue, 16 Jul 2024 at 16:22, Valentine Krasnobaeva <vkrasnoba...@haproxy.com> wrote: > > It is obscure for some users 'fd-hard-limit'. And a lot of them > may ask: "What is the best value, according to my environment, > which I should put here ?", "What will be the impact?" This is completely true and why I prefer users think about maxconn instead, but like I said, even that is hard. Regards, Lukas
