Hi Lukas ! Hi All!
Below are some links and comments just to provide more information on
"fd-hard-limit" default value subject :).
On 11/07/2024 12:17, Lukas Tribus wrote:
Hi,
I will get back to this for further research and discussion in about a week.
In the meantime, do we agree that the environment we are developing the fix
for is the following:
the hard limit is always set to the maximum available in the kernel which
on amd64 is one billion with a B, whether the systems has 128M or 2T of
memory is irrelevant.
Yes, we have a user's bug report, you could check all details here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1074374
It's on DebianTrixie, amd64
Our issue in GITHUB: https://github.com/haproxy/haproxy/issues/2621
You agree that this is the environment systemd sets us up with, right?
Yes, as it was investigated by Apollon systemd/256~rc3-3 now sets the
file descriptor hard limit to kernel max on boot.
https://lists.debian.org/debian-devel/2024/06/msg00041.html
And here is the link to this systemd patch:
https://github.com/systemd/systemd/pull/28968/commits/f470dafddcd688c3ea6031d4bbcbf934fd094711
https://github.com/systemd/systemd/pull/28968
1. We are trying to anticipate this change for haproxy packages
mainteners and for the new users, who have big servers and just want to
discover haproxy, compile and start it with a very simple config.
2. We try to avoid the situation, when haproxy installed from package in
the new distro with systemd >= 256~rc3-3, refuses to start with such
kind of errors at stderr:
[ALERT] (968) : fd hard limit is very high ~1B, Please, set fd-hard-limit in
your global section
Or:
[ALERT] (30116) : Not enough memory to allocate 1000000000 entries for fdtab!
[ALERT] (30116) : No polling mechanism available. Please set fd-hard-limit
in your global section...
It is obscure for some users 'fd-hard-limit'. And a lot of them may ask: "What is the best
value, according to my environment, which I should put here ?", "What will be the impact
?".
3. In case of small systems with less than 16 GB of RAM and fs.nr_open=32768,
you could do the following test with the current master branch in order to
confirm, that:
a) process can't not bump its runtime fd hard limit to 1048576, if the current
fd hard limit is lower;
b) process does not exits at a startup with an error in this case.
root@corei7:# git diff
diff --git a/examples/quick-test.cfg b/examples/quick-test.cfg
index f27eeff43..abda4172d 100644
--- a/examples/quick-test.cfg
+++ b/examples/quick-test.cfg
@@ -2,6 +2,7 @@
# support for HTTP/1 and 2.
global
+ stats socket /tmp/haproxy.sock
strict-limits # refuse to start if insufficient FDs/memory
# add some process-wide tuning here if required
diff --git a/src/limits.c b/src/limits.c
index bb7451060..70477d86b 100644
--- a/src/limits.c
+++ b/src/limits.c
@@ -148,6 +148,7 @@ int compute_ideal_maxconn()
* if only one of these ha-specific limits is presented in config or in
* the cmdline.
*/
+ ha_notice(">>> DEFAULT_MAXFD=%d\n", DEFAULT_MAXFD);
if (!is_any_limit_configured())
global.fd_hard_limit = DEFAULT_MAXFD;
root@corei7:#
root@corei7:# sysctl -w fs.nr_open=32768
fs.nr_open = 32768
root@corei7:# ulimit -n 32768
root@corei7:# ulimit -n
32768
root@corei7:# ulimit -n -H
32768
root@corei7:# cd haproxy && make -j 8 TARGET=linux-glibc USE_LUA=1 USE_PCRE2=1
root@corei7:/home/haproxy/haproxy# ./haproxy -D -f examples/quick-test.cfg
[NOTICE] (30970) : config : >>> DEFAULT_MAXFD=1048576
root@corei7:# echo 'show dev' | socat /tmp/haproxy.sock stdio
HAProxy version 3.1-dev3-e8799d-67
Features
-51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS
+DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT
+LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER
+NS -OBSOLETE_LINKER -OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2
-PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC
-QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ -SSL -STATIC_PCRE -STATIC_PCRE2
+SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Build options
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=8).
Built with Lua version : Lua 5.4.4
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0
Platform info
machine vendor: Supermicro
machine family: Default string
machine model: SYS-5038AD-I
board model: C7X99-OCE
soc vendor: GenuineIntel
soc model: Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz
cpu model: Fam 6 Model 79 Step 1
virtual machine: no
container: no
OS name: Linux
OS release: 6.9.0-rc2-custom+
OS version: #2 SMP PREEMPT_DYNAMIC Mon Apr 15 15:43:16 UTC 2024
OS architecture: x86_64
node name: corei7
Process info
pid: 31006
cmdline: ./haproxy -D -f examples/quick-test.cfg
boot uid: 0
runtime uid: 0
boot gid: 0
runtime gid: 0
boot capabilities:
CapEff: 0x000001ffffffffff
CapPrm: 0x000001ffffffffff
CapInh: 0x0000000000000000
runtime capabilities:
CapEff: 0x000001ffffffffff
CapPrm: 0x000001ffffffffff
CapInh: 0x0000000000000000
boot limits:
fd limit (soft): 32768
fd limit (hard): 32768
ram limit (soft): unlimited
ram limit (hard): unlimited
runtime limits:
fd limit (soft): 32768
fd limit (hard): 32768
ram limit (soft): unlimited
ram limit (hard): unlimited
root@corei7:#
Kind regards,
Valentine
Thanks,
Lukas
--
Bien cordialement,
Valentine
