Re: Active session count drop after HAProxy upgrade from 2.0 to 2.4
Hi Wily, That's a bug and it shouldn't be like this. > You can find information about this here : https://www.mail-archive.com/haproxy@formilux.org/msg43291.html But don't waste too much time on this. > > For those interested, the (small) necessary config changes were : > > - option httpchk syntax (use http-check) > > - some healthchecks not working anymore on servers with > > "send-proxy-v2-ssl-cn ssl-check", due to an unresolved bug in Apache 2.4 > ( > > https://bz.apache.org/bugzilla/show_bug.cgi?id=63893). > > But why were they working previously ? Yes, I confirm this was working previously with the exact same haproxy config file. > Maybe they were sent as dummy > PROXY commands ? If so maybe we could implement a workaround for such > broken implementations if that's a big problem (not sure if this is > feasible, just trying to figure what the desired behavior should be). > I don't know what changed in HAProxy 2.2 or 2.4 about this. The configuration was the following : listen x:443 mode tcp bind x.x.x.x:443 option httpchk GET /test.php HTTP/1.0 # to be updated to new format with 2.4 server sx 192.168.1.19:443 id 12 check weight 5 send-proxy-v2-ssl-cn check-ssl verify none server sx2 192.168.1.22:443 id 13 check weight 5 send-proxy-v2-ssl-cn check-ssl verify none The error reported was L6RSP (+ the above error in Apache log files). Same error with "mode http" instead of tcp. Removing "check-ssl" leads to L7RSP, but this is expected (talking plain text when SSL is required). Right now, I'm avoiding this issue by making the test on port 80 (http-check connect port 80). > > Everything seems to run smoothly, but on the monitoring, the number of > > active sessions (scur) dropped significantly (only one third active > > sessions compared to before), even after several hours. I did not make > any > > change on keep alive or timeouts, that's why I'm wondering if any > > modifications between 2.0 and 2.4 may explain this behaviour. > > If you were running without HTX mode it's very likely because in the > past it was indicating the number of established sessions while now > it's reporting the number of active requests (since technically it's > always a stream that is being accounted for, but in the past they used > to remain present while in idle state, using all the resources between > two requests). > That's it. I was indeed NOT using HTX in 2.0. Thanks for the explanation. Olivier
Active session count drop after HAProxy upgrade from 2.0 to 2.4
Hello, I've finally updated our load balancer, using HAProxy 2.0, to HAProxy 2.4 \o/ I was motivated by both the EOL on 2.0, and by a recurring segfault everytime we reloaded. btw, that segfault is now gone with 2.4 :) I did not update to a newer version because we are still heavy users of "nbproc" that we still need to convert to thread usage (and replace bind-process by 'process'). For those interested, the (small) necessary config changes were : - option httpchk syntax (use http-check) - some healthchecks not working anymore on servers with "send-proxy-v2-ssl-cn ssl-check", due to an unresolved bug in Apache 2.4 ( https://bz.apache.org/bugzilla/show_bug.cgi?id=63893). Everything seems to run smoothly, but on the monitoring, the number of active sessions (scur) dropped significantly (only one third active sessions compared to before), even after several hours. I did not make any change on keep alive or timeouts, that's why I'm wondering if any modifications between 2.0 and 2.4 may explain this behaviour. Cheers, Olivier
Segfault with HAProxy 2.0.31
Hello,
We are experiencing for the past weeks a segfault on haproxy processes when
reloading haproxy.
Each thread generates a coredump. Fortunately, this is the old process that
crashes, so there is no production impact.
The same behaviour happens with haproxy 2.0.25 compiled with OpenSSL 1.1.1l
Core was generated by `/root/haproxy -sf 42184 42185 42186 42187 42188
42189 42190 42191 42192 42193 4'.
Program terminated with signal 6, Aborted.
#0 0x7f414c4d4495 in raise () from /lib64/libc.so.6
(gdb) bt full
#0 0x7f414c4d4495 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x7f414c4d5c75 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x7f414c5123a7 in __libc_message () from /lib64/libc.so.6
No symbol table info available.
#3 0x7f414c517dee in malloc_printerr () from /lib64/libc.so.6
No symbol table info available.
#4 0x7f414c51ac3d in _int_free () from /lib64/libc.so.6
No symbol table info available.
#5 0x0047149c in ssl_sock_free_ssl_conf (conf=0x6f3d440) at
src/ssl_sock.c:3766
No locals.
#6 0x00479f3d in ssl_sock_free_ssl_conf (conf=) at
src/ssl_sock.c:3764
No locals.
#7 ssl_sock_free_all_ctx (bind_conf=bind_conf@entry=0x6f3d160) at
src/ssl_sock.c:5144
node = 0x6f4ac60
sni = 0x6f4ac40
#8 0x0047a4a1 in ssl_sock_destroy_bind_conf (bind_conf=0x6f3d160)
at src/ssl_sock.c:5176
No locals.
#9 0x0050aae8 in deinit () at src/haproxy.c:2673
haproxy -vv output :
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=
Feature list : -51DEGREES +ACCEPT4 -CLOSEFROM +CPU_AFFINITY +CRYPT_H
-DEVICEATLAS +DL +EPOLL -EVPORTS +FUTEX +GETADDRINFO -KQUEUE +LIBCRYPT
+LINUX_SPLICE +LINUX_TPROXY +LUA -MY_ACCEPT4 -MY_EPOLL -MY_SPLICE
+NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT
+POLL +PRCTL -PRIVATE_CACHE -PTHREAD_PSHARED -REGPARM +RT -SLZ +STATIC_PCRE
-STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -VSYSCALL -WURFL
+ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=32).
Built with OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
Running on OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.13
Running on zlib version : 1.2.13
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.45 2021-06-15
Running on PCRE version : 8.45 2021-06-15
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE mux=H2
h2 : mode=HTTP side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
I understand this is linked to some SSL config file, but fail to find what.
The haproxy config file is very large (> 1000 lines) with dozens of
frontend/backend. It will help me to know which one is it. Any "gdb" advice
how to narrow the issue down ?
Thanks,
Olivier
Re: Blocking log4j CVE with HAProxy
Hi,
Le lun. 13 déc. 2021 à 19:38, John Lauro a écrit :
> http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or
> hdr_sub(user-agent) -i "\$\{jndi:" }
> was not catching the bad traffic. I think the escapes were causing issues
> in the matching.
>
> The following did work:
> http-request deny deny_status 405 if { url_sub -i -f
> /etc/haproxy/bad_header.lst }
> http-request deny deny_status 405 if { hdr_sub(user-agent)
> -i -f /etc/haproxy/bad_header.lst }
> and in bad_header.lst
> ${jndi:
>
I tried
http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or
hdr_sub(user-agent) -i "\$\{jndi:" }
and
http-request deny deny_status 405 if { url_sub -i ${jndi: or
hdr_sub(user-agent) -i ${jndi: }
without success. Can anyone tell what's wrong with both syntaxes ? And how
to escape special chars correctly ?
Olivier
Blocking log4j CVE with HAProxy
Hello there,
If you don't know yet, a CVE was published on friday about library log4j,
allowing a remote code execution with a crafted HTTP request.
We would like to filter these requests on HAProxy to lower the exposition.
At peak times, 20% of our web traffic is scanners about this bug !
The offended string is "${jndi:". It must be filtered on any fields that
could go to log servers:
- URL
- User-Agent
- User name
What would be the easier way to do that ? If I give it a try :
http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or
hdr_sub(user-agent) -i "\$\{jndi:" }
What do you think ?
Olivier
HTTP2 concurrent streams and connection count
Hello,
I'm investigating an issue on specific rules for a customer.
The rules are the following :
stick-table type ipv6 size 6 expire 1h store conn_cur,conn_rate(10s)
http-request deny deny_status 429 if { src_conn_cur ge 100 }
http-request deny deny_status 429 if { src_conn_rate ge 600 }
The expected behaviour is to throw an error if a single IP has more than
100 connections or if it tries to open more than 600 connections in 10s.
So first, can you confirm the rules are written correctly ? :)
If so, the issue here is that the customer is reporting having 429 errors
himself. He was able to troubleshoot these errors to a specific page on his
website, with hundreds of images loaded simultaneously. The connection is
performed with HTTP2.
In my mind, src_conn_cur and src_conn_rate are incremented only when a new
TCP connection is triggered on the frontend. But maybe I dont understand it
correctly and the hundreds of simultaneous streams in a single http2
connection triggers the limit ?
Any hint would help to understand what's happening here. It's difficult as
I don't have direct access to rules or the website of course :)
Olivier
Re: range queries (my favourite)
Le jeu. 28 mai 2020 à 09:48, Willy Tarreau a écrit : > No you're not :-) hdr_cnt() counts *values*. So : > > Range: bytes=0-,0-,0-,0- > > decomposes as the following values around the comma delimiter: > > "bytes=0-", "0-", "0-", "0-" > > And actually if you'd send several Range headers with such values they > could be remerged and interpreted as above. So in this case it's quite > convenient for us. > My bad :( You made me realize I never used correctly this function. I was "counting" duplicate headers with it, and it worked because headers are merged and final behaviour matches what I was expecting. Thank you ! Olivier
Re: range queries (my favourite)
Hello,
Le jeu. 28 mai 2020 à 09:17, Willy Tarreau a écrit :
> http-request del-header range if { req.hdr_cnt(range) gt 1 }
>
This will only filter if header "Range" is present multiple times, not this
one :
Range: bytes=0-,0-,0-,0-
Am I correct ?
Olivier
Re: raise() on HAProxy 2.0
Hello Willy, Le ven. 15 mai 2020 à 17:33, Willy Tarreau a écrit : > > Is it 100% reproducible and if so can you please share a minimal config > and reproducer so that we can quickly focus on it ? > Unfortunately I was unable to reproduce it. It only happens for several hours in a row, then stop (for no reason), then triggers again. And the config file contains ~ 1000 certificates... It did not trigger before upgrade to 2.0.14 (I was using 2.0.13 before). let me know if I can dig into some of the coredump to provide any useful information. Olivier
raise() on HAProxy 2.0
Hello, I'm spamming a lot these days :) I found a strange coredump on HAProxy 2.0.14 that started a few days ago for no reason. It's not a coredump but a raise(). Stacktrace : #0 0x7fde8c9f8495 in raise () from /lib64/libc.so.6 #1 0x7fde8c9f9c75 in abort () from /lib64/libc.so.6 #2 0x7fde8ca363a7 in __libc_message () from /lib64/libc.so.6 #3 0x7fde8ca3bdee in malloc_printerr () from /lib64/libc.so.6 #4 0x7fde8ca3ec3d in _int_free () from /lib64/libc.so.6 #5 0x0047a885 in ssl_sock_free_ssl_conf () at src/ssl_sock.c:3740 #6 0x0047bdd2 in ssl_sock_free_all_ctx () at src/ssl_sock.c:5063 #7 0x0047c301 in ssl_sock_destroy_bind_conf () at src/ssl_sock.c:5095 #8 0x0050c8fb in deinit () at src/haproxy.c:2533 #9 0x0050dc3f in main () at src/haproxy.c:3449 This seems to happen when issuing the following command : 'set ssl ocsp-response xxx' |socat stdio /var/run/haproxy.sock This is the first time I see such a behaviour :/ I can provide a "bt full" output privately if needed. HAProxy build options: Build options : TARGET = linux-glibc CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_NS= Built with OpenSSL version : OpenSSL 1.1.1g 21 Apr 2020 Olivier
Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Hi again, Le mer. 6 mai 2020 à 17:47, Willy Tarreau a écrit : > Hi Olivier, > > On Wed, May 06, 2020 at 05:29:59PM +0200, Olivier D wrote: > > > Try applying this commit: > > > > > > > https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94 > > > > > > So this patch is not working for me, with or without patching Apache2 > with > > https://bz.apache.org/bugzilla/show_bug.cgi?id=63893 > > > > But "good news" : reverting 7f26391bc51 did the trick. > > This is sad. So this means that we've trained external components to > get used to our bugs and consider them to be the default behavior > despite what the doc says. > > > To make sure we are talking about the same things, I've attached both > > commits as patch files. > > - applying 7f26391bc.patch did not fix the issue > > - reverting 02c88036a.patch fixed the issue > > Then this is confusing because 7f2639 was already applied to your tree > and is supposed to be the one causing you the issue, and 02c88 was not > yet so you couldn't revert it. That's why I'd like to have a precise > description of the starting state and what operations you did which > worked and those which didn't. > My bad, I've inverted 7f26391bc and 02c88036a ... I should have used named patches instead of dealing with commit number. So to be clear : I'm using 2.0.14 source code. In this version, patch 7f26391bc is already applied and 02c88036a is not. So applying 02c88036a did nothing (well, it triggers two different non-working behaviour with Apache 2.4 patched, and a single behaviour without the patch on remoteip). And with clean source again, reverting 7f26391bc did the trick (on both Apache 2.4 versions). > > How safe is it to use 02c88036a reverted in production ? > > Either choice is safe for a given component. It's just that we send > wrong information on health checks, forcing other implementations to > implement bugs (see bug #511), that at least Dovecot doesn't support > a "relaxed" approach combining LOCAL with an address, and that from > what you're saying, Apache2 doesn't support LOCAL without an address. > > At this point I guess we'll have to revert all this from older branches > and provide a config option for 2.2+ to re-enable the old behavior for > compatibility with servers that got it wrong the first time. > Yes I understand. That's the price of fame :) HAProxy is now so widely used that softwares are implemented based on what HAProxy does instead of what the specs says. Olivier
Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Hello,
Le mer. 6 mai 2020 à 15:30, Tim Düsterhus a écrit :
> Olivier,
>
> > I was not aware there were any change in the way HAProxy was doing its
> > checks over proxy-protocol in 2.0.14 ... any hint ?
>
> This sounds like this issue we've seen with Dovecot:
> https://www.mail-archive.com/haproxy@formilux.org/msg36890.html
>
> Try applying this commit:
>
> https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94
So this patch is not working for me, with or without patching Apache2 with
https://bz.apache.org/bugzilla/show_bug.cgi?id=63893
But "good news" : reverting 7f26391bc51 did the trick.
To make sure we are talking about the same things, I've attached both
commits as patch files.
- applying 7f26391bc.patch did not fix the issue
- reverting 02c88036a.patch fixed the issue
How safe is it to use 02c88036a reverted in production ?
Olivier
--- src/connection.c
+++ src/connection.c
@@ -1247,6 +1247,7 @@ int make_proxy_line_v2(char *buf, int buf_len, struct
server *srv, struct connec
/* At least one of src or dst is not of AF_INET or AF_INET6 */
if ( !src
|| !dst
+ || conn_is_back(remote)
|| (src->ss_family != AF_INET && src->ss_family != AF_INET6)
|| (dst->ss_family != AF_INET && dst->ss_family != AF_INET6)) {
if (buf_len < PP2_HDR_LEN_UNSPEC)
@@ -1256,14 +1257,7 @@ int make_proxy_line_v2(char *buf, int buf_len, struct
server *srv, struct connec
ret = PP2_HDR_LEN_UNSPEC;
}
else {
- /* Note: due to historic compatibility with V1 which required
-* to send "PROXY" with local addresses for local connections,
-* we can end up here with the remote in fact being our outgoing
-* connection. We still want to send real addresses and LOCAL on
-* it.
-*/
- hdr->ver_cmd = PP2_VERSION;
- hdr->ver_cmd |= conn_is_back(remote) ? PP2_CMD_LOCAL :
PP2_CMD_PROXY;
+ hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
/* IPv4 for both src and dst */
if (src->ss_family == AF_INET && dst->ss_family == AF_INET) {
if (buf_len < PP2_HDR_LEN_INET)
--- src/connection.c
+++ src/connection.c
@@ -1318,11 +1318,18 @@ int make_proxy_line_v2(char *buf, int buf_len, struct
server *srv, struct connec
ret = PP2_HDR_LEN_UNSPEC;
}
else {
+ /* Note: due to historic compatibility with V1 which required
+* to send "PROXY" with local addresses for local connections,
+* we can end up here with the remote in fact being our outgoing
+* connection. We still want to send real addresses and LOCAL on
+* it.
+*/
+ hdr->ver_cmd = PP2_VERSION;
+ hdr->ver_cmd |= conn_is_back(remote) ? PP2_CMD_LOCAL :
PP2_CMD_PROXY;
/* IPv4 for both src and dst */
if (src->ss_family == AF_INET && dst->ss_family == AF_INET) {
if (buf_len < PP2_HDR_LEN_INET)
return 0;
- hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
hdr->fam = PP2_FAM_INET | PP2_TRANS_STREAM;
hdr->addr.ip4.src_addr = ((struct sockaddr_in
*)src)->sin_addr.s_addr;
hdr->addr.ip4.src_port = ((struct sockaddr_in
*)src)->sin_port;
@@ -1336,7 +1343,6 @@ int make_proxy_line_v2(char *buf, int buf_len, struct
server *srv, struct connec
if (buf_len < PP2_HDR_LEN_INET6)
return 0;
- hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
hdr->fam = PP2_FAM_INET6 | PP2_TRANS_STREAM;
if (src->ss_family == AF_INET) {
v4tov6(, &((struct sockaddr_in
*)src)->sin_addr);
Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Hello,
This morning I tried to upgrade HAProxy 2.0.13 to 2.0.14 but had to
rollback immediately : some backends checks started to fail.
Error reported was : SOCKERR - SSL handshake failure
The backends failing have a specific configuration as follows (I removed
anything unnecessary to trigger the issue)
listen webtruc:443
mode tcp
bind X.X.X.X:443
server xxx X.X.X.X:443 check weight 5 send-proxy-v2-ssl-cn check-ssl
verify none
Backend is an Apache 2.4 with "RemoteIPProxyProtocol On".
In apache logs I have :
[remoteip:error] [pid 1067 [client :26847] AH03507:
RemoteIPProxyProtocol: unsupported command 20
I can link this error to this bugreport :
https://bz.apache.org/bugzilla/show_bug.cgi?id=63893
So I applied this patch to Apache 2.4 and then get this error :
HAproxy side: L7STS Bad request
Apache side : RemoteIPProxyProtocol data is missing, but required! Aborting
request.
I was not aware there were any change in the way HAProxy was doing its
checks over proxy-protocol in 2.0.14 ... any hint ?
HAProxy -vv output :
HA-Proxy version 2.0.14 2020/04/02 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE
-PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
-REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
+LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
-MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO -NS +DL +RT -DEVICEATLAS
-51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=20).
Built with OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.44 2020-02-12
Running on PCRE version : 8.44 2020-02-12
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE mux=H2
h2 : mode=HTTP side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Understanding rate-limit sessions
Hello, I was creating counter-measures against a DOS attack, but I failed to understand some numbers I received. I'm using HAProxy 2.0.14 My (expurged) frontend config is : listen test bind X.X.X.X:443 maxconn 65536 rate-limit sessions 128 But during the attack, the following numbers are reported on HAProxy status page : FRONTEND: Session rate: max=1821 ; limit=128 Putting cursor on "max" shows : max connection rate: 2368/s max session rate: 1821/s max request rate: 3901/s I wondered why session-rate exceeded my maximal number of 128 I set on config file. I'm probably not understanding something correctly here. The documentation seems quite clear : "Since the session rate is measured every millisecond, it is extremely accurate" Any clue ? Unfortunately I was not quick enough to record the traffic received, or dump internal HAProxy counters when this happens :( Thank you ! Olivier
Re: [PATCH] Minor improvements to doc "http-request set-src"
Hi,
Le mar. 21 avr. 2020 à 12:56, Tim Düsterhus a écrit :
> Olivier,
>
> PS: Personal opinion, but I prefer quotes in replies to be shortened as
> much as possible, while still providing context. I don't want to scroll
> through kilobytes of stuff I've already seen :-)
>
;)
Patch updated attached.
From e6b11f3a795ec40c8b802d9d1190f3f6bbd15f5d Mon Sep 17 00:00:00 2001
From: Olivier Doucet
Date: Tue, 21 Apr 2020 09:32:56 +0200
Subject: [PATCH] DOC: Improve documentation on http-request set-src
This patch adds more explanation on how to use "http-request set-src"
and a link to "option forwardfor".
This patch can be applied to all previous version starting at 1.6
---
doc/configuration.txt | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git doc/configuration.txt doc/configuration.txt
index 5d01835d7..e695ab7f5 100644
--- doc/configuration.txt
+++ doc/configuration.txt
@@ -5114,16 +5114,23 @@ http-request set-src [ { if | unless }
]
This is used to set the source IP address to the value of specified
expression. Useful when a proxy in front of HAProxy rewrites source IP, but
provides the correct IP in a HTTP header; or you want to mask source IP for
- privacy.
+ privacy. All subsequent calls to "src" fetch will return this value
+ (see example).
Arguments :
Is a standard HAProxy expression formed by a sample-fetch followed
by some converters.
+ See also "option forwardfor".
+
Example:
http-request set-src hdr(x-forwarded-for)
http-request set-src src,ipmask(24)
+# After the masking this will track connections
+# based on the IP address with the last byte zeroed out.
+http-request track-sc0 src
+
When possible, set-src preserves the original source port as long as the
address family allows it, otherwise the source port is set to 0.
--
2.18.0.windows.1
Re: [PATCH] Minor improvements to doc "http-request set-src"
Hello,
Le lun. 20 avr. 2020 à 20:37, Tim Düsterhus a écrit :
> Olivier,
>
> Am 20.04.20 um 20:03 schrieb Olivier D:
> > I'm using gmail so I add to attach patches and was not able to send them
> > directly. If format is wrong, tell me :)
> >
>
> Format looks good to me. Your commit message however does not (fully)
> follow the instructions within the CONTRIBUTING file
> (
> https://github.com/haproxy/haproxy/blob/dfad6a41ad9f012671b703788dd679cf24eb8c5a/CONTRIBUTING#L562-L567
> ):
>
> >As a rule of thumb, your patch MUST NEVER be made only of a subject
> line,
> >it *must* contain a description. Even one or two lines, or indicating
> >whether a backport is desired or not. It turns out that single-line
> commits
> >are so rare in the Git world that they require special manual (hence
> >painful) handling when they are backported, and at least for this
> reason
> >it's important to keep this in mind.
>
> Regarding the patch itself:
>
> > diff --git doc/configuration.txt doc/configuration.txt
> > index 5d01835d7..ddfabcd92 100644
> > --- doc/configuration.txt
> > +++ doc/configuration.txt
> > @@ -6735,7 +6735,8 @@ option forwardfor [ except ] [ header
> ] [ if-none ]
> >header for a known source address or network by adding the "except"
> keyword
> >followed by the network address. In this case, any source IP matching
> the
> >network will not cause an addition of this header. Most common uses
> are with
> > - private networks or 127.0.0.1.
> > + private networks or 127.0.0.1. Another way to do it is to tell
> HAProxy to
> > + trust a custom header with "http-request set-src".
>
> This change looks incorrect to me. "option forwardfor" is for sending,
> not "receiving" IP addresses.
>
> >Alternatively, the keyword "if-none" states that the header will only
> be
> >added if it is not present. This should only be used in perfectly
> trusted
> > @@ -6760,6 +6761,14 @@ option forwardfor [ except ] [ header
> ] [ if-none ]
> > mode http
> > option forwardfor header X-Client
> >
> > + Example :
> > +# Trust a specific header and use it as origin IP.
> > +# If not found, source IP will be used.
> > +frontend www
> > +mode http
> > +http-request set-src CF-Connecting-IP
>
> I believe this should read `http-request set-src
> %[req.hdr(CF-Connecting-IP)]`. However:
>
> 1. I don't like having company specific headers in there. Especially
> since Cloudflare supports the standard XFF.
> 2. I don't consider that a useful addition.
>
> > +option forwardfor
> > +
> >See also : "option httpclose", "option http-server-close",
> > "option http-keep-alive"
> >
>
> Patch 2:
>
> > diff --git doc/configuration.txt doc/configuration.txt
> > index ddfabcd92..49324fa53 100644
> > --- doc/configuration.txt
> > +++ doc/configuration.txt
> > @@ -5114,7 +5114,8 @@ http-request set-src [ { if | unless }
> ]
> >This is used to set the source IP address to the value of specified
> >expression. Useful when a proxy in front of HAProxy rewrites source
> IP, but
> >provides the correct IP in a HTTP header; or you want to mask source
> IP for
> > - privacy.
> > + privacy. All subsequent calls to src field will return this value
> > + (see example).
>
> This change looks good to me.
>
> >Arguments :
> >Is a standard HAProxy expression formed by a sample-fetch
> followed
> > @@ -5124,6 +5125,11 @@ http-request set-src [ { if | unless }
> ]
> > http-request set-src hdr(x-forwarded-for)
> > http-request set-src src,ipmask(24)
> >
> > + Example:
>
> Only a single "Example:" heading is used throughout the documentation.
> As the first line can be shared with the previous example you could
> write something like: # After the masking this will track connections
> based on the IP address with the last octet zeroed out.
>
> > +# This will track connection based on header IP
> > +http-request set-src hdr(x-forwarded-for)
> > +http-request track-sc0 src
> > +
> >When possible, set-src preserves the original source port as long as
> the
> >address family allows it, otherwise the source port is set to 0.
>
Thank you for your valuable feedback. Find attached a new patch will all
your comments taken into account.
Olivier
>
> Best regards
> Tim Düster
[PATCH] Minor improvements to doc "http-request set-src"
Hello,
Find attached two small patches to improve documentation on "option
forwardfor" and "http-request set-src".
I'm using gmail so I add to attach patches and was not able to send them
directly. If format is wrong, tell me :)
Olivier
From efbc320861c9c5a43219983cfc1073070b3e6622 Mon Sep 17 00:00:00 2001
From: Olivier Doucet
Date: Mon, 20 Apr 2020 19:39:27 +0200
Subject: [DOC] This patch adds example on how to use "http-request
set-src" with "option forwardfor".
---
doc/configuration.txt | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git doc/configuration.txt doc/configuration.txt
index 5d01835d7..ddfabcd92 100644
--- doc/configuration.txt
+++ doc/configuration.txt
@@ -6735,7 +6735,8 @@ option forwardfor [ except ] [ header ]
[ if-none ]
header for a known source address or network by adding the "except" keyword
followed by the network address. In this case, any source IP matching the
network will not cause an addition of this header. Most common uses are with
- private networks or 127.0.0.1.
+ private networks or 127.0.0.1. Another way to do it is to tell HAProxy to
+ trust a custom header with "http-request set-src".
Alternatively, the keyword "if-none" states that the header will only be
added if it is not present. This should only be used in perfectly trusted
@@ -6760,6 +6761,14 @@ option forwardfor [ except ] [ header ]
[ if-none ]
mode http
option forwardfor header X-Client
+ Example :
+# Trust a specific header and use it as origin IP.
+# If not found, source IP will be used.
+frontend www
+mode http
+http-request set-src CF-Connecting-IP
+option forwardfor
+
See also : "option httpclose", "option http-server-close",
"option http-keep-alive"
--
2.18.0.windows.1
From 34efa737cf09753301787dde7dc77df2041b3288 Mon Sep 17 00:00:00 2001
From: Olivier Doucet
Date: Mon, 20 Apr 2020 19:59:43 +0200
Subject: [DOC] add useful informations on "http-request set-src"
---
doc/configuration.txt | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git doc/configuration.txt doc/configuration.txt
index ddfabcd92..49324fa53 100644
--- doc/configuration.txt
+++ doc/configuration.txt
@@ -5114,7 +5114,8 @@ http-request set-src [ { if | unless }
]
This is used to set the source IP address to the value of specified
expression. Useful when a proxy in front of HAProxy rewrites source IP, but
provides the correct IP in a HTTP header; or you want to mask source IP for
- privacy.
+ privacy. All subsequent calls to src field will return this value
+ (see example).
Arguments :
Is a standard HAProxy expression formed by a sample-fetch followed
@@ -5124,6 +5125,11 @@ http-request set-src [ { if | unless }
]
http-request set-src hdr(x-forwarded-for)
http-request set-src src,ipmask(24)
+ Example:
+# This will track connection based on header IP
+http-request set-src hdr(x-forwarded-for)
+http-request track-sc0 src
+
When possible, set-src preserves the original source port as long as the
address family allows it, otherwise the source port is set to 0.
--
2.18.0.windows.1
Re: HAProxy concurrent HTTP query limit based on header
Le ven. 17 avr. 2020 à 20:49, Tim Düsterhus a écrit :
> Olivier,
>
> Am 17.04.20 um 20:22 schrieb Olivier D:
> > My first tries are based on something like this :
> >stick-table type ipv6 size 100k expire 30s store http_req_rate(10s)
> Not sure whether that's just an error in your email, but: You store a
> http_req_rate, not a number of connections.
>
You are correct, last test was
stick-table type ipv6 size 100k expire 30s store conn_cur
but It seems to not do what I want.
I'll check again on monday with some rest :)
Olivier
>
> >http-request track-sc0 req.hdr( X-Forwarded-For )
> >http-request deny deny_status 429 if { sc0_conn_cur ge 20 }
> >
>
> Best regards
> Tim Düsterhus
>
HAProxy concurrent HTTP query limit based on header
Hello everyone,
I would like to implement a "max concurrent connection" in HAProxy. This is
easy to do at TCP level :
stick-table type ipv6 size 100k expire 30s store conn_cur
http-request track-sc0 src
http-request deny deny_status 429 if { src_conn_cur ge 20 }
But now, I want to do the same for concurrent HTTP queries, based on header
'X-Forwarded-For'. For example, I want to send a 429 error code if someone
is sending an HTTP query when he already have 20 ongoing.
My first tries are based on something like this :
stick-table type ipv6 size 100k expire 30s store http_req_rate(10s)
http-request track-sc0 req.hdr( X-Forwarded-For )
http-request deny deny_status 429 if { sc0_conn_cur ge 20 }
but it doesn't seem to work the way I want ...
Now I'm a bit lost, but maybe someone already implemented this ?
Thank you !
Olivier
Segfault with HAProxy 2.0 with peers
Hello,
With latest haproxy 2.0, you can generate a simple segfault with only
configuration test (haproxy -f test.cfg -c)
Config content :
--
defaults
mode http
backend test
stick-table type ip size 10k expire 1h store http_req_rate(1h) peers
mypeers
peers mypeers
peer toto 127.0.0.1:1024
-
Running :
[WARNING] 083/173758 (2599) : Removing incomplete section 'peers mypeers'
(no peer named 'myhostname.localhost').
Segmentation fault
peers_register_table (peers=0xc8a110, table=table@entry=0xc8c6f0) at
src/peers.c:3028
(gdb) bt
#0 peers_register_table (peers=0xc8a110, table=table@entry=0xc8c6f0) at
src/peers.c:3028
#1 0x00522bec in stktable_init (t=0xc8c6f0) at
src/stick_table.c:644
#2 0x0050fa67 in check_config_validity () at src/cfgparse.c:4048
#3 0x00505311 in init (argc=, argc@entry=4,
argv=, argv@entry=0x7fffe848) at src/haproxy.c:1760
#4 0x0046475f in main (argc=4, argv=0x7fffe848) at
src/haproxy.c:2727
I know my peer entry is not correct (my hostname is not "toto") but we
should not end with a segfault ^^
HAPRoxy -vv :
HA-Proxy version 2.0.13 2020/02/13 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE
-PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
-REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
+LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
-MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO -NS +DL +RT -DEVICEATLAS
-51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=20).
Built with OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.44 2020-02-12
Running on PCRE version : 8.44 2020-02-12
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE mux=H2
h2 : mode=HTTP side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Olivier
Re: Segfault on HAProxy 2.0.11 on HTX mode
Le mer. 19 févr. 2020 à 16:24, Christopher Faulet a
écrit :
> Le 19/02/2020 à 16:05, Olivier D a écrit :
> > A bug was fixed in 2.0.12 that could explain such of crashes. The
> upstream
> > commit id is eec7f8ac0 (or 0ed1e8963 in the 2.0 tree). It is related
> to the
> > GitHub issue #420.
> >
> > But I don't know if it is the same bug because I don't know how it
> is possible
> > to apply an HTTP load-balancing algo on a TCP backend. I must take a
> look at
> > your configuration. You can send it to me in private. Maybe I'll
> found
> > something
> > explaining your crashes.
> >
> >
> > I have hundreds of frontend/backends in this config. What made you think
> this is
> > related to a tcp backend ? That would help me a lot.
> >
> >
>
> Because the mentioned commit fixes a bug where it was possible to assign a
> TCP
> backend to an HTX stream. It is possible to hit this bug when dynamic
> rules are
> used to choose the backend. In such case, we are unable to detect bad
> configuration during HAProxy startup.
>
We do use some use_backend if {}, but only on http frontends (I checked).
Never on tcp.
We have a mix between "listen" blocks with "server" defined inside, and
some frontend/backend blocks. So one "listen" block may also have a
"use_backend if".
Yes, it's bad, but it has been auto-generated since we use HAProxy 1.5 and
we never rewrite this part.
So, if you have TCP frontends that can be dynamically routed to HTTP or TCP
> backends, you may hit the bug. See github issue #420.
>
I don't think it is this one. Our only tcp frontends are all formated like
this :
listen x
id 20609
bind-process 18
balance source
hash-type consistent
mode tcp
bind X.X.X.X:443
server s1 X.X.X.X:443 id 4567 check weight 5 send-proxy-v2-ssl-cn
check-ssl verify none
server s2 X.X.X.X:443 id 1234 check weight 5 send-proxy-v2-ssl-cn
check-ssl verify none
> There is another source of bugs. In HAProxy 2.0, the HTX mode is not
> enabled by
> default. If you have dynamic routing rules, be careful to have the same
> mode
> (legacy or HTX) everywhere. I will do some tests to be sure this case it
> properly handled.
>
I thought HTX was default mode since 2.0-dev3 (
https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#no%20option%20http-use-htx
)
We don't have custom config on this, so default mode was used everywhere.
>
>
> > Did you make any recent changes on HAproxy or your servers ? I'm
> surprised the
> > segaults appear spontaneously after 2 months without any problem.
> >
> >
> > Only minor modifications in the last few days ...
>
> minor modifications may have huge impact especially if you hit an hidden
> bug :)
>
Config file is auto-generated from a central server, so we always add
frontends, backends or certificates. That's all.
I can send you the config file, but it's 8k lines, so it wont help you much
I guess. Can the coredump help you more, with the binary used ?
Olivier
>
> --
> Christopher Faulet
>
Re: Segfault on HAProxy 2.0.11 on HTX mode
Hello, Le mer. 19 févr. 2020 à 15:27, Christopher Faulet a écrit : > Le 19/02/2020 à 11:35, Olivier D a écrit : > > Hello, > > > > I would like to report a segfault on HAProxy 2.0.11 ; this version has > been > > running fine for two months, and this morning starting segfaulting over > and over. > > Mitigation was performed by adding "no option http-use-htx" on > 'defaults' block. > > > > I know it's not the latest version :) I'll update to 2.0.13 this evening. > > > > Program terminated with signal 11, Segmentation fault. > > #0 htx_sl_p2 (sl=) at include/common/htx.h:293 > > 293 include/common/htx.h: No such file or directory. > > (gdb) bt > > #0 htx_sl_p2 (sl=) at include/common/htx.h:293 > > #1 htx_sl_req_uri (sl=) at include/common/htx.h:308 > > #2 assign_server (s=0xdc139f0) at src/backend.c:746 > > #3 0x00552114 in assign_server_and_queue (s=s@entry=0xdc139f0) > at > > src/backend.c:977 > > #4 0x005556f8 in assign_server_and_queue (s=0xdc139f0) at > > src/backend.c:1772 > > #5 srv_redispatch_connect (s=s@entry=0xdc139f0) at src/backend.c:1705 > > #6 0x004c2cf8 in sess_prepare_conn_req (s=) at > > src/stream.c:1250 > > #7 process_stream (t=t@entry=0xd1db790, context=0xdc139f0, > state= > out>) at src/stream.c:2414 > > #8 0x00594865 in process_runnable_tasks () at src/task.c:412 > > #9 0x005038f7 in run_poll_loop () at src/haproxy.c:2520 > > #10 run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2641 > > #11 0x004653b0 in main (argc=, > argv=0x7fff848ae498) at > > src/haproxy.c:3318 > > > > > > > > Config file is very long ... If needed, a coredump + binary can be sent > on private. > > > > Hi, > > A bug was fixed in 2.0.12 that could explain such of crashes. The upstream > commit id is eec7f8ac0 (or 0ed1e8963 in the 2.0 tree). It is related to > the > GitHub issue #420. > > But I don't know if it is the same bug because I don't know how it is > possible > to apply an HTTP load-balancing algo on a TCP backend. I must take a look > at > your configuration. You can send it to me in private. Maybe I'll found > something > explaining your crashes. > I have hundreds of frontend/backends in this config. What made you think this is related to a tcp backend ? That would help me a lot. > > Did you make any recent changes on HAproxy or your servers ? I'm surprised > the > segaults appear spontaneously after 2 months without any problem. > Only minor modifications in the last few days ... I'll update to latest haproxy version to check. Olivier > > > -- > Christopher Faulet >
Segfault on HAProxy 2.0.11 on HTX mode
Hello,
I would like to report a segfault on HAProxy 2.0.11 ; this version has been
running fine for two months, and this morning starting segfaulting over and
over.
Mitigation was performed by adding "no option http-use-htx" on 'defaults'
block.
I know it's not the latest version :) I'll update to 2.0.13 this evening.
Program terminated with signal 11, Segmentation fault.
#0 htx_sl_p2 (sl=) at include/common/htx.h:293
293 include/common/htx.h: No such file or directory.
(gdb) bt
#0 htx_sl_p2 (sl=) at include/common/htx.h:293
#1 htx_sl_req_uri (sl=) at include/common/htx.h:308
#2 assign_server (s=0xdc139f0) at src/backend.c:746
#3 0x00552114 in assign_server_and_queue (s=s@entry=0xdc139f0) at
src/backend.c:977
#4 0x005556f8 in assign_server_and_queue (s=0xdc139f0) at
src/backend.c:1772
#5 srv_redispatch_connect (s=s@entry=0xdc139f0) at src/backend.c:1705
#6 0x004c2cf8 in sess_prepare_conn_req (s=) at
src/stream.c:1250
#7 process_stream (t=t@entry=0xd1db790, context=0xdc139f0,
state=) at src/stream.c:2414
#8 0x00594865 in process_runnable_tasks () at src/task.c:412
#9 0x005038f7 in run_poll_loop () at src/haproxy.c:2520
#10 run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2641
#11 0x004653b0 in main (argc=, argv=0x7fff848ae498)
at src/haproxy.c:3318
haproxy -vv:
HA-Proxy version 2.0.11 2019/12/11 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE
-PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
-REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
+LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
-MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO -NS +DL +RT -DEVICEATLAS
-51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=20).
Built with OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.43 2019-02-23
Running on PCRE version : 8.43 2019-02-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE mux=H2
h2 : mode=HTTP side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Config file is very long ... If needed, a coredump + binary can be sent on
private.
Olivier
Re: PROXY protocol and check port
Hello, Le mar. 17 déc. 2019 à 11:11, Willy Tarreau a écrit : > Hi Olivier, > > On Tue, Dec 17, 2019 at 09:20:21AM +0100, Olivier D wrote: > > That's not what I was saying. I'm already using "show server state", and > > that's exactly what leads me to hours of debugging : between two versions > > of my haproxy config file, I changed backend server port from 80 to 443. > > When HAProxy reloaded, it loaded server state file and loaded both the > > previous state(up/down) of the server, but also the server port. That's > why > > HAProxy never used port 443 as backend port and was still using old port > 80. > > This is not clearly stated in the configuration (or maybe I missed it ?), > > that's why I was asking whether it is an expected behaviour. In my mind, > > the server state was only used to get server status between two reload, > not > > more informations. > > Bah, state-file strikes again :-( > > We predicted exactly this when it was designed. Some people complain that > their changes are lost and others that their changes are kept. We discussed > this already with Baptiste and we figured that the only solution to this is > to change the format to include, for each and every field, *both* the > config > config state and the current state, so that upon reload, the state file > parser can detect what changed from previous running instance: if the > config > changed, you want the config change to be taken into account. If the config > didn't change, then very likely you want to keep the previous state. > > It's a really tricky thing because most of the time a reload is made in > order to apply a change, but this change may conflict with what is in the > state file in a way that's not that determinist :-/ Personally I don't like > the idea of having two authoritative sources for a single configuration, > which is why we must make this state file processing stricter and its > contents more detailed. > > It's worth noting that some people are already abusing it by writing > their changes directly to the state file and avoid touching the config > (typically just have a server-template directive in the config and all > the details in the state file). This shows how tricky the situation can > become every time we try to address one's needs and break someonee else's > :-/ > > Willy > Ah! thank you for the explanation ! Just explaining this in doc should be enough. I always thought that only the state (UP/DOWN/MAINT) was loaded from that file. What would you think of the following modification ? server-state-file Specifies the path to the file containing state of servers. [...] Please note that when we say "state of servers", it means not only server status (up/down/maint) but also IP/port. [...] if "state-only" is present, then only server state (up/down/maint) will be reloaded. IP/port will be filled with what is provided in config file. Well, like this it's quite ugly but you see my point :) It's backward-compatible, and you can enforce the behaviour you expect. Olivier
Re: PROXY protocol and check port
Hello Igor, Le lun. 16 déc. 2019 à 23:41, Igor Cicimov a écrit : > Hi, > > On Tue, Dec 17, 2019 at 2:55 AM Olivier D wrote: > >> Hello, >> >> I found what was wrong : I was using "load-server-state-from-file" and >> previous config file was using port 80 as server port. >> It seems using this instruction loads previous server state but also >> previous srv_port. >> Is this an expected behaviour ? >> > > Yes, basically it is your responsibility to dump the current state into > the file otherwise you'll gate outdated data as you noticed. For example I > add: > > ExecReload=/bin/echo "show servers state" | /usr/bin/socat stdio > /run/haproxy/admin.sock > /var/lib/haproxy/state > > in the systemd service file. > That's not what I was saying. I'm already using "show server state", and that's exactly what leads me to hours of debugging : between two versions of my haproxy config file, I changed backend server port from 80 to 443. When HAProxy reloaded, it loaded server state file and loaded both the previous state(up/down) of the server, but also the server port. That's why HAProxy never used port 443 as backend port and was still using old port 80. This is not clearly stated in the configuration (or maybe I missed it ?), that's why I was asking whether it is an expected behaviour. In my mind, the server state was only used to get server status between two reload, not more informations. Olivier
Re: PROXY protocol and check port
Hello, I found what was wrong : I was using "load-server-state-from-file" and previous config file was using port 80 as server port. It seems using this instruction loads previous server state but also previous srv_port. Is this an expected behaviour ? Olivier Le ven. 13 déc. 2019 à 18:32, Olivier D a écrit : > Hello all, > I struggle with what seemed a very easy config : > > listen test:443 > id 20609 > bind-process 16 > balance source > hash-type consistent > mode tcp > bind x.x.x.x:443 > server s1 192.168.x.x:443 id 2158 check weight 5 send-proxy port 80 > server s2 192.168.x.x:443 id 2168 check weight 5 send-proxy port 80 > > I expect traffic to be routed to port 443 of backend server, and checks to > be performed without proxy-protocol on port 80. tests seem ok, but > traffic seems also routed to port 80 (I checked with tcpdump) instead of > 443. > > Can you at least confirm that my expectations are correct ? Or am I just > very tired on a friday evening ? > > I'm using HAProxy 1.9.12 > > Thank you ! >
PROXY protocol and check port
Hello all, I struggle with what seemed a very easy config : listen test:443 id 20609 bind-process 16 balance source hash-type consistent mode tcp bind x.x.x.x:443 server s1 192.168.x.x:443 id 2158 check weight 5 send-proxy port 80 server s2 192.168.x.x:443 id 2168 check weight 5 send-proxy port 80 I expect traffic to be routed to port 443 of backend server, and checks to be performed without proxy-protocol on port 80. tests seem ok, but traffic seems also routed to port 80 (I checked with tcpdump) instead of 443. Can you at least confirm that my expectations are correct ? Or am I just very tired on a friday evening ? I'm using HAProxy 1.9.12 Thank you !
Setting SSL/TLS options but still allow some exceptions
Hello, I'm rewriting a complex HAProxy config file and would like to be sure how ssl-default-bind-options and bind options work together. I would like to configure safe options by default, but still allow less-safe protocols on some frontend. I'm puzzled by "force-X" documentation (does it really "force" protocol or just allow it ? What if I use several force-X options all together ?) and want to be sure of the behaviour. Here is what I would like to do : frontend foo : supports TLS 1.2 and TLS 1.3 frontend foo-unsecure : supports everything from sslv3 to TLS 1.3 frontend foo-unsecure2 : supports TLS 1.1 to TLS 1.3 And here is how I would write it down : # Default (safe) config : ssl-default-bind-options no-sslv3 no-tls10 no-tls11 frontend foo bind 127.0.0.1:8080 ssl frontend foo-unsecure bind 127.0.0.1:1234 ssl force-sslv3 force-tls10 force-tls11 frontend foo-unsecure2 bind 127.0.0.1:4321 ssl force-tls11 I dont want to use 'ssl-min-ver' or 'ssl-max-ver' because the config file is auto-generated from a database, and it would make the code more difficult. Thank you for your feedback. Olivier
Segfaults with 1.9.6
Hello,
I know I'm reporting an issue with an old version, but I got 2 segfaults
in 48h.
As I only got 3 segfaults with HAProxy in +10 years, I just wanted to make
sure these bugs have been caught and are now fixed.
haproxy -vv output:
HA-Proxy version 1.9.6 2019/03/29 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-format-truncation -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers
-Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019
Running on OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.41 2017-07-05
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE
h2 : mode=HTTP side=FE
: mode=HTXside=FE|BE
: mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
### First segfault : ###
Program terminated with signal 11, Segmentation fault.
#0 0x004cba32 in h2_process_mux (h2c=0x9b4b300) at
src/mux_h2.c:2588
(gdb) bt full
#0 0x004cba32 in h2_process_mux (h2c=0x9b4b300) at
src/mux_h2.c:2588
h2s = 0x98edf50
#1 h2_send (h2c=h2c@entry=0x9b4b300) at src/mux_h2.c:2716
flags =
conn = 0x9aef030
done = 0
sent = 0
#2 0x004d3918 in h2_io_cb (t=, ctx=0x9b4b300,
status=) at src/mux_h2.c:2778
h2c = 0x9b4b300
ret = 0
#3 0x00584456 in process_runnable_tasks () at src/task.c:437
t = 0x9e15170
state =
ctx =
process =
t =
max_processed = 194
#4 0x00503fd4 in run_poll_loop () at src/haproxy.c:2642
next =
exp =
#5 run_thread_poll_loop (data=data@entry=0x19a32b0) at src/haproxy.c:2707
ptif =
ptdf =
start_lock = 0
#6 0x004648d8 in main (argc=, argv=0x7ffccfb0cba8)
at src/haproxy.c:3343
tids = 0x19a32b0
threads = 0x19a2750
i =
old_sig = {__val = {68097, 0, 64, 206158430210, 532575944795,
472446402679, 0, 139791683256608, 24, 11381472, 335544638, 11392704,
26776016, 139791680031404, 0, 26699504}}
blocked_sig = {__val = {1844674406710583, 18446744073709551615
}}
err =
retry =
limit = {rlim_cur = 801167, rlim_max = 801167}
errmsg =
"\000\000\000\000\000\000\000\000\220Ap\312#\177\000\000\000\357\200\000\000\000\000\000(\357\200\000\000\000\000\000\231\353\200\000\000\000\000\000\000\000\000\000\002",
'\000' "\350,
Dp\312#\177\000\000p\311\260\317\374\177\000\000\035\000\000\000\000\000\000\000\210\311\260\317\374\177\000\000
\326\230\001\001\000\000\000\000v\000"
pidfd =
### Second segfault ###
Program terminated with signal 11, Segmentation fault.
#0 0x005808b5 in __pendconn_unlink (p=p@entry=0x7fff694b0730) at
src/queue.c:138
(gdb) bt full
#0 0x005808b5 in __pendconn_unlink (p=p@entry=0x7fff694b0730) at
src/queue.c:138
No locals.
#1 0x00581507 in pendconn_redistribute (s=s@entry=0x6b01cd0) at
src/queue.c:413
p = 0x7fff694b0730
node = 0xb781a88
#2 0x004ee2b2 in srv_update_status (s=s@entry=0x6b01cd0) at
src/server.c:4805
next_admin =
check = 0x6b02170
xferred =
px = 0x6a357e0
prev_srv_count = 2
srv_was_stopping =
log_level =
tmptrash = 0x0
#3 0x004eef04 in srv_set_stopped (s=0x6b01cd0,
reason=reason@entry=0x0,
check=) at src/server.c:1016
srv =
#4 0x004eefc1 in srv_set_stopped (s=,
reason=reason@entry=0x0, check=) at
Setting SSL/TLS options but still allow some exceptions
Hello, I'm rewriting a complex HAProxy config file and would like to be sure how ssl-default-bind-options and bind options work together. I would like to configure safe options by default, but still allow less-safe protocols on some frontend. I'm puzzled by "force-X" documentation (does it really "force" protocol or just allow it ? What if I use several force-X options all together ?) and want to be sure of the behaviour. Here is what I would like to do : frontend foo : supports TLS 1.2 and TLS 1.3 frontend foo-unsecure : supports everything from sslv3 to TLS 1.3 frontend foo-unsecure2 : supports TLS 1.1 to TLS 1.3 And here is how I would write it down : # Default (safe) config : ssl-default-bind-options no-sslv3 no-tls10 no-tls11 frontend foo bind 127.0.0.1:8080 ssl frontend foo-unsecure bind 127.0.0.1:1234 ssl force-sslv3 force-tls10 force-tls11 frontend foo-unsecure2 bind 127.0.0.1:4321 ssl force-tls11 I dont want to use 'ssl-min-ver' or 'ssl-max-ver' because the config file is auto-generated from a database, and it would make the code more difficult. Thank you for your feedback. Olivier
Re: Idea + question regarding the build targets
Hi,
Le mer. 12 juin 2019 à 19:19, Willy Tarreau a écrit :
> Hi guys,
>
> On Wed, Jun 12, 2019 at 04:27:42PM +0200, Lukas Tribus wrote:
> (...)
> > I think it's a bad idea.
> >
> > Basically what Tim says (I was interrupted several times while writing
> > this email).
>
> OK, and this morning William told me he thought the same for the same
> reasons, so definitely I'm the one wrong here, which justifies that I
> asked. And I totally agree with your arguments.
>
> (...)
> > Those will be the interactions, with a lot of round-trips back and
> > forth. At least the user understands that USE_LUA=1 means that LUA
> > will be included. With TARGET=ubuntu18 the user may not even know what
> > LUA is, let alone that it's a dependency.
>
> I was actually thinking about enabling what I expect to be installed
> by default, i.e. zlib & openssl mainly. But at first I didn't want to
> address libraries as much as the default OS which involves a minimal
> kernel version and a libc. In the past we've had to guess quite a few
> times some settings, which were nicely addressed by the split on kernel
> 2.6.28, but it seems cleaner to support a kernel+libc combination, which
> is what made me think about distros, hence their respective libs.
>
> > So do we brake the build by
> > default and just document in INSTALL that in Ubuntu 18.04 the LUA
> > dependency must be fulfilled by installing the source package, which
> > can be either liblua5.1-0-dev, liblua5.2-dev or liblua5.3-dev?
>
> I agree.
>
> > INSTALL already contains suggestions about what USE_ flags to include,
> > as a matter of fact all 5 external libraries mentioned above are
> > suggested:
> >
> >
> http://git.haproxy.org/?p=haproxy.git;a=blob;f=INSTALL;h=9df17caf68c4f00cdaaaec2ec929f0af66e1d297;hb=HEAD#l35
> >
> >
> > To be honest, I don't think this is very common in OSS projects; most
> > of them use configure scripts that just include the library if the
> > headers are detected, or not link against it at all if it isn't there.
> > But we would brake the build here, which is different.
>
> OK. When discussing this with William, we figured it could be interesting
> instead to have some aliases which are maybe more symbolic, such as :
>
> - linux-complete : full set of supported features, will simply fail
> if you don't have all libs installed, useful primarily for devs ;
>
> - linux-recent : common set of features present by default on latest
> release of LTS distros at the time of releasing. I.e. could be the
> default if you have a reasonably up-to-date system ;
>
> - linux-ancient : common set of features present by default on oldest
> supported LTS distros at the time of releasing. Should be a safe
> fallback for everyone who doesn't want to care more ;
>
> - linux-minimal : basically just oldest supported LTS kernel + equivalent
> libc, would serve as a starting point to manually add USE_*..
>
> In fact I'm still a bit concerned by the fact that we continue to
> advertise 2.6.28 as the split point while no older kernels are still
> supported, and that some of the features placed there very likely still
> don't work well in embedded environments (openwrt for example).
>
> I'd like very much to get rid of this laughing "2.6.28" now but if we
> only use "linux" nobody will update it and we'll be back to the same
> situation in one or two versions. With the split above we can have
> fast moving targets ("complete", updated during dev; "recent", updated
> with new distro announces) and slow moving ones ("ancient", "minimal")
> and that might be a sweet spot.
>
> > > just like we already have for solaris/openbsd/freebsd for example
> >
> > None of these include external libraries though, like OpenSSL, LUA,
> > zlib, PCRE1/2. Only kernel features and "very core" libraries are
> > included with the solaris/BSD targets (just like linux2628). So the
> > build may brake because a basic and crucial threading lib is missing,
> > but not because LUA is not there. That's very different kind of build
> > failure.
>
> Agreed, probably that I was a bit too enthousiast by this idea and
> conflated several things. We should probably get back to platform and
> features separately. openssl, lua, zlib, pcre are in fact features if
> they are not present by default. I was happy to place some of them by
> default but if they are not necessarily present, let's not force :-)
>
> thanks guys for fueling the discussion.
>
> Willy
>
Sorry if I'm totally "out" on this point, but I was wondering why HAProxy
did not provide a simple "configure" script : all available options would
be listed (I know it's in the doc, but hey, how many people do actually
read it ? :p). and script will check if options chosen can be compiled (if
pcre2 is available, if openssl-devel is available, and so on ...).
You may then pick a default config, for example SSL always compiled in, LUA
not, ... if no other option are provided.
Olivier
Re: HAProxy compilation issue
Hello, Le sam. 12 janv. 2019 à 13:19, Willy Tarreau a écrit : > Hi Olivier, > > On Wed, Jan 09, 2019 at 07:23:42PM +0100, Olivier D wrote: > > Hello folks, > > > > Just wanted to raise an issue with a compilation error on HAProxy that I > > was able to solve by myself. Just wanted to know if this issue is > > haproxy-related or compiler-related (and if a fix should be provided in > the > > future) > > > > Compiling haproxy (1.8.17) failed with this error : > (...) > > /usr/src/haproxy/opensslbin/lib/libcrypto.a(threads_pthread.o): In > function > > `fork_once_func': > > threads_pthread.c:(.text+0x16): undefined reference to `pthread_atfork' > > collect2: error: ld returned 1 exit status > > It's been a while since we've got such linking issues. Usually they come > from libpthread or libdl, which are most always shared. Can you please > try the attached patch ? > Tested and approved :) compilation is working correctly \o/ Olivier > > Thanks, > Willy >
HAProxy compilation issue
Hello folks, Just wanted to raise an issue with a compilation error on HAProxy that I was able to solve by myself. Just wanted to know if this issue is haproxy-related or compiler-related (and if a fix should be provided in the future) Compiling haproxy (1.8.17) failed with this error : make TARGET=linux2628 USE_STATIC_PCRE=1 USE_ZLIB=1 USE_OPENSSL=1 ZLIB_LIB=/usr/src/haproxy/zlibbin/lib ZLIB_INC=/usr/src/haproxy/zlibbin/include SSL_INC=/usr/src/haproxy/opensslbin/include SSL_LIB=/usr/src/haproxy/opensslbin/lib ADDLIB=-ldl -lzlib PCREDIR=/usr/src/haproxy/pcrebin USE_LUA=1 LUA_LIB=/usr/src/haproxy/lua/lib LUA_INC=/usr/src/haproxy/lua/include [...] gcc -g -o haproxy src/ev_poll.o src/ev_epoll.o src/ssl_sock.o src/hlua.o src/hlua_fcn.o ebtree/ebtree.o ebtree/eb32sctree.o ebtree/eb32tree.o ebtree/eb64tree.o ebtree/ebmbtree.o ebtree/ebsttree.o ebtree/ebimtree.o ebtree/ebistree.o src/proto_http.o src/cfgparse.o src/server.o src/stream.o src/flt_spoe.o src/stick_table.o src/stats.o src/mux_h2.o src/checks.o src/haproxy.o src/log.o src/dns.o src/peers.o src/standard.o src/sample.o src/cli.o src/stream_interface.o src/proto_tcp.o src/backend.o src/proxy.o src/tcp_rules.o src/listener.o src/flt_http_comp.o src/pattern.o src/cache.o src/filters.o src/vars.o src/acl.o src/payload.o src/connection.o src/raw_sock.o src/proto_uxst.o src/flt_trace.o src/session.o src/ev_select.o src/channel.o src/task.o src/queue.o src/applet.o src/map.o src/frontend.o src/freq_ctr.o src/lb_fwlc.o src/mux_pt.o src/auth.o src/fd.o src/hpack-dec.o src/memory.o src/lb_fwrr.o src/lb_chash.o src/lb_fas.o src/hathreads.o src/chunk.o src/lb_map.o src/xxhash.o src/regex.o src/shctx.o src/buffer.o src/action.o src/h1.o src/compression.o src/pipe.o src/namespace.o src/sha1.o src/hpack-tbl.o src/hpack-enc.o src/uri_auth.o src/time.o src/proto_udp.o src/arg.o src/signal.o src/protocol.o src/lru.o src/hdr_idx.o src/hpack-huff.o src/mailers.o src/h2.o src/base64.o src/hash.o -lcrypt -L/usr/src/haproxy/zlibbin/lib -lz -ldl -lpthread -L/usr/src/haproxy/opensslbin/lib -lssl -lcrypto -ldl -Wl,--export-dynamic -L/usr/src/haproxy/lua/lib -llua -lm -ldl -L/usr/src/haproxy/pcrebin/lib -Wl,-Bstatic -lpcreposix -lpcre -Wl,-Bdynamic -ldl /usr/src/haproxy/opensslbin/lib/libcrypto.a(threads_pthread.o): In function `fork_once_func': threads_pthread.c:(.text+0x16): undefined reference to `pthread_atfork' collect2: error: ld returned 1 exit status Seems this error already happened to someone here : https://github.com/openssl/openssl/issues/3884 So I modified by hand gcc line to move -lpthread at the end of the line like this : gcc -g -o haproxy src/ev_poll.o src/ev_epoll.o src/ssl_sock.o src/hlua.o src/hlua_fcn.o ebtree/ebtree.o ebtree/eb32sctree.o ebtree/eb32tree.o ebtree/eb64tree.o ebtree/ebmbtree.o ebtree/ebsttree.o ebtree/ebimtree.o ebtree/ebistree.o src/proto_http.o src/cfgparse.o src/server.o src/stream.o src/flt_spoe.o src/stick_table.o src/stats.o src/mux_h2.o src/checks.o src/haproxy.o src/log.o src/dns.o src/peers.o src/standard.o src/sample.o src/cli.o src/stream_interface.o src/proto_tcp.o src/backend.o src/proxy.o src/tcp_rules.o src/listener.o src/flt_http_comp.o src/pattern.o src/cache.o src/filters.o src/vars.o src/acl.o src/payload.o src/connection.o src/raw_sock.o src/proto_uxst.o src/flt_trace.o src/session.o src/ev_select.o src/channel.o src/task.o src/queue.o src/applet.o src/map.o src/frontend.o src/freq_ctr.o src/lb_fwlc.o src/mux_pt.o src/auth.o src/fd.o src/hpack-dec.o src/memory.o src/lb_fwrr.o src/lb_chash.o src/lb_fas.o src/hathreads.o src/chunk.o src/lb_map.o src/xxhash.o src/regex.o src/shctx.o src/buffer.o src/action.o src/h1.o src/compression.o src/pipe.o src/namespace.o src/sha1.o src/hpack-tbl.o src/hpack-enc.o src/uri_auth.o src/time.o src/proto_udp.o src/arg.o src/signal.o src/protocol.o src/lru.o src/hdr_idx.o src/hpack-huff.o src/mailers.o src/h2.o src/base64.o src/hash.o -lcrypt -L/usr/src/haproxy/zlibbin/lib -lz -ldl -L/usr/src/haproxy/opensslbin/lib -lssl -lcrypto -ldl -Wl,--export-dynamic -L/usr/src/haproxy/lua/lib -llua -lm -ldl -L/usr/src/haproxy/pcrebin/lib -Wl,-Bstatic -lpcreposix -lpcre -Wl,-Bdynamic -ldl -lpthread ... and compilation was OK ! I tried with gcc 4.8.2 (centos6) and gcc 7.2.1 with exactly the same behaviour. Additional softwares used : OpenSSL 1.1.1a Lua 5.3.5 PCRE version : 8.41 zlib version : 1.2.11 Olivier
