Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour
Hello, On 23 May 2018 at 22:17, Jim Freeman wrote: > Or kludge around it with eg; http://www.issihosts.com/haveged/ ? No, it's not about insufficient entropy in the kernel. It's about interfacing with that entropy while in chroot. Lukas
Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour
Or kludge around it with eg; http://www.issihosts.com/haveged/ ? On Wed, May 23, 2018 at 1:48 PM, Lukas Tribus wrote: > Hello, > > > On 23 May 2018 at 18:29, Emeric Brun wrote: > > This issue was due to openssl-1.1.1 which re-seed after an elapsed time > or number of request. > > > > If /dev/urandom is used as seeding source when haproxy is chrooted it > fails to re-open /dev/urandom > > > > By defaut the openssl-1.1.1 configure script uses the syscall getrandom > as seeding source and fallback on /dev/urandom if not available. > > > > So you don't face the issue if your openssl-1.1.1 is compiled to use > getrandom > > > > But getrandom syscall is available only since kernel > 3.17 and the main > point: for glibc > 2.25. > > > > With openssl-1.1.1 you can check this this way: > > # ./openssl-1.1.1/openssl version -r > > Seeding source: getrandom-syscall > > I have glibc 2.23 (Ubuntu 16.04) and openssl shows "os-specific", even > if kernel headers are installed while compiling, yet -pre6 does not > hang for me in chroot (-pre4 did): > > lukas@dev:~/libsslbuild/bin$ uname -r > 4.4.0-109-generic > lukas@dev:~/libsslbuild/bin$ ./openssl version > OpenSSL 1.1.1-pre6 (beta) 1 May 2018 > lukas@dev:~/libsslbuild/bin$ ./openssl version -r > Seeding source: os-specific > lukas@dev:~/libsslbuild/bin$ > > > But, stracing haproxy shows that the library IS ACTUALLY using > getrandom(). So the "Seeding source" output of the executable is > wrong. Gonna dig into this as well, but seeing how my haproxy > executable uses getrandom() calls, this perfectly explains why I did > not see this in -pre6 (which has the build-workaround for < libc 2.25, > while pre4 did not, so it did not use the getrandom() call). > > > @Sander it looks like openssl folks won't change their mind about > this. You have to either upgrade to a kernel more recent than 3.17 so > that getrandom() can be used, or make /dev/xrandom available within > your chroot. > > > > Lukas > >
Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour
Hello, On 23 May 2018 at 18:29, Emeric Brun wrote: > This issue was due to openssl-1.1.1 which re-seed after an elapsed time or > number of request. > > If /dev/urandom is used as seeding source when haproxy is chrooted it fails > to re-open /dev/urandom > > By defaut the openssl-1.1.1 configure script uses the syscall getrandom as > seeding source and fallback on /dev/urandom if not available. > > So you don't face the issue if your openssl-1.1.1 is compiled to use getrandom > > But getrandom syscall is available only since kernel > 3.17 and the main > point: for glibc > 2.25. > > With openssl-1.1.1 you can check this this way: > # ./openssl-1.1.1/openssl version -r > Seeding source: getrandom-syscall I have glibc 2.23 (Ubuntu 16.04) and openssl shows "os-specific", even if kernel headers are installed while compiling, yet -pre6 does not hang for me in chroot (-pre4 did): lukas@dev:~/libsslbuild/bin$ uname -r 4.4.0-109-generic lukas@dev:~/libsslbuild/bin$ ./openssl version OpenSSL 1.1.1-pre6 (beta) 1 May 2018 lukas@dev:~/libsslbuild/bin$ ./openssl version -r Seeding source: os-specific lukas@dev:~/libsslbuild/bin$ But, stracing haproxy shows that the library IS ACTUALLY using getrandom(). So the "Seeding source" output of the executable is wrong. Gonna dig into this as well, but seeing how my haproxy executable uses getrandom() calls, this perfectly explains why I did not see this in -pre6 (which has the build-workaround for < libc 2.25, while pre4 did not, so it did not use the getrandom() call). @Sander it looks like openssl folks won't change their mind about this. You have to either upgrade to a kernel more recent than 3.17 so that getrandom() can be used, or make /dev/xrandom available within your chroot. Lukas
Re: Dynamically adding/deleting SSL certificates
Hello Emeric, On Tue, May 22, 2018 at 05:37:58PM +0200, Emeric Brun wrote: > Hi Auréline > > I see that you're using the domain to known the certificate to delete. > > If you take a look to crt-list, you will see that the identifier of the > certificate > is customizable and is not necessarily the domain. > > I think to perform the adds/delete operation we should use the same > identifiers than the crt-list option > Right, I forgot about these. It's now possible to use one of the identifiers (if any) of a certificate from a crt-list in the deletion. I added a refcount in the struct sni_ctx, added wildcard support that I forgot and corrected a typo in an error message. It's still RFC patches. -- Aurélien. >From 9612ff9dcfda0611e5287ea0e9d9959d972e0061 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aur=C3=A9lien=20Nephtali?= Date: Wed, 18 Apr 2018 15:34:33 +0200 Subject: [PATCH 1/2] MINOR/WIP: ssl: Refactor ssl_sock_load_cert_file() into ssl_sock_load_cert2() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ssl_sock_get_dh_from_file() -> ssl_sock_get_dh() - ssl_sock_load_dh_params() takes a BIO * instead of a char * - ssl_sock_load_cert_chain_file() -> ssl_sock_load_cert_chain() + takes a BIO * instead of a char * Signed-off-by: Aurélien Nephtali --- src/ssl_sock.c | 210 + 1 file changed, 136 insertions(+), 74 deletions(-) diff --git a/src/ssl_sock.c b/src/ssl_sock.c index 7a602ad57..eb0d43ded 100644 --- a/src/ssl_sock.c +++ b/src/ssl_sock.c @@ -2583,43 +2583,39 @@ static DH *ssl_get_tmp_dh(SSL *ssl, int export, int keylen) return dh; } -static DH * ssl_sock_get_dh_from_file(const char *filename) +static DH * ssl_sock_get_dh(BIO *in) { - DH *dh = NULL; - BIO *in = BIO_new(BIO_s_file()); + return PEM_read_bio_DHparams(in, NULL, NULL, NULL); +} - if (in == NULL) - goto end; +int ssl_sock_load_global_dh_param_from_file(const char *filename) +{ + BIO *in; + int ret = -1; + + in = BIO_new(BIO_s_file()); + if (!in) + return -1; if (BIO_read_filename(in, filename) <= 0) goto end; - dh = PEM_read_bio_DHparams(in, NULL, NULL, NULL); + global_dh = ssl_sock_get_dh(in); + if (global_dh) + ret = 0; end: -if (in) -BIO_free(in); + BIO_free(in); - return dh; -} - -int ssl_sock_load_global_dh_param_from_file(const char *filename) -{ - global_dh = ssl_sock_get_dh_from_file(filename); - - if (global_dh) { - return 0; - } - - return -1; + return ret; } /* Loads Diffie-Hellman parameter from a file. Returns 1 if loaded, else -1 if an error occured, and 0 if parameter not found. */ -int ssl_sock_load_dh_params(SSL_CTX *ctx, const char *file) +int ssl_sock_load_dh_params(SSL_CTX *ctx, BIO *in) { int ret = -1; - DH *dh = ssl_sock_get_dh_from_file(file); + DH *dh = ssl_sock_get_dh(in); if (dh) { ret = 1; @@ -3192,10 +3188,9 @@ static int ssl_sock_load_multi_cert(const char *path, struct bind_conf *bind_con /* Loads a certificate key and CA chain from a file. Returns 0 on error, -1 if * an early error happens and the caller must call SSL_CTX_free() by itelf. */ -static int ssl_sock_load_cert_chain_file(SSL_CTX *ctx, const char *file, struct bind_conf *s, - struct ssl_bind_conf *ssl_conf, char **sni_filter, int fcount) +static int ssl_sock_load_cert_chain(SSL_CTX *ctx, BIO *in, struct bind_conf *s, +struct ssl_bind_conf *ssl_conf, char **sni_filter, int fcount) { - BIO *in; X509 *x = NULL, *ca; int i, err; int ret = -1; @@ -3211,14 +3206,6 @@ static int ssl_sock_load_cert_chain_file(SSL_CTX *ctx, const char *file, struct STACK_OF(GENERAL_NAME) *names; #endif - in = BIO_new(BIO_s_file()); - if (in == NULL) - goto end; - - if (BIO_read_filename(in, file) <= 0) - goto end; - - passwd_cb = SSL_CTX_get_default_passwd_cb(ctx); passwd_cb_userdata = SSL_CTX_get_default_passwd_cb_userdata(ctx); @@ -3308,44 +3295,110 @@ end: if (x) X509_free(x); - if (in) - BIO_free(in); - return ret; } -static int ssl_sock_load_cert_file(const char *path, struct bind_conf *bind_conf, struct ssl_bind_conf *ssl_conf, - char **sni_filter, int fcount, char **err) +struct ssl_input { +const char *cert_path; +struct chunk *cert_buf; +}; + +static int ssl_sock_load_cert2(struct ssl_input *si, struct bind_conf *bind_conf, struct ssl_bind_conf *ssl_conf, + char **sni_filter, int fcount, char **err) { int ret; SSL_CTX *ctx; + BIO *in; + + if (!si->cert_path && (!si->cert_buf || !si->cert_buf->len)) + return 1; ctx = SSL_CTX_new(SSLv23_server_method()); if (!ctx) { - memprintf(err, "%sunable to allocate SSL context for cert '%s'.\n", - err && *err ? *err : "", path); + if (si->cert_path) + memprintf(err, "%sunable to allocate SSL context for cert '%s'.\n", + err && *err ? *err : "", si->cert_path); + else + memprintf(err, "%sunable to allocate SSL
Re: remaining process after (seamless) reload
On Wed, May 23, 2018 at 06:49:09PM +0200, William Dauchy wrote: > We do frequent reloads (approximatively every 10s). > After a while some processes remains alive and seem to never exit (waited >24 > hours). While stracing them, some of them are still handling traffic and > doing healthchecks. Some of them are exiting normally after the reload. > I was wondering how I can help to debug this issue. I assume I won't > have any other info through the stats socket, since it concerns older > processes but maybe I missed something. > Do you have any hint to help me understand what is going on? More details which could help understand what is going on: ps output: root 15928 0.3 0.0 255216 185268 ? Ss May21 10:11 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 16988 16912 6340 28271 30590 30334 -x /var/lib/haproxy/stats haproxy 6340 2.0 0.0 526172 225476 ? Ssl May22 35:03 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 6328 6315 -x /var/lib/haproxy/stats haproxy 28271 1.8 0.0 528720 229508 ? Ssl May22 27:13 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 28258 28207 28232 6340 -x /var/lib/haproxy/stats haproxy 30590 265 0.0 527268 225032 ? Rsl 04:35 2188:55 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 30578 28271 6340 -x /var/lib/haproxy/stats haproxy 30334 197 0.0 526704 224544 ? Rsl 09:17 1065:59 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 30322 30295 27095 6340 28271 30590 -x /var/lib/haproxy/stats haproxy 16912 1.7 0.0 527544 216552 ? Ssl 18:14 0:03 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 16899 28271 30590 30334 6340 -x /var/lib/haproxy/stats haproxy 17001 2.2 0.0 528392 214656 ? Ssl 18:17 0:00 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 16988 16912 6340 28271 30590 30334 -x /var/lib/haproxy/stats lsof output: haproxy6340haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy6340 6341 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy6340 6342 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy6340 6343 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy 17020haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy 17020 17021 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy 17020 17022 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy 17020 17023 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy 28271haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy 28271 28272 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy 28271 28273 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp haproxy 28271 28274 haproxy5u unix 0x883feec97000 0t0 679289634 /var/lib/haproxy/stats.15928.tmp (So on unhealthy nodes, I find old processes which are still linked to the socket.) The provisioning part is also seeing data which are supposed to be already updated through the runtime API. I suspect I am getting old data when connecting to the unix socket. The later being still attached to an old process? Indeed, if I try for i in {1..500}; do sudo echo "show info" | sudo socat stdio /var/lib/haproxy/stats | grep Pid; done I get "Pid: 17001" most of the time, which is the last process but I sometimes get: "Pid: 28271"(!) which is a > 24 hours old process. Is there something we are doing wrongly? Best, -- William
full Requesting URL (with path, scheme, etc)
Hi, I really liked haproxy-auth-request's idea[0] () but needed some extra stuff, so I forked it and implemented what I needed/wanted[1], by doing a sub-request against hashicorp vault. One of the things I need is the full requesting URL, with scheme, etc. So far I've come up with: http-request set-header X-Requesting-URL https%%3A//%[hdr("host")]%[url] which seems to mostly work, but is there a better way? I use this so that when forwarding to the authentication website, it can redirect them back to where they were going when they successfully login. 0: https://github.com/TimWolla/haproxy-auth-request/ 1: https://github.com/csawyerYumaed/hapvault/
Re: Rewrite image path based on HTTP_REQUEST
Hi, On Sat, May 19, Aleksandar Lazic wrote: > On 17/05/2018, Lotic Lists wrote: > > How can I rewrite a image path based on URL? > > > > Example, users request the url www.example.com/images/logo.png, haproxy just > > balance to backend servers normally. > > > > Now users request www.newdomain.com, I need rewrite URI to > > /images/new-logo.png > > Well what have you already tried? > > I would try, untested. > > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-reqrep > > acl old_dom hdr(host) -i www.example.com > acl old_path path_beg -i /images/logo.png > > reqrep "^([^ :]*) /images/logo.png /images/new-logo.png" if old_dom && > old_path > reqirep "^Host: www.example.com "Host: www.newdomain.com" if old_dom && > old_path If you just need to change the path then http-request set-path should be enough(https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-http-request). Something like: acl new_dom hdr_dom(host) -i www.newdomain.com acl old_path path_beg -i /images/logo.png http-request set-path "/images/new-logo.png" if new_dom old_path -Jarno -- Jarno Huuskonen - System Administrator | jarno.huuskonen atsign uef.fi
remaining process after (seamless) reload
Hello, I am trying to understand a possible issue we have regarding haproxy (seamless) reloads. I am using haproxy v1.8.9 with the following config (using nbthread): global log 127.0.0.1 local0 info maxconn 262144 user haproxy group haproxy nbproc 1 daemon stats socket /var/lib/haproxy/stats level admin mode 644 expose-fd listeners stats timeout 2m tune.bufsize 33792 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-server-verify none crt-base /etc/haproxy/tls/ nbthread 4 cpu-map auto:1/1-4 0-3 defaults log global mode http retries 3 timeout connect 10s timeout client 180s timeout server 180s timeout http-keep-alive 10s timeout http-request 10s timeout queue 1s timeout check 5s option httplog option dontlognull option redispatch option prefer-last-server option dontlog-normal option http-keep-alive option forwardfor except 127.0.0.0/8 balance roundrobin maxconn 262134 http-reuse safe default-server inter 5s fastinter 1s fall 3 slowstart 20s observe layer7 error-limit 1 on-error fail-check http-check send-state http-check expect status 200 string passing listen stats bind *:8080 stats enable stats uri /haproxy_stats frontend fe_main # arbitrary split in two for http/https traffic bind *:80 name http_1 process 1/1 bind *:80 name http_2 process 1/2 bind *:443 name https_3 ssl crt /etc/haproxy/tls/fe_main process 1/3 bind *:443 name https_4 ssl crt /etc/haproxy/tls/fe_main process 1/4 [...] The rest of the config contains lost of acl/backends (> 1000) We do frequent reloads (approximatively every 10s). After a while some processes remains alive and seem to never exit (waited >24 hours). While stracing them, some of them are still handling traffic and doing healthchecks. Some of them are exiting normally after the reload. I was wondering how I can help to debug this issue. I assume I won't have any other info through the stats socket, since it concerns older processes but maybe I missed something. Do you have any hint to help me understand what is going on? Best regards, -- William
Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour
Hi Sander, Lukas, On 05/23/2018 02:32 PM, Lukas Tribus wrote: > Hello, > > On 23 May 2018 at 13:10, Sander Hoentjen wrote: >> I can confirm the issue is gone when I don't use chroot. I will try to >> see if I can get more info like a strace soon. I won't be able to today >> though. Thanks Lucas and Emeric! > > 1.8.9 with 1.1.1-pre6 chrooted is now running for me for more than 22 > hours. I don't see the same issue. > > So are probably we are facing a different issue. > > > > Lukas > This issue was due to openssl-1.1.1 which re-seed after an elapsed time or number of request. If /dev/urandom is used as seeding source when haproxy is chrooted it fails to re-open /dev/urandom By defaut the openssl-1.1.1 configure script uses the syscall getrandom as seeding source and fallback on /dev/urandom if not available. So you don't face the issue if your openssl-1.1.1 is compiled to use getrandom But getrandom syscall is available only since kernel > 3.17 and the main point: for glibc > 2.25. With openssl-1.1.1 you can check this this way: # ./openssl-1.1.1/openssl version -r Seeding source: getrandom-syscall R, Emeric
Re: DNS resolver + threads, 100% cpu usage / hang 1.9dev
Hi Olivier, On Wed, May 23, 2018 at 04:13:20PM +0200, Olivier Houchard wrote: > Thanks a lot for testing, and your usual investigation work ! > > Willy, can you please apply this ? Sure, now done, thanks! Willy
Re: [RFC PATCH] MINOR: ssl: set SSL_OP_PRIORITIZE_CHACHA
Hi Lukas, On Wed, May 23, 2018 at 03:30:55PM +0200, Lukas Tribus wrote: > You can merge as-is, it was just RFC to see if there are any > objections to enable this flag unconditionally. OK now merged, thank you! Willy
Re: DNS resolver + threads, 100% cpu usage / hang 1.9dev
Hi Pieter, On Tue, May 22, 2018 at 09:00:24PM +0200, PiBa-NL wrote: > Hi Olivier, > > Op 22-5-2018 om 18:46 schreef Olivier Houchard: > > Hi Pieter, > > > > Does the attached patch fix it for you ? It's been generated from master, > > but will probably apply against 1.8 as well. > > > > Thanks ! > > > > Olivier > > Patch works for me (on master, didn't try with 1.8). Or at least i'm running > the same testbox for an hour now without issue. > Thanks ! > Thanks a lot for testing, and your usual investigation work ! Willy, can you please apply this ? Thanks ! Olivier >From b938f86e1fe51e95adc73f9e583dd225f5ecf88d Mon Sep 17 00:00:00 2001 From: Olivier Houchard Date: Tue, 22 May 2018 18:40:07 +0200 Subject: [PATCH] BUG/MEDIUM: dns: Delay the attempt to run a DNS resolution on check failure. When checks fail, the code tries to run a dns resolution, in case the IP changed. The old way of doing that was to check, in case the last dns resolution hadn't expired yet, if there were an applicable IP, which should be useless, because it has already be done when the resolution was first done, or to run a new resolution. Both are a locking nightmare, and lead to deadlocks, so instead, just wake the resolvers task, that should do the trick. This should be backported to 1.8. --- src/dns.c | 9 - 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/src/dns.c b/src/dns.c index e42950212..e2d46a658 100644 --- a/src/dns.c +++ b/src/dns.c @@ -348,11 +348,10 @@ void dns_trigger_resolution(struct dns_requester *req) /* The resolution must not be triggered yet. Use the cached response, if * valid */ exp = tick_add(res->last_resolution, resolvers->hold.valid); - if (res->status == RSLV_STATUS_VALID && - tick_isset(res->last_resolution) && !tick_is_expired(exp, now_ms)) - req->requester_cb(req, NULL); - else - dns_run_resolution(res); + if (resolvers->t && (res->status != RSLV_STATUS_VALID || + !tick_isset(res->last_resolution) || tick_is_expired(exp, now_ms))) + task_wakeup(resolvers->t, TASK_WOKEN_OTHER); + } -- 2.14.3
Re: [RFC PATCH] MINOR: ssl: set SSL_OP_PRIORITIZE_CHACHA
Hi Willy, On 22 May 2018 at 18:54, Willy Tarreau wrote: > On Tue, May 22, 2018 at 04:28:38PM +0200, Emeric Brun wrote: >> I agree, we could merge it as it is. > > OK thanks Emeric. > > So Lukas, just let me know if you want me to merge it as-is or if you > still have some polishing to do on it, since you marked it RFC. You can merge as-is, it was just RFC to see if there are any objections to enable this flag unconditionally. Thanks, Lukas
Re: SSL certs loading performance regression
Hi Hervé, > Le 22 mai 2018 à 10:31, Hervé Commowick a > écrit : > > Hello HAProxy ML, > > I tracked down a performance regression about loading bunch of > certificates, at least 3x to 5x more time for loading 10 certs since > this commit > http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=f6b37c67be277b5f0ae60438d796ff29ef19be40 > > This regression is 1.8 specific, (no issue in 1.6 or 1.7 branch) > > my bind line : > bind 127.0.0.1:1443 ssl crt ssl10k ecdhe secp384r1 > > After some tests with William, it looks like it is also related to > "ecdhe secp384r1" parameter, i don't really understand why, but without > this i don't see any regression (and it looks like secp384r1 was > effectively working in old version) > can you try with « curves » parameter and not the old « ecdhe » ? > Let me know if i can test something, from 1min30s to 5min has some > impacts as you can understand :-) > Manu.
Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour
Hello, On 23 May 2018 at 13:10, Sander Hoentjen wrote: > I can confirm the issue is gone when I don't use chroot. I will try to > see if I can get more info like a strace soon. I won't be able to today > though. Thanks Lucas and Emeric! 1.8.9 with 1.1.1-pre6 chrooted is now running for me for more than 22 hours. I don't see the same issue. So are probably we are facing a different issue. Lukas
Process crash on reload with TLS tickets
Hi, this seems harmless, but haproxy processes crash on reload when using TLS tickets with multiple sockets per port. Following configuration reproduces the problem: global nbproc 2 user haproxy group haproxy daemon defaults timeout connect 5000 timeout client 5 timeout server 5 frontend test bind 127.0.0.1:8443 process 1 ssl crt file tls-ticket-keys file bind 127.0.0.1:8443 process 2 ssl crt file tls-ticket-keys file Every reload results with the following warning in logs: [WARNING] 142/134019 (23389) : Former worker 23404 exited with code 134 gdb shows following: Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x7fc87c8cc42a in __GI_abort () at abort.c:89 #2 0x7fc87c908c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fc87c9fdd98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x7fc87c90efc6 in malloc_printerr (action=3, str=0x7fc87c9fde10 "double free or corruption (!prev)", ptr=, ar_ptr=) at malloc.c:5049 #4 0x7fc87c90f80e in _int_free (av=0x7fc87cc31b00 , p=0x55f389a1a4a0, have_lock=0) at malloc.c:3905 #5 0x55f387b9aa16 in ssl_sock_destroy_bind_conf (bind_conf=0x55f389a1c400) at src/ssl_sock.c:4818 #6 0x55f387c25280 in deinit () at src/haproxy.c:2240 #7 0x55f387b8846e in main (argc=, argv=0x7ffed2261cc8) at src/haproxy.c:3070 src/ssl_sock.c:4818 contains: free(bind_conf->keys_ref->tlskeys); haproxy version (1.8.9 recompiled on Debian stretch with USE_TFO): $ /usr/sbin/haproxy -vv HA-Proxy version 1.8.9-1~tsg9+1 2018/05/21 Copyright 2000-2018 Willy Tarreau Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -g -O2 -fdebug-prefix-map=/root/haproxy-1.8.9=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1 USE_NS=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.1.0f 25 May 2017 Running on OpenSSL version : OpenSSL 1.1.0f 25 May 2017 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.3 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Encrypted password support via crypt(3): yes Built with multi-threading support. Built with PCRE version : 8.39 2016-06-14 Running on PCRE version : 8.39 2016-06-14 PCRE library supports JIT : yes Built with zlib version : 1.2.8 Running on zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with network namespace support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available filters : [SPOE] spoe [COMP] compression [TRACE] trace -- Janusz Dziemidowicz
Re: Fwd: [haproxy/haproxy] BUG/MAJOR: server: Segfault after parsing server state file. (0bedb8a)
Willy,(writing from my phone, blame it, if I mess up the quotes) Am 23.05.2018 11:20 vorm. schrieb Willy Tarreau : Well, please post here instead, it's where people are present and follow the activity. I'm adding Fred in CC since he's the one who fixed the crash, and Baptiste as well since he's the state file designer. Please note that the initial comment on GitHub is not written by me. I'm very well aware that the list is to be used.I'm just the messenger, I don't know the original author and I suspect that the original author will not read this mailing list thread, because GitHub does not give out their email address. Someone should probably comment on the GitHub commit, hinting to the mailing list. I can do so tomorrow when I'm back at my computer. Best regardsTim Düsterhus
gRPC protocol
Hi. Any plans to add the gRPC Protocol into haproxy? Have anyone used https://github.com/jinq0123/grpc-lua or similar in haproxy? In some areas is the gRPC Protocol more and more famous due to this fact I ask just ask what's the plan for this protocol is like mqtt? Best regards Aleks
Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour
On 05/22/2018 04:31 PM, Sander Hoentjen wrote: > On 05/22/2018 04:19 PM, Emeric Brun wrote: >> Hi Sander, >> >> On 05/22/2018 02:04 PM, Sander Hoentjen wrote: >>> On 05/22/2018 12:04 PM, Lukas Tribus wrote: Hello, On 22 May 2018 at 11:48, Sander Hoentjen wrote: > I did, but I still experience the same issues. What is your exact > haproxy version you tested with? Mine is 1.8.8 > Built with OpenSSL version : OpenSSL 1.1.1-pre6 (beta) 1 May 2018 > Running on OpenSSL version : OpenSSL 1.1.1-pre6 (beta) 1 May 2018 I'm using the development tree. So if it doesn't depend on openssl, it may be bug that has been fixed in haproxy itself. Can you try 1.8.9? >>> I just tried HA-Proxy version 1.8.9-83616ec 2018/05/18, it has the same >>> issue unfortunately. >>> >> Do you use chroot ? >> >> I remember a behavior change between 1.1.0 and 1.1.1. After an elapsed time >> (or a number of new sess) openssl >> try to re-open some /dev/Xrandom sources. And it could fail in a chroot. >> >> https://github.com/openssl/openssl/issues/5330 >> >> > Thanks, I do use chroot. In that issue it is mentioned that this problem > has been fixed, but I will test without chroot anyway. Will report > tomorrow if this worked or not. I can confirm the issue is gone when I don't use chroot. I will try to see if I can get more info like a strace soon. I won't be able to today though. Thanks Lucas and Emeric!
Re: SPOE and modsecurity contrib
On Sun, May 20, 2018 at 10:59:02AM -0400, Daniel Corbett wrote: > While I haven't been able to get 'tcp-request content reject' to work with > this configuration -- I am able to get 'http-request deny' to work: > > http-request deny if { var(txn.modsec.code) -m int gt 0 } This is expected. The "tcp-request content" rules are evaluated before HTTP rules, hence before the filters as well. Thus here the rule says that it will reject the request based on a variable that holds its initial value and never had a chance to be modified yet. Unfortunately there is little we can do to detect this better, except improving the doc and providing better config examples. Willy
Re: Fwd: [haproxy/haproxy] BUG/MAJOR: server: Segfault after parsing server state file. (0bedb8a)
Hi Tim, On Tue, May 22, 2018 at 09:52:16PM +0200, Tim Düsterhus wrote: > Hi list > > the following comment has been posted to GitHub on commit > 0bedb8ac90ffdf1498a999c44d1c91556fb726ee > > https://github.com/haproxy/haproxy/commit/0bedb8ac90ffdf1498a999c44d1c91556fb726ee#commitcomment-29087381 Well, please post here instead, it's where people are present and follow the activity. I'm adding Fred in CC since he's the one who fixed the crash, and Baptiste as well since he's the state file designer. So it's about commit 0bedb8a ("BUG/MAJOR: server: Segfault after parsing server state file."). > @wtarreau This patch causes the state of the haproxy to be discarded by > the haproxy unless the server id of the backend server is not fixed. > Normally they are auto-generated and removing e.g. 1st server on the > list of backends results in renumerating all server ids :( So if I understand right, the problem is in fact that servers from the state file which don't exist in the config file are reported as mismatches, thus the code for the ID/name mismatch was modified instead of modifying the special case. I don't know what could be done to detect such cases. At the very least, given that the previous code used to crash in this case, it means this condition differs in a detectable way (possibly that a server lookup returns null or something like this). I'll let Fred and Baptiste take a look. Thanks, Willy
Re: [PATCH] MINOR: http: Log warning if (add|set)-header fails
Hi Tim, On Sun, May 20, 2018 at 05:55:06PM +0200, Tim Duesterhus wrote: > Willy, > > attached is a first attempt at a patch that adds logging (without any rate > limiting). I have a few questions regarding the whole counters and logging > infrastructure: > > 1. I noticed that there is ha_warning(...) and send_log(p, LOG_WARNING, ...), >usually both are used in the same place. What's the difference between >them? Are the log lines in my patch going into the correct place? ha_warning() is used to emit warnings on stderr during startup, mainly when parsing the config. Example of such warnings are "missing timeouts for proxy foobar". send_log() is used at run time to either report activity (regular logs) or send notification about noticeable events like servers going up/down, or risks for the reliability due to external events (no more FDs) or issues like this one. Historically we've accumulated a few ha_warning() in runtime code paths, but each time you see this it's an error. Sometimes due to a thinko, sometimes due to code being moved or reused. > 2. How do you prefer the rate limiting of the log lines? >- Is there an existing "flag" for rate limited logs? Not that I'm aware of. >- Where should the information about the rate limited logging be stored > (in a static variable?)? At a few places (which ones, I don't remember) we have a local static variable to report certain elements only once. It could be one option. But the problem is that if the first event was missed, the new ones will be missed as well. >- Log only every Xth failed call to http_header_add_tail2? It could be the easiest and most convenient solution for the mid term I think. E.g. just log once every 256 using "if (!(counter++ & 255)) ..." >- By time? This would require an additional call to gettimeofday, no? It's not welcome there. We could rely on a freq_counter though, but I'm not sure it's worth the extra pain (enthough it's likely that over the long term that's what it will evolve to). If you prefer to do it anyway, it will look roughly like this : #define LIMIT 10 // burst of 10 messages max #define PERIOD 1// over 10 seconds static struct freq_ctr_period warn_limiter; static unsigned int skipped; if (freq_ctr_remain_period(&warn_limiter, PERIOD, LIMIT, 0)) { update_freq_ctr_period(&warn_limiter, PERIOD, 1); send_log("log: blah (skipped %d messages)", skipped); skipped=0; } else { skipped++; } >- Should the code have separate rate limiting per header (I guess no, > because the only meaningful thing a administrator can do is increase > tune.maxrewrite and that fixes all headers at once)? It would really be overkill and would provide little to no value. As you say, what matters is to alert against too low maxrewrite. However using a dedicated log function to warn about such problems and maintain its own counter definitely makes sense to me. > 3. Regarding the counters I noticed frontends don't really seem to use >error counters yet. Do I need to add a counter to both >struct fe_counters as well as struct be_counters and increment depending >on whether the (add|set)-header line is in a backend or frontend? I'm not sure what you mean about frontends not using error counters, because the request parsing issues are reported there, just like some of the errors causing an error to be reported. The frontend currently tracks failed_req, denied_{conn,sess,req,resp}, and {cli,srv}_aborts. > diff --git a/src/proto_http.c b/src/proto_http.c > index 3adb54f2..9739fc7a 100644 > --- a/src/proto_http.c > +++ b/src/proto_http.c > @@ -2631,7 +2631,11 @@ resume_execution: > } > } > > - http_header_add_tail2(&txn->req, &txn->hdr_idx, > replace->str, replace->len); > + if (http_header_add_tail2(&txn->req, &txn->hdr_idx, > replace->str, replace->len) < 0) { > + replace->str[rule->arg.hdr_add.name_len] = 0; > + ha_warning("Proxy %s failed to add or set the > request header '%s'. You might need to increase tune.maxrewrite.\n", px->id, > replace->str); > + send_log(px, LOG_WARNING, "Proxy %s failed to > add or set the request header '%s'. You might need to increase > tune.maxrewrite.", px->id, replace->str); > + } In my opinion it could be useful to add "for request #%u",s->uniq_id. This ID can be retrieved using "%rt" in the logs, so for those who need to understand what request broke in order to know a bit more about it, it will definitely help. thanks, Willy