unsubscribe
Hug
Re: DNS Resolver Issues
Hello! Friendly bump :) I'd be willing to amend the documentation once I understand what's going on :D Cheers, Daniel > On 18. Mar 2019, at 20:28, Daniel Schneller > wrote: > > Hi everyone! > > I assume I am misunderstanding something, but I cannot figure out what it is. > We are using haproxy in AWS, in this case as sidecars to applications so they > need not > know about changing backend addresses at all, but can always talk to > localhost. > > Haproxy listens on localhost and then forwards traffic to an ELB instance. > This works great, but there have been two occasions now, where due to a > change in the > ELB's IP addresses, our services went down, because the backends could not be > reached > anymore. I don't understand why haproxy sticks to the old IP address instead > of going > to one of the updated ones. > > There is a resolvers section which points to the local dnsmasq instance > (there to send > some requests to consul, but that's not used here). All other traffic is > forwarded on > to the AWS DNS server set via DHCP. > > I managed to get timely updates and updated backend servers when using > server-template, > but form what I understand this should not really be necessary for this. > > This is the trimmed down sidecar config. I have not made any changes to dns > timeouts etc. > > resolvers default > # dnsmasq > nameserver local 127.0.0.1:53 > > listen regular > bind 127.0.0.1:9300 > option dontlog-normal > server lb-internal loadbalancer-internal.xxx.yyy:9300 resolvers default > check addr loadbalancer-internal.xxx.yyy port 9300 > > listen templated > bind 127.0.0.1:9200 > option dontlog-normal > option httpchk /haproxy-simple-healthcheck > server-template lb-internal 2 loadbalancer-internal.xxx.yyy:9200 resolvers > default check port 9299 > > > To simulate changing ELB adresses, I added entries for > loadbalancer-internal.xxx.yyy in /etc/hosts > and to be able to control them via dnsmasq. > > I tried different scenarios, but could not reliably predict what would happen > in all cases. > > The address ending in 52 (marked as "valid" below) is a currently (as of the > time of testing) > valid IP for the ELB. The one ending in 199 (marked "invalid") is an unused > private IP address > in my VPC. > > > Starting with /etc/hosts: > > 10.205.100.52 loadbalancer-internal.xxx.yyy# valid > 10.205.100.199 loadbalancer-internal.xxx.yyy# invalid > > haproxy starts and reports: > > regular: lb-internal UP/L7OK > templated: lb-internal1 DOWN/L4TOUT > lb-internal2UP/L7OK > > That's expected. Now when I edit /etc/hosts to _only_ contain the _invalid_ > address > and restart dnsmasq, I would expect both proxies to go fully down. But only > the templated > proxy behaves like that: > > regular: lb-internal UP/L7OK > templated: lb-internal1 DOWN/L4TOUT > lb-internal2 MAINT (resolution) > > Reloading haproxy in this state leads to: > > regular: lb-internal DOWN/L4TOUT > templated: lb-internal1 MAINT (resolution) > lb-internal2 DOWN/L4TOUT > > After fixing /etc/hosts to include the valid server again and restarting > dnsmasq: > > regular: lb-internal DOWN/L4TOUT > templated: lb-internal1UP/L7OK > lb-internal2 DOWN/L4TOUT > > > Shouldn't the regular proxy also recognize the change and bring the backend > up or down > depending on the DNS change? I have waited for several health check rounds > (seeing > "* L4TOUT" and "L4TOUT") toggle, but it still never updates. > > I also tried to have _only_ the invalid address in /etc/hosts, then > restarting haproxy. > The regular backends will never recognize it when I add the valid one back in. > > The templated one does, _unless_ I set it up to have only 1 instead of 2 > server slots. > In that case it behaves will also only pick up the valid server when reloaded. > > On the other hand, it _will_ recognize when I remove the valid server without > a reload > on the next health check, but _not_ bring them back in and make the proxy UP > when it > comes back. > > > I assume my understanding of something here is broken, and I would gladly be > told > about it :) > > > Thanks a lot! > Daniel > > > Version Info: > -- > $ haproxy -vv > HA-Proxy version 1.8.19-1ppa1~trusty 2019/02/12 > Copyright 2000-2019 Willy Tarreau > > Build options : > TARGET = linux2628 > CPU = generic > CC = gcc > CFLAGS = -O2 -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 > -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-strict-aliasing > -Wdeclaration-after-statement -fwrapv -Wno-unused-label > OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 > USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014 > Running on OpenSSL versio
[PATCH] MINOR: ssl: Add aes_gcm_decrypt converter
The converted can be used to decrypt the raw byte input using the
AES-GCM algorithm, using provided nonce, key and AEAD tag. This can
be useful to decrypt encrypted cookies for example and make decisions
based on the content.
---
doc/configuration.txt | 11
src/ssl_sock.c| 140 ++
2 files changed, 151 insertions(+)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index 88148c3b..a4242843 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -13205,6 +13205,17 @@ add()
This prefix is followed by a name. The separator is a '.'. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
+aes_gcm_decrypt(,,,)
+ Decrypts the raw byte input using the AES128-GCM, AES192-GCM or
+ AES256-GCM algorithm, depending on the parameter. All other parameters
+ need to be base64 encoded and the returned result is in raw byte format.
+ If the validation fails, the converter doesn't return any data.
+ The , and can either be strings or variables.
+
+ Example:
+http-response set-header X-Decrypted-Text %[var(txn.enc),\
+ aes_gcm_decrypt(128,txn.nonce,Zm9vb2Zvb29mb29wZm9vbw==,txn.aead_tag)]
+
and()
Performs a bitwise "AND" between and the input value of type signed
integer, and returns the result as an signed integer. can be a
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 138b1c58..ba75b94a 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -118,6 +118,7 @@
#include
#include
#include
+#include
/* Warning, these are bits, not integers! */
#define SSL_SOCK_ST_FL_VERIFY_DONE 0x0001
@@ -9075,6 +9076,137 @@ static int cli_parse_set_ocspresponse(char **args, char
*payload, struct appctx
}
+static inline int sample_conv_var2smp_str(const struct arg *arg, struct sample
*smp)
+{
+ switch (arg->type) {
+ case ARGT_STR:
+ smp->data.type = SMP_T_STR;
+ smp->data.u.str = arg->data.str;
+ return 1;
+ case ARGT_VAR:
+ if (!vars_get_by_desc(&arg->data.var, smp))
+ return 0;
+ if (!sample_casts[smp->data.type][SMP_T_STR])
+ return 0;
+ if (!sample_casts[smp->data.type][SMP_T_STR](smp))
+ return 0;
+ return 1;
+ default:
+ return 0;
+ }
+}
+
+static int check_aes_gcm(struct arg *args, struct sample_conv *conv,
+ const char *file, int line,
char **err)
+{
+ switch(args[0].data.sint) {
+ case 128:
+ case 192:
+ case 256:
+ break;
+ default:
+ memprintf(err, "key size must be 128, 192 or 256 (bits).");
+ return 0;
+ }
+ /* Try to decode a variable. */
+ vars_check_arg(&args[1], NULL);
+ vars_check_arg(&args[2], NULL);
+ vars_check_arg(&args[3], NULL);
+ return 1;
+}
+
+/* Arguements: AES size in bits, nonce, key, tag. The last three arguments are
base64 encoded */
+static int sample_conv_aes_gcm_decrypt(const struct arg *arg_p, struct sample
*smp, void *private)
+{
+ struct sample nonce, key, aead_tag;
+ struct buffer *smp_trash, *smp_trash_alloc;
+ EVP_CIPHER_CTX *ctx;
+ int dec_size, ret;
+
+ smp_set_owner(&nonce, smp->px, smp->sess, smp->strm, smp->opt);
+ if (!sample_conv_var2smp_str(&arg_p[1], &nonce))
+ return 0;
+
+ smp_set_owner(&key, smp->px, smp->sess, smp->strm, smp->opt);
+ if (!sample_conv_var2smp_str(&arg_p[2], &key))
+ return 0;
+
+ smp_set_owner(&aead_tag, smp->px, smp->sess, smp->strm, smp->opt);
+ if (!sample_conv_var2smp_str(&arg_p[3], &aead_tag))
+ return 0;
+
+ smp_trash = get_trash_chunk();
+ smp_trash_alloc = alloc_trash_chunk();
+ if (!smp_trash_alloc)
+ return 0;
+
+ ctx = EVP_CIPHER_CTX_new();
+
+ if (!ctx)
+ goto err;
+
+ dec_size = base64dec(nonce.data.u.str.area, nonce.data.u.str.data,
smp_trash->area, smp_trash->size);
+ if (dec_size < 0)
+ goto err;
+ smp_trash->data = dec_size;
+
+ /* Set cipher type and mode */
+ switch(arg_p[0].data.sint) {
+ case 128:
+ EVP_DecryptInit_ex(ctx, EVP_aes_128_gcm(), NULL, NULL, NULL);
+ break;
+ case 192:
+ EVP_DecryptInit_ex(ctx, EVP_aes_192_gcm(), NULL, NULL, NULL);
+ break;
+ case 256:
+ EVP_DecryptInit_ex(ctx, EVP_aes_256_gcm(), NULL, NULL, NULL);
+ break;
+ }
+
+ EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_AEAD_SET_IVLEN, smp_trash->data,
NULL);
+
+ /* Initialise IV */
+ if(!EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, (unsigned char *)
smp_trash->area))
+ goto err;
+
+ dec_size = base64dec(key.data.u.str.area,
Re: DNS Resolver Issues
Hello Daniel, You might be missing the hold-valid directive in your resolvers section: https://www.haproxy.com/documentation/hapee/1-9r1/onepage/#5.3.2-timeout This should force HAProxy to fetch the DNS record values from the resolver. A reload of the HAProxy instance also forces the instances to query all records from the resolver. Can you please retest with the updated configuration and report back the results? Best regards, Bruno Henc ‐‐‐ Original Message ‐‐‐ On Thursday, March 21, 2019 12:09 PM, Daniel Schneller wrote: > Hello! > > Friendly bump :) > I'd be willing to amend the documentation once I understand what's going on :D > > Cheers, > Daniel > > > On 18. Mar 2019, at 20:28, Daniel Schneller > > daniel.schnel...@centerdevice.com wrote: > > Hi everyone! > > I assume I am misunderstanding something, but I cannot figure out what it > > is. > > We are using haproxy in AWS, in this case as sidecars to applications so > > they need not > > know about changing backend addresses at all, but can always talk to > > localhost. > > Haproxy listens on localhost and then forwards traffic to an ELB instance. > > This works great, but there have been two occasions now, where due to a > > change in the > > ELB's IP addresses, our services went down, because the backends could not > > be reached > > anymore. I don't understand why haproxy sticks to the old IP address > > instead of going > > to one of the updated ones. > > There is a resolvers section which points to the local dnsmasq instance > > (there to send > > some requests to consul, but that's not used here). All other traffic is > > forwarded on > > to the AWS DNS server set via DHCP. > > I managed to get timely updates and updated backend servers when using > > server-template, > > but form what I understand this should not really be necessary for this. > > This is the trimmed down sidecar config. I have not made any changes to dns > > timeouts etc. > > resolvers default > > > > dnsmasq > > > > > > > > nameserver local 127.0.0.1:53 > > listen regular > > bind 127.0.0.1:9300 > > option dontlog-normal > > server lb-internal loadbalancer-internal.xxx.yyy:9300 resolvers default > > check addr loadbalancer-internal.xxx.yyy port 9300 > > listen templated > > bind 127.0.0.1:9200 > > option dontlog-normal > > option httpchk /haproxy-simple-healthcheck > > server-template lb-internal 2 loadbalancer-internal.xxx.yyy:9200 resolvers > > default check port 9299 > > To simulate changing ELB adresses, I added entries for > > loadbalancer-internal.xxx.yyy in /etc/hosts > > and to be able to control them via dnsmasq. > > I tried different scenarios, but could not reliably predict what would > > happen in all cases. > > The address ending in 52 (marked as "valid" below) is a currently (as of > > the time of testing) > > valid IP for the ELB. The one ending in 199 (marked "invalid") is an unused > > private IP address > > in my VPC. > > Starting with /etc/hosts: > > 10.205.100.52 loadbalancer-internal.xxx.yyy # valid > > 10.205.100.199 loadbalancer-internal.xxx.yyy # invalid > > haproxy starts and reports: > > regular: lb-internal UP/L7OK > > templated: lb-internal1 DOWN/L4TOUT > > lb-internal2 UP/L7OK > > That's expected. Now when I edit /etc/hosts to only contain the invalid > > address > > and restart dnsmasq, I would expect both proxies to go fully down. But only > > the templated > > proxy behaves like that: > > regular: lb-internal UP/L7OK > > templated: lb-internal1 DOWN/L4TOUT > > lb-internal2 MAINT (resolution) > > Reloading haproxy in this state leads to: > > regular: lb-internal DOWN/L4TOUT > > templated: lb-internal1 MAINT (resolution) > > lb-internal2 DOWN/L4TOUT > > After fixing /etc/hosts to include the valid server again and restarting > > dnsmasq: > > regular: lb-internal DOWN/L4TOUT > > templated: lb-internal1 UP/L7OK > > lb-internal2 DOWN/L4TOUT > > Shouldn't the regular proxy also recognize the change and bring the backend > > up or down > > depending on the DNS change? I have waited for several health check rounds > > (seeing > > "* L4TOUT" and "L4TOUT") toggle, but it still never updates. > > I also tried to have only the invalid address in /etc/hosts, then > > restarting haproxy. > > The regular backends will never recognize it when I add the valid one back > > in. > > The templated one does, unless I set it up to have only 1 instead of 2 > > server slots. > > In that case it behaves will also only pick up the valid server when > > reloaded. > > On the other hand, it will recognize when I remove the valid server without > > a reload > > on the next health check, but not bring them back in and make the proxy UP > > when it > > comes back. > > I assume my understanding of something here is broken, and I would gladly > > be told > > about it :) > > Thanks a lot! > > Daniel > > > > Version Info: > > > > -- > > > > $ haproxy -vv > > HA-Proxy version 1.8.19-1ppa1~trusty 2019/02/12 > >
[PATCH] ssl: ability to set TLS 1.3 ciphers using ssl-default-server-ciphersuites
Any attempt to put TLS 1.3 ciphers on servers failed with output 'unable
to set TLS 1.3 cipher suites'.
This was due to usage of SSL_CTX_set_cipher_list instead of
SSL_CTX_set_ciphersuites in the TLS 1.3 block (protected by
OPENSSL_VERSION_NUMBER >= 0x10101000L & so).
Signed-off-by: Pierre Cheynier
Reported-by: Damien Claisse
---
src/ssl_sock.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 138b1c58c..47548edc1 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -4785,7 +4785,7 @@ int ssl_sock_prepare_srv_ctx(struct server *srv)
#if (OPENSSL_VERSION_NUMBER >= 0x10101000L && !defined OPENSSL_IS_BORINGSSL &&
!defined LIBRESSL_VERSION_NUMBER)
if (srv->ssl_ctx.ciphersuites &&
- !SSL_CTX_set_cipher_list(srv->ssl_ctx.ctx,
srv->ssl_ctx.ciphersuites)) {
+ !SSL_CTX_set_ciphersuites(srv->ssl_ctx.ctx,
srv->ssl_ctx.ciphersuites)) {
ha_alert("Proxy '%s', server '%s' [%s:%d] : unable to set TLS
1.3 cipher suites to '%s'.\n",
curproxy->id, srv->id,
srv->conf.file, srv->conf.line,
srv->ssl_ctx.ciphersuites);
--
2.20.1
AW: 400 SC on h2 xhr post
Hello Jarno,
thanks for your suggestions. It was not successful.
However, I managed to make it reproductible. I would be really happy, if
someone more experienced would take a look on this.
Setup
Client (Chrome) -> Haproxy (Docker) -> Jetty (Docker)
The client executes following script, it can be saved on the local disk, we can
ignore the CORS logging. Do not change the '3', otherwise it will not occur.
function loopMe () {
setTimeout(function () {
var xhr = new XMLHttpRequest();
xhr.open('POST', 'https://[DOMAIN]/app/docker-jetty.json');
xhr.send();
loopMe();
}, 3)
}
loopMe();
Haproxy.cfg
global
daemon
tune.ssl.default-dh-param 2048
stats socket /var/run/haproxy.stat
defaults
mode http
option httplog
log stdout format raw daemon
timeout connect 5m
timeout client 5m
timeout server 5m
frontend frontend_h2
bind *:443 ssl crt /usr/local/etc/haproxy/ssl/ alpn h2,http/1.1
use_backend backend_jetty
backend backend_jetty
server web850 127.0.0.1:81
Commands for starting the container
1) docker run -p 443:443 -v [LOCAL_DIR_HAPROXY]:/usr/local/etc/haproxy/ -it
haproxy:1.9.4
2) docker run -d --name jetty -v [LOCAL_DIR_JETTY]:/var/lib/jetty/webapps -p
81:8080 jetty
Substitute the following variables:
1) [DOMAIN]: Domain you have a certificate for or generate one
2) [LOCAL_DIR_HAPROXY]: Local directory where you need to put the "haproxy.cfg"
and your certificate (subdirectory "ssl")
3) [LOCAL_DIR_JETTY]: Local directory, create a subdirectory called "app" and
create an empty file named "docker-jetty.json").
Substitute the variables, start the container and open the script in the
browser. After 10-15 requests you should get a SC 400
At first sight, it looks like jetty is doing something terribly wrong. But, and
that's the problem, it does not occur if I have just http/1.1 enabled between
the client and haproxy. Any ideas?
Thanks,
Max
-Ursprüngliche Nachricht-
Von: Jarno Huuskonen
Gesendet: Mittwoch, 20. März 2019 12:59
An: Maximilian Böhm
Cc: haproxy@formilux.org
Betreff: Re: 400 SC on h2 xhr post
Hi Max,
On Wed, Mar 20, Maximilian Böhm wrote:
> >> If the 400 errors happen within 3mins, have you tried changing
> >> client/keep-alive timeouts to see if anything changes ?
> They do most often happen in the first 3 mins. But that's not always the
> case. And if it's really a timeout, shouldn't it be more clearly recurring?
> Like every tenth request fails. But that's also not the case. Sometimes it's
> the 3rd request, sometimes the 20th or even later.
> However, I did increase the previously set timeouts (40min). But this did not
> change anything at all. Is there another timeout which explicitly only
> affects h2 on the client side?
I'm not aware of any more timeouts to test. I think possible timeouts are in
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.1
Have you tested different values for http-reuse (never to always) ?
(https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.2-http-reuse)
(Probably doesn't make any difference).
This could be related to
https://www.mail-archive.com/haproxy@formilux.org/msg32959.html
that test case also returns 400 error with state CH-- with http2.
-Jarno
> -Ursprüngliche Nachricht-
> Von: Jarno Huuskonen
> Gesendet: Dienstag, 19. März 2019 17:34
> An: Maximilian Böhm
> Cc: haproxy@formilux.org
> Betreff: Re: 400 SC on h2 xhr post
>
> Hi,
>
> On Tue, Mar 19, Maximilian Böhm wrote:
> > The problem I experience is within a legacy javascript application which
> > periodically checks if the user is still logged in. It does so by sending
> > an xhr request every 30 seconds (I said, it's a legacy app, right? It does
> > so by POST not GET...). As you may guess, this behavior works using http1.1
> > quasi infinitely. But as soon as I activate HTTP/2, I'll get the following
> > output (sooner or later):
> > 172.17.0.1:46372 [19/Mar/2019:12:10:13.465] [fntnd] [bknd] 0/0/0/14/14 200
> > 368 - - 1/1/0/1/0 0/0 "POST [URL] HTTP/1.1"
> > 172.17.0.1:46372 [19/Mar/2019:12:10:43.465] [fntnd] [bknd] 0/0/0/-1/8 400
> > 187 - - CH-- 1/1/0/0/0 0/0 "POST [URL] HTTP/1.1"
> >
> > Which means, the developer toolbar announces a response code "400" and
> > "400 Bad requestYour browser sent an invalid
> > request.". I was not yet successful reproduce this behavior
> > with OkHttp (java http2-capable library). Jetty - on the backend site -
> > does not report any requests in its ncsa request log.
>
> I've seen some(very few (maybe one-two a day)) 400 bad requests with haproxy
> 1.9.4 (http2) to apache+php (http/1.1) backend. These requests alos have CH..
> state in logs.
> (400 errors have also happened for GET requests).
>
> > It is not directly reproducible (like every second time) but it usually
http-keep-alive timeout and client unexpected socket hangup
Hi, I am encountering a problem with http 1.1 clients that reuse their connection just at the same moment that HAProxy is closing it, resulting in socket hangup ECONNRESET style errors. Version is 1.9.4. In our haproxy config we have something like : timeout http-keep-alive 5s A client who: opens a connection, does a number of requests, then idles for 5 seconds before firing off another requests will sometimes encounter an error. The server is close the socket while the client is sending the last request. The client must retry/handle the error gracefully. We can avoid the problem by setting a similar timeout client side so that they do not try to reuse a connection that has been idle for say, 4 seconds, but this is difficult to manage when we cannot control all the clients. The timeout parameter of the Keep-Alive header seems like it could help : https://tools.ietf.org/id/draft-thomson-hybi-http-timeout-01.html#p-timeout I'd almost expect HAProxy to add the Keep-Alive header when it sees Connection: Keep-Alive in the request, but maybe that's going too far. Is it a good practice to do some header manipulation to communicate this timeout from HAProxy to the client ? I.e. : timeout http-keep-alive 5s http-response set-header Connection Keep-Alive http-response set-header Keep-Alive timeout=4 Having HAProxy indicate the correct timeout value would be more manageable that implementing it in each of our clients ... Regards, Robert Higgins
Re: DNS Resolver Issues
Hi! Thanks for the response. I had looked at the "hold" directives, but since they all seem to have reasonable defaults, I did not touch them. I specified 10s explictly, but it did not make a difference. I did some more tests, however, and it seems to have more to do with the number of responses for the initial(?) DNS queries. Hopefully these three tables make sense and don't get mangled in the mail. The "templated" proxy is defined via "server-template" with 3 "slots". The "regular" one just as "server". Test 1: Start out with both "valid" and "broken" DNS entries. Then comment out/add back one at a time as described in (1)-(5). Each time after changing /etc/hosts, restart dnsmasq and check haproxy via hatop. Haproxy started fresh once dnsmasq was set up to (1). | state state /etc/hosts | regular templated |- (1) BRK| UP/L7OK DOWN/L4TOUT VALID | MAINT/resolution | UP/L7OK | (2) BRK| DOWN/L4TOUT DOWN/L4TOUT #VALID | MAINT/resolution | MAINT/resolution | (3) #BRK | UP/L7OK UP/L7OK VALID | MAINT/resolution | MAINT/resolution | (4) BRK| UP/L7OK UP/L7OK VALID | DOWN/L4TOUT | MAINT/resolution | (5) BRK| DOWN/L4TOUT DOWN/L4TOUT #VALID | MAINT/resolution | MAINT/resolution This all looks normal and as expected. As soon as the "VALID" DNS entry is present, the UP state follows within a few seconds. Test 2: Start out "valid only" (1) and proceed as described in (2)-(5), again restarting dnsmasq each time, and haproxy reloaded after dnsmasq was set up to (1). | state state /etc/hosts | regular templated | (1) #BRK | UP/L7OK MAINT/resolution VALID | MAINT/resolution | UP/L7OK | (2) BRK| UP/L7OK DOWN/L4TOUT VALID | MAINT/resolution | UP/L7OK | (3) #BRK | UP/L7OK MAINT/resolution VALID | MAINT/resolution | UP/L7OK | (4) BRK| UP/L7OK DOWN/L4TOUT VALID | MAINT/resolution | UP/L7OK | (5) BRK| DOWN/L4TOUT DOWN/L4TOUT #VALID | MAINT/resolution | MAINT/resolution Everything good here, too. Adding the broken DNS entry does not bring the proxies down until only the broken one is left. Test 3: Start out "broken only" (1). Again, same as before, haproxy restarted once dnsmasq was initialized to (1). | state state /etc/hosts | regular templated | (1) BRK| DOWN/L4TOUT DOWN/L4TOUT #VALID | MAINT/resolution | MAINT/resolution | (2) BRK| DOWN/L4TOUT UP/L7OK VALID | MAINT/resolution | MAINT/resolution | (3) #BRK | UP/L7OK MAINT/resolution VALID | UP/L7OK | MAINT/resolution | (4) BRK| UP/L7OK DOWN/L4TOUT VALID | UP/L7OK | MAINT/resolution |--
Re: http-keep-alive timeout and client unexpected socket hangup
This is an unfortunate part of the http/1 spec (http/2 makes this much better with the goaway messages so the client knows for sure if the server processed the request) Including the time out in the header is nice, although I'm not sure how many clients/client libraries will pick up on that. I've added it in other work with pretty good results; especially if you advertise one second less than you'll actually wait, that would eliminate the request and close crossing in transit except in case of missed packets and retransmits. -- Richard Russo to...@enslaves.us On Thu, Mar 21, 2019, at 10:37 AM, Robert Higgins wrote: > Hi, > > I am encountering a problem with http 1.1 clients that reuse their > connection just at the same moment that HAProxy is closing it, > resulting in socket hangup ECONNRESET style errors. Version is 1.9.4. > > In our haproxy config we have something like : > > timeout http-keep-alive 5s > > A client who: opens a connection, does a number of requests, then idles > for 5 seconds before firing off another requests will sometimes > encounter an error. The server is close the socket while the client is > sending the last request. The client must retry/handle the error > gracefully. > > We can avoid the problem by setting a similar timeout client side so > that they do not try to reuse a connection that has been idle for say, > 4 seconds, but this is difficult to manage when we cannot control all > the clients. > > The timeout parameter of the Keep-Alive header seems like it could help > : > https://tools.ietf.org/id/draft-thomson-hybi-http-timeout-01.html#p-timeout > > I'd almost expect HAProxy to add the Keep-Alive header when it sees > Connection: Keep-Alive in the request, but maybe that's going too far. > > Is it a good practice to do some header manipulation to communicate > this timeout from HAProxy to the client ? I.e. : > > timeout http-keep-alive 5s > http-response set-header Connection Keep-Alive > http-response set-header Keep-Alive timeout=4 > > Having HAProxy indicate the correct timeout value would be more > manageable that implementing it in each of our clients ... > > Regards, > > Robert Higgins > >
