[chkfail] what went wrong?
Hi, I have checked the documentation but cannot find much information on chkfail's in there. Is there any way to check what went wrong when a chkfail has occurred? We use a monitoring-plugin that has thresholds set on chkfail's which shows me that some machines have multiple chkfail's every day. However, no problem is logged both on the HAProxy side and on the webserver side, and show errors on the stats-socket shows 0 errors. I'd like to know what happened, and if there has been any impact from this failure. Is a chkfail always because the server didn't respond within 10 seconds or can it be something else? Met vriendelijke groet, kind regards, Michel Hoogervorst
Re: [chkfail] what went wrong?
On Tue, Jun 3, 2014 at 11:28 AM, Michel Hoogervorst m.hoogervo...@uniserver.nl wrote: Hi, I have checked the documentation but cannot find much information on chkfail’s in there. Is there any way to check what went wrong when a “chkfail” has occurred? We use a monitoring-plugin that has thresholds set on chkfail’s which shows me that some machines have multiple chkfail’s every day. However, no problem is logged both on the HAProxy side and on the webserver side, and “show errors” on the stats-socket shows 0 errors. I’d like to know what happened, and if there has been any impact from this failure. Is a chkfail always because the server didn’t respond within 10 seconds or can it be something else? Met vriendelijke groet, kind regards, Michel Hoogervorst Hi Michel, Maybe you could start by sharing your configuration, then it will be easier to help you. HAProxy setup log severity to notice when a server fails move to status DOWN due to wrong health check response. If you don't retrieve this info in your log it's either because your HAProxy or your syslog server is not well configured (or both of them in the mean time). Baptiste
RE: [chkfail] what went wrong?
Hi, Thanks for your reply. This is how our config looks like (I stripped some non-relevant comments and names): global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 2 userhaproxy group haproxy daemon tune.maxrewrite 2048 stats socket /var/lib/haproxy/stats defaults modehttp log global option httplog option dontlognull option dontlog-normal option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request10s timeout queue 1m timeout connect 10s timeout client 30m timeout server 30m timeout http-keep-alive 10s timeout check 10s maxconn 2 listen webcluster_prod 127.0.0.1:8081 mode http balance hdr(X-Forwarded-For) option contstats option httpchk HEAD /check.txt HTTP/1.0 option httpclose server xxx 10.20.30.1:80 check server yyy 10.20.30.2:80 check When a server goes down I do get a message in the log, so loggins seems to work fine. I just would like to know what the chkfail is about since I only see it in the stats and nowhere else. Kind regards, Michel Hoogervorst -Oorspronkelijk bericht- Van: Baptiste [mailto:bed...@gmail.com] Verzonden: dinsdag 3 juni 2014 11:41 Aan: Michel Hoogervorst CC: haproxy@formilux.org Onderwerp: Re: [chkfail] what went wrong? On Tue, Jun 3, 2014 at 11:28 AM, Michel Hoogervorst m.hoogervo...@uniserver.nl wrote: Hi, I have checked the documentation but cannot find much information on chkfail’s in there. Is there any way to check what went wrong when a “chkfail” has occurred? We use a monitoring-plugin that has thresholds set on chkfail’s which shows me that some machines have multiple chkfail’s every day. However, no problem is logged both on the HAProxy side and on the webserver side, and “show errors” on the stats-socket shows 0 errors. I’d like to know what happened, and if there has been any impact from this failure. Is a chkfail always because the server didn’t respond within 10 seconds or can it be something else? Met vriendelijke groet, kind regards, Michel Hoogervorst Hi Michel, Maybe you could start by sharing your configuration, then it will be easier to help you. HAProxy setup log severity to notice when a server fails move to status DOWN due to wrong health check response. If you don't retrieve this info in your log it's either because your HAProxy or your syslog server is not well configured (or both of them in the mean time). Baptiste
Multiple/non-standard ssl ports on one frontend?
Using 1.5-dev25. Sometimes for transitional support, we need to temporarily
use a non-standard port for frontend SSL traffic. Is there anything that
might prevent identical requests from matching the below frontend/backend
if they're sent to :8080 instead of :443?
frontend https-in
bind *:443 ssl crt /etc/haproxy/sslcerts/tmpcert.pem
bind *:8080 ssl crt /etc/haproxy/sslcerts/tmpcert.pem
use_backend ssl_app if { hdr_sub(host) -i app.prod }
backend ssl_app
option httpchk GET /health
server app1 app1.prod:8080 ssl check
server app2 app2.prod:8080 ssl check
The following is the output and log entry from a request to port 443:
$ curl -k https://proxy.prod/health
OK
haproxy[4424]: 10.20.30.40:53283 [03/Jun/2014:15:49:03.097] https-in~
app.prod/app1
455/0/97/4/556 200 147 - - 0/0/0/0/0 0/0 GET /health HTTP/1.1
The following is an identical curl/output/log, only using port 8080 instead
$ curl -k https://proxy.prod:8080/health
htmlbodyh1503 Service Unavailable/h1
No server is available to handle this request.
/body/html
haproxy[4424]: 10.20.30.40:53291 [03/Jun/2014:15:49:09.750] https-in~
https-in/NOSRV -1/-1/-1/-1/453 503 213 - - SC-- 0/0/0/0/0 0/0 GET
/health HTTP/1.1
reqidel/forwardfor
Quick question (after writing the email, I realized that there's nothing
quick about it) about the order of processing for reqidel and option
forwardfor options. First and foremost, we're running HA-Proxy version
1.5-dev24-8860dcd 2014/04/26
Up until a few days ago, we were doing pretty straightforward things with
our X-Forwarded-For headers. Basically, we just removed them altogether,
and then used forwardfor to add a new one:
reqidel ^X-Forwarded-For:.*
option forwardfor
There were no problems with that, AFAIK. Now, we have some traffic coming
through Distil, as a test. These, of course, have client IPs that we don't
really care about, but we'd like to trust the X-Forwarded-For header that
Distil is tacking on. I tried using an acl to only remove X-Forwarded-For
if the request isn't coming from Distil, and then use if-none for option
forwardfor:
capture request header X-Forwarded-For len 49
acl from-distil hdr(X-Distil) some_string_that_we_can_trust
reqidel ^X-Forwarded-For:.* unless from-distil
option forwardfor if-none
At first glance, this seemed to do the trick -- in all cases, it looks like
Distil is sending X-Forwarded-For headers with only one single IP address
in them, as seen in the haproxy logs: (I redacted the client IP, and it's
shown as 11.22.33.44)
Jun 2 00:00:16 haproxy2a haproxy[4373]: 192.225.212.21:3612
[02/Jun/2014:00:00:15.828] frontend backend/srvr 0/0/0/337/337 200 823 - -
1716/937/24/4/0 0/0 {e4d077fcc15ade3dfa99ca86ccf01084|0|11.22.33.44}
POST /my/url.cgi HTTP/1.0
However, on the server side, the X-Forwarded-For header that's coming
through is:
X-Forwarded-For: 11.22.33.44, 192.225.212.21
Therefore, I'm curious when/how the logic for option forwardfor's if-none
keyword is determined. Also, am I correct in assuming that reqidel always
happens before option forwardfor? I tried flipping them around in the
config file to see if a warning would pop up, and none did, but I'd assume
that reqidel happens first, otherwise one would never see any
X-Forwarded-For headers on the backend servers at all.
--
Andy Walker
System Administrator
FBS - creators of flexmls
3415 39th St S
Fargo, ND 58104
701-235-7300
Re: Multiple/non-standard ssl ports on one frontend?
Hi Justin, On Tue, 03 Jun 2014, 17:56:49 +0200, Justin Rush wrote: [...] The following is the output and log entry from a request to port 443: $ curl -k https://proxy.prod/health OK haproxy[4424]: 10.20.30.40:53283 [03/Jun/2014:15:49:03.097] https-in~ app.prod/app1 455/0/97/4/556 200 147 - - 0/0/0/0/0 0/0 GET /health HTTP/1.1 The following is an identical curl/output/log, only using port 8080 instead $ curl -k https://proxy.prod:8080/health Can you try if curl -k http://proxy.prod:8080/health; works? If I'm not mistaken, https:// implicitly uses port 443, but don't know how the explicit :8080 might interfere with that. HTH, cheers. l8er manfred
Re: Multiple/non-standard ssl ports on one frontend?
Hi Manfred, On Tue, Jun 3, 2014 at 11:12 AM, Manfred Hollstein mhollst...@t-online.de wrote: Can you try if curl -k http://proxy.prod:8080/health; works? If I'm not mistaken, https:// implicitly uses port 443, but don't know how the explicit :8080 might interfere with that. As I expected, this gets an empty reply: $ curl -k http://app.prod:8080/health curl: (52) Empty reply from server So, haproxy is definitely listening on 8080 and expecting an SSL client. (Also, I just realized my sanitizing above was wrong, my curl commands are all to app.prod, not proxy.prod.)
Re: Multiple/non-standard ssl ports on one frontend?
Justin Rush schreef op 3-6-2014 18:19:
Hi Manfred,
On Tue, Jun 3, 2014 at 11:12 AM, Manfred Hollstein
mhollst...@t-online.de mailto:mhollst...@t-online.de wrote:
Can you try if curl -k http://proxy.prod:8080/health; works? If
I'm not
mistaken, https:// implicitly uses port 443, but don't know how the
explicit :8080 might interfere with that.
As I expected, this gets an empty reply:
$ curl -k http://app.prod:8080/health
curl: (52) Empty reply from server
So, haproxy is definitely listening on 8080 and expecting an SSL client.
(Also, I just realized my sanitizing above was wrong, my curl commands
are all to app.prod, not proxy.prod.)
Can you give it a try with:
use_backend ssl_app if { hdr_sub(host) -i app.prod:8080 }
Think ive seen at least with sni requests that the (non standard) port
is part of the sni name indication, not sure about how plain http is
handled.
Greets PiBa-NL
Re: Multiple/non-standard ssl ports on one frontend?
Justin Rush schreef op 3-6-2014 18:19:
Hi Manfred,
On Tue, Jun 3, 2014 at 11:12 AM, Manfred Hollstein
mhollst...@t-online.de mailto:mhollst...@t-online.de wrote:
Can you try if curl -k http://proxy.prod:8080/health; works? If
I'm not
mistaken, https:// implicitly uses port 443, but don't know how the
explicit :8080 might interfere with that.
As I expected, this gets an empty reply:
$ curl -k http://app.prod:8080/health
curl: (52) Empty reply from server
So, haproxy is definitely listening on 8080 and expecting an SSL client.
(Also, I just realized my sanitizing above was wrong, my curl commands
are all to app.prod, not proxy.prod.)
Can you give it a try with:
use_backend ssl_app if { hdr_sub(host) -i app.prod:8080 }
Think ive seen at least with sni requests that the (non standard) port
is part of the sni name indication, not sure about how plain http is
handled.
Greets PiBa-NL
RE: Multiple/non-standard ssl ports on one frontend?
Hi guys,
Hi Manfred,
On Tue, Jun 3, 2014 at 11:12 AM, Manfred Hollstein
mhollst...@t-online.demailto:mhollst...@t-online.de wrote:
Can you try if curl -k http://proxy.prod:8080/health; works? If I'm not
mistaken, https:// implicitly uses port 443, but don't know how the
explicit :8080 might interfere with that.
As I expected, this gets an empty reply:
$ curl -k http://app.prod:8080/health
curl: (52) Empty reply from server
So, haproxy is definitely listening on 8080 and expecting an SSL client.
(Also, I just realized my sanitizing above was wrong, my curl commands
are all to app.prod, not proxy.prod.)
Can you give it a try with:
use_backend ssl_app if { hdr_sub(host) -i app.prod:8080 }
Think ive seen at least with sni requests that the (non standard) port
is part of the sni name indication, not sure about how plain http is
handled.
Yes, the host header in the request will look like:
Host: app.prod:8080
but still, since hdr_sub matches a substring and not an exact
string, haproxy should be able to match it and use that particular backend.
Justin, can you double check that you are in fact using hdr_sub?
Please use curl with the -v argument to see all the headers and try
what happens if you overwrite the request header with -H Host: app.prod.
Regards,
Lukas
Re: Multiple/non-standard ssl ports on one frontend?
On Tue, Jun 3, 2014 at 12:25 PM, PiBa-NL piba.nl@gmail.com wrote:
Can you give it a try with:
use_backend ssl_app if { hdr_sub(host) -i app.prod:8080 }
Think ive seen at least with sni requests that the (non standard) port is
part of the sni name indication, not sure about how plain http is handled.
Same result, 503 the log indicates that it didn't even match the backend
rule.
Re: Multiple/non-standard ssl ports on one frontend?
Hi there! On Tue, Jun 3, 2014 at 12:44 PM, Lukas Tribus luky...@hotmail.com wrote: Justin, can you double check that you are in fact using hdr_sub? My apologies: Through a layer of human-abstraction and another layer of config sanitization, I discovered that we were using hdr_end instead of hdr_sub. I didn't know that the :8080 would be included in the Host header (thanks for the tip, Lukas). After replacing all the hdr_end rules with hdr_sub, I no longer need the :8080 in the rule, and everything is working as expected. Thanks for all the help, guys. -Justin
Re: reqidel/forwardfor
On Tue, Jun 3, 2014 at 8:57 AM, Andy Walker a...@fbsdata.com wrote: Quick question (after writing the email, I realized that there's nothing quick about it) about the order of processing for reqidel and option forwardfor options. First and foremost, we're running HA-Proxy version 1.5-dev24-8860dcd 2014/04/26 There were no problems with that, AFAIK. Now, we have some traffic coming through Distil, as a test. These, of course, have client IPs that we don't really care about, but we'd like to trust the X-Forwarded-For header that Distil is tacking on. I tried using an acl to only remove X-Forwarded-For if the request isn't coming from Distil, and then use if-none for option forwardfor: capture request header X-Forwarded-For len 49 acl from-distil hdr(X-Distil) some_string_that_we_can_trust reqidel ^X-Forwarded-For:.* unless from-distil option forwardfor if-none At first glance, this seemed to do the trick -- in all cases, it looks like Distil is sending X-Forwarded-For headers with only one single IP address in them, as seen in the haproxy logs: (I redacted the client IP, and it's shown as 11.22.33.44) I have a similar setup and it's working well and doesn't add the client-ip if from-distil is set -- maybe another option forwardfor without the if-none was left in a default, frontend or backend? Also, if I were do build the config starting with 1.5, I think that the http-request set-header configuration would be the way to go as it should be simpler and more flexible. Something like acl from-distil hdr(X-Distil) some_string_that_we_can_trust http-request set-header X-Forwarded-For %[src] unless from-distil I've not tried that yet but it seems like it should work. -Bryan
Re: reqidel/forwardfor
Oh for dumb... I actually do have an: option forwardfor except 127.0.0.0/8 in the defaults section, despite the fact that I'd defined it in all of the frontends that we have. Big oversight! My apologies :) So, removing that line from the defaults section actually did fix the issue, but I'll probably still take your suggestion and use http-request set-header, since it can be set conditionally by acl. Thanks much, Bryan! -- Andy Walker System Administrator FBS - creators of flexmls 3415 39th St S Fargo, ND 58104 701-235-7300 On Tue, Jun 3, 2014 at 2:25 PM, Bryan Talbot bryan.tal...@playnext.com wrote: On Tue, Jun 3, 2014 at 8:57 AM, Andy Walker a...@fbsdata.com wrote: Quick question (after writing the email, I realized that there's nothing quick about it) about the order of processing for reqidel and option forwardfor options. First and foremost, we're running HA-Proxy version 1.5-dev24-8860dcd 2014/04/26 There were no problems with that, AFAIK. Now, we have some traffic coming through Distil, as a test. These, of course, have client IPs that we don't really care about, but we'd like to trust the X-Forwarded-For header that Distil is tacking on. I tried using an acl to only remove X-Forwarded-For if the request isn't coming from Distil, and then use if-none for option forwardfor: capture request header X-Forwarded-For len 49 acl from-distil hdr(X-Distil) some_string_that_we_can_trust reqidel ^X-Forwarded-For:.* unless from-distil option forwardfor if-none At first glance, this seemed to do the trick -- in all cases, it looks like Distil is sending X-Forwarded-For headers with only one single IP address in them, as seen in the haproxy logs: (I redacted the client IP, and it's shown as 11.22.33.44) I have a similar setup and it's working well and doesn't add the client-ip if from-distil is set -- maybe another option forwardfor without the if-none was left in a default, frontend or backend? Also, if I were do build the config starting with 1.5, I think that the http-request set-header configuration would be the way to go as it should be simpler and more flexible. Something like acl from-distil hdr(X-Distil) some_string_that_we_can_trust http-request set-header X-Forwarded-For %[src] unless from-distil I've not tried that yet but it seems like it should work. -Bryan
Re: [PATCH] replace-header/modify-header option for http-request/http-response directives
Hello, I don't read all the content of your patch, but a little thing get my mind: you're using a regex_t. Haproxy can be compiled with standard regex, pcre regex or pcre regex with JIT. You must use the struct my_regex defined in include/common/regex.h to use compliant system. regcomp is replaced by regex_comp regexec is replaced by regex_exec regfree is replaced by regex_free Thierry On Fri, 30 May 2014 22:56:47 -0600 Sasha Pachev sa...@asksasha.com wrote: Per our discussion with Willy earlier, attached is the patch for the feature. I noticed that while we had free_http_req_rules() there was no free_http_res_rules() even though the memory for each response rule entry was being allocated with calloc() just like with the request rule and should have been getting free during deinit() but was not. No big deal really, only affect memory leak detection test runs, but I felt bad about allocating memory for a regex and not freeing it, so I added free_http_res_rules(). -- Sasha Pachev Fast Running Blog. http://fastrunningblog.com Run. Blog. Improve. Repeat.
Re: [PATCH] replace-header/modify-header option for http-request/http-response directives
Therry: I noticed that both regex_t and my_regex were used throughout the code, and was not sure what the policy was. So I take it new code should be using my_regex. Should we also change the old code to my_regex? On Tue, Jun 3, 2014 at 4:12 PM, Thierry FOURNIER tfourn...@exceliance.fr wrote: Hello, I don't read all the content of your patch, but a little thing get my mind: you're using a regex_t. Haproxy can be compiled with standard regex, pcre regex or pcre regex with JIT. You must use the struct my_regex defined in include/common/regex.h to use compliant system. regcomp is replaced by regex_comp regexec is replaced by regex_exec regfree is replaced by regex_free Thierry On Fri, 30 May 2014 22:56:47 -0600 Sasha Pachev sa...@asksasha.com wrote: Per our discussion with Willy earlier, attached is the patch for the feature. I noticed that while we had free_http_req_rules() there was no free_http_res_rules() even though the memory for each response rule entry was being allocated with calloc() just like with the request rule and should have been getting free during deinit() but was not. No big deal really, only affect memory leak detection test runs, but I felt bad about allocating memory for a regex and not freeing it, so I added free_http_res_rules(). -- Sasha Pachev Fast Running Blog. http://fastrunningblog.com Run. Blog. Improve. Repeat. -- Sasha Pachev Fast Running Blog. http://fastrunningblog.com Run. Blog. Improve. Repeat.
