Websocket metrics
Hi, I'm looking for recommendations on best practices for collecting metrics and performing analytics on websocket connections through HAProxy. I'm already capturing the http log which serves me well for http traffic, but since websockets get upgraded to TCP connections I'm losing the kind of atomicity I'm used to. Is there anything I can do with HAProxy to sample frames from a websocket, or am I going to have to go more low-level and do layer 2 traffic capture?
Re: Odd behaviour with option forwardfor.
Hi Aleks, Patrick's solution is correct. What I was expecting was that HAProxy would take X-Forwarded-For: 1.1.1.1 and produce X-Forwarded-For: 1.1.1.1,2.2.2.2 but what it actually does is produce X-Forwarded-For: 1.1.1.1 X-Forwarded-For: 2.2.2.2 The backend concatenates the lines and treats them like what I was expecting, but when I do a header capture in the second HAProxy it only reads the second line. And yes, my snippet is missing the header capture on the http frontend, thanks for spotting that. On Jul 23, 2017 03:50, "Aleksandar Lazic" wrote: Hi Patrick Hemmer, Patrick Hemmer wrote on 22.07.2017: > On 2017/7/22 11:11, Claus Strommer wrote: > > Hi all, I'm seeing some odd behaviour with our > haproxy balancer and am looking for some insights. > >The setup: > >I have a webserver that is behind two haproxy > balancers (version 1.5.18 on EL7), which are > behind CloudFlare. In effect the request goes > > > client->CF->haproxy1->haproxy2->server. > >On both haproxy balancers I have "option > forwardfor" and "capture request header > X-Forwarded-For len 128" set. > On the server I also capture X-Forwarded-For > > Now here is where the odd behaviour > (highlighted) happens: > > * haproxy1 logs the full X-Forwarded-For header. > * haproxy2 only logs the IP of the CF proxy (the last address in X-Forwarded-For) > * server logs the full X-Forwarded-For header. > * If I turn off "option forwardfor" on haproxy1, then > haproxy2 logs the full header as received by CF. > * Changing the length of the capture request does not > seem to make a difference. > > * I noticed that haproxy uses spaces after the comma > between the header entries, but CF does not. I tried > replicating this issue with a direct curl request to haproxy2 > replicating the x-forwarded-for header that haproxy1 would > have sent, and I cannot reproduce the issue. > > The only thing that I notice is that CF > > >Am I missing something obvious here? Below are the full > options I'm using on haproxy1 and haproxy2. Everything after that is ACLs > > defaults >modehttp >log global >option httplog >option dontlognull >option http-server-close >option forwardfor except 127.0.0.0/8 >option redispatch >retries 3 > >frontend http *:80 >mode http >reqadd X-Forwarded-Proto:\ https I miss here the capture request header X-Forwarded-For len 15 >redirect scheme https code 301 > >frontend https >bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem >mode http >capture request header Host len 50 I miss here the capture request header X-Forwarded-For len 15 Can you try to run haproxy for a short time in debug mode? http://cbonte.github.io/haproxy-dconv/1.7/management.html#3 -d > The "option forwardfor" setting appends a complete new header, > not appends the value to an existing header. From the docs on "option forwardfor": > > Enable insertion of the X-Forwarded-For header to requests sent to servers > ... > > this header is always appended at the end of the existing header list > > Your header capture is grabbing the last X-Forwarded-For header. > On issues like this, you should perform a packet capture. It > would make the issue immediately apparent. > > Personally I use 2 rules similar to the following to append to X-Forwarded-For: > >http-request set-header X-Forwarded-For > %[req.fhdr(X-Forwarded-For)],\ %[src] if { req.fhdr(X-Forwarded-For) -m found } >http-request set-header X-Forwarded-For %[src] if !{ > req.fhdr(X-Forwarded-For) -m found } > > -Patrick But doesn't haproxy do this already? http://git.haproxy.org/?p=haproxy-1.7.git;a=blob;f=src/proto_http.c;h= 94c8d639f6f777241109f605e1e1742f9a39bf33;hb=HEAD#l4639 -- Best Regards Aleks
Re: Odd behaviour with option forwardfor.
Thanks Patrick, your explanation makes sense. On Sat, Jul 22, 2017 at 12:28 PM, Patrick Hemmer wrote: > On 2017/7/22 11:11, Claus Strommer wrote: > > Hi all, I'm seeing some odd behaviour with our haproxy balancer and am > looking for some insights. > > The setup: > > I have a webserver that is behind two haproxy balancers (version 1.5.18 on > EL7), which are behind CloudFlare. In effect the request goes > > client->CF->haproxy1->haproxy2->server. > > On both haproxy balancers I have "option forwardfor" and "capture request > header X-Forwarded-For len 128" set. On the server I also capture > X-Forwarded-For > > Now here is where the odd behaviour (*highlighted*) happens: > > * haproxy1 logs the full X-Forwarded-For header. > * *haproxy2 only logs the IP of the CF proxy (the last address in > X-Forwarded-For)* > * server logs the full X-Forwarded-For header. > * If I turn off "option forwardfor" on haproxy1, then haproxy2 logs the > full header as received by CF. > * Changing the length of the capture request does not seem to make a > difference. > * I noticed that haproxy uses spaces after the comma between the header > entries, but CF does not. I tried replicating this issue with a direct > curl request to haproxy2 replicating the x-forwarded-for header that > haproxy1 would have sent, and I cannot reproduce the issue. > > The only thing that I notice is that CF > > Am I missing something obvious here? Below are the full options I'm using > on haproxy1 and haproxy2. Everything after that is ACLs > > defaults > modehttp > log global > option httplog > option dontlognull > option http-server-close > option forwardfor except 127.0.0.0/8 > option redispatch > retries 3 > > frontend http *:80 > mode http > reqadd X-Forwarded-Proto:\ https > redirect scheme https code 301 > > frontend https > bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem > mode http > capture request header Host len 50 > > > The "option forwardfor" setting appends a complete new header, not appends > the value to an existing header. From the docs on "option forwardfor": > > Enable insertion of the X-Forwarded-For header to requests sent to > servers > ... > > this header is always appended at the end of the existing header list > > Your header capture is grabbing the last X-Forwarded-For header. > On issues like this, you should perform a packet capture. It would make > the issue immediately apparent. > > Personally I use 2 rules similar to the following to append to > X-Forwarded-For: > > http-request set-header X-Forwarded-For %[req.fhdr(X-Forwarded-For)],\ > %[src] if { req.fhdr(X-Forwarded-For) -m found } > http-request set-header X-Forwarded-For %[src] if !{ > req.fhdr(X-Forwarded-For) -m found } > > -Patrick >
Odd behaviour with option forwardfor.
Hi all, I'm seeing some odd behaviour with our haproxy balancer and am looking for some insights. The setup: I have a webserver that is behind two haproxy balancers (version 1.5.18 on EL7), which are behind CloudFlare. In effect the request goes client->CF->haproxy1->haproxy2->server. On both haproxy balancers I have "option forwardfor" and "capture request header X-Forwarded-For len 128" set. On the server I also capture X-Forwarded-For Now here is where the odd behaviour (*highlighted*) happens: * haproxy1 logs the full X-Forwarded-For header. * *haproxy2 only logs the IP of the CF proxy (the last address in X-Forwarded-For)* * server logs the full X-Forwarded-For header. * If I turn off "option forwardfor" on haproxy1, then haproxy2 logs the full header as received by CF. * Changing the length of the capture request does not seem to make a difference. * I noticed that haproxy uses spaces after the comma between the header entries, but CF does not. I tried replicating this issue with a direct curl request to haproxy2 replicating the x-forwarded-for header that haproxy1 would have sent, and I cannot reproduce the issue. The only thing that I notice is that CF Am I missing something obvious here? Below are the full options I'm using on haproxy1 and haproxy2. Everything after that is ACLs defaults modehttp log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 frontend http *:80 mode http reqadd X-Forwarded-Proto:\ https redirect scheme https code 301 frontend https bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem mode http capture request header Host len 50
Re: Mixed-mode frontend
By the way, the ssl option for the server did the trick. Based on the documentation for the frontend mode option ("tcp ... This is the default mode. It should be used for SSL, SSH, SMTP, ...") I had assumed that no such backend option was available. But I'm glad I was wrong! Thanks again, Baptiste! On Wed, Jun 25, 2014 at 3:49 PM, Claus Strommer wrote: > Whoops! > > Just to be safe, here's the whole thing again, with additions > > > 8< snip > frontend httpweb > bind *:80 > bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem > mode http > option httplog > > acl host_about hdr_end(host) -i about.site.com > > acl host_api hdr_end(host) -i api.site.com > acl require_ssl hdr_end(host) -i api.site.com > > acl host_error hdr_end(host) -i error.site.com > acl require_nossl hdr_end(host) -i error.site.com > > > redirect scheme https if !{ ssl_fc } require_ssl > redirect scheme http if { ssl_fc } require_nossl > > use backend about:3000 if host_about > use backend api:80 if host_api > use_backend nginx:8080 if host_error > > > > backend about:3000 > mode http > balance roundrobin > option httplog > #option httpclose > option forwardfor > server about.backend.com about.backend.com:3000 check inter 5000 > > backend api:80 > mode http > balance roundrobin > option httplog > #option httpclose > option forwardfor > server api.backend.com api.backend.com:80 check inter 5000 > > backend about:3001 > mode http > balance roundrobin > option httplog > #option httpclose > option forwardfor > server about.backend.com about.backend.com:3001 check inter 5000 > > backend nginx:8080 > > mode http > balance roundrobin > option httplog > #option httpclose > option forwardfor > server localhost localhost:8080 check inter 5000 > 8< snip > > > > Basically, all our backends except host_about and host_error go through > SSL. host_error is forced to HTTP, host_about is kept at whatever the > client requests. > > > <http://api.site.com> > > > On Wed, Jun 25, 2014 at 3:37 PM, Baptiste wrote: > >> On Wed, Jun 25, 2014 at 5:47 PM, Claus Strommer >> wrote: >> > Hello all, >> > >> > For reasons that I'll spare you I'm working on replacing a Pound >> balancer >> > with HAProxy 1.5. I am mostly happy with my configuration, except for >> one >> > thing: >> > >> > All of my backends accept http, except for a Node.js server which >> accepts >> > mixed http and https. This server has a login page that explicitly >> requires >> > an SSL connection by checking the local socket used for the connection. >> In >> > Pound this was done by setting the HTTPS parameter on the backend, >> however >> > from my understanding HAProxy requires that I use TCP passthrough to >> let the >> > backend handle SSL. I am uncertain as to how I should shape the HAProxy >> > configuration to achieve this, as I would like the TCP backend to >> listen on >> > the same port as on the HTTP backend. My (simplified) config looks >> thus: >> > >> > >> > 8< snip >> > frontend httpweb >> > bind *:80 >> > bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem >> > mode http >> > option httplog >> > >> > acl host_about hdr_end(host) -i about.site.com >> > acl host_api hdr_end(host) -i api.site.com >> > >> > redirect scheme https if !{ ssl_fc } require_ssl >> > redirect scheme http if { ssl_fc } require_nossl >> > >> > use backend about:3000 if host_about >> > use backend api:80 if host_api >> > >> > backend about:3000 >> > mode http >> > balance roundrobin >> > option httplog >> > #option httpclose >> > option forwardfor >> > server about.backend.com about.backend.com:3000 check inter 5000 >> > >> > backend api:80 >> > mode http >> > balance roundrobin >> > option httplog >> > #option httpclose >> > option forwardfor >> > server api.backend.com api.backend.com:80 check inter 5000 >> > >> > backend about:3001 >> > mode http >> > balance roundrobin >
Re: Mixed-mode frontend
Whoops! Just to be safe, here's the whole thing again, with additions 8< snip frontend httpweb bind *:80 bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem mode http option httplog acl host_about hdr_end(host) -i about.site.com acl host_api hdr_end(host) -i api.site.com acl require_ssl hdr_end(host) -i api.site.com acl host_error hdr_end(host) -i error.site.com acl require_nossl hdr_end(host) -i error.site.com redirect scheme https if !{ ssl_fc } require_ssl redirect scheme http if { ssl_fc } require_nossl use backend about:3000 if host_about use backend api:80 if host_api use_backend nginx:8080 if host_error backend about:3000 mode http balance roundrobin option httplog #option httpclose option forwardfor server about.backend.com about.backend.com:3000 check inter 5000 backend api:80 mode http balance roundrobin option httplog #option httpclose option forwardfor server api.backend.com api.backend.com:80 check inter 5000 backend about:3001 mode http balance roundrobin option httplog #option httpclose option forwardfor server about.backend.com about.backend.com:3001 check inter 5000 backend nginx:8080 mode http balance roundrobin option httplog #option httpclose option forwardfor server localhost localhost:8080 check inter 5000 8< snip Basically, all our backends except host_about and host_error go through SSL. host_error is forced to HTTP, host_about is kept at whatever the client requests. <http://api.site.com> On Wed, Jun 25, 2014 at 3:37 PM, Baptiste wrote: > On Wed, Jun 25, 2014 at 5:47 PM, Claus Strommer > wrote: > > Hello all, > > > > For reasons that I'll spare you I'm working on replacing a Pound balancer > > with HAProxy 1.5. I am mostly happy with my configuration, except for > one > > thing: > > > > All of my backends accept http, except for a Node.js server which accepts > > mixed http and https. This server has a login page that explicitly > requires > > an SSL connection by checking the local socket used for the connection. > In > > Pound this was done by setting the HTTPS parameter on the backend, > however > > from my understanding HAProxy requires that I use TCP passthrough to let > the > > backend handle SSL. I am uncertain as to how I should shape the HAProxy > > configuration to achieve this, as I would like the TCP backend to listen > on > > the same port as on the HTTP backend. My (simplified) config looks thus: > > > > > > 8< snip > > frontend httpweb > > bind *:80 > > bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem > > mode http > > option httplog > > > > acl host_about hdr_end(host) -i about.site.com > > acl host_api hdr_end(host) -i api.site.com > > > > redirect scheme https if !{ ssl_fc } require_ssl > > redirect scheme http if { ssl_fc } require_nossl > > > > use backend about:3000 if host_about > > use backend api:80 if host_api > > > > backend about:3000 > > mode http > > balance roundrobin > > option httplog > > #option httpclose > > option forwardfor > > server about.backend.com about.backend.com:3000 check inter 5000 > > > > backend api:80 > > mode http > > balance roundrobin > > option httplog > > #option httpclose > > option forwardfor > > server api.backend.com api.backend.com:80 check inter 5000 > > > > backend about:3001 > > mode http > > balance roundrobin > > option httplog > > #option httpclose > > option forwardfor > > server about.backend.com about.backend.com:3001 check inter 5000 > > 8< snip > > > > This of course sends the client into a redirect loop (301) if I hit e.g. > > https://about.site.com/login , because the connection between HAProxy > and > > Node is non-SSL, so it redirects me back to that URL expecting the > > subsequent connection to be HTTPS. If I add an about:3001 backend (3001 > is > > Node's SSL port) I of course get a 502 error because HAProxy connects to > it > > via non-SSL protocol. I also tried to set the backend to tcp mode but > that > > failed because the frontend is http. > > > > So my guess is that I need to add a tcp frontend to handle specifically > > HTTPS connections for about.site.com. How would I go about doing that? > Can > > I have both a
Mixed-mode frontend
Hello all, For reasons that I'll spare you I'm working on replacing a Pound balancer with HAProxy 1.5. I am mostly happy with my configuration, except for one thing: All of my backends accept http, except for a Node.js server which accepts mixed http and https. This server has a login page that explicitly requires an SSL connection by checking the local socket used for the connection. In Pound this was done by setting the HTTPS parameter on the backend, however from my understanding HAProxy requires that I use TCP passthrough to let the backend handle SSL. I am uncertain as to how I should shape the HAProxy configuration to achieve this, as I would like the TCP backend to listen on the same port as on the HTTP backend. My (simplified) config looks thus: 8< snip frontend httpweb bind *:80 bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem mode http option httplog acl host_about hdr_end(host) -i about.site.com acl host_api hdr_end(host) -i api.site.com redirect scheme https if !{ ssl_fc } require_ssl redirect scheme http if { ssl_fc } require_nossl use backend about:3000 if host_about use backend api:80 if host_api backend about:3000 mode http balance roundrobin option httplog #option httpclose option forwardfor server about.backend.com about.backend.com:3000 check inter 5000 backend api:80 mode http balance roundrobin option httplog #option httpclose option forwardfor server api.backend.com api.backend.com:80 check inter 5000 backend about:3001 mode http balance roundrobin option httplog #option httpclose option forwardfor server about.backend.com about.backend.com:3001 check inter 5000 8< snip This of course sends the client into a redirect loop (301) if I hit e.g. https://about.site.com/login , because the connection between HAProxy and Node is non-SSL, so it redirects me back to that URL expecting the subsequent connection to be HTTPS. If I add an about:3001 backend (3001 is Node's SSL port) I of course get a 502 error because HAProxy connects to it via non-SSL protocol. I also tried to set the backend to tcp mode but that failed because the frontend is http. So my guess is that I need to add a tcp frontend to handle specifically HTTPS connections for about.site.com. How would I go about doing that? Can I have both a TCP and HTTP frontend bind to the same port? How would I shape the ACLs to direct https://about.site.com to the TCP frontend, and everything else to the HTTP frontend? -- Claus Strommer, Dev/Ops Engineering Specialist