Websocket metrics
Hi, I'm looking for recommendations on best practices for collecting metrics and performing analytics on websocket connections through HAProxy. I'm already capturing the http log which serves me well for http traffic, but since websockets get upgraded to TCP connections I'm losing the kind of atomicity I'm used to. Is there anything I can do with HAProxy to sample frames from a websocket, or am I going to have to go more low-level and do layer 2 traffic capture?
Re: Odd behaviour with option forwardfor.
Hi Aleks,
Patrick's solution is correct. What I was expecting was that HAProxy would
take
X-Forwarded-For: 1.1.1.1
and produce
X-Forwarded-For: 1.1.1.1,2.2.2.2
but what it actually does is produce
X-Forwarded-For: 1.1.1.1
X-Forwarded-For: 2.2.2.2
The backend concatenates the lines and treats them like what I was
expecting, but when I do a header capture in the second HAProxy it only
reads the second line.
And yes, my snippet is missing the header capture on the http frontend,
thanks for spotting that.
On Jul 23, 2017 03:50, "Aleksandar Lazic" <al-hapr...@none.at> wrote:
Hi Patrick Hemmer,
Patrick Hemmer wrote on 22.07.2017:
> On 2017/7/22 11:11, Claus Strommer wrote:
>
> Hi all, I'm seeing some odd behaviour with our
> haproxy balancer and am looking for some insights.
>
>The setup:
>
>I have a webserver that is behind two haproxy
> balancers (version 1.5.18 on EL7), which are
> behind CloudFlare. In effect the request goes
>
>
> client->CF->haproxy1->haproxy2->server.
>
>On both haproxy balancers I have "option
> forwardfor" and "capture request header
> X-Forwarded-For len 128" set.
> On the server I also capture X-Forwarded-For
>
> Now here is where the odd behaviour
> (highlighted) happens:
>
> * haproxy1 logs the full X-Forwarded-For header.
> * haproxy2 only logs the IP of the CF proxy (the last
address in X-Forwarded-For)
> * server logs the full X-Forwarded-For header.
> * If I turn off "option forwardfor" on haproxy1, then
> haproxy2 logs the full header as received by CF.
> * Changing the length of the capture request does not
> seem to make a difference.
>
> * I noticed that haproxy uses spaces after the comma
> between the header entries, but CF does not. I tried
> replicating this issue with a direct curl request to haproxy2
> replicating the x-forwarded-for header that haproxy1 would
> have sent, and I cannot reproduce the issue.
>
> The only thing that I notice is that CF
>
>
>Am I missing something obvious here? Below are the full
> options I'm using on haproxy1 and haproxy2. Everything after
that is ACLs
>
> defaults
>modehttp
>log global
>option httplog
>option dontlognull
>option http-server-close
>option forwardfor except 127.0.0.0/8
>option redispatch
>retries 3
>
>frontend http *:80
>mode http
>reqadd X-Forwarded-Proto:\ https
I miss here the
capture request header X-Forwarded-For len 15
>redirect scheme https code 301
>
>frontend https
>bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
>mode http
>capture request header Host len 50
I miss here the
capture request header X-Forwarded-For len 15
Can you try to run haproxy for a short time in debug mode?
http://cbonte.github.io/haproxy-dconv/1.7/management.html#3
-d
> The "option forwardfor" setting appends a complete new header,
> not appends the value to an existing header. From the docs on
"option forwardfor":
> > Enable insertion of the X-Forwarded-For header to requests sent
to servers
> ...
> > this header is always appended at the end of the existing
header list
>
> Your header capture is grabbing the last X-Forwarded-For header.
> On issues like this, you should perform a packet capture. It
> would make the issue immediately apparent.
>
> Personally I use 2 rules similar to the following to append to
X-Forwarded-For:
>
>http-request set-header X-Forwarded-For
> %[req.fhdr(X-Forwarded-For)],\ %[src] if { req.fhdr(X-Forwarded-For) -m
found }
>http-request set-header X-Forwarded-For %[src] if !{
> req.fhdr(X-Forwarded-For) -m found }
>
> -Patrick
But doesn't haproxy do this already?
http://git.haproxy.org/?p=haproxy-1.7.git;a=blob;f=src/proto_http.c;h=
94c8d639f6f777241109f605e1e1742f9a39bf33;hb=HEAD#l4639
--
Best Regards
Aleks
Re: Odd behaviour with option forwardfor.
Thanks Patrick, your explanation makes sense.
On Sat, Jul 22, 2017 at 12:28 PM, Patrick Hemmer <hapr...@stormcloud9.net>
wrote:
> On 2017/7/22 11:11, Claus Strommer wrote:
>
> Hi all, I'm seeing some odd behaviour with our haproxy balancer and am
> looking for some insights.
>
> The setup:
>
> I have a webserver that is behind two haproxy balancers (version 1.5.18 on
> EL7), which are behind CloudFlare. In effect the request goes
>
> client->CF->haproxy1->haproxy2->server.
>
> On both haproxy balancers I have "option forwardfor" and "capture request
> header X-Forwarded-For len 128" set. On the server I also capture
> X-Forwarded-For
>
> Now here is where the odd behaviour (*highlighted*) happens:
>
> * haproxy1 logs the full X-Forwarded-For header.
> * *haproxy2 only logs the IP of the CF proxy (the last address in
> X-Forwarded-For)*
> * server logs the full X-Forwarded-For header.
> * If I turn off "option forwardfor" on haproxy1, then haproxy2 logs the
> full header as received by CF.
> * Changing the length of the capture request does not seem to make a
> difference.
> * I noticed that haproxy uses spaces after the comma between the header
> entries, but CF does not. I tried replicating this issue with a direct
> curl request to haproxy2 replicating the x-forwarded-for header that
> haproxy1 would have sent, and I cannot reproduce the issue.
>
> The only thing that I notice is that CF
>
> Am I missing something obvious here? Below are the full options I'm using
> on haproxy1 and haproxy2. Everything after that is ACLs
>
> defaults
> modehttp
> log global
> option httplog
> option dontlognull
> option http-server-close
> option forwardfor except 127.0.0.0/8
> option redispatch
> retries 3
>
> frontend http *:80
> mode http
> reqadd X-Forwarded-Proto:\ https
> redirect scheme https code 301
>
> frontend https
> bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
> mode http
> capture request header Host len 50
>
>
> The "option forwardfor" setting appends a complete new header, not appends
> the value to an existing header. From the docs on "option forwardfor":
> > Enable insertion of the X-Forwarded-For header to requests sent to
> servers
> ...
> > this header is always appended at the end of the existing header list
>
> Your header capture is grabbing the last X-Forwarded-For header.
> On issues like this, you should perform a packet capture. It would make
> the issue immediately apparent.
>
> Personally I use 2 rules similar to the following to append to
> X-Forwarded-For:
>
> http-request set-header X-Forwarded-For %[req.fhdr(X-Forwarded-For)],\
> %[src] if { req.fhdr(X-Forwarded-For) -m found }
> http-request set-header X-Forwarded-For %[src] if !{
> req.fhdr(X-Forwarded-For) -m found }
>
> -Patrick
>
Odd behaviour with option forwardfor.
Hi all, I'm seeing some odd behaviour with our haproxy balancer and am looking for some insights. The setup: I have a webserver that is behind two haproxy balancers (version 1.5.18 on EL7), which are behind CloudFlare. In effect the request goes client->CF->haproxy1->haproxy2->server. On both haproxy balancers I have "option forwardfor" and "capture request header X-Forwarded-For len 128" set. On the server I also capture X-Forwarded-For Now here is where the odd behaviour (*highlighted*) happens: * haproxy1 logs the full X-Forwarded-For header. * *haproxy2 only logs the IP of the CF proxy (the last address in X-Forwarded-For)* * server logs the full X-Forwarded-For header. * If I turn off "option forwardfor" on haproxy1, then haproxy2 logs the full header as received by CF. * Changing the length of the capture request does not seem to make a difference. * I noticed that haproxy uses spaces after the comma between the header entries, but CF does not. I tried replicating this issue with a direct curl request to haproxy2 replicating the x-forwarded-for header that haproxy1 would have sent, and I cannot reproduce the issue. The only thing that I notice is that CF Am I missing something obvious here? Below are the full options I'm using on haproxy1 and haproxy2. Everything after that is ACLs defaults modehttp log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 frontend http *:80 mode http reqadd X-Forwarded-Proto:\ https redirect scheme https code 301 frontend https bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem mode http capture request header Host len 50
Mixed-mode frontend
Hello all,
For reasons that I'll spare you I'm working on replacing a Pound balancer
with HAProxy 1.5. I am mostly happy with my configuration, except for one
thing:
All of my backends accept http, except for a Node.js server which accepts
mixed http and https. This server has a login page that explicitly requires
an SSL connection by checking the local socket used for the connection. In
Pound this was done by setting the HTTPS parameter on the backend, however
from my understanding HAProxy requires that I use TCP passthrough to let
the backend handle SSL. I am uncertain as to how I should shape the
HAProxy configuration to achieve this, as I would like the TCP backend to
listen on the same port as on the HTTP backend. My (simplified) config
looks thus:
8 snip
frontend httpweb
bind *:80
bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem
mode http
option httplog
acl host_about hdr_end(host) -i about.site.com
acl host_api hdr_end(host) -i api.site.com
redirect scheme https if !{ ssl_fc } require_ssl
redirect scheme http if { ssl_fc } require_nossl
use backend about:3000 if host_about
use backend api:80 if host_api
backend about:3000
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3000 check inter 5000
backend api:80
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server api.backend.com api.backend.com:80 check inter 5000
backend about:3001
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3001 check inter 5000
8 snip
This of course sends the client into a redirect loop (301) if I hit e.g.
https://about.site.com/login , because the connection between HAProxy and
Node is non-SSL, so it redirects me back to that URL expecting the
subsequent connection to be HTTPS. If I add an about:3001 backend (3001 is
Node's SSL port) I of course get a 502 error because HAProxy connects to it
via non-SSL protocol. I also tried to set the backend to tcp mode but that
failed because the frontend is http.
So my guess is that I need to add a tcp frontend to handle specifically
HTTPS connections for about.site.com. How would I go about doing that?
Can I have both a TCP and HTTP frontend bind to the same port? How would I
shape the ACLs to direct https://about.site.com to the TCP frontend, and
everything else to the HTTP frontend?
--
Claus Strommer, Dev/Ops Engineering Specialist
Re: Mixed-mode frontend
Whoops!
Just to be safe, here's the whole thing again, with additions
8 snip
frontend httpweb
bind *:80
bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem
mode http
option httplog
acl host_about hdr_end(host) -i about.site.com
acl host_api hdr_end(host) -i api.site.com
acl require_ssl hdr_end(host) -i api.site.com
acl host_error hdr_end(host) -i error.site.com
acl require_nossl hdr_end(host) -i error.site.com
redirect scheme https if !{ ssl_fc } require_ssl
redirect scheme http if { ssl_fc } require_nossl
use backend about:3000 if host_about
use backend api:80 if host_api
use_backend nginx:8080 if host_error
backend about:3000
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3000 check inter 5000
backend api:80
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server api.backend.com api.backend.com:80 check inter 5000
backend about:3001
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3001 check inter 5000
backend nginx:8080
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server localhost localhost:8080 check inter 5000
8 snip
Basically, all our backends except host_about and host_error go through
SSL. host_error is forced to HTTP, host_about is kept at whatever the
client requests.
http://api.site.com
On Wed, Jun 25, 2014 at 3:37 PM, Baptiste bed...@gmail.com wrote:
On Wed, Jun 25, 2014 at 5:47 PM, Claus Strommer
claus.strom...@primal.com wrote:
Hello all,
For reasons that I'll spare you I'm working on replacing a Pound balancer
with HAProxy 1.5. I am mostly happy with my configuration, except for
one
thing:
All of my backends accept http, except for a Node.js server which accepts
mixed http and https. This server has a login page that explicitly
requires
an SSL connection by checking the local socket used for the connection.
In
Pound this was done by setting the HTTPS parameter on the backend,
however
from my understanding HAProxy requires that I use TCP passthrough to let
the
backend handle SSL. I am uncertain as to how I should shape the HAProxy
configuration to achieve this, as I would like the TCP backend to listen
on
the same port as on the HTTP backend. My (simplified) config looks thus:
8 snip
frontend httpweb
bind *:80
bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem
mode http
option httplog
acl host_about hdr_end(host) -i about.site.com
acl host_api hdr_end(host) -i api.site.com
redirect scheme https if !{ ssl_fc } require_ssl
redirect scheme http if { ssl_fc } require_nossl
use backend about:3000 if host_about
use backend api:80 if host_api
backend about:3000
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3000 check inter 5000
backend api:80
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server api.backend.com api.backend.com:80 check inter 5000
backend about:3001
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3001 check inter 5000
8 snip
This of course sends the client into a redirect loop (301) if I hit e.g.
https://about.site.com/login , because the connection between HAProxy
and
Node is non-SSL, so it redirects me back to that URL expecting the
subsequent connection to be HTTPS. If I add an about:3001 backend (3001
is
Node's SSL port) I of course get a 502 error because HAProxy connects to
it
via non-SSL protocol. I also tried to set the backend to tcp mode but
that
failed because the frontend is http.
So my guess is that I need to add a tcp frontend to handle specifically
HTTPS connections for about.site.com. How would I go about doing that?
Can
I have both a TCP and HTTP frontend bind to the same port? How would I
shape the ACLs to direct https://about.site.com to the TCP frontend, and
everything else to the HTTP frontend?
--
Claus Strommer, Dev/Ops Engineering Specialist
Hi Claus,
first, I don't understand your configuration.
You may have cut some part of it, but you cut too much :)
We need to know the content of require_ssl and require_nossl.
Also, the backend 3001 is not used.
Maybe this could help: if you want to cipher a connection to a server,
then simply add the keyword ssl on the server line.
(check also
Re: Mixed-mode frontend
By the way, the ssl option for the server did the trick. Based on the
documentation for the frontend mode option (tcp ... This is the default
mode. It should be used for SSL, SSH, SMTP, ...) I had assumed that no
such backend option was available. But I'm glad I was wrong!
Thanks again, Baptiste!
On Wed, Jun 25, 2014 at 3:49 PM, Claus Strommer claus.strom...@primal.com
wrote:
Whoops!
Just to be safe, here's the whole thing again, with additions
8 snip
frontend httpweb
bind *:80
bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem
mode http
option httplog
acl host_about hdr_end(host) -i about.site.com
acl host_api hdr_end(host) -i api.site.com
acl require_ssl hdr_end(host) -i api.site.com
acl host_error hdr_end(host) -i error.site.com
acl require_nossl hdr_end(host) -i error.site.com
redirect scheme https if !{ ssl_fc } require_ssl
redirect scheme http if { ssl_fc } require_nossl
use backend about:3000 if host_about
use backend api:80 if host_api
use_backend nginx:8080 if host_error
backend about:3000
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3000 check inter 5000
backend api:80
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server api.backend.com api.backend.com:80 check inter 5000
backend about:3001
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3001 check inter 5000
backend nginx:8080
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server localhost localhost:8080 check inter 5000
8 snip
Basically, all our backends except host_about and host_error go through
SSL. host_error is forced to HTTP, host_about is kept at whatever the
client requests.
http://api.site.com
On Wed, Jun 25, 2014 at 3:37 PM, Baptiste bed...@gmail.com wrote:
On Wed, Jun 25, 2014 at 5:47 PM, Claus Strommer
claus.strom...@primal.com wrote:
Hello all,
For reasons that I'll spare you I'm working on replacing a Pound
balancer
with HAProxy 1.5. I am mostly happy with my configuration, except for
one
thing:
All of my backends accept http, except for a Node.js server which
accepts
mixed http and https. This server has a login page that explicitly
requires
an SSL connection by checking the local socket used for the connection.
In
Pound this was done by setting the HTTPS parameter on the backend,
however
from my understanding HAProxy requires that I use TCP passthrough to
let the
backend handle SSL. I am uncertain as to how I should shape the HAProxy
configuration to achieve this, as I would like the TCP backend to
listen on
the same port as on the HTTP backend. My (simplified) config looks
thus:
8 snip
frontend httpweb
bind *:80
bind *:443 ssl crt /etc/ssl/private/primal_bundle_2014.pem
mode http
option httplog
acl host_about hdr_end(host) -i about.site.com
acl host_api hdr_end(host) -i api.site.com
redirect scheme https if !{ ssl_fc } require_ssl
redirect scheme http if { ssl_fc } require_nossl
use backend about:3000 if host_about
use backend api:80 if host_api
backend about:3000
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3000 check inter 5000
backend api:80
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server api.backend.com api.backend.com:80 check inter 5000
backend about:3001
mode http
balance roundrobin
option httplog
#option httpclose
option forwardfor
server about.backend.com about.backend.com:3001 check inter 5000
8 snip
This of course sends the client into a redirect loop (301) if I hit e.g.
https://about.site.com/login , because the connection between HAProxy
and
Node is non-SSL, so it redirects me back to that URL expecting the
subsequent connection to be HTTPS. If I add an about:3001 backend
(3001 is
Node's SSL port) I of course get a 502 error because HAProxy connects
to it
via non-SSL protocol. I also tried to set the backend to tcp mode but
that
failed because the frontend is http.
So my guess is that I need to add a tcp frontend to handle specifically
HTTPS connections for about.site.com. How would I go about doing
that? Can
I have both a TCP and HTTP frontend bind to the same port? How would I
shape the ACLs to direct https://about.site.com to the TCP frontend,
and
everything else
