Re: Odd behaviour with option forwardfor.

[Claus Strommer]

Thanks Patrick, your explanation makes sense.

On Sat, Jul 22, 2017 at 12:28 PM, Patrick Hemmer 
wrote:

> On 2017/7/22 11:11, Claus Strommer wrote:
>
> Hi all, I'm seeing some odd behaviour with our haproxy balancer and am
> looking for some insights.
>
> The setup:
>
> I have a webserver that is behind two haproxy balancers (version 1.5.18 on
> EL7), which are behind CloudFlare.   In effect the request goes
>
> client->CF->haproxy1->haproxy2->server.
>
> On both haproxy balancers I have "option forwardfor" and "capture request
> header X-Forwarded-For len 128" set.  On the server I also capture
> X-Forwarded-For
>
> Now here is where the odd behaviour (*highlighted*) happens:
>
> * haproxy1 logs the full X-Forwarded-For header.
> * *haproxy2 only logs the IP of the CF proxy (the last address in
> X-Forwarded-For)*
> * server logs the full X-Forwarded-For header.
> * If I turn off "option forwardfor" on haproxy1, then haproxy2 logs the
> full header as received by CF.
> * Changing the length of the capture request does not seem to make a
> difference.
> * I noticed that haproxy uses spaces after the comma between the header
> entries, but CF does not.  I tried replicating this issue with a direct
> curl request to haproxy2 replicating the x-forwarded-for header that
> haproxy1 would have sent, and I cannot reproduce the issue.
>
> The only thing that I notice is that CF
>
> Am I missing something obvious here?  Below are the full options I'm using
> on haproxy1 and haproxy2.  Everything after that is ACLs
>
> defaults
> modehttp
> log global
> option  httplog
> option  dontlognull
> option http-server-close
> option forwardfor   except 127.0.0.0/8
> option  redispatch
> retries 3
>
> frontend  http *:80
> mode http
> reqadd X-Forwarded-Proto:\ https
> redirect scheme https code 301
>
> frontend https
> bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
> mode http
> capture request header Host len 50
>
>
> The "option forwardfor" setting appends a complete new header, not appends
> the value to an existing header. From the docs on "option forwardfor":
> > Enable insertion of the X-Forwarded-For header to requests sent to
> servers
> ...
> > this header is always appended at the end of the existing header list
>
> Your header capture is grabbing the last X-Forwarded-For header.
> On issues like this, you should perform a packet capture. It would make
> the issue immediately apparent.
>
> Personally I use 2 rules similar to the following to append to
> X-Forwarded-For:
>
>   http-request set-header X-Forwarded-For %[req.fhdr(X-Forwarded-For)],\
> %[src] if { req.fhdr(X-Forwarded-For) -m found }
>   http-request set-header X-Forwarded-For %[src] if !{
> req.fhdr(X-Forwarded-For) -m found }
>
> -Patrick
>

Re: Odd behaviour with option forwardfor.

[Patrick Hemmer]

On 2017/7/22 11:11, Claus Strommer wrote:
> Hi all, I'm seeing some odd behaviour with our haproxy balancer and am
> looking for some insights.
>
> The setup:
>
> I have a webserver that is behind two haproxy balancers (version
> 1.5.18 on EL7), which are behind CloudFlare.   In effect the request goes
>
> client->CF->haproxy1->haproxy2->server. 
>
> On both haproxy balancers I have "option forwardfor" and "capture
> request header X-Forwarded-For len 128" set.  On the server I also
> capture X-Forwarded-For
>
> Now here is where the odd behaviour (*highlighted*) happens:
>
> * haproxy1 logs the full X-Forwarded-For header.
> * *haproxy2 only logs the IP of the CF proxy (the last address in
> X-Forwarded-For)*
> * server logs the full X-Forwarded-For header.
> * If I turn off "option forwardfor" on haproxy1, then haproxy2 logs
> the full header as received by CF. 
> * Changing the length of the capture request does not seem to make a
> difference.
> * I noticed that haproxy uses spaces after the comma between the
> header entries, but CF does not.  I tried replicating this issue with
> a direct curl request to haproxy2 replicating the x-forwarded-for
> header that haproxy1 would have sent, and I cannot reproduce the issue.
>
> The only thing that I notice is that CF
>
> Am I missing something obvious here?  Below are the full options I'm
> using on haproxy1 and haproxy2.  Everything after that is ACLs
>
> defaults
> modehttp
> log global
> option  httplog
> option  dontlognull
> option http-server-close
> option forwardfor   except 127.0.0.0/8 
> option  redispatch
> retries 3
>
> frontend  http *:80
> mode http
> reqadd X-Forwarded-Proto:\ https
> redirect scheme https code 301
>
> frontend https
> bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
> mode http
> capture request header Host len 50
>

The "option forwardfor" setting appends a complete new header, not
appends the value to an existing header. From the docs on "option
forwardfor":
> Enable insertion of the X-Forwarded-For header to requests sent to servers
...
> this header is always appended at the end of the existing header list

Your header capture is grabbing the last X-Forwarded-For header.
On issues like this, you should perform a packet capture. It would make
the issue immediately apparent.

Personally I use 2 rules similar to the following to append to
X-Forwarded-For:

  http-request set-header X-Forwarded-For %[req.fhdr(X-Forwarded-For)],\
%[src] if { req.fhdr(X-Forwarded-For) -m found }
  http-request set-header X-Forwarded-For %[src] if !{
req.fhdr(X-Forwarded-For) -m found }

-Patrick

Odd behaviour with option forwardfor.

[Claus Strommer]

Hi all, I'm seeing some odd behaviour with our haproxy balancer and am
looking for some insights.

The setup:

I have a webserver that is behind two haproxy balancers (version 1.5.18 on
EL7), which are behind CloudFlare.   In effect the request goes

client->CF->haproxy1->haproxy2->server.

On both haproxy balancers I have "option forwardfor" and "capture request
header X-Forwarded-For len 128" set.  On the server I also capture
X-Forwarded-For

Now here is where the odd behaviour (*highlighted*) happens:

* haproxy1 logs the full X-Forwarded-For header.
* *haproxy2 only logs the IP of the CF proxy (the last address in
X-Forwarded-For)*
* server logs the full X-Forwarded-For header.
* If I turn off "option forwardfor" on haproxy1, then haproxy2 logs the
full header as received by CF.
* Changing the length of the capture request does not seem to make a
difference.
* I noticed that haproxy uses spaces after the comma between the header
entries, but CF does not.  I tried replicating this issue with a direct
curl request to haproxy2 replicating the x-forwarded-for header that
haproxy1 would have sent, and I cannot reproduce the issue.

The only thing that I notice is that CF

Am I missing something obvious here?  Below are the full options I'm using
on haproxy1 and haproxy2.  Everything after that is ACLs

defaults
modehttp
log global
option  httplog
option  dontlognull
option http-server-close
option forwardfor   except 127.0.0.0/8
option  redispatch
retries 3

frontend  http *:80
mode http
reqadd X-Forwarded-Proto:\ https
redirect scheme https code 301

frontend https
bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
mode http
capture request header Host len 50

Re: AWS ELB as a backend

[Aleksandar Lazic]

Title: Re: AWS ELB as a backend


Hi DHAVAL JAISWAL,

DHAVAL JAISWAL wrote on 21.07.2017:





I have used ELB (public) as a front of Haproxy and ELB (internal) as a backend for the apps server. 

so structure is like as follows. Currently using Haproxy 1.7. However, request is not going to the backend server. 

ELB -> HAPROXY -> ELB -> APPS server. 



Following config in my haproxy.  Let me know what i am doing wrong. 


backend mybackend

server server1 internal-testinelbtomcat-193184.ap-southeast-1.elb.amazonaws.com



Do you have a set resolver?

http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.3

Please post also the following data, thanks.

haproxy -vv
anonymized haproxy conf
some error and access logs

-- 
Best Regards
Aleks