Re: Odd behaviour with option forwardfor.

[Aleksandar Lazic]

Hi Willy Tarreau,

Willy Tarreau wrote on 24.07.2017:

> Hi Aleks,

> On Sun, Jul 23, 2017 at 09:50:41AM +0200, Aleksandar Lazic wrote:
>> >  Personally I use 2 rules similar to the following to append to   
>> > X-Forwarded-For:
>> >   
>> >    http-request set-header X-Forwarded-For
>> > %[req.fhdr(X-Forwarded-For)],\ %[src] if { req.fhdr(X-Forwarded-For)   -m 
>> > found }
>> >    http-request set-header X-Forwarded-For %[src] if !{
>> > req.fhdr(X-Forwarded-For) -m found }
>> >   
>> >  -Patrick
>> 
>> But doesn't haproxy do this already?
>> 
>> http://git.haproxy.org/?p=haproxy-1.7.git;a=blob;f=src/proto_http.c;h=94c8d639f6f777241109f605e1e1742f9a39bf33;hb=HEAD#l4639

> It always adds a new header field. But this is strictly equivalent to
> adding a new value to an existing entry, it's just much less expensive
> (no need to scan the list to find one, nor to move bytes around to insert
> a new value).

Thanks willy for confirmation.

> Willy

-- 
Best Regards
Aleks

Re: Odd behaviour with option forwardfor.

[Willy Tarreau]

Hi Aleks,

On Sun, Jul 23, 2017 at 09:50:41AM +0200, Aleksandar Lazic wrote:
> >  Personally I use 2 rules similar to the following to append to   
> > X-Forwarded-For:
> >   
> >    http-request set-header X-Forwarded-For
> > %[req.fhdr(X-Forwarded-For)],\ %[src] if { req.fhdr(X-Forwarded-For)   -m 
> > found }
> >    http-request set-header X-Forwarded-For %[src] if !{
> > req.fhdr(X-Forwarded-For) -m found }
> >   
> >  -Patrick
> 
> But doesn't haproxy do this already?
> 
> http://git.haproxy.org/?p=haproxy-1.7.git;a=blob;f=src/proto_http.c;h=94c8d639f6f777241109f605e1e1742f9a39bf33;hb=HEAD#l4639

It always adds a new header field. But this is strictly equivalent to
adding a new value to an existing entry, it's just much less expensive
(no need to scan the list to find one, nor to move bytes around to insert
a new value).

Willy

Re: Odd behaviour with option forwardfor.

[Aleksandar Lazic]

Hi Claus,

Claus Strommer wrote on 23.07.2017:

> Hi Aleks,
>
> Patrick's solution is correct.  What I was expecting was that HAProxy would 
> take
>
> X-Forwarded-For: 1.1.1.1
>
> and produce
>
> X-Forwarded-For: 1.1.1.1,2.2.2.2
>
> but what it actually does is produce
>
> X-Forwarded-For: 1.1.1.1
> X-Forwarded-For: 2.2.2.2
>
> The backend concatenates the lines and treats them like what I was
> expecting, but when I do a header capture in the second HAProxy it only reads 
> the second line.

Thanks for clarification.
I thought that haproxy produce the concated header.

Best regards
Aleks

> And yes, my snippet is missing the header capture on the http
> frontend, thanks for spotting that. 
>
> On Jul 23, 2017 03:50, "Aleksandar Lazic"  wrote:

> Hi Patrick Hemmer,


>  Patrick Hemmer wrote on 22.07.2017:

 >> On 2017/7/22 11:11, Claus Strommer wrote:
 >>
 >> Hi all, I'm seeing some odd behaviour with our
 >> haproxy balancer and am looking for some insights.
 >>
 >>                        The setup:
 >>
 >>                        I have a webserver that is behind two haproxy
 >> balancers (version 1.5.18 on EL7), which are
 >> behind CloudFlare.   In effect the request goes
 >>
 >>                           
 >> client->CF->haproxy1->haproxy2->server. 
 >>
 >>                        On both haproxy balancers I have "option
 >> forwardfor" and "capture request header
 >> X-Forwarded-For len 128" set.
 >> On the server I also capture X-Forwarded-For
 >>
 >> Now here is where the odd behaviour
 >> (highlighted)                     happens:
 >>
 >>              * haproxy1 logs the full X-Forwarded-For header.
 >>              * haproxy2 only logs the IP of the CF proxy (the   last 
 >>address in X-Forwarded-For)
 >>              * server logs the full X-Forwarded-For header.
 >>              * If I turn off "option forwardfor" on haproxy1, then
 >> haproxy2 logs the full header as received by CF. 
 >>              * Changing the length of the capture request does not
 >> seem to   make a difference.
 >>
 >> * I noticed that haproxy uses spaces after the comma
 >> between the header entries, but CF does not.  I tried
 >> replicating this issue with a direct curl request to haproxy2
 >> replicating the x-forwarded-for header that haproxy1 would
 >> have sent, and I cannot reproduce the issue.
 >>
 >> The only thing that I notice is that CF
 >>
 >>
 >>            Am I missing something obvious here?  Below are the full
 >> options         I'm using on haproxy1 and haproxy2.  Everything after that 
 >> is         ACLs
 >>
 >>   defaults
 >>                mode    http
 >>                log global
 >>                option  httplog
 >>                option  dontlognull
 >>                option http-server-close
 >>                option forwardfor   except 127.0.0.0/8
 >>                option  redispatch
 >>                retries 3
 >>
 >>            frontend  http *:80
 >>                mode http
 >>                reqadd X-Forwarded-Proto:\ https

> I miss here the

>  capture request header X-Forwarded-For len 15


 >>                redirect scheme https code 301
 >>
 >>            frontend https
 >>                bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
 >>                mode http
 >>                capture request header Host len 50

> I miss here the

>  capture request header X-Forwarded-For len 15

>  Can you try to run haproxy for a short time in debug mode?

> http://cbonte.github.io/haproxy-dconv/1.7/management.html#3

>  -d


 >>      The "option forwardfor" setting appends a complete new header,
 >> not     appends the value to an existing header. From the docs on "option   
 >>   forwardfor":
 >>      > Enable insertion of the X-Forwarded-For header to requests sent     
 >>to servers
 >>      ...
 >>      > this header is always appended at the end of the existing     header 
 >>list
 >>
 >>      Your header capture is grabbing the last X-Forwarded-For header.
 >>      On issues like this, you should perform a packet capture. It
 >> would     make the issue immediately apparent.
 >>
 >>      Personally I use 2 rules similar to the following to append to   
 >>X-Forwarded-For:
 >>
 >>        http-request set-header X-Forwarded-For
 >> %[req.fhdr(X-Forwarded-For)],\ %[src] if { req.fhdr(X-Forwarded-For)   -m 
 >> found }
 >>        http-request set-header X-Forwarded-For %[src] if !{
 >> req.fhdr(X-Forwarded-For) -m found }
 >>
 >>      -Patrick

> But doesn't haproxy do this already?

> http://git.haproxy.org/?p=haproxy-1.7.git;a=blob;f=src/proto_http.c;h=94c8d639f6f777241109f605e1e1742f9a39bf33;hb=HEAD#l4639


>  --
>  Best Regards
>  Aleks






-- 
Best Regards
Aleks

Re: Odd behaviour with option forwardfor.

[Claus Strommer]

Hi Aleks,

Patrick's solution is correct.  What I was expecting was that HAProxy would
take

X-Forwarded-For: 1.1.1.1

and produce

X-Forwarded-For: 1.1.1.1,2.2.2.2

but what it actually does is produce

X-Forwarded-For: 1.1.1.1
X-Forwarded-For: 2.2.2.2

The backend concatenates the lines and treats them like what I was
expecting, but when I do a header capture in the second HAProxy it only
reads the second line.

And yes, my snippet is missing the header capture on the http frontend,
thanks for spotting that.



On Jul 23, 2017 03:50, "Aleksandar Lazic"  wrote:

Hi Patrick Hemmer,

Patrick Hemmer wrote on 22.07.2017:

> On 2017/7/22 11:11, Claus Strommer wrote:
>
> Hi all, I'm seeing some odd behaviour with our
> haproxy balancer and am looking for some insights.
>
>The setup:
>
>I have a webserver that is behind two haproxy
> balancers (version 1.5.18 on EL7), which are
> behind CloudFlare.   In effect the request goes
>
>
> client->CF->haproxy1->haproxy2->server.
>
>On both haproxy balancers I have "option
> forwardfor" and "capture request header
> X-Forwarded-For len 128" set.
> On the server I also capture X-Forwarded-For
>
> Now here is where the odd behaviour
> (highlighted) happens:
>
>  * haproxy1 logs the full X-Forwarded-For header.
>  * haproxy2 only logs the IP of the CF proxy (the   last
address in X-Forwarded-For)
>  * server logs the full X-Forwarded-For header.
>  * If I turn off "option forwardfor" on haproxy1, then
> haproxy2 logs the full header as received by CF.
>  * Changing the length of the capture request does not
> seem to   make a difference.
>
> * I noticed that haproxy uses spaces after the comma
> between the header entries, but CF does not.  I tried
> replicating this issue with a direct curl request to haproxy2
> replicating the x-forwarded-for header that haproxy1 would
> have sent, and I cannot reproduce the issue.
>
> The only thing that I notice is that CF
>
>
>Am I missing something obvious here?  Below are the full
> options I'm using on haproxy1 and haproxy2.  Everything after
that is ACLs
>
>   defaults
>modehttp
>log global
>option  httplog
>option  dontlognull
>option http-server-close
>option forwardfor   except 127.0.0.0/8
>option  redispatch
>retries 3
>
>frontend  http *:80
>mode http
>reqadd X-Forwarded-Proto:\ https

I miss here the

capture request header X-Forwarded-For len 15

>redirect scheme https code 301
>
>frontend https
>bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
>mode http
>capture request header Host len 50

I miss here the

capture request header X-Forwarded-For len 15

Can you try to run haproxy for a short time in debug mode?

http://cbonte.github.io/haproxy-dconv/1.7/management.html#3

-d

>  The "option forwardfor" setting appends a complete new header,
> not appends the value to an existing header. From the docs on
"option forwardfor":
>  > Enable insertion of the X-Forwarded-For header to requests sent
 to servers
>  ...
>  > this header is always appended at the end of the existing
 header list
>
>  Your header capture is grabbing the last X-Forwarded-For header.
>  On issues like this, you should perform a packet capture. It
> would make the issue immediately apparent.
>
>  Personally I use 2 rules similar to the following to append to
 X-Forwarded-For:
>
>http-request set-header X-Forwarded-For
> %[req.fhdr(X-Forwarded-For)],\ %[src] if { req.fhdr(X-Forwarded-For)   -m
found }
>http-request set-header X-Forwarded-For %[src] if !{
> req.fhdr(X-Forwarded-For) -m found }
>
>  -Patrick

But doesn't haproxy do this already?

http://git.haproxy.org/?p=haproxy-1.7.git;a=blob;f=src/proto_http.c;h=
94c8d639f6f777241109f605e1e1742f9a39bf33;hb=HEAD#l4639


--
Best Regards
Aleks

Re: Odd behaviour with option forwardfor.

[Aleksandar Lazic]

Hi Patrick Hemmer,

Patrick Hemmer wrote on 22.07.2017:

> On 2017/7/22 11:11, Claus Strommer wrote:
>   
> Hi all, I'm seeing some odd behaviour with our  
> haproxy balancer and am looking for some insights.
>   
>The setup:
>   
>I have a webserver that is behind two haproxy 
> balancers (version 1.5.18 on EL7), which are  
> behind CloudFlare.   In effect the request goes
>   
>     
> client->CF->haproxy1->haproxy2->server.   
>   
>On both haproxy balancers I have "option  
> forwardfor" and "capture request header  
> X-Forwarded-For len 128" set.
> On the server I also capture X-Forwarded-For
>   
> Now here is where the odd behaviour
> (highlighted) happens:
>   
>  * haproxy1 logs the full X-Forwarded-For header. 
>  * haproxy2 only logs the IP of the CF proxy (the   last address 
> in X-Forwarded-For)
>  * server logs the full X-Forwarded-For header.
>  * If I turn off "option forwardfor" on haproxy1, then 
> haproxy2 logs the full header as received by CF.  
>  * Changing the length of the capture request does not
> seem to   make a difference.
> 
> * I noticed that haproxy uses spaces after the comma  
> between the header entries, but CF does not.  I tried  
> replicating this issue with a direct curl request to haproxy2 
> replicating the x-forwarded-for header that haproxy1 would  
> have sent, and I cannot reproduce the issue. 
> 
> The only thing that I notice is that CF 
> 
>
>Am I missing something obvious here?  Below are the full
> options I'm using on haproxy1 and haproxy2.  Everything after that is 
> ACLs
>   
>   defaults
>    mode    http
>    log global
>    option  httplog
>    option  dontlognull
>    option http-server-close
>    option forwardfor   except 127.0.0.0/8
>    option  redispatch
>    retries 3
>   
>frontend  http *:80
>    mode http
>    reqadd X-Forwarded-Proto:\ https

I miss here the

capture request header X-Forwarded-For len 15

>    redirect scheme https code 301
>   
>frontend https
>    bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
>    mode http
>    capture request header Host len 50

I miss here the

capture request header X-Forwarded-For len 15

Can you try to run haproxy for a short time in debug mode?

http://cbonte.github.io/haproxy-dconv/1.7/management.html#3

-d
   
>  The "option forwardfor" setting appends a complete new header,
> not appends the value to an existing header. From the docs on "option 
> forwardfor":
>  > Enable insertion of the X-Forwarded-For header to requests sent to 
> servers
>  ...
>  > this header is always appended at the end of the existing header 
> list
>   
>  Your header capture is grabbing the last X-Forwarded-For header.
>  On issues like this, you should perform a packet capture. It
> would make the issue immediately apparent.
>   
>  Personally I use 2 rules similar to the following to append to   
> X-Forwarded-For:
>   
>    http-request set-header X-Forwarded-For
> %[req.fhdr(X-Forwarded-For)],\ %[src] if { req.fhdr(X-Forwarded-For)   -m 
> found }
>    http-request set-header X-Forwarded-For %[src] if !{
> req.fhdr(X-Forwarded-For) -m found }
>   
>  -Patrick

But doesn't haproxy do this already?

http://git.haproxy.org/?p=haproxy-1.7.git;a=blob;f=src/proto_http.c;h=94c8d639f6f777241109f605e1e1742f9a39bf33;hb=HEAD#l4639


-- 
Best Regards
Aleks

Re: Odd behaviour with option forwardfor.

[Claus Strommer]

Thanks Patrick, your explanation makes sense.

On Sat, Jul 22, 2017 at 12:28 PM, Patrick Hemmer 
wrote:

> On 2017/7/22 11:11, Claus Strommer wrote:
>
> Hi all, I'm seeing some odd behaviour with our haproxy balancer and am
> looking for some insights.
>
> The setup:
>
> I have a webserver that is behind two haproxy balancers (version 1.5.18 on
> EL7), which are behind CloudFlare.   In effect the request goes
>
> client->CF->haproxy1->haproxy2->server.
>
> On both haproxy balancers I have "option forwardfor" and "capture request
> header X-Forwarded-For len 128" set.  On the server I also capture
> X-Forwarded-For
>
> Now here is where the odd behaviour (*highlighted*) happens:
>
> * haproxy1 logs the full X-Forwarded-For header.
> * *haproxy2 only logs the IP of the CF proxy (the last address in
> X-Forwarded-For)*
> * server logs the full X-Forwarded-For header.
> * If I turn off "option forwardfor" on haproxy1, then haproxy2 logs the
> full header as received by CF.
> * Changing the length of the capture request does not seem to make a
> difference.
> * I noticed that haproxy uses spaces after the comma between the header
> entries, but CF does not.  I tried replicating this issue with a direct
> curl request to haproxy2 replicating the x-forwarded-for header that
> haproxy1 would have sent, and I cannot reproduce the issue.
>
> The only thing that I notice is that CF
>
> Am I missing something obvious here?  Below are the full options I'm using
> on haproxy1 and haproxy2.  Everything after that is ACLs
>
> defaults
> modehttp
> log global
> option  httplog
> option  dontlognull
> option http-server-close
> option forwardfor   except 127.0.0.0/8
> option  redispatch
> retries 3
>
> frontend  http *:80
> mode http
> reqadd X-Forwarded-Proto:\ https
> redirect scheme https code 301
>
> frontend https
> bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
> mode http
> capture request header Host len 50
>
>
> The "option forwardfor" setting appends a complete new header, not appends
> the value to an existing header. From the docs on "option forwardfor":
> > Enable insertion of the X-Forwarded-For header to requests sent to
> servers
> ...
> > this header is always appended at the end of the existing header list
>
> Your header capture is grabbing the last X-Forwarded-For header.
> On issues like this, you should perform a packet capture. It would make
> the issue immediately apparent.
>
> Personally I use 2 rules similar to the following to append to
> X-Forwarded-For:
>
>   http-request set-header X-Forwarded-For %[req.fhdr(X-Forwarded-For)],\
> %[src] if { req.fhdr(X-Forwarded-For) -m found }
>   http-request set-header X-Forwarded-For %[src] if !{
> req.fhdr(X-Forwarded-For) -m found }
>
> -Patrick
>

Re: Odd behaviour with option forwardfor.

[Patrick Hemmer]

On 2017/7/22 11:11, Claus Strommer wrote:
> Hi all, I'm seeing some odd behaviour with our haproxy balancer and am
> looking for some insights.
>
> The setup:
>
> I have a webserver that is behind two haproxy balancers (version
> 1.5.18 on EL7), which are behind CloudFlare.   In effect the request goes
>
> client->CF->haproxy1->haproxy2->server. 
>
> On both haproxy balancers I have "option forwardfor" and "capture
> request header X-Forwarded-For len 128" set.  On the server I also
> capture X-Forwarded-For
>
> Now here is where the odd behaviour (*highlighted*) happens:
>
> * haproxy1 logs the full X-Forwarded-For header.
> * *haproxy2 only logs the IP of the CF proxy (the last address in
> X-Forwarded-For)*
> * server logs the full X-Forwarded-For header.
> * If I turn off "option forwardfor" on haproxy1, then haproxy2 logs
> the full header as received by CF. 
> * Changing the length of the capture request does not seem to make a
> difference.
> * I noticed that haproxy uses spaces after the comma between the
> header entries, but CF does not.  I tried replicating this issue with
> a direct curl request to haproxy2 replicating the x-forwarded-for
> header that haproxy1 would have sent, and I cannot reproduce the issue.
>
> The only thing that I notice is that CF
>
> Am I missing something obvious here?  Below are the full options I'm
> using on haproxy1 and haproxy2.  Everything after that is ACLs
>
> defaults
> modehttp
> log global
> option  httplog
> option  dontlognull
> option http-server-close
> option forwardfor   except 127.0.0.0/8 
> option  redispatch
> retries 3
>
> frontend  http *:80
> mode http
> reqadd X-Forwarded-Proto:\ https
> redirect scheme https code 301
>
> frontend https
> bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
> mode http
> capture request header Host len 50
>

The "option forwardfor" setting appends a complete new header, not
appends the value to an existing header. From the docs on "option
forwardfor":
> Enable insertion of the X-Forwarded-For header to requests sent to servers
...
> this header is always appended at the end of the existing header list

Your header capture is grabbing the last X-Forwarded-For header.
On issues like this, you should perform a packet capture. It would make
the issue immediately apparent.

Personally I use 2 rules similar to the following to append to
X-Forwarded-For:

  http-request set-header X-Forwarded-For %[req.fhdr(X-Forwarded-For)],\
%[src] if { req.fhdr(X-Forwarded-For) -m found }
  http-request set-header X-Forwarded-For %[src] if !{
req.fhdr(X-Forwarded-For) -m found }

-Patrick

Odd behaviour with option forwardfor.

[Claus Strommer]

Hi all, I'm seeing some odd behaviour with our haproxy balancer and am
looking for some insights.

The setup:

I have a webserver that is behind two haproxy balancers (version 1.5.18 on
EL7), which are behind CloudFlare.   In effect the request goes

client->CF->haproxy1->haproxy2->server.

On both haproxy balancers I have "option forwardfor" and "capture request
header X-Forwarded-For len 128" set.  On the server I also capture
X-Forwarded-For

Now here is where the odd behaviour (*highlighted*) happens:

* haproxy1 logs the full X-Forwarded-For header.
* *haproxy2 only logs the IP of the CF proxy (the last address in
X-Forwarded-For)*
* server logs the full X-Forwarded-For header.
* If I turn off "option forwardfor" on haproxy1, then haproxy2 logs the
full header as received by CF.
* Changing the length of the capture request does not seem to make a
difference.
* I noticed that haproxy uses spaces after the comma between the header
entries, but CF does not.  I tried replicating this issue with a direct
curl request to haproxy2 replicating the x-forwarded-for header that
haproxy1 would have sent, and I cannot reproduce the issue.

The only thing that I notice is that CF

Am I missing something obvious here?  Below are the full options I'm using
on haproxy1 and haproxy2.  Everything after that is ACLs

defaults
modehttp
log global
option  httplog
option  dontlognull
option http-server-close
option forwardfor   except 127.0.0.0/8
option  redispatch
retries 3

frontend  http *:80
mode http
reqadd X-Forwarded-Proto:\ https
redirect scheme https code 301

frontend https
bind *:443 ssl crt /etc/pki/tls/certs/hacert.pem
mode http
capture request header Host len 50