Re: ACL randomly failing

2017-02-14 Thread Willy Tarreau 
On Tue, Feb 14, 2017 at 11:36:14AM +0100, Cyril Bonté wrote:
> > De: "Willy Tarreau" 
> > À: "Mathieu Poussin" 
> > Cc: "Daniel Schneller" , 
> > "haproxy+h...@formilux.org" 
> > Envoyé: Mardi 14 Février 2017 10:18:30
> > Objet: Re: ACL randomly failing
> > 
> > On Tue, Feb 14, 2017 at 10:04:30AM +0100, Mathieu Poussin wrote:
> > > Ok I just found why, ports can be shared because of SO_REUSEADDR...
> > > What's a surprise :)
> > 
> > SO_REUSEPORT to be precise :-)
> > 
> > > Is there a way to force disable the use of this flag by HAProxy ?
> > 
> > We merged a patch for this recently, the option is "noreuseport" on
> > the
> > "bind" line, it's in all 1.6 and backported into 1.6.10.
> 
> in all 1.7, to be precise :-P

Oops sorry, that's what I wanted to say but it seems that with aging
my fingers get stronger than my brain :-)

Willy

Re: ACL randomly failing

2017-02-14 Thread Cyril Bonté 
> De: "Willy Tarreau" 
> À: "Mathieu Poussin" 
> Cc: "Daniel Schneller" , 
> "haproxy+h...@formilux.org" 
> Envoyé: Mardi 14 Février 2017 10:18:30
> Objet: Re: ACL randomly failing
> 
> On Tue, Feb 14, 2017 at 10:04:30AM +0100, Mathieu Poussin wrote:
> > Ok I just found why, ports can be shared because of SO_REUSEADDR...
> > What's a surprise :)
> 
> SO_REUSEPORT to be precise :-)
> 
> > Is there a way to force disable the use of this flag by HAProxy ?
> 
> We merged a patch for this recently, the option is "noreuseport" on
> the
> "bind" line, it's in all 1.6 and backported into 1.6.10.

in all 1.7, to be precise :-P

Cheers !
Cyril Bonté

Re: ACL randomly failing

2017-02-14 Thread Willy Tarreau 
On Tue, Feb 14, 2017 at 10:04:30AM +0100, Mathieu Poussin wrote:
> Ok I just found why, ports can be shared because of SO_REUSEADDR... What's a 
> surprise :)

SO_REUSEPORT to be precise :-)

> Is there a way to force disable the use of this flag by HAProxy ?

We merged a patch for this recently, the option is "noreuseport" on the
"bind" line, it's in all 1.6 and backported into 1.6.10.

Regards,
Willy

Re: ACL randomly failing

2017-02-14 Thread Mathieu Poussin 
Ok I just found why, ports can be shared because of SO_REUSEADDR… What’s a 
surprise :)

Is there a way to force disable the use of this flag by HAProxy ?
 
> On 14 Feb 2017, at 09:58, Mathieu Poussin  wrote:
> 
> Hello Daniel.
> 
> I just checked that and you are right.
> I had both the legacy haproxy 1.4 (/usr/sbin/haproxy) and the compiled 1.7 
> (/usr/local/sbin/haproxy) running for an unknown reason…
> 
> The most surprising thing is that they were both listening to the same ports… 
> Aren’t socket binding supposed to be exclusive ? o_O
> 
> tcp0  0 0.0.0.0:0.0.0.0:*   LISTEN
>   18493/haproxy
> tcp0  0 0.0.0.0:0.0.0.0:*   LISTEN
>   20611/haproxy
> tcp0  0 0.0.0.0:80000.0.0.0:*   LISTEN
>   18493/haproxy
> tcp0  0 0.0.0.0:80000.0.0.0:*   LISTEN
>   20611/haproxy
> tcp0  0 0.0.0.0:80010.0.0.0:*   LISTEN
>   18493/haproxy
> tcp0  0 0.0.0.0:80010.0.0.0:*   LISTEN
>   20611/haproxy
> tcp0  0 0.0.0.0:80020.0.0.0:*   LISTEN
>   18493/haproxy
> tcp0  0 0.0.0.0:80020.0.0.0:*   LISTEN
>   20611/haproxy
> tcp0  0 0.0.0.0:80030.0.0.0:*   LISTEN
>   18493/haproxy
> tcp0  0 0.0.0.0:80030.0.0.0:*   LISTEN
>   20611/haproxy
> tcp0  0 0.0.0.0:80040.0.0.0:*   LISTEN
>   18493/haproxy
> tcp0  0 0.0.0.0:80040.0.0.0:*   LISTEN
>   20611/haproxy
> 
> Thank you :)
> Best regards,
> Mathieu
> 
>> On 13 Feb 2017, at 17:38, Daniel Schneller 
>> > > wrote:
>> 
>> Mathieu, 
>> 
>> I have often been fooled like this by multiple haproxy instances running at 
>> the same time.
>> Whenever I had restarted them with config changes there were sometimes open 
>> client connections keeping instances with older configs alive. Those would 
>> respond to a random set of the connections.
>> So I suggest you make sure first you have exactly one instance running, e. 
>> g. with “ps aux | grep haproxy”.
>> 
>> Daniel
>> 
>> -- 
>> Daniel Schneller
>> Principal Cloud Engineer
>>  
>> CenterDevice GmbH  | Hochstraße 11
>>| 42697 Solingen
>> tel: +49 1754155711| Deutschland
>> daniel.schnel...@centerdevice.de    
>> | www.centerdevice.de 
>> 
>> Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
>> Michael Rosbach, Handelsregister-Nr.: HRB 18655,
>> HR-Gericht: Bonn, USt-IdNr.: DE-815299431
>> 
>> 
>>> On 13. Feb. 2017, at 15:45, Mathieu Poussin >> > wrote:
>>> 
>>> Hello.
>>> 
>>> I have setup HAProxy on our environment and I can see a very strange 
>>> behaviour. 
>>> 
>>> I have the following configuration (Just a part of it) :
>>> 
>>> global
>>> chroot /var/lib/haproxy
>>> user haproxy
>>> group haproxy
>>> daemon
>>> tune.maxrewrite 4096
>>> 
>>> 
>>> ### Defaults ###
>>> 
>>> 
>>> defaults
>>> modehttp
>>> option  httplog
>>> option  dontlog-normal
>>> option  dontlognull
>>> option  log-health-checks
>>> option  redispatch
>>> option http-server-close
>>> unique-id-header X-LB-Request-ID
>>> log-format %{+Q}r\ %ST\ "%CC"\ "%hr"\ "%CS"\ "%hs"\ %ID
>>> 
>>> timeout connect 5000
>>> timeout client 5
>>> timeout server 5
>>> 
>>> frontend websitemanager
>>> bind *:8004
>>> log global
>>> capture request header Host len 128
>>> capture request header X-Real-IP len 128
>>> capture request header X-LB-Request-ID len 128
>>> capture request header X-HAProxy-Key len 128
>>> http-request set-var(txn.x_haproxy_key) req.hdr(X-HAProxy-Key)
>>> http-request set-var(txn.x_real_ip) req.hdr(X-Real-IP)
>>> http-request set-var(txn.url) url
>>> mode http
>>> default_backend websitemanager
>>> 
>>> backend websitemanager
>>> mode http
>>> log global
>>> balance roundrobin
>>> option httpchk GET /health/ HTTP/1.0
>>> http-check expect ! string false
>>> acl debug_headers var(txn.x_real_ip) xxx.xxx.xxx.xxx
>>> acl debug_headers var(txn.x_haproxy_key) -m str -i xxx
>>> acl debug_headers var(txn.referer) -m sub -i haproxy-key=xxx
>>> acl debug_headers var(txn.url) -m sub -i haproxy-key=xxx
>>> http-response set-header X-HAProxy-Frontend-Name "%f" if 
>>> debug_headers
>>> http-response set-header X-HAProxy-Frontend-Socket "%fi:%fp" if

Re: ACL randomly failing

2017-02-14 Thread Mathieu Poussin 
Hello Daniel.

I just checked that and you are right.
I had both the legacy haproxy 1.4 (/usr/sbin/haproxy) and the compiled 1.7 
(/usr/local/sbin/haproxy) running for an unknown reason…

The most surprising thing is that they were both listening to the same ports… 
Aren’t socket binding supposed to be exclusive ? o_O

tcp0  0 0.0.0.0:0.0.0.0:*   LISTEN  
18493/haproxy
tcp0  0 0.0.0.0:0.0.0.0:*   LISTEN  
20611/haproxy
tcp0  0 0.0.0.0:80000.0.0.0:*   LISTEN  
18493/haproxy
tcp0  0 0.0.0.0:80000.0.0.0:*   LISTEN  
20611/haproxy
tcp0  0 0.0.0.0:80010.0.0.0:*   LISTEN  
18493/haproxy
tcp0  0 0.0.0.0:80010.0.0.0:*   LISTEN  
20611/haproxy
tcp0  0 0.0.0.0:80020.0.0.0:*   LISTEN  
18493/haproxy
tcp0  0 0.0.0.0:80020.0.0.0:*   LISTEN  
20611/haproxy
tcp0  0 0.0.0.0:80030.0.0.0:*   LISTEN  
18493/haproxy
tcp0  0 0.0.0.0:80030.0.0.0:*   LISTEN  
20611/haproxy
tcp0  0 0.0.0.0:80040.0.0.0:*   LISTEN  
18493/haproxy
tcp0  0 0.0.0.0:80040.0.0.0:*   LISTEN  
20611/haproxy

Thank you :)
Best regards,
Mathieu

> On 13 Feb 2017, at 17:38, Daniel Schneller 
>  wrote:
> 
> Mathieu, 
> 
> I have often been fooled like this by multiple haproxy instances running at 
> the same time.
> Whenever I had restarted them with config changes there were sometimes open 
> client connections keeping instances with older configs alive. Those would 
> respond to a random set of the connections.
> So I suggest you make sure first you have exactly one instance running, e. g. 
> with “ps aux | grep haproxy”.
> 
> Daniel
> 
> -- 
> Daniel Schneller
> Principal Cloud Engineer
>  
> CenterDevice GmbH  | Hochstraße 11
>| 42697 Solingen
> tel: +49 1754155711| Deutschland
> daniel.schnel...@centerdevice.de    
> | www.centerdevice.de 
> 
> Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
> Michael Rosbach, Handelsregister-Nr.: HRB 18655,
> HR-Gericht: Bonn, USt-IdNr.: DE-815299431
> 
> 
>> On 13. Feb. 2017, at 15:45, Mathieu Poussin > > wrote:
>> 
>> Hello.
>> 
>> I have setup HAProxy on our environment and I can see a very strange 
>> behaviour. 
>> 
>> I have the following configuration (Just a part of it) :
>> 
>> global
>> chroot /var/lib/haproxy
>> user haproxy
>> group haproxy
>> daemon
>> tune.maxrewrite 4096
>> 
>> 
>> ### Defaults ###
>> 
>> 
>> defaults
>> modehttp
>> option  httplog
>> option  dontlog-normal
>> option  dontlognull
>> option  log-health-checks
>> option  redispatch
>> option http-server-close
>> unique-id-header X-LB-Request-ID
>> log-format %{+Q}r\ %ST\ "%CC"\ "%hr"\ "%CS"\ "%hs"\ %ID
>> 
>> timeout connect 5000
>> timeout client 5
>> timeout server 5
>> 
>> frontend websitemanager
>> bind *:8004
>> log global
>> capture request header Host len 128
>> capture request header X-Real-IP len 128
>> capture request header X-LB-Request-ID len 128
>>  capture request header X-HAProxy-Key len 128
>>  http-request set-var(txn.x_haproxy_key) req.hdr(X-HAProxy-Key)
>>  http-request set-var(txn.x_real_ip) req.hdr(X-Real-IP)
>> http-request set-var(txn.url) url
>> mode http
>> default_backend websitemanager
>> 
>> backend websitemanager
>> mode http
>> log global
>> balance roundrobin
>> option httpchk GET /health/ HTTP/1.0
>> http-check expect ! string false
>>  acl debug_headers var(txn.x_real_ip) xxx.xxx.xxx.xxx
>> acl debug_headers var(txn.x_haproxy_key) -m str -i xxx
>> acl debug_headers var(txn.referer) -m sub -i haproxy-key=xxx
>> acl debug_headers var(txn.url) -m sub -i haproxy-key=xxx
>> http-response set-header X-HAProxy-Frontend-Name "%f" if 
>> debug_headers
>> http-response set-header X-HAProxy-Frontend-Socket "%fi:%fp" if 
>> debug_headers
>> http-response set-header X-HAProxy-Backend-Group "%b" if 
>> debug_headers
>> http-response set-header X-HAProxy-Backend-Name "%s" if debug_headers
>> http-response set-header X-HAProxy-Backend-Socket "%si:%sp" if 
>> debug_headers
>> http-response set-header X-HAProxy-Via "%H" if debug_headers
>> http-response set-header X-HAProxy-TerminationState "%ts" if

Re: ACL randomly failing

2017-02-13 Thread Daniel Schneller 
Mathieu, 

I have often been fooled like this by multiple haproxy instances running at the 
same time.
Whenever I had restarted them with config changes there were sometimes open 
client connections keeping instances with older configs alive. Those would 
respond to a random set of the connections.
So I suggest you make sure first you have exactly one instance running, e. g. 
with “ps aux | grep haproxy”.

Daniel

-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431


> On 13. Feb. 2017, at 15:45, Mathieu Poussin  wrote:
> 
> Hello.
> 
> I have setup HAProxy on our environment and I can see a very strange 
> behaviour. 
> 
> I have the following configuration (Just a part of it) :
> 
> global
> chroot /var/lib/haproxy
> user haproxy
> group haproxy
> daemon
> tune.maxrewrite 4096
> 
> 
> ### Defaults ###
> 
> 
> defaults
> modehttp
> option  httplog
> option  dontlog-normal
> option  dontlognull
> option  log-health-checks
> option  redispatch
> option http-server-close
> unique-id-header X-LB-Request-ID
> log-format %{+Q}r\ %ST\ "%CC"\ "%hr"\ "%CS"\ "%hs"\ %ID
> 
> timeout connect 5000
> timeout client 5
> timeout server 5
> 
> frontend websitemanager
> bind *:8004
> log global
> capture request header Host len 128
> capture request header X-Real-IP len 128
> capture request header X-LB-Request-ID len 128
>   capture request header X-HAProxy-Key len 128
>   http-request set-var(txn.x_haproxy_key) req.hdr(X-HAProxy-Key)
>   http-request set-var(txn.x_real_ip) req.hdr(X-Real-IP)
> http-request set-var(txn.url) url
> mode http
> default_backend websitemanager
> 
> backend websitemanager
> mode http
> log global
> balance roundrobin
> option httpchk GET /health/ HTTP/1.0
> http-check expect ! string false
>   acl debug_headers var(txn.x_real_ip) xxx.xxx.xxx.xxx
> acl debug_headers var(txn.x_haproxy_key) -m str -i xxx
> acl debug_headers var(txn.referer) -m sub -i haproxy-key=xxx
> acl debug_headers var(txn.url) -m sub -i haproxy-key=xxx
> http-response set-header X-HAProxy-Frontend-Name "%f" if debug_headers
> http-response set-header X-HAProxy-Frontend-Socket "%fi:%fp" if 
> debug_headers
> http-response set-header X-HAProxy-Backend-Group "%b" if debug_headers
> http-response set-header X-HAProxy-Backend-Name "%s" if debug_headers
> http-response set-header X-HAProxy-Backend-Socket "%si:%sp" if 
> debug_headers
> http-response set-header X-HAProxy-Via "%H" if debug_headers
> http-response set-header X-HAProxy-TerminationState "%ts" if 
> debug_headers
> http-response set-header X-Real-IP "%[var(txn.x_real_ip)]"
> server gc-certmgr-live-1 10.0.0.49:80 check observe layer7 on-error 
> mark-down slowstart 10s weight 100
> server gc-certmgr-live-2 10.0.0.50:80 check observe layer7 on-error 
> mark-down slowstart 10s weight 100
> server gc-certmgr-live-3 10.0.0.51:80 check observe layer7 on-error 
> mark-down slowstart 10s weight 100
> 
> And many other fronted/backend combo with the same configuration (The same 
> ACL).
> Basically, I want the X-HAProxy headers to appears in any of the following 
> condition :
> - The connection is coming from HQ (Specific X-Real-IP header)
> - The header X-HAProxy-Key header is present and set to the correct key
> - The Referer contains the key
> - The URL contains the key as parameter
> 
> I have a nginx in front of this setup, that is setting up the X-Real-IP.
> I’ve checked the logs, and the connection is forwarded to HAProxy in all the 
> cases, so nginx is not the cause of the issue (Or at least it’s still 
> forwarding to HAProxy)
> 
> 
> Almost half of the requests are failing the ACL where they should work 
> without issue (Because the source IP matches or because of the connection 
> string.
> 
> It’s completely random, I have no idea why it’s doing that.
> 
> What could be the cause ? I could not find much googling for this issue.
> 
> My version is HA-Proxy version 1.7.2-6edf8f-4
> 
> Thank you.
> Best regards,
> Mathieu
>

ACL randomly failing

2017-02-13 Thread Mathieu Poussin 
Hello.

I have setup HAProxy on our environment and I can see a very strange behaviour. 

I have the following configuration (Just a part of it) :

global
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
tune.maxrewrite 4096


### Defaults ###


defaults
modehttp
option  httplog
option  dontlog-normal
option  dontlognull
option  log-health-checks
option  redispatch
option http-server-close
unique-id-header X-LB-Request-ID
log-format %{+Q}r\ %ST\ "%CC"\ "%hr"\ "%CS"\ "%hs"\ %ID

timeout connect 5000
timeout client 5
timeout server 5

frontend websitemanager
bind *:8004
log global
capture request header Host len 128
capture request header X-Real-IP len 128
capture request header X-LB-Request-ID len 128
capture request header X-HAProxy-Key len 128
http-request set-var(txn.x_haproxy_key) req.hdr(X-HAProxy-Key)
http-request set-var(txn.x_real_ip) req.hdr(X-Real-IP)
http-request set-var(txn.url) url
mode http
default_backend websitemanager

backend websitemanager
mode http
log global
balance roundrobin
option httpchk GET /health/ HTTP/1.0
http-check expect ! string false
acl debug_headers var(txn.x_real_ip) xxx.xxx.xxx.xxx
acl debug_headers var(txn.x_haproxy_key) -m str -i xxx
acl debug_headers var(txn.referer) -m sub -i haproxy-key=xxx
acl debug_headers var(txn.url) -m sub -i haproxy-key=xxx
http-response set-header X-HAProxy-Frontend-Name "%f" if debug_headers
http-response set-header X-HAProxy-Frontend-Socket "%fi:%fp" if 
debug_headers
http-response set-header X-HAProxy-Backend-Group "%b" if debug_headers
http-response set-header X-HAProxy-Backend-Name "%s" if debug_headers
http-response set-header X-HAProxy-Backend-Socket "%si:%sp" if 
debug_headers
http-response set-header X-HAProxy-Via "%H" if debug_headers
http-response set-header X-HAProxy-TerminationState "%ts" if 
debug_headers
http-response set-header X-Real-IP "%[var(txn.x_real_ip)]"
server gc-certmgr-live-1 10.0.0.49:80 check observe layer7 on-error 
mark-down slowstart 10s weight 100
server gc-certmgr-live-2 10.0.0.50:80 check observe layer7 on-error 
mark-down slowstart 10s weight 100
server gc-certmgr-live-3 10.0.0.51:80 check observe layer7 on-error 
mark-down slowstart 10s weight 100

And many other fronted/backend combo with the same configuration (The same ACL).
Basically, I want the X-HAProxy headers to appears in any of the following 
condition :
- The connection is coming from HQ (Specific X-Real-IP header)
- The header X-HAProxy-Key header is present and set to the correct key
- The Referer contains the key
- The URL contains the key as parameter

I have a nginx in front of this setup, that is setting up the X-Real-IP.
I’ve checked the logs, and the connection is forwarded to HAProxy in all the 
cases, so nginx is not the cause of the issue (Or at least it’s still 
forwarding to HAProxy)


Almost half of the requests are failing the ACL where they should work without 
issue (Because the source IP matches or because of the connection string.

It’s completely random, I have no idea why it’s doing that.

What could be the cause ? I could not find much googling for this issue.

My version is HA-Proxy version 1.7.2-6edf8f-4

Thank you.
Best regards,
Mathieu