New commitment invitation - haproxy@formilux.org

[Ervin Marino]

  

  



  


  

  

  

Sarah Smith


  
  
Hi
Namailu User, 


You
have a commitment invitation from Sarah Smith. To view your
commitment invitation please follow this link:


View
Invitation


  

  

  


  

  

  
Copyright
© 2015, Namailu Online Ltd


  
  
|


  
  
User 
Agreement


  
  
|


  
  
Privacy
Policy


  
  
|


  
  
Privacy
Settings


  

  

  

  


 

Re: "proxy haproxy has no server available!"

[Krishna Kumar Unnikrishnan (Engineering)]

Thanks Igor, you pointed me to the correct answer. I removed the check.txt.
I am not sure how the file got missing, my systems were down for a few days
and I moved to KVM last night.

Regards,
- KK

On Tue, Apr 7, 2015 at 11:56 AM, Igor Cicimov <
ig...@encompasscorporation.com> wrote:

> Forgot to cc the list.
>
> -- Forwarded message --
> From: Igor Cicimov 
> Date: Tue, Apr 7, 2015 at 4:25 PM
> Subject: Re: "proxy haproxy has no server available!"
> To: "Krishna Kumar Unnikrishnan (Engineering)" 
>
>
>
>
> On Tue, Apr 7, 2015 at 3:58 PM, Krishna Kumar Unnikrishnan (Engineering) <
> krishna...@flipkart.com> wrote:
>
>> Thanks Igor for the suggestion. I get:
>>
>> root@haproxy-2:/var/www# curl --http1.0 -X HEAD
>> 192.168.122.101:80/check.txt
>> curl: (18) transfer closed with 168 bytes remaining to read
>> root@haproxy-2:/var/www# curl --http1.0 -X HEAD
>> 192.168.122.102:80/check.txt
>> curl: (18) transfer closed with 168 bytes remaining to read
>>
>> And without the flags:
>>
>> root@haproxy-2:/var/www# curl 192.168.122.102:80/check.txt
>> 
>> 404 Not Found
>> 
>> 404 Not Found
>> nginx/1.6.2
>> 
>> 
>>
>> Is this the problem? I am not sure how to fix it.
>>
>>
> Obviously the given txt file does not exist in your nginx document root
> directory. You said you are migrating the setup so wonder how did this use
> to work till now?
>
>
>> Thanks,
>> - KK
>>
>> On Tue, Apr 7, 2015 at 11:10 AM, Igor Cicimov <
>> ig...@encompasscorporation.com> wrote:
>>
>>>
>>>
>>> On Tue, Apr 7, 2015 at 3:24 PM, Krishna Kumar Unnikrishnan (Engineering)
>>>  wrote:
>>>
 Sorry, forgot to mention, this is haproxy version 1.5.11


 On Tue, Apr 7, 2015 at 10:52 AM, Krishna Kumar Unnikrishnan
 (Engineering)  wrote:

> Hi all,
>
> I am moving from using LXC to KVM for haproxy on my Debian 7 system.
> When I
> start haproxy, I get this error:
> _
> Apr  7 10:38:22 localhost haproxy[3418]: Proxy haproxy started.
> Apr  7 10:38:24 localhost haproxy[3420]: Server haproxy/nginx-1 is
> DOWN, reason Layer4 timeout, check duration: 2000ms. 1 active and 0 backup
> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
> Apr  7 10:38:24 localhost haproxy[3419]: Server haproxy/nginx-1 is
> DOWN, reason Layer4 timeout, check duration: 2001ms. 1 active and 0 backup
> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
> Apr  7 10:38:25 localhost haproxy[3420]: Server haproxy/nginx-2 is
> DOWN, reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
> Apr  7 10:38:25 localhost haproxy[3420]: proxy haproxy has no server
> available!
> Apr  7 10:38:25 localhost haproxy[3419]: Server haproxy/nginx-2 is
> DOWN, reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
> Apr  7 10:38:25 localhost haproxy[3419]: proxy haproxy has no server
> available!
>
> From outside the haproxy, I get the error:
> # wget 192.168.122.112:80
> --2015-04-07 10:48:47--  http://192.168.122.112/
> Connecting to 192.168.122.112:80... connected.
> HTTP request sent, awaiting response... 503 Service Unavailable
> 2015-04-07 10:48:47 ERROR 503: Service Unavailable.
> ___
>
> The config file is:
> global
> log 127.0.0.1   local0
> log 127.0.0.1   local1 notice
> maxconn  65536
> daemon
> quiet
> nbproc 2
> debug
> user haproxy
> group haproxy
>
> defaults
> log global
> modehttp
> option  dontlognull
> retries 3
> option redispatch
> maxconn 65536
> timeout connect 5000
> timeout client  5
> timeout server  5
>
> #listen haproxy 192.168.122.112:80
> listen haproxy *:80
> mode http
> stats enable
> stats auth someuser:somepassword
> balance roundrobin
> option prefer-last-server
> option forwardfor
> option httpchk HEAD /check.txt HTTP/1.0
>

>>> Check if the above health check is really working, you show that
>>> requesting the root page works but we don't see you checking the /check.txt
>>> file (does it exist at all?). Run:
>>>
>>> $ curl --http1.0 -X HEAD 192.168.122.101:80 
>>> /check.txt
>>> $ curl --http1.0 -X HEAD 192.168.122.102:80 
>>> /check.txt
>>>
>>> from the HAP server.
>>>
>>> server nginx-1 192.168.122.101:80 check
> server nginx-2 192.168.122.102:80 check
>
> BTW, I could not use "listen haproxy 192.168.122.112:80", but had to
> use *:80
> as haproxy does not start up with the f

Fwd: "proxy haproxy has no server available!"

[Igor Cicimov]

Forgot to cc the list.

-- Forwarded message --
From: Igor Cicimov 
Date: Tue, Apr 7, 2015 at 4:25 PM
Subject: Re: "proxy haproxy has no server available!"
To: "Krishna Kumar Unnikrishnan (Engineering)" 




On Tue, Apr 7, 2015 at 3:58 PM, Krishna Kumar Unnikrishnan (Engineering) <
krishna...@flipkart.com> wrote:

> Thanks Igor for the suggestion. I get:
>
> root@haproxy-2:/var/www# curl --http1.0 -X HEAD
> 192.168.122.101:80/check.txt
> curl: (18) transfer closed with 168 bytes remaining to read
> root@haproxy-2:/var/www# curl --http1.0 -X HEAD
> 192.168.122.102:80/check.txt
> curl: (18) transfer closed with 168 bytes remaining to read
>
> And without the flags:
>
> root@haproxy-2:/var/www# curl 192.168.122.102:80/check.txt
> 
> 404 Not Found
> 
> 404 Not Found
> nginx/1.6.2
> 
> 
>
> Is this the problem? I am not sure how to fix it.
>
>
Obviously the given txt file does not exist in your nginx document root
directory. You said you are migrating the setup so wonder how did this use
to work till now?


> Thanks,
> - KK
>
> On Tue, Apr 7, 2015 at 11:10 AM, Igor Cicimov <
> ig...@encompasscorporation.com> wrote:
>
>>
>>
>> On Tue, Apr 7, 2015 at 3:24 PM, Krishna Kumar Unnikrishnan (Engineering)
>>  wrote:
>>
>>> Sorry, forgot to mention, this is haproxy version 1.5.11
>>>
>>>
>>> On Tue, Apr 7, 2015 at 10:52 AM, Krishna Kumar Unnikrishnan
>>> (Engineering)  wrote:
>>>
 Hi all,

 I am moving from using LXC to KVM for haproxy on my Debian 7 system.
 When I
 start haproxy, I get this error:
 _
 Apr  7 10:38:22 localhost haproxy[3418]: Proxy haproxy started.
 Apr  7 10:38:24 localhost haproxy[3420]: Server haproxy/nginx-1 is
 DOWN, reason Layer4 timeout, check duration: 2000ms. 1 active and 0 backup
 servers left. 0 essions active, 0 requeued, 0 remaining in queue.
 Apr  7 10:38:24 localhost haproxy[3419]: Server haproxy/nginx-1 is
 DOWN, reason Layer4 timeout, check duration: 2001ms. 1 active and 0 backup
 servers left. 0 essions active, 0 requeued, 0 remaining in queue.
 Apr  7 10:38:25 localhost haproxy[3420]: Server haproxy/nginx-2 is
 DOWN, reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
 servers left. 0 essions active, 0 requeued, 0 remaining in queue.
 Apr  7 10:38:25 localhost haproxy[3420]: proxy haproxy has no server
 available!
 Apr  7 10:38:25 localhost haproxy[3419]: Server haproxy/nginx-2 is
 DOWN, reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
 servers left. 0 essions active, 0 requeued, 0 remaining in queue.
 Apr  7 10:38:25 localhost haproxy[3419]: proxy haproxy has no server
 available!

 From outside the haproxy, I get the error:
 # wget 192.168.122.112:80
 --2015-04-07 10:48:47--  http://192.168.122.112/
 Connecting to 192.168.122.112:80... connected.
 HTTP request sent, awaiting response... 503 Service Unavailable
 2015-04-07 10:48:47 ERROR 503: Service Unavailable.
 ___

 The config file is:
 global
 log 127.0.0.1   local0
 log 127.0.0.1   local1 notice
 maxconn  65536
 daemon
 quiet
 nbproc 2
 debug
 user haproxy
 group haproxy

 defaults
 log global
 modehttp
 option  dontlognull
 retries 3
 option redispatch
 maxconn 65536
 timeout connect 5000
 timeout client  5
 timeout server  5

 #listen haproxy 192.168.122.112:80
 listen haproxy *:80
 mode http
 stats enable
 stats auth someuser:somepassword
 balance roundrobin
 option prefer-last-server
 option forwardfor
 option httpchk HEAD /check.txt HTTP/1.0

>>>
>> Check if the above health check is really working, you show that
>> requesting the root page works but we don't see you checking the /check.txt
>> file (does it exist at all?). Run:
>>
>> $ curl --http1.0 -X HEAD 192.168.122.101:80 
>> /check.txt
>> $ curl --http1.0 -X HEAD 192.168.122.102:80 
>> /check.txt
>>
>> from the HAP server.
>>
>> server nginx-1 192.168.122.101:80 check
 server nginx-2 192.168.122.102:80 check

 BTW, I could not use "listen haproxy 192.168.122.112:80", but had to
 use *:80
 as haproxy does not start up with the former. It seems like haproxy
 startup is
 happening ahead of networking.
 __

 I also stopped/restarted haproxy, but I still get the same error at
 start.

 root@haproxy-2:~# netstat -apn | grep :80
 tcp0  0 0.0.0.0:80  0.0.0.0:*
 LISTEN  3558/haproxy
 ___

Re: "proxy haproxy has no server available!"

[Krishna Kumar Unnikrishnan (Engineering)]

It seems to be a problem with my configuration file. I tried the one from
Section 2.3
of haproxy.org/download/1.5/doc/configuration.txt, and it works good now:

global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
daemon
maxconn 256

defaults
log global
mode http
timeout connect 5000ms
timeout client 5ms
timeout server 5ms

frontend http-in
bind *:80
default_backend servers

backend servers
server nginx-1 192.168.122.101:80 maxconn 32
server nginx-2 192.168.122.102:80 maxconn 32

Thanks,
- KK

On Tue, Apr 7, 2015 at 11:28 AM, Krishna Kumar Unnikrishnan (Engineering) <
krishna...@flipkart.com> wrote:

> Thanks Igor for the suggestion. I get:
>
> root@haproxy-2:/var/www# curl --http1.0 -X HEAD
> 192.168.122.101:80/check.txt
> curl: (18) transfer closed with 168 bytes remaining to read
> root@haproxy-2:/var/www# curl --http1.0 -X HEAD
> 192.168.122.102:80/check.txt
> curl: (18) transfer closed with 168 bytes remaining to read
>
> And without the flags:
>
> root@haproxy-2:/var/www# curl 192.168.122.102:80/check.txt
> 
> 404 Not Found
> 
> 404 Not Found
> nginx/1.6.2
> 
> 
>
> Is this the problem? I am not sure how to fix it.
>
> Thanks,
> - KK
>
> On Tue, Apr 7, 2015 at 11:10 AM, Igor Cicimov <
> ig...@encompasscorporation.com> wrote:
>
>>
>>
>> On Tue, Apr 7, 2015 at 3:24 PM, Krishna Kumar Unnikrishnan (Engineering)
>>  wrote:
>>
>>> Sorry, forgot to mention, this is haproxy version 1.5.11
>>>
>>>
>>> On Tue, Apr 7, 2015 at 10:52 AM, Krishna Kumar Unnikrishnan
>>> (Engineering)  wrote:
>>>
 Hi all,

 I am moving from using LXC to KVM for haproxy on my Debian 7 system.
 When I
 start haproxy, I get this error:
 _
 Apr  7 10:38:22 localhost haproxy[3418]: Proxy haproxy started.
 Apr  7 10:38:24 localhost haproxy[3420]: Server haproxy/nginx-1 is
 DOWN, reason Layer4 timeout, check duration: 2000ms. 1 active and 0 backup
 servers left. 0 essions active, 0 requeued, 0 remaining in queue.
 Apr  7 10:38:24 localhost haproxy[3419]: Server haproxy/nginx-1 is
 DOWN, reason Layer4 timeout, check duration: 2001ms. 1 active and 0 backup
 servers left. 0 essions active, 0 requeued, 0 remaining in queue.
 Apr  7 10:38:25 localhost haproxy[3420]: Server haproxy/nginx-2 is
 DOWN, reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
 servers left. 0 essions active, 0 requeued, 0 remaining in queue.
 Apr  7 10:38:25 localhost haproxy[3420]: proxy haproxy has no server
 available!
 Apr  7 10:38:25 localhost haproxy[3419]: Server haproxy/nginx-2 is
 DOWN, reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
 servers left. 0 essions active, 0 requeued, 0 remaining in queue.
 Apr  7 10:38:25 localhost haproxy[3419]: proxy haproxy has no server
 available!

 From outside the haproxy, I get the error:
 # wget 192.168.122.112:80
 --2015-04-07 10:48:47--  http://192.168.122.112/
 Connecting to 192.168.122.112:80... connected.
 HTTP request sent, awaiting response... 503 Service Unavailable
 2015-04-07 10:48:47 ERROR 503: Service Unavailable.
 ___

 The config file is:
 global
 log 127.0.0.1   local0
 log 127.0.0.1   local1 notice
 maxconn  65536
 daemon
 quiet
 nbproc 2
 debug
 user haproxy
 group haproxy

 defaults
 log global
 modehttp
 option  dontlognull
 retries 3
 option redispatch
 maxconn 65536
 timeout connect 5000
 timeout client  5
 timeout server  5

 #listen haproxy 192.168.122.112:80
 listen haproxy *:80
 mode http
 stats enable
 stats auth someuser:somepassword
 balance roundrobin
 option prefer-last-server
 option forwardfor
 option httpchk HEAD /check.txt HTTP/1.0

>>>
>> Check if the above health check is really working, you show that
>> requesting the root page works but we don't see you checking the /check.txt
>> file (does it exist at all?). Run:
>>
>> $ curl --http1.0 -X HEAD 192.168.122.101:80 
>> /check.txt
>> $ curl --http1.0 -X HEAD 192.168.122.102:80 
>> /check.txt
>>
>> from the HAP server.
>>
>> server nginx-1 192.168.122.101:80 check
 server nginx-2 192.168.122.102:80 check

 BTW, I could not use "listen haproxy 192.168.122.112:80", but had to
 use *:80
 as haproxy does not start up with the former. It seems like haproxy
 startup is
 happening ahead of networking.
 __

 I also stopped/restarted haproxy, but I still get t

Re: "proxy haproxy has no server available!"

[Krishna Kumar Unnikrishnan (Engineering)]

Thanks Igor for the suggestion. I get:

root@haproxy-2:/var/www# curl --http1.0 -X HEAD 192.168.122.101:80/check.txt
curl: (18) transfer closed with 168 bytes remaining to read
root@haproxy-2:/var/www# curl --http1.0 -X HEAD 192.168.122.102:80/check.txt
curl: (18) transfer closed with 168 bytes remaining to read

And without the flags:

root@haproxy-2:/var/www# curl 192.168.122.102:80/check.txt

404 Not Found

404 Not Found
nginx/1.6.2



Is this the problem? I am not sure how to fix it.

Thanks,
- KK

On Tue, Apr 7, 2015 at 11:10 AM, Igor Cicimov <
ig...@encompasscorporation.com> wrote:

>
>
> On Tue, Apr 7, 2015 at 3:24 PM, Krishna Kumar Unnikrishnan (Engineering) <
> krishna...@flipkart.com> wrote:
>
>> Sorry, forgot to mention, this is haproxy version 1.5.11
>>
>>
>> On Tue, Apr 7, 2015 at 10:52 AM, Krishna Kumar Unnikrishnan (Engineering)
>>  wrote:
>>
>>> Hi all,
>>>
>>> I am moving from using LXC to KVM for haproxy on my Debian 7 system.
>>> When I
>>> start haproxy, I get this error:
>>> _
>>> Apr  7 10:38:22 localhost haproxy[3418]: Proxy haproxy started.
>>> Apr  7 10:38:24 localhost haproxy[3420]: Server haproxy/nginx-1 is DOWN,
>>> reason Layer4 timeout, check duration: 2000ms. 1 active and 0 backup
>>> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
>>> Apr  7 10:38:24 localhost haproxy[3419]: Server haproxy/nginx-1 is DOWN,
>>> reason Layer4 timeout, check duration: 2001ms. 1 active and 0 backup
>>> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
>>> Apr  7 10:38:25 localhost haproxy[3420]: Server haproxy/nginx-2 is DOWN,
>>> reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
>>> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
>>> Apr  7 10:38:25 localhost haproxy[3420]: proxy haproxy has no server
>>> available!
>>> Apr  7 10:38:25 localhost haproxy[3419]: Server haproxy/nginx-2 is DOWN,
>>> reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
>>> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
>>> Apr  7 10:38:25 localhost haproxy[3419]: proxy haproxy has no server
>>> available!
>>>
>>> From outside the haproxy, I get the error:
>>> # wget 192.168.122.112:80
>>> --2015-04-07 10:48:47--  http://192.168.122.112/
>>> Connecting to 192.168.122.112:80... connected.
>>> HTTP request sent, awaiting response... 503 Service Unavailable
>>> 2015-04-07 10:48:47 ERROR 503: Service Unavailable.
>>> ___
>>>
>>> The config file is:
>>> global
>>> log 127.0.0.1   local0
>>> log 127.0.0.1   local1 notice
>>> maxconn  65536
>>> daemon
>>> quiet
>>> nbproc 2
>>> debug
>>> user haproxy
>>> group haproxy
>>>
>>> defaults
>>> log global
>>> modehttp
>>> option  dontlognull
>>> retries 3
>>> option redispatch
>>> maxconn 65536
>>> timeout connect 5000
>>> timeout client  5
>>> timeout server  5
>>>
>>> #listen haproxy 192.168.122.112:80
>>> listen haproxy *:80
>>> mode http
>>> stats enable
>>> stats auth someuser:somepassword
>>> balance roundrobin
>>> option prefer-last-server
>>> option forwardfor
>>> option httpchk HEAD /check.txt HTTP/1.0
>>>
>>
> Check if the above health check is really working, you show that
> requesting the root page works but we don't see you checking the /check.txt
> file (does it exist at all?). Run:
>
> $ curl --http1.0 -X HEAD 192.168.122.101:80 
> /check.txt
> $ curl --http1.0 -X HEAD 192.168.122.102:80 
> /check.txt
>
> from the HAP server.
>
> server nginx-1 192.168.122.101:80 check
>>> server nginx-2 192.168.122.102:80 check
>>>
>>> BTW, I could not use "listen haproxy 192.168.122.112:80", but had to
>>> use *:80
>>> as haproxy does not start up with the former. It seems like haproxy
>>> startup is
>>> happening ahead of networking.
>>> __
>>>
>>> I also stopped/restarted haproxy, but I still get the same error at
>>> start.
>>>
>>> root@haproxy-2:~# netstat -apn | grep :80
>>> tcp0  0 0.0.0.0:80  0.0.0.0:*
>>> LISTEN  3558/haproxy
>>> ___
>>> From outside haproxy, I can do a wget/curl" to either of the two servers:
>>>
>>> # wget 192.168.122.101:80
>>> --2015-04-07 10:42:28--  http://192.168.122.101/
>>> Connecting to 192.168.122.101:80... connected.
>>> HTTP request sent, awaiting response... 200 OK
>>> Length: 867 [text/html]
>>> Saving to: `index.html'
>>>
>>> 100%[==>] 867 --.-K/s   in
>>> 0s
>>>
>>> 2015-04-07 10:42:28 (104 MB/s) - `index.html' saved [867/867]
>>> ___
>>>
>>> And I can do the same from haproxy:
>>> root@haproxy-2:~# wget 192.168.12

Re: "proxy haproxy has no server available!"

[Igor Cicimov]

On Tue, Apr 7, 2015 at 3:24 PM, Krishna Kumar Unnikrishnan (Engineering) <
krishna...@flipkart.com> wrote:

> Sorry, forgot to mention, this is haproxy version 1.5.11
>
>
> On Tue, Apr 7, 2015 at 10:52 AM, Krishna Kumar Unnikrishnan (Engineering)
>  wrote:
>
>> Hi all,
>>
>> I am moving from using LXC to KVM for haproxy on my Debian 7 system. When
>> I
>> start haproxy, I get this error:
>> _
>> Apr  7 10:38:22 localhost haproxy[3418]: Proxy haproxy started.
>> Apr  7 10:38:24 localhost haproxy[3420]: Server haproxy/nginx-1 is DOWN,
>> reason Layer4 timeout, check duration: 2000ms. 1 active and 0 backup
>> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
>> Apr  7 10:38:24 localhost haproxy[3419]: Server haproxy/nginx-1 is DOWN,
>> reason Layer4 timeout, check duration: 2001ms. 1 active and 0 backup
>> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
>> Apr  7 10:38:25 localhost haproxy[3420]: Server haproxy/nginx-2 is DOWN,
>> reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
>> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
>> Apr  7 10:38:25 localhost haproxy[3420]: proxy haproxy has no server
>> available!
>> Apr  7 10:38:25 localhost haproxy[3419]: Server haproxy/nginx-2 is DOWN,
>> reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
>> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
>> Apr  7 10:38:25 localhost haproxy[3419]: proxy haproxy has no server
>> available!
>>
>> From outside the haproxy, I get the error:
>> # wget 192.168.122.112:80
>> --2015-04-07 10:48:47--  http://192.168.122.112/
>> Connecting to 192.168.122.112:80... connected.
>> HTTP request sent, awaiting response... 503 Service Unavailable
>> 2015-04-07 10:48:47 ERROR 503: Service Unavailable.
>> ___
>>
>> The config file is:
>> global
>> log 127.0.0.1   local0
>> log 127.0.0.1   local1 notice
>> maxconn  65536
>> daemon
>> quiet
>> nbproc 2
>> debug
>> user haproxy
>> group haproxy
>>
>> defaults
>> log global
>> modehttp
>> option  dontlognull
>> retries 3
>> option redispatch
>> maxconn 65536
>> timeout connect 5000
>> timeout client  5
>> timeout server  5
>>
>> #listen haproxy 192.168.122.112:80
>> listen haproxy *:80
>> mode http
>> stats enable
>> stats auth someuser:somepassword
>> balance roundrobin
>> option prefer-last-server
>> option forwardfor
>> option httpchk HEAD /check.txt HTTP/1.0
>>
>
Check if the above health check is really working, you show that requesting
the root page works but we don't see you checking the /check.txt file (does
it exist at all?). Run:

$ curl --http1.0 -X HEAD 192.168.122.101:80 
/check.txt
$ curl --http1.0 -X HEAD 192.168.122.102:80 
/check.txt

from the HAP server.

server nginx-1 192.168.122.101:80 check
>> server nginx-2 192.168.122.102:80 check
>>
>> BTW, I could not use "listen haproxy 192.168.122.112:80", but had to use
>> *:80
>> as haproxy does not start up with the former. It seems like haproxy
>> startup is
>> happening ahead of networking.
>> __
>>
>> I also stopped/restarted haproxy, but I still get the same error at start.
>>
>> root@haproxy-2:~# netstat -apn | grep :80
>> tcp0  0 0.0.0.0:80  0.0.0.0:*
>> LISTEN  3558/haproxy
>> ___
>> From outside haproxy, I can do a wget/curl" to either of the two servers:
>>
>> # wget 192.168.122.101:80
>> --2015-04-07 10:42:28--  http://192.168.122.101/
>> Connecting to 192.168.122.101:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 867 [text/html]
>> Saving to: `index.html'
>>
>> 100%[==>] 867 --.-K/s   in
>> 0s
>>
>> 2015-04-07 10:42:28 (104 MB/s) - `index.html' saved [867/867]
>> ___
>>
>> And I can do the same from haproxy:
>> root@haproxy-2:~# wget 192.168.122.101
>> --2015-04-07 10:43:48--  http://192.168.122.101/
>> Connecting to 192.168.122.101:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 867 [text/html]
>> Saving to: `index.html'
>>
>> 100%[==>] 867 --.-K/s   in
>> 0s
>>
>> 2015-04-07 10:43:48 (80.3 MB/s) - `index.html' saved [867/867]
>> ___
>>
>> How do I fix this problem?
>>
>> Thank you,
>> - KK
>>
>
>

Re: "proxy haproxy has no server available!"

[Krishna Kumar Unnikrishnan (Engineering)]

Sorry, forgot to mention, this is haproxy version 1.5.11

On Tue, Apr 7, 2015 at 10:52 AM, Krishna Kumar Unnikrishnan (Engineering) <
krishna...@flipkart.com> wrote:

> Hi all,
>
> I am moving from using LXC to KVM for haproxy on my Debian 7 system. When I
> start haproxy, I get this error:
> _
> Apr  7 10:38:22 localhost haproxy[3418]: Proxy haproxy started.
> Apr  7 10:38:24 localhost haproxy[3420]: Server haproxy/nginx-1 is DOWN,
> reason Layer4 timeout, check duration: 2000ms. 1 active and 0 backup
> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
> Apr  7 10:38:24 localhost haproxy[3419]: Server haproxy/nginx-1 is DOWN,
> reason Layer4 timeout, check duration: 2001ms. 1 active and 0 backup
> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
> Apr  7 10:38:25 localhost haproxy[3420]: Server haproxy/nginx-2 is DOWN,
> reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
> Apr  7 10:38:25 localhost haproxy[3420]: proxy haproxy has no server
> available!
> Apr  7 10:38:25 localhost haproxy[3419]: Server haproxy/nginx-2 is DOWN,
> reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
> servers left. 0 essions active, 0 requeued, 0 remaining in queue.
> Apr  7 10:38:25 localhost haproxy[3419]: proxy haproxy has no server
> available!
>
> From outside the haproxy, I get the error:
> # wget 192.168.122.112:80
> --2015-04-07 10:48:47--  http://192.168.122.112/
> Connecting to 192.168.122.112:80... connected.
> HTTP request sent, awaiting response... 503 Service Unavailable
> 2015-04-07 10:48:47 ERROR 503: Service Unavailable.
> ___
>
> The config file is:
> global
> log 127.0.0.1   local0
> log 127.0.0.1   local1 notice
> maxconn  65536
> daemon
> quiet
> nbproc 2
> debug
> user haproxy
> group haproxy
>
> defaults
> log global
> modehttp
> option  dontlognull
> retries 3
> option redispatch
> maxconn 65536
> timeout connect 5000
> timeout client  5
> timeout server  5
>
> #listen haproxy 192.168.122.112:80
> listen haproxy *:80
> mode http
> stats enable
> stats auth someuser:somepassword
> balance roundrobin
> option prefer-last-server
> option forwardfor
> option httpchk HEAD /check.txt HTTP/1.0
> server nginx-1 192.168.122.101:80 check
> server nginx-2 192.168.122.102:80 check
>
> BTW, I could not use "listen haproxy 192.168.122.112:80", but had to use
> *:80
> as haproxy does not start up with the former. It seems like haproxy
> startup is
> happening ahead of networking.
> __
>
> I also stopped/restarted haproxy, but I still get the same error at start.
>
> root@haproxy-2:~# netstat -apn | grep :80
> tcp0  0 0.0.0.0:80  0.0.0.0:*
> LISTEN  3558/haproxy
> ___
> From outside haproxy, I can do a wget/curl" to either of the two servers:
>
> # wget 192.168.122.101:80
> --2015-04-07 10:42:28--  http://192.168.122.101/
> Connecting to 192.168.122.101:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 867 [text/html]
> Saving to: `index.html'
>
> 100%[==>] 867 --.-K/s   in
> 0s
>
> 2015-04-07 10:42:28 (104 MB/s) - `index.html' saved [867/867]
> ___
>
> And I can do the same from haproxy:
> root@haproxy-2:~# wget 192.168.122.101
> --2015-04-07 10:43:48--  http://192.168.122.101/
> Connecting to 192.168.122.101:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 867 [text/html]
> Saving to: `index.html'
>
> 100%[==>] 867 --.-K/s   in
> 0s
>
> 2015-04-07 10:43:48 (80.3 MB/s) - `index.html' saved [867/867]
> ___
>
> How do I fix this problem?
>
> Thank you,
> - KK
>

"proxy haproxy has no server available!"

[Krishna Kumar Unnikrishnan (Engineering)]

Hi all,

I am moving from using LXC to KVM for haproxy on my Debian 7 system. When I
start haproxy, I get this error:
_
Apr  7 10:38:22 localhost haproxy[3418]: Proxy haproxy started.
Apr  7 10:38:24 localhost haproxy[3420]: Server haproxy/nginx-1 is DOWN,
reason Layer4 timeout, check duration: 2000ms. 1 active and 0 backup
servers left. 0 essions active, 0 requeued, 0 remaining in queue.
Apr  7 10:38:24 localhost haproxy[3419]: Server haproxy/nginx-1 is DOWN,
reason Layer4 timeout, check duration: 2001ms. 1 active and 0 backup
servers left. 0 essions active, 0 requeued, 0 remaining in queue.
Apr  7 10:38:25 localhost haproxy[3420]: Server haproxy/nginx-2 is DOWN,
reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
servers left. 0 essions active, 0 requeued, 0 remaining in queue.
Apr  7 10:38:25 localhost haproxy[3420]: proxy haproxy has no server
available!
Apr  7 10:38:25 localhost haproxy[3419]: Server haproxy/nginx-2 is DOWN,
reason Layer4 timeout, check duration: 2001ms. 0 active and 0 backup
servers left. 0 essions active, 0 requeued, 0 remaining in queue.
Apr  7 10:38:25 localhost haproxy[3419]: proxy haproxy has no server
available!

>From outside the haproxy, I get the error:
# wget 192.168.122.112:80
--2015-04-07 10:48:47--  http://192.168.122.112/
Connecting to 192.168.122.112:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2015-04-07 10:48:47 ERROR 503: Service Unavailable.
___

The config file is:
global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
maxconn  65536
daemon
quiet
nbproc 2
debug
user haproxy
group haproxy

defaults
log global
modehttp
option  dontlognull
retries 3
option redispatch
maxconn 65536
timeout connect 5000
timeout client  5
timeout server  5

#listen haproxy 192.168.122.112:80
listen haproxy *:80
mode http
stats enable
stats auth someuser:somepassword
balance roundrobin
option prefer-last-server
option forwardfor
option httpchk HEAD /check.txt HTTP/1.0
server nginx-1 192.168.122.101:80 check
server nginx-2 192.168.122.102:80 check

BTW, I could not use "listen haproxy 192.168.122.112:80", but had to use
*:80
as haproxy does not start up with the former. It seems like haproxy startup
is
happening ahead of networking.
__

I also stopped/restarted haproxy, but I still get the same error at start.

root@haproxy-2:~# netstat -apn | grep :80
tcp0  0 0.0.0.0:80  0.0.0.0:*
LISTEN  3558/haproxy
___
>From outside haproxy, I can do a wget/curl" to either of the two servers:

# wget 192.168.122.101:80
--2015-04-07 10:42:28--  http://192.168.122.101/
Connecting to 192.168.122.101:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 867 [text/html]
Saving to: `index.html'

100%[==>] 867 --.-K/s   in
0s

2015-04-07 10:42:28 (104 MB/s) - `index.html' saved [867/867]
___

And I can do the same from haproxy:
root@haproxy-2:~# wget 192.168.122.101
--2015-04-07 10:43:48--  http://192.168.122.101/
Connecting to 192.168.122.101:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 867 [text/html]
Saving to: `index.html'

100%[==>] 867 --.-K/s   in
0s

2015-04-07 10:43:48 (80.3 MB/s) - `index.html' saved [867/867]
___

How do I fix this problem?

Thank you,
- KK

Re: Health check for backend constituted with multiple socks proxies.

[Hongyi Zhao]

On Fri, 03 Apr 2015 11:10:31 +0200, Baptiste wrote:

> I mean what happens if you point your browser directly to one of the Ip
> address?
> Cause, what you're doing with your HAProxy configuration currently, is
> only forwarding the TCP connection from a browser client to a socks5
> server.
> If your browser client don't know how to speak to the socks5 server,
> HAProxy won't do it on behalf of it.
> So please confirm first the browser can use any of the listed IP without
> using HAProxy.

Because these all are * free * socks5 proxy servers found on the 
internet, the stability cann't be ensured.  Some of them can be used to 
let my browser directly access the internet, some may not; or if do the 
test in different time, the one cann't becomes good, and vice versa.  
Based on the above fact, I want to use all of them in one group of the 
backend of haproxy to get the redundancy and the loadbalance capability.  
The only issue for my case it that I want let haproxy to determine the 
correspoing server are up or down at the specific port and then direct me 
the up ones at the testing time done by haproxy.

Regards  

> 
> Then we'll dig into your issue...
> 
> Baptiste





-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

global maxconn limit in pure TCP mode

[Florin Andrei]

HAproxy 1.5 on CentOS 7.
5 front-ends, 5 backends, mapped 1:1. Pure TCP mode. I'm basically 
port-forwarding the backends from a different subnet.
The actual servers behind HAproxy are custom Tomcat apps capable of 
handling lots of connections.


I suspect I cannot increase the global maxconn indefinitely. At some 
point, I'll run into some limits. What will dictate those limits? In 
other words, how should I design the instance running HAproxy to make 
sure I can increase maxconn to a very high value?


--
Florin Andrei
http://florin.myip.org/

Re: 1.5, reload and zero downtime

[Dennis Jacobfeuerborn]

On 06.04.2015 22:45, Pavlos Parissis wrote:
> On 06/04/2015 08:41 μμ, Brian Fleming wrote:
>> I can do reload and there will be no downtime?
> 
> Yes, reload is a safe operation. But, don't be surprised if you see the
> old process alive for long time(days). This behavior is caused by insane
> timeout values on the client-side used by some people(including my self).

In case you use sticky sessions keep this in mind:

If you are running with nbproc > 1 (e.g. if you are doing ssl
offloading) and stick tables then you will lose the content of these
stick tables.
If you are running with nbproc = 1 then you have to configure the
localhost as a peer so the stick table gets picked up by the new process.

Regards,
  Dennis

Re: 1.5, reload and zero downtime

[Pavlos Parissis]

On 06/04/2015 08:41 μμ, Brian Fleming wrote:
> I can do reload and there will be no downtime?

Yes, reload is a safe operation. But, don't be surprised if you see the
old process alive for long time(days). This behavior is caused by insane
timeout values on the client-side used by some people(including my self).

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

server-side connection pool manager

[Pavlos Parissis]

Hoi,

While I was reading commit descriptions I saw in
REORG/MAJOR: session: rename the "session" entity to "stream"

[..snip..]
Some more cleanup is needed because some code was already far from
being clean. The server queue management still refers to sessions at
many places while comments talk about connections. This will have to
be cleaned up once we have a server-side connection pool manager.

I was wondering if we are going to see server-side connection pooling in
1.6. I know that HTTP/2 will bring in it at the client-side.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

1.5, reload and zero downtime

[Brian Fleming]

I can do reload and there will be no downtime?

Re: CPU saturated with 250Mbps traffic on frontend

[Willy Tarreau]

On Mon, Apr 06, 2015 at 02:54:13PM +0200, Evgeniy Sudyr wrote:
> this is server with 2x Intel I350-T4 1G Quad port NICs, where on first
> card each NIC is connected to uplink provider and 2nd NIC 4 ports are
> used for trunk interface with lacp connected to internal 1Gb switch
> with lacp configured as well. I've tested uplinks and internal link
> with iperf and was able to see at least 900Mbps for TCP tests.

You may want to retry without LACP. A long time ago on Linux, the bonding
driver used not to propagate NIC-specific optimizations and resulted in
worse performance sometimes than without. Also I don't know if you're
using VLANs, and I don't know if openbsd supports checksum offloading
on VLANs, but that could as well be something which limits the list of
possible optimizations/offloadings that normally result in lower CPU
usage.

> Card seems to be OK. Haproxy definitely needs to be moved to separate
> servers in inside network.

Makes sense. Then make sure to use a distro with a kernel 3.10 or above,
that's where you'll get the best performance.

> Btw, where Pavlos reported his test results? There in list or somewhere else?

It was posted one or two weeks ago on this list, yes. I must say I was
quite happy to see someone else post results in the order of magnitude
I encounter in my own tests, because at least I won't be suspected of
cheating anymore :-)

Cheers,
Willy

Re: CPU saturated with 250Mbps traffic on frontend

[Baptiste]

On Mon, Apr 6, 2015 at 2:54 PM, Evgeniy Sudyr  wrote:
> Btw, where Pavlos reported his test results? There in list or somewhere else?

On this ML.
Pavlos was running Linux ;)

Baptiste

Re: CPU saturated with 250Mbps traffic on frontend

[Evgeniy Sudyr]

this is server with 2x Intel I350-T4 1G Quad port NICs, where on first
card each NIC is connected to uplink provider and 2nd NIC 4 ports are
used for trunk interface with lacp connected to internal 1Gb switch
with lacp configured as well. I've tested uplinks and internal link
with iperf and was able to see at least 900Mbps for TCP tests.

Card seems to be OK. Haproxy definitely needs to be moved to separate
servers in inside network.

Btw, where Pavlos reported his test results? There in list or somewhere else?

Thanks again!

--
Evgeniy


On Mon, Apr 6, 2015 at 12:48 PM, Willy Tarreau  wrote:
> On Mon, Apr 06, 2015 at 12:34:05PM +0200, Evgeniy Sudyr wrote:
>> Hi Willy,
>>
>> pleasure for me to get answer from you!
>>
>> 1) I've tested with OpenBSD's SP kernel and single process (no nbproc)
>> in haproxy.conf and it was no significant difference in load.
>
> OK, I was not sure whether it was the SP kernel or just no nbproc.
>
>> I can't test to disable PF to test, because it's some kind of production 
>> router.
>
> I can understand, the test needs to be run on a test machine.
>
>> 2) I guess solution is to get separated loadbalancing servers with
>> Debian on it and better CPUs and run testing.
>
> I wouldn't give up too fast with openbsd. It's an excellent OS when you
> want a "drop and forget" solution. It's just that it's not very fast. If
> you manage to find what is causing this important load, maybe you can
> work around it or find some tunables.
>>
>> 3) What are "good numbers" - I've tried to find some recent benchmarks
>> for haproxy on commodity hardware, but not much available.
>
> Pavlos recently reported 438000 requests/s. I'm used to see about 110-120k
> end-to-end connections per second on high frequency Xeon CPUs. Bandwidth
> is really cheap these days with the proper NICs : with moderately large
> objects (250kB or more) today it's not hard to reach 40 Gbps on a recent
> machine equipped with one 40G or four 10G NICs.
>
> Just thinking about something since you're reporting 250-300 Mbps, I
> guess you're running on 1 Gbps NICs. Are you using good quality NICs ?
> By good I mean, aren't you running on low-end realteks or similar which
> can require significant work on the driver side and thus explain the
> high CPU usage in interrupt ?
>
> Willy
>



-- 
--
With regards,
Eugene Sudyr

Re: CPU saturated with 250Mbps traffic on frontend

[Willy Tarreau]

On Mon, Apr 06, 2015 at 12:34:05PM +0200, Evgeniy Sudyr wrote:
> Hi Willy,
> 
> pleasure for me to get answer from you!
> 
> 1) I've tested with OpenBSD's SP kernel and single process (no nbproc)
> in haproxy.conf and it was no significant difference in load.

OK, I was not sure whether it was the SP kernel or just no nbproc.

> I can't test to disable PF to test, because it's some kind of production 
> router.

I can understand, the test needs to be run on a test machine.

> 2) I guess solution is to get separated loadbalancing servers with
> Debian on it and better CPUs and run testing.

I wouldn't give up too fast with openbsd. It's an excellent OS when you
want a "drop and forget" solution. It's just that it's not very fast. If
you manage to find what is causing this important load, maybe you can
work around it or find some tunables.
> 
> 3) What are "good numbers" - I've tried to find some recent benchmarks
> for haproxy on commodity hardware, but not much available.

Pavlos recently reported 438000 requests/s. I'm used to see about 110-120k
end-to-end connections per second on high frequency Xeon CPUs. Bandwidth
is really cheap these days with the proper NICs : with moderately large
objects (250kB or more) today it's not hard to reach 40 Gbps on a recent
machine equipped with one 40G or four 10G NICs.

Just thinking about something since you're reporting 250-300 Mbps, I
guess you're running on 1 Gbps NICs. Are you using good quality NICs ?
By good I mean, aren't you running on low-end realteks or similar which
can require significant work on the driver side and thus explain the
high CPU usage in interrupt ?

Willy

Re: CPU saturated with 250Mbps traffic on frontend

[Evgeniy Sudyr]

Hi Willy,

pleasure for me to get answer from you!

1) I've tested with OpenBSD's SP kernel and single process (no nbproc)
in haproxy.conf and it was no significant difference in load.

I can't test to disable PF to test, because it's some kind of production router.

2) I guess solution is to get separated loadbalancing servers with
Debian on it and better CPUs and run testing.

3) What are "good numbers" - I've tried to find some recent benchmarks
for haproxy on commodity hardware, but not much available.

--
Evgeniy



--
Evgeniy

On Mon, Apr 6, 2015 at 11:59 AM, Willy Tarreau  wrote:
> Hi Evgeniy,
>
> On Sun, Apr 05, 2015 at 06:29:53PM +0200, Evgeniy Sudyr wrote:
>> Nenad,
>>
>> thank your answer!
>>
>> 1) this is only Haproxy server active (active/passive config exists,
>> but using carp on OpenBSD).
>>
>> 2) As I understand with nbcproc 4 I can't get stats working correctly ...
>>
>> however at the moment I see that for https frontend I have :
>> Current connection rate:58/s
>> Current session rate:53/s
>> Current request rate:124/s
>>
>> For http frontend:
>> Current connection rate:240/s
>> Current session rate:240/s
>> Current request rate:542/s
>
> These numbers are really low.
>
>>
>> 3) current top output (total in/out for HTTP/HTTPs traffic on external
>> interfaces is avg 300 Mbps and this is only Haproxy traffic):
>>
>> load averages:  4.02,  3.92,  3.88
>> router2 19:28:18
>> 32 processes: 1 running, 27 idle, 4 on processor
>> CPU0 states: 12.6% user,  0.0% nice, 11.2% system, 60.9% interrupt, 15.4% 
>> idle
>> CPU1 states: 25.2% user,  0.0% nice, 47.0% system,  0.2% interrupt, 27.6% 
>> idle
>> CPU2 states: 25.1% user,  0.0% nice, 43.3% system,  0.6% interrupt, 30.9% 
>> idle
>> CPU3 states: 21.6% user,  0.0% nice, 48.2% system,  0.2% interrupt, 30.0% 
>> idle
>> Memory: Real: 1017M/1709M act/tot Free: 14G Cache: 111M Swap: 0K/16G
>
> This huge CPU usage in interrupt definitely reminds me of performance issues
> related to pf I used to face a long time ago. The performance would double
> or triple just after issuing "pfctl -d" (to disable it). At least it's easy
> to test. I've never tested openbsd's network stack in SMP yet, it could be
> possible that it comes with some extra cost (for locking or whatever), but
> it might be something else as well.
>
> Regards,
> Willy
>



-- 
--
With regards,
Eugene Sudyr

Re: Trouble with getting ocsp response to work

[Vasileios Tzimourtos]

Hello Jarno,

thanks for the response. First of all, it worked!

It was the issue that you mentioned with the 300sec SKEW. I compiled 
haproxy with smaller value (30 :) ) and id returns the response :)


The test with the openssl that toy mentioned returns Verified OK. The 
problem was the refferrence to the past


Finally, to ease your curiosity, the CA is HARICA ( harica.gr )

Thanks again!


On 6/4/2015 12:50 μμ, Jarno Huuskonen wrote:

Hi,

On Mon, Apr 06, Vasileios Tzimourtos wrote:

**/usr/bin/openssl ocsp -noverify -issuer $ROOT_CERT_FILE -cert
$SERVER_CERT_FILE -url "$OCSP_URL" -no_nonce -header Host `echo
"$OCSP_URL" | cut -d"/" -f3` -respout $OCSP_FILE**
**echo "set ssl ocsp-response $(/usr/bin/base64 -w 1
$OCSP_FILE)" | socat $HAPROXY_SOCKET stdio**
*

Can you run openssl ocsp w/out -noverify (and maybe -VAfile) ?
So something like:
/usr/bin/openssl ocsp -issuer $ROOT_CERT_FILE \
  -cert $SERVER_CERT_FILE -url "$OCSP_URL" -no_nonce \
  -header Host `echo "$OCSP_URL" | cut -d"/" -f3` -respout $OCSP_FILE \
  [ -VAfile $ROOT_CERT_FILE [-validity_period 300] ]


Running the above script returns that all is OK and that ocsp
response was updated

Do you get any messages about ocsp response if you reload haproxy/check
configuration sometime after creating the ocsp response ?
  

*/etc/haproxy/certs/mycertificate.crt.pem: good**
**This Update: Apr  6 08:28:46 2015 GMT**
**Next Update: Apr  6 08:33:46 2015 GMT**
**OCSP Response updated!**

Out of curiosity which CA issues responses for only 5min ?

Haproxy defaults.h has:
#define OCSP_MAX_RESPONSE_TIME_SKEW 300

In commit 4f3c87a5d942d4d0649c35805ff4e335970b87d4 there's:
"   Haproxy stops serving OCSP response if nextupdate date minus
 the supported time skew (#define OCSP_MAX_RESPONSE_TIME_SKEW) is
 in the past.
"

Your problem maybe be that the ocsp response is valid for 5min(300s)
Quick check to test this could be to compile haproxy with
different OCSP_MAX_RESPONSE_TIME_SKEW (< 300) ?

-Jarno



--
Vassilis Tzimourtos

Re: CPU saturated with 250Mbps traffic on frontend

[Willy Tarreau]

Hi Evgeniy,

On Sun, Apr 05, 2015 at 06:29:53PM +0200, Evgeniy Sudyr wrote:
> Nenad,
> 
> thank your answer!
> 
> 1) this is only Haproxy server active (active/passive config exists,
> but using carp on OpenBSD).
> 
> 2) As I understand with nbcproc 4 I can't get stats working correctly ...
> 
> however at the moment I see that for https frontend I have :
> Current connection rate:58/s
> Current session rate:53/s
> Current request rate:124/s
> 
> For http frontend:
> Current connection rate:240/s
> Current session rate:240/s
> Current request rate:542/s

These numbers are really low.

> 
> 3) current top output (total in/out for HTTP/HTTPs traffic on external
> interfaces is avg 300 Mbps and this is only Haproxy traffic):
> 
> load averages:  4.02,  3.92,  3.88
> router2 19:28:18
> 32 processes: 1 running, 27 idle, 4 on processor
> CPU0 states: 12.6% user,  0.0% nice, 11.2% system, 60.9% interrupt, 15.4% idle
> CPU1 states: 25.2% user,  0.0% nice, 47.0% system,  0.2% interrupt, 27.6% idle
> CPU2 states: 25.1% user,  0.0% nice, 43.3% system,  0.6% interrupt, 30.9% idle
> CPU3 states: 21.6% user,  0.0% nice, 48.2% system,  0.2% interrupt, 30.0% idle
> Memory: Real: 1017M/1709M act/tot Free: 14G Cache: 111M Swap: 0K/16G

This huge CPU usage in interrupt definitely reminds me of performance issues
related to pf I used to face a long time ago. The performance would double
or triple just after issuing "pfctl -d" (to disable it). At least it's easy
to test. I've never tested openbsd's network stack in SMP yet, it could be
possible that it comes with some extra cost (for locking or whatever), but
it might be something else as well.

Regards,
Willy

Re: Trouble with getting ocsp response to work

[Jarno Huuskonen]

Hi,

On Mon, Apr 06, Vasileios Tzimourtos wrote:
> **/usr/bin/openssl ocsp -noverify -issuer $ROOT_CERT_FILE -cert
> $SERVER_CERT_FILE -url "$OCSP_URL" -no_nonce -header Host `echo
> "$OCSP_URL" | cut -d"/" -f3` -respout $OCSP_FILE**
> **echo "set ssl ocsp-response $(/usr/bin/base64 -w 1
> $OCSP_FILE)" | socat $HAPROXY_SOCKET stdio**
> *

Can you run openssl ocsp w/out -noverify (and maybe -VAfile) ?
So something like:
/usr/bin/openssl ocsp -issuer $ROOT_CERT_FILE \
 -cert $SERVER_CERT_FILE -url "$OCSP_URL" -no_nonce \
 -header Host `echo "$OCSP_URL" | cut -d"/" -f3` -respout $OCSP_FILE \
 [ -VAfile $ROOT_CERT_FILE [-validity_period 300] ]

> Running the above script returns that all is OK and that ocsp
> response was updated

Do you get any messages about ocsp response if you reload haproxy/check
configuration sometime after creating the ocsp response ?
 
> */etc/haproxy/certs/mycertificate.crt.pem: good**
> **This Update: Apr  6 08:28:46 2015 GMT**
> **Next Update: Apr  6 08:33:46 2015 GMT**
> **OCSP Response updated!**

Out of curiosity which CA issues responses for only 5min ?

Haproxy defaults.h has:
#define OCSP_MAX_RESPONSE_TIME_SKEW 300

In commit 4f3c87a5d942d4d0649c35805ff4e335970b87d4 there's:
"   Haproxy stops serving OCSP response if nextupdate date minus
the supported time skew (#define OCSP_MAX_RESPONSE_TIME_SKEW) is
in the past.
"

Your problem maybe be that the ocsp response is valid for 5min(300s)
Quick check to test this could be to compile haproxy with
different OCSP_MAX_RESPONSE_TIME_SKEW (< 300) ?

-Jarno

-- 
Jarno Huuskonen

Trouble with getting ocsp response to work

[Vasileios Tzimourtos]

Hello to all,

i have trouble on getting a ocsp response from haproxy when i ask him 
with openssl. I will be detailing bellow all the steps i have taken and 
i get a 'OCSP Response updated!' when i pass the setting to the haproxy 
socket


The version of the haproxy that i use is as follows:
===
*>**haproxy -vv*
HA-Proxy version 1.5.11 2015/01/31
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing
  OPTIONS = USE_OPENSSL=1 USE_STATIC_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND


Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
=

In the haproxy.conf i have set the follwing directives in the global 
section to enable the socket function with socat


*   stats socket /var/run/haproxy.stat mode 600 level admin**
**stats timeout 2m*

the socket is working properly and i cat query it with success

In the certificate that i use (e.g.: mycertificate.crt.pem) the 
structure is as follows


*-BEGIN CERTIFICATE (public mycertificate.crt.pem)-**
**-END CERTIFICATE (public)-**
**-BEGIN CERTIFICATE (_chain 1_)-**
**-END CERTIFICATE (_chain 1_)-**
**-BEGIN CERTIFICATE (_chain 2_)-**
**-END CERTIFICATE (_chain 2_)-**
**-BEGIN RSA PRIVATE KEY (of mycertificate.crt.pem)-**
**-END RSA PRIVATE KEY (of mycertificate.crt.pem)-**
**-BEGIN DH PARAMETERS-**
**-END DH PARAMETERS-**
***

The following script runs with cron every 5minutes and creates the .ocsp 
file and passes it to the haproxy socket. The .issuer file contains the 
previously mentioned *chain 1* and *chain 2 *certificates


*#!/bin/sh**
**SERVER_CERT_FILE=/etc/haproxy/certs/mycertificate.crt.pem**
**ROOT_CERT_FILE=${SERVER_CERT_FILE}.issuer**
**HAPROXY_SOCKET=/var/run/haproxy.stat**
**OCSP_URL=`/usr/bin/openssl x509 -in $SERVER_CERT_FILE -text | grep -i 
ocsp | cut -d":" -f2-2,3`**

**OCSP_FILE=${SERVER_CERT_FILE}.ocsp**
**
**/usr/bin/openssl ocsp -noverify -issuer $ROOT_CERT_FILE -cert 
$SERVER_CERT_FILE -url "$OCSP_URL" -no_nonce -header Host `echo 
"$OCSP_URL" | cut -d"/" -f3` -respout $OCSP_FILE**
**echo "set ssl ocsp-response $(/usr/bin/base64 -w 1 $OCSP_FILE)" | 
socat $HAPROXY_SOCKET stdio**

*

Running the above script returns that all is OK and that ocsp response 
was updated


*/etc/haproxy/certs/mycertificate.crt.pem: good**
**This Update: Apr  6 08:28:46 2015 GMT**
**Next Update: Apr  6 08:33:46 2015 GMT**
**OCSP Response updated!**
*
After all the configuration and having restarted haproxy when i run the 
command


*openssl s_client -connect myservice_URL:443 -tls1 -tlsextdebug -status*

i still get:

*OCSP response: no response sent*

I tried also not to use the haproxy socket and just have the files .ocsp 
and .issuer but with no luck. Any ideas on what is going wrong or what 
to try next?


Thanks in advance! :)

--
Vassilis Tzimourtos