Re: Meeting with LML Studios

[M Sami Kerrouche]

Hi Guys,

Was this something you needed?

I'd appreciate an answer. Happy to help.

Best wishes,

Sami

On Mon, May 6, 2024 at 1:10 PM M Sami Kerrouche <
s...@londonmedialounge.co.uk> wrote:

> Hi,
>
> I am waiting for you on our call that you booked.
>
> Let me know if you'd like to reschedule.
>
> Here's my phone number, call me for a quicker response.
>
> +44 7577 295184
>
> Best wishes,
>
> *Sami Kerrouche*
> Managing Director
>
> London Media Lounge Ltd
> 16a/17a Windsor St
> Uxbridge
> Middx
> UB8 1AB
> www.LondonMediaLounge.co.uk <http://www.londonmedialounge.co.uk/>
> https://www.linkedin.com/in/msamikerrouche/
> <https://www.linkedin.com/in/msamikerrouche/>
>

Meeting with LML Studios

[M Sami Kerrouche]

Hi,

I am waiting for you on our call that you booked.

Let me know if you'd like to reschedule.

Here's my phone number, call me for a quicker response.

+44 7577 295184

Best wishes,

*Sami Kerrouche*
Managing Director

London Media Lounge Ltd
16a/17a Windsor St
Uxbridge
Middx
UB8 1AB
www.LondonMediaLounge.co.uk 
https://www.linkedin.com/in/msamikerrouche/

I need guest post on your website-https://us-cert.cisa.gov

[M Afzal]

What is guest post price at-https://us-cert.cisa.gov

I am waiting for your good reply

Thanks

Re: OpenSSL Security Advisory

[Fox, Kevin M]

That would be unfortunate. Some clusters won't be able to distinguish if there 
is an update or not.

That's one reason I typically follow the distro convention of packaging, of 
tacking onto the tag a -1, so if I need to bump them, it can be. -2 when a new 
release of the same version comes out. -3 next, etc. Could something like that 
be adopted?

Thanks,
Kevin


From: Tim Düsterhus 
Sent: Thursday, March 25, 2021 11:32 AM
To: Paul Lockaby
Cc: Lukas Tribus; haproxy
Subject: Re: OpenSSL Security Advisory

Check twice before you click! This email originated from outside PNNL.


Paul,

On 3/25/21 7:31 PM, Paul Lockaby wrote:
> Thanks for all of the responses! So the image version number for HAProxy 
> stays the same but the hash will update?
>

Yes exactly.

Best regards
Tim Düsterhus

Request for mentioning your brand

[Virginia M Faulkner]

I’m a collaborator of GoodNoon a PR agency (in cc. my collegue Donna).
Completely free of charge, I can pitch your brand to my network to see if
any of them would be interested in publishing your story in the outlets
they work for.

My outreach service is offered completely free of charges, and although I
might even be able to offer you magazine interviews and coverage for free,
at times journalists and outlets do ask for a publication fee. This however
will be visible in the offers we will be sending you and that's completely
up to you to decide whether you’d like to move forward with the offer or
not.

Thanks in advance

Sponsored contribution for haproxy.org

[Rick M.]

 Sponsored contribution for haproxy.org
Hi there,

My name is Rick Manarauskis. I create & curate new content at PiketMedia.

I was looking for relevant people to potentially work with and haproxy.org
popped up on my radar, which is why I'm reaching out to you.

I would love to send you a couple of article titles I've came up with or
discuss sponsoring a relevant resource link placement.

Please, let me know what you think.

Best wishes,
Rick Manarauskis

*Content Creator*
PiketMedia 

P.S. Don’t want to hear from me again? Reply with the word ‘unsubscribe’
and I will *never* reach out to you again.

Feature request: smtpchk additional output check

[Stu M]

Hi,

I have a small but hopefully simple request that would be useful in
Exchange SMTP load balancing situations.

When Exchange 2013+ hub transport service is put into maintenance mode it
keeps the SMTP service running but responds to mail from commands with "421
4.3.2 Service not active" - I believe this is for "smart redirect" of
Outlook clients that can relay thru another server in a cluster.

However, where other MTAs are concerned, when relaying thru a load balanced
Exchange this becomes problematic because they simply fail to submit
messages - HAproxy still sees the backend server is up because SMTP is
still alive, despite using smtpchk (Exchange still responds to the ehlo
command, see below).

E.g. typical output when a hub transport host is in service mode..

220 xyzserver Microsoft ESMTP MAIL Service ready at Fri, 9 Feb 2018
09:32:20 +
EHLO
250-xyzserver Hello [x.x.x.x]
250-SIZE 37748736
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-8BITMIME
250-BINARYMIME
250 CHUNKING
mail from: m...@hotmail.com
421 4.3.2 Service not active

My question is, would it be possible to have add additional option for
smtpchk that causes HAP to look for additional data in the SMTP response -
specifically the last line above "service not active"?

So after doing ehlo, if HAP then does a dummy "mail from:" it will get the
result code at that point, perhaps something like the following would
suffice?

   option smtpchk ehlo 
   option smtpchk-exch 

Appreciate any thoughts on this.

Best regards,
Stuart.

Re: help for configuration between http and tcp mode

[M]

Hi,

> 
> Because your Host header is certainly "influxdb-drp.example.net:8086", not 
> "influxdb-drp.example.net". You can verify this with this acl instead :
>  acl host_influxdb-drp.example.net hdr(host) -i influxdb-drp.example.net:8086
> 
> Or you can even capture the header in your logs, it's quite useful to debug 
> acls ;-)
> 

You are right. The port was missing from acl :(

Using "capture request header Host len 150"  was effectively showing it in log:

Jul  9 18:54:59 kalinga haproxy[46185]: 192.168.246.17:59204 
[09/Jul/2017:18:54:59.275] https_influxdb~ influxdb-drp.example.net/https_8086 
0/0/1/0/1 200 332 - -  1/1/0/0/0 0/0 {influxdb-drp.example.net:8086} "GET 
/query?q=SHOW%20DATABASES HTTP/1.1"

Thank you.

Regards,
M.

Re: help for configuration between http and tcp mode

[M]

Hi,

It seems the error is related to acl and I don’t yet understand why.

I have done additional tests with configuration: 

———

frontend https
  bind 0.0.0.0:443 ssl crt /data/ssl_certs no-sslv3 ciphers 
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:DES-CBC3-SHA:!NULL:!aNULL:!RC4:!RC2:!MEDIUM:!LOW:!EXPORT:!DES:!MD5:!PSK:!3DES
  mode http
  option dontlognull

  reqadd X-Forwarded-Proto:\ https

  option http-server-close
  option forwardfor

  acl host_piwigo.example.org hdr(host) -i piwigo.example.org
  use_backend piwigo.example.org if host_piwigo.example.org

  acl host_wordpress.example.com hdr(host) -i wordpress.example.com
  use_backend wordpress.example.com if host_wordpress.example.com

  # adding redirection on https tcp/443 fronted:
  acl host_influxdb-drp.example.net hdr(host) -i influxdb-drp.example.net
  use_backend influxdb-drp.example.net if host_influxdb-drp.example.net


frontend https_influxdb
  bind 192.168.246.17:8086 ssl crt /data/ssl_certs no-sslv3 ciphers 
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:DES-CBC3-SHA:!NULL:!aNULL:!RC4:!RC2:!MEDIUM:!LOW:!EXPORT:!DES:!MD5:!PSK:!3DES
  mode http
  option dontlognull

  reqadd X-Forwarded-Proto:\ https

  option http-server-close
  option forwardfor

  #default_backend influxdb-drp.example.net # to test later with a default 
backend as acl seems not working 

  acl host_influxdb-drp.example.net hdr(host) -i influxdb-drp.example.net
  use_backend influxdb-drp.example.net if host_influxdb-drp.example.net

backend influxdb-drp.example.net
  mode http
  server https_8086 127.0.0.1:8086 check ssl verify none

———

Backend is answering on https on tcp/8086. 

Testing against frontend https on tcp/443, it is working:

#curl -G https://influxdb-drp.example.net:443/query -u admin:'xxx' 
--data-urlencode "q=SHOW DATABASES"
{"results":[{"statement_id":0,"series":[{"name":"databases","columns":["name"],"values":[["_internal"]]}]}]}

Jul  9 15:46:32 kalinga haproxy[50375]: 192.168.246.17:59154 
[09/Jul/2017:15:46:31.900] https~ influxdb-drp.example.net/https_8086 0/1/132 
332 -- 5/5/0/0/0 0/0

The acl has matched to forward to the expected backend.

Testing against frontend https_influxdb on tcp/8086, it is failing:

#curl -G https://influxdb-drp.example.net:8086/query -u admin:'xxx' 
--data-urlencode "q=SHOW DATABASES"
503 Service Unavailable
No server is available to handle this request.


Jul  9 15:46:16 kalinga haproxy[50375]: 192.168.246.17:57242 
[09/Jul/2017:15:46:16.665] https_influxdb~ https_influxdb/ -1/-1/135 212 
SC 4/0/0/0/0 0/0

The acl is not matching under this frontend :-(

When testing by uncommenting following line:
  #default_backend influxdb-drp.example.net
in order to add a default backend, it is working:

#curl -G https://influxdb-drp.example.net:8086/query -u admin:'xxx' 
--data-urlencode "q=SHOW DATABASES"
{"results":[{"statement_id":0,"series":[{"name":"databases","columns":["name"],"values":[["_internal"]]}]}]}

Jul  9 15:46:32 kalinga haproxy[50375]: 192.168.246.17:59154 
[09/Jul/2017:15:46:31.900] https~ influxdb-drp.example.net/https_8086 0/1/132 
332 -- 5/5/0/0/0 0/0

Why acl is matching only on frontend https and not on frontend https_influxdb?

Haproxy version is the latest stable 1.7.8.


Regards,
M.

> Le 8 juil. 2017 à 22:19, M  a écrit :
> 
> Hi,
> 
> I don't understand why http mode is no able provide a backend server whereas 
> at same time tcp mode is able to do it. 
> 
> I am trying to setup Haproxy in front of an Influxdb database running on 
> HTTPS.
> 
> When frontend is configured on http mode, requests are failing with NOSRV 
> error.
> When fronted is configure on tcp mod, requests are working. 
> 
> Example below with http mode on tcp/8086 port and tcp mode on tcp/8087 with 
> same backend:
> 
> #curl -G https://influxdb-drp.example.net:8086/query -u admin:'' 
> --data-urlencode "q=SHOW DATABASES"
> 503 Service Unavailable
> No server is available to handle this request.
> 
> 
> Haproxy log showing NOSRV and SC:
> Jul  8 19:59:44 kalinga haproxy[26228]: 192.168.246.17:52946 
> [08/Jul/2017:19:59:44.661] https_influxdb~ https_influxdb/ -1/-1/136 
> 212 SC 0/0/0/0/0 0/0
> 
> 
> #curl -G https://influxdb-drp.example.net:8087/query -u admin:'' 
> --data-ur

help for configuration between http and tcp mode

[M]

Hi,

I don't understand why http mode is no able provide a backend server whereas at 
same time tcp mode is able to do it. 

I am trying to setup Haproxy in front of an Influxdb database running on HTTPS.

When frontend is configured on http mode, requests are failing with NOSRV error.
When fronted is configure on tcp mod, requests are working. 

Example below with http mode on tcp/8086 port and tcp mode on tcp/8087 with 
same backend:

#curl -G https://influxdb-drp.example.net:8086/query -u admin:'' 
--data-urlencode "q=SHOW DATABASES"
503 Service Unavailable
No server is available to handle this request.


Haproxy log showing NOSRV and SC:
Jul  8 19:59:44 kalinga haproxy[26228]: 192.168.246.17:52946 
[08/Jul/2017:19:59:44.661] https_influxdb~ https_influxdb/ -1/-1/136 212 
SC 0/0/0/0/0 0/0


#curl -G https://influxdb-drp.example.net:8087/query -u admin:'' 
--data-urlencode "q=SHOW DATABASES"
{"results":[{"statement_id":0,"series":[{"name":"databases","columns":["name"],"values":[["_internal"]]}]}]}

Haproxy log:
Jul  8 20:00:16 kalinga haproxy[26228]: 192.168.246.17:37142 
[08/Jul/2017:20:00:16.672] https_influxdb_tcp~ 
influxdb-drp.example.net/https_8086 0/2/123 332 -- 1/1/0/0/0 0/0

Haproxy configuration file:

---

global
log /dev/log local1 debug
maxconn 4096
debug
tune.ssl.default-dh-param 4096

defaults
log global
modetcp
option  tcplog
retries 3
option redispatch
maxconn 4000
timeout connect 5000ms
timeout client 5ms
timeout server 5ms

frontend https_influxdb
  bind 192.168.246.17:8086 ssl crt /data/ssl_certs no-sslv3 ciphers 
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:DES-CBC3-SHA:!NULL:!aNULL:!RC4:!RC2:!MEDIUM:!LOW:!EXPORT:!DES:!MD5:!PSK:!3DES
  mode http
  option dontlognull

  reqadd X-Forwarded-Proto:\ https

  option http-server-close
  option forwardfor

  acl host_influxdb-drp.example.net hdr(host) -i influxdb-drp.example.net
  use_backend influxdb-drp.example.net if host_influxdb-drp.example.net

frontend https_influxdb_tcp
  bind 192.168.246.17:8087 ssl crt /data/ssl_certs no-sslv3 ciphers 
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:DES-CBC3-SHA:!NULL:!aNULL:!RC4:!RC2:!MEDIUM:!LOW:!EXPORT:!DES:!MD5:!PSK:!3DES
  mode tcp
  option dontlognull
  option http-server-close
  option tcplog

  default_backend influxdb-drp.example.net

backend influxdb-drp.example.net
  mode http
  server https_8086 127.0.0.1:8086 check ssl verify none

---

How can I configure Haproxy to work on http mode?

M.

Re: Possible bug with haproxy 1.6.9/1.7.0: multiproc + resolvers cause DNS timeouts

[Joshua M. Boniface]

Sorry here is my haproxy command information as well:

| u...@elb2.domain.net ~ $ sudo haproxy -vv 
| HA-Proxy version 1.7.0-1 2016/11/27
| Copyright 2000-2016 Willy Tarreau 
| 
| Build options :
|   TARGET  = linux2628
|   CPU = generic
|   CC  = gcc 
|   CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat 
-Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
|   OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 
USE_NS=1
| 
| Default settings :
|   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 
| 
| Encrypted password support via crypt(3): yes 
| Built with zlib version : 1.2.8
| Running on zlib version : 1.2.8
| Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
| Built with OpenSSL version : OpenSSL 1.0.2j  26 Sep 2016
| Running on OpenSSL version : OpenSSL 1.0.2j  26 Sep 2016
| OpenSSL library supports TLS extensions : yes 
| OpenSSL library supports SNI : yes 
| OpenSSL library supports prefer-server-ciphers : yes 
| Built with PCRE version : 8.35 2014-04-04
| Running on PCRE version : 8.35 2014-04-04
| PCRE library supports JIT : no (USE_PCRE_JIT not set)
| Built with Lua version : Lua 5.3.1
| Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
| Built with network namespace support
| 
| Available polling systems :
|   epoll : pref=300,  test result OK
|poll : pref=200,  test result OK
|  select : pref=150,  test result OK
| Total: 3 (3 usable), will use epoll.
| 
| Available filters :
| [COMP] compression
| [TRACE] trace
| [SPOE] spoe 

Thanks,
Joshua M. Boniface
Linux System Ærchitect - Boniface Labs
Sigmentation fault: core dumped

On 29/11/16 02:17 AM, Joshua M. Boniface wrote:
> Hello list!
> 
> I believe I've found a bug in haproxy related to multiproc and a set of DNS 
> resolvers. What happens is, when combining these two features (multiproc and 
> dynamic resolvers), I get the following problem: the DNS resolvers, one per 
> process it seems, will fail intermittently and independently for no obvious 
> reason, and this triggers a DOWN event in the backend; a short time later, 
> the resolution succeeds and the backend goes back UP for a short time, before 
> repeating indefinitely. This bug also seems to have a curious effect of 
> causing the active record type to switch from A to  and then back to A 
> repeatedly in a dual-stack setup, though the test below shows that this bug 
> occurs in an IPv4-only environment as well, and this failure is not 
> documented in my tests.
> 
> First, some background. I'm attempting to set up an haproxy instance with 
> multiple processes for SSL termination. At the same time, I'm also trying to 
> use a IPv6 backend managed by DNS, so I set up a "resolvers" section so I 
> could use resolved IPv6 addresses from  records. As an aside, I've 
> noticed that haproxy will not start up if the only record for a host is an 
>  record, reporting that the address can't be resolved. However since I 
> run a dual-stack [A + ] record setup normally, this is not a huge deal to 
> me, though I think supporting an IPv6/-only backend should definitely be 
> a future goal!
> 
> First my config (figure 1). This is the most basic config I can construct 
> that triggers the bug while keeping most of my important settings; note the 
> host resolves to an A record only (figure 2); this record is provided by a 
> dnsmasq process which has read it out of /etc/hosts, so the resolution here 
> should be 100% stable.
> 
> (figure 1)
> | global
> | log ::1:514 daemon debug
> | log-send-hostname
> | chroot /var/lib/haproxy
> | pidfile /run/haproxy/haproxy.pid
> | nbproc 2
> | cpu-map 1 0
> | cpu-map 2 1
> | stats socket /var/lib/haproxy/admin-1.sock mode 660 level admin 
> process 1
> | stats socket /var/lib/haproxy/admin-2.sock mode 660 level admin 
> process 2
> | stats timeout 30s
> | user haproxy
> | group haproxy
> | daemon
> | maxconn 1
> | resolvers dns
> | nameserver dnsmasq 127.0.0.1:53
> | resolve_retries 1
> | hold valid 1s
> | hold timeout 1s
> | timeout retry 1s
> | defaults
> | log global
> | option  http-keep-alive
> | option  forwardfor except 127.0.0.0/8
> | option  redispatch
> | option  dontlognull
> | option  forwardfor
> | timeout connect 5s
> | timeout client  24h
> | timeout server  60m
> | listen b

Possible bug with haproxy 1.6.9/1.7.0: multiproc + resolvers cause DNS timeouts

[Joshua M. Boniface]

S resolution 
requests to dnsmask. The most curious thing I see in the DNS pcap is requests 
for  records that don't exist, but I don't know if that's the cause, though 
this would explain the swapping-A-and--records I mentioned earlier.

I've noticed this bug both on haproxy 1.6.9 (from the Debian jessie-backports 
repo) and also on my own self-built 1.7.0 package as well, and all the above 
testing was on 1.7.0. Please let me know if I can provide any further 
information!

Thanks,
Joshua M. Boniface
Linux System Ærchitect - Boniface Labs
Sigmentation fault: core dumped


haproxy.pcap
Description: application/vnd.tcpdump.pcap
Process 29888 attached
01:46:54 epoll_wait(0, {}, 200, 983)= 0
01:46:55 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:55 recvfrom(4, 
"\320)\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL, 
NULL) = 32
01:46:55 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
01:46:55 sendto(4, "\320)\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 
32, 0, NULL, 0) = 32
01:46:55 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:55 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:55 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:55 connect(1, {sa_family=AF_INET, sin_port=htons(80), 
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in 
progress)
01:46:55 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:55 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
01:46:55 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:55 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily 
unavailable)
01:46:55 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
01:46:55 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLOUT, {u32=1, u64=1}}) = 0
01:46:55 epoll_wait(0, {{EPOLLOUT, {u32=1, u64=1}}}, 200, 1000) = 1
01:46:55 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:55 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:55 epoll_ctl(0, EPOLL_CTL_MOD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:55 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 999) = 1
01:46:55 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL, 
NULL) = 243
01:46:55 close(1)   = 0
01:46:55 epoll_wait(0, {}, 200, 998)= 0
01:46:56 epoll_wait(0, {}, 200, 0)  = 0
01:46:56 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:56 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:56 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:56 connect(1, {sa_family=AF_INET, sin_port=htons(80), 
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in 
progress)
01:46:56 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:56 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
01:46:56 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:56 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:56 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
01:46:56 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:56 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 999) = 1
01:46:56 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL, 
NULL) = 243
01:46:56 close(1)   = 0
01:46:56 epoll_wait(0, {}, 200, 998)= 0
01:46:57 epoll_wait(0, {}, 200, 0)  = 0
01:46:57 sendto(4, "\223\311\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 
32, 0, NULL, 0) = 32
01:46:57 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:57 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:57 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:57 connect(1, {sa_family=AF_INET, sin_port=htons(80), 
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in 
progress)
01:46:57 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:57 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
01:46:57 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:57 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:57 recvfrom(4, 
"n\256\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL, 
NULL) = 32
01:46:57 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
01:46:57 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:57 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 1000) = 1
01:46:57 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384,

Getting 502 Gateway Timeout for BOSH over HAPRoxy

[Vijayalakshmi Devi A M]

Hi,

Our web application uses XMPP for chat. We are using ejabberd xmpp server . We 
have configured xmpp BOSH( http://xmpp.org/extensions/xep-0206.html ) via 
HAProxy. But sometimes haproxy is throwing error with Status Code:504 Gateway 
Time-out. Our application shows error as No 'Access-Control-Allow-Origin' 
header is present on the requested resource. Origin '. But the same request 
initially it works without any error. Can you please guide us on how to resolve 
this issue

Regards
Vijaya

Unsubscribe

[Pedro M. S. Oliveira]

Re: Missing anchor log format

[m...@felixsanz.com]

Actually anchors are not send to the server. Those are just client 
fragments. My bad sorry.


Thanks people at IRC for pointing this.


On jue, jun 30, 2016 at 9:52 , m...@felixsanz.com  
wrote:
Poor #anchors. They got replaced by hashtags and no one remembers 
them.

Meanwhile, Custom log formats:
  |   | %H   | hostname  | string 
 |
  | H | %HM  | HTTP method (ex: POST)| string 
 |
  | H | %HP  | HTTP request URI without query string (path)  | string 
 |
  | H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string 
 |
  | H | %HU  | HTTP request URI (ex: /foo?bar=baz)   | string 
 |
  | H | %HV  | HTTP version (ex: HTTP/1.0)   | string 
 |


We need something like:
  | H | %HA  | HTTP request URI anchor (ex: #title)   | 
string  |

Missing anchor log format

[m...@felixsanz.com]

Poor #anchors. They got replaced by hashtags and no one remembers them.
Meanwhile, Custom log formats:
 |   | %H   | hostname  | string   
  |
 | H | %HM  | HTTP method (ex: POST)| string   
  |
 | H | %HP  | HTTP request URI without query string (path)  | string   
  |
 | H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string   
  |
 | H | %HU  | HTTP request URI (ex: /foo?bar=baz)   | string   
  |
 | H | %HV  | HTTP version (ex: HTTP/1.0)   | string   
  |


We need something like:
 | H | %HA  | HTTP request URI anchor (ex: #title)   | string  
   |

RE: KA-BOOM! Hit MaxConn despite higher setting in config file

[Fox, Kevin M]

Except with systemd based distros where its a unit file setting.

Thanks,
Kevin


From: CJ Ess
Sent: Saturday, April 02, 2016 6:48:56 PM
To: PiBa-NL
Cc: HAProxy
Subject: Re: KA-BOOM! Hit MaxConn despite higher setting in config file

I'm on Linux so I think that /etc/security/limits.d and 
/etc/security/limits.conf are where I would change the default settings for a 
user - however the ulimit-n setting in haproxy is a fraction of what the user's 
current ulimit -n is, and I'm not sure why.


On Sat, Apr 2, 2016 at 4:46 PM, PiBa-NL 
mailto:piba.nl@gmail.com>> wrote:
Op 2-4-2016 om 22:32 schreef CJ Ess:
So in my config file I have:

maxconn 65535
fullconn 64511

However, "show info" still has a maxconn 2000 limit and that caused a blow up 
because I exceeded the limit =(

So my questions are 1)  is there a way to raise maxconn without restarting 
haproxy with the -P parameter (can I add -P when I do a reload?) 2) Are there 
any other related gotchas I need to take care of?

I notice that ulimit-n and maxsock both show 4495 despite "ulimit -n" for the 
user showing 65536 (which is probably half of what I really want since each 
"session" is going to consume two sockets)
as for ulimit-n on freebsd i need to set these two system flags: kern.maxfiles  
 kern.maxfilesperprocwhat OS are you using?

I'm using haproxy 1.5.12

Re: Wrestling with rewrites

[M. Lebbink]

Hi Baptiste,

Thank you for your response, it helped somewhat. but I'm starting to 
think I'm to stupid for this


My issue
I have multiple websites on multiple servers running behind 2 haproxy 
servers.


One of the websites servers photo's using fotoplayer. In order for people to 
link to photo's I get

the following request:
   sub.domain.com/?folder=Harmelen%2F2014-09-27%2F&file=_DSC0001.jpg

Using https rewrite rules:
   RewriteCond %{QUERY_STRING} ^(.*)?folder=(.*)&file=(.*).jpg(.*)$
   RewriteRule / /%1%2slides/%3.jpg

I can rebuild this to
   sub.domain.com/Harmelen%2F2014-09-27/slides/_DSC0001.jpg

But my webserver is not understanding the %2F and issues a html 404. 
Resubmitting the same request directly onto the
server will produce the requested photo (tried playing with 
AllowEncodedSlashes but that does not help).


So, I thought, let HAproxy 1.5.x do the url rebuild I have tried all 
sorts of combinations and itterations of this:

   acl has_jpg path_sub jpg
   reqirep ^([^\ ]*)\ /?folder=(.*)&file=(.*).jpg(.*) \1\2/slides/\3.jpg 
if has_jpg


But sofar, no dice

Any hint's or tips on getting this to actually work?




-Original Message- 
From: Baptiste

Sent: Wednesday, October 29, 2014 1:48 PM
To: M. Lebbink
Cc: HAProxy
Subject: Re: Wrestling with rewrites

On Wed, Oct 29, 2014 at 11:07 AM, M. Lebbink  
wrote:

Hi list,

I've been wrestling with rewrite rules withing haproxy and httpd, but I
can't find the docs I would like to read.

I keep reading examples with all sorts of rules containing hdr_dom(host) &
hdr_beg(host). But I can't find
any description of what is actually contained in these values.

Does anyone have a link or pdf listing and describing these headers?

All I want to do is check if there is a specific string in the URL for one
of the backends and if it is present, rewrite
that rule by replacing parts of it to create something the httpd server 
will

actually understand.


Michiel




Hi,

What you want to do is a reqirep conditionned by an ACL on path.
The doc:
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#reqirep

The example:
 reqrep ^([^\ :]*)\ /static/(.*) \1\ /\2 if {path_beg /static/ }

The doc about path acl:
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#path

other ACLs are defined in the same chapter.

Baptiste 

Wrestling with rewrites

[M. Lebbink]

Hi list,

I’ve been wrestling with rewrite rules withing haproxy and httpd, but I can’t 
find the docs I would like to read.

I keep reading examples with all sorts of rules containing hdr_dom(host) & 
hdr_beg(host). But I can’t find 
any description of what is actually contained in these values.

Does anyone have a link or pdf listing and describing these headers?

All I want to do is check if there is a specific string in the URL for one of 
the backends and if it is present, rewrite
that rule by replacing parts of it to create something the httpd server will 
actually understand.


Michiel

Fwd: Error Logs in Haproxy

[Peter M Souter]

Yeah, I looked in the logs before and I couldn't find the errors. But
either I didn't look hard enough or someone's changed the settings since I
last looked:

cat /var/log/capd/haproxy.log | grep 'has no server available!' | wc -l
46

But at least I can confirm I got the correct setup! :)


On Wed, Jul 16, 2014 at 4:08 PM, Baptiste  wrote:

> On Wed, Jul 16, 2014 at 4:22 PM, Peter M Souter 
> wrote:
> > Hello All!
> >
> > I'm fairly new to Haproxy and I'm configurting it with puppet as a
> reverse
> > proxy for several web apps.
> >
> > An issue I'm running into right now is that we're not getting error logs
> in
> > a file, they're just sent to stdout like so:
> >
> > Message from syslogd@localhost at Jul 16 11:17:06 ...
> > HAPROXY[8271]: backend foo has no server available!
> >
> > Right now the haproxy.cfg looks something like this:
> >
> > global
> > maxconn 4096
> > user haproxy
> > group haproxy
> > daemon
> > log 127.0.0.1 local0 debug
> > log-tag HAPROXY
> >
> > And my rsyslog config looks like this:
> >
> > $ModLoad imudp
> > $UDPServerRun 514
> >
> > local0.* -/var/log/capd/haproxy.log
> >
> > I read through the documentation and saw the log-separate-errors flag,
> if I
> > turn this on will those errors that normally go to stdout go to the
> file? Or
> > will it go to a seperate file that I need to configure in Rsyslog? Thanks
> > Regards
> >
>
> Hi Peter,
>
> Have you looked into /var/log/capd/haproxy.log and look for the errors
> here?
> They should be there ;)
> This is your syslog server which is printing this error on your
> console, not HAProxy.
>
> Second, this is not a parsing error, but an error related to
> load-balancing: HAProxy tells you that there is no servers available
> in your farm .
> none of them where able to positively answer to health checks.
>
> Baptiste
>

Re: Binaries for HAProxy.

[Peter M Souter]

> I was trying to compile HAProxy 1.5.2 with SSL support on my box but was
facing below error.Am i missing some pre requisites.Could anybody help:

Pre-reqs are:

sudo yum -y install pcre-devel gcc make openssl-devel


Taken from https://github.com/nmilford/rpm-haproxy

On Wed, Jul 16, 2014 at 3:22 PM, Kuldip Madnani 
wrote:

> Thanks Ryan/Mathew.
>
>
> Step 1 : tar xzvf haproxy-1.5.2.tar.gz
>
> Step 2 : cd haproxy-1.5.2
>
> Step 3 : make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
>
> gcc -Iinclude -Iebtree -Wall  -O2 -g -fno-strict-aliasing
> -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY
> -DCONFIG_HAP_CRYPT -DUSE_ZLIB  -DENABLE_POLL -DENABLE_EPOLL
> -DUSE_CPU_AFFINITY -DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER
> -DUSE_GETSOCKNAME -DUSE_OPENSSL  -DUSE_SYSCALL_FUTEX -DUSE_PCRE
> -I/usr/local/include  -DCONFIG_HAPROXY_VERSION=\"1.5.2\"
> -DCONFIG_HAPROXY_DATE=\"2014/07/12\" \
>   -DBUILD_TARGET='"linux2628"' \
>   -DBUILD_ARCH='""' \
>   -DBUILD_CPU='"generic"' \
>   -DBUILD_CC='"gcc"' \
>   -DBUILD_CFLAGS='"-O2 -g -fno-strict-aliasing"' \
>   -DBUILD_OPTIONS='"USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1"' \
>-c -o src/haproxy.o src/haproxy.c
> In file included from include/types/proxy.h:34,
>  from include/proto/log.h:32,
>  from include/common/cfgparse.h:29,
>  from src/haproxy.c:61:
> include/common/regex.h:30:18: error: pcre.h: No such file or directory
> include/common/regex.h:31:23: error: pcreposix.h: No such file or directory
> In file included from include/types/proxy.h:34,
>  from include/proto/log.h:32,
>  from include/common/cfgparse.h:29,
>  from src/haproxy.c:61:
> include/common/regex.h:38: error: expected specifier-qualifier-list before
> âpcreâ
> include/common/regex.h:67: error: expected â=â, â,â, â;â, âasmâ or
> â__attribute__â before âpmatchâ
> include/common/regex.h:80: warning: type defaults to âintâ in declaration
> of âregmatch_tâ
> include/common/regex.h:80: error: expected â;â, â,â or â)â before â*â token
> include/common/regex.h: In function âregex_execâ:
> include/common/regex.h:89: warning: implicit declaration of function
> âpcre_execâ
> include/common/regex.h:89: error: âconst struct my_regexâ has no member
> named âregâ
> include/common/regex.h:89: error: âconst struct my_regexâ has no member
> named âextraâ
> include/common/regex.h: In function âregex_exec2â:
> include/common/regex.h:109: error: âconst struct my_regexâ has no member
> named âregâ
> include/common/regex.h:109: error: âconst struct my_regexâ has no member
> named âextraâ
> include/common/regex.h: At top level:
> include/common/regex.h:125: error: expected declaration specifiers or
> â...â before âregmatch_tâ
> include/common/regex.h:127: error: expected declaration specifiers or
> â...â before âregmatch_tâ
> include/common/regex.h: In function âregex_freeâ:
> include/common/regex.h:131: warning: implicit declaration of function
> âpcre_freeâ
> include/common/regex.h:131: error: âstruct my_regexâ has no member named
> âregâ
> In file included from include/types/acl.h:33,
>  from include/types/proxy.h:40,
>  from include/proto/log.h:32,
>  from include/common/cfgparse.h:29,
>  from src/haproxy.c:61:
> include/types/server.h:29:25: error: openssl/ssl.h: No such file or
> directory
> In file included from include/types/connection.h:30,
>  from include/types/server.h:36,
>  from include/types/acl.h:33,
>  from include/types/proxy.h:40,
>  from include/proto/log.h:32,
>  from include/common/cfgparse.h:29,
>  from src/haproxy.c:61:
> include/types/listener.h: At top level:
> include/types/listener.h:127: error: expected specifier-qualifier-list
> before âSSL_CTXâ
> In file included from include/types/session.h:35,
>  from include/types/queue.h:29,
>  from include/types/server.h:41,
>  from include/types/acl.h:33,
>  from include/types/proxy.h:40,
>  from include/proto/log.h:32,
>  from include/common/cfgparse.h:29,
>  from src/haproxy.c:61:
> include/types/compression.h:28:18: error: zlib.h: No such file or directory
> In file included from include/types/session.h:35,
>  from include/types/queue.h:29,
>  from include/types/server.h:41,
>  from include/types/acl.h:33,
>  from include/types/proxy.h:40,
>  from include/proto/log.h:32,
>  from include/common/cfgparse.h:29,
>  from src/haproxy.c:61:
> include/types/compression.h:40: error: expected specifier-qualifier-list
> before âz_streamâ
> In file included from include/types/acl.h:33,
>  

Re: Binaries for HAProxy.

[Peter M Souter]

I've made a custom RPM I've been using for version 1.5.1 and I've hosted it
here:

https://bitbucket.org/PeteMS/haproxy-rpm-vagrant/downloads

I adapted the spec from https://github.com/nmilford/rpm-haproxy


On Wed, Jul 16, 2014 at 3:18 PM, Ryan O'Hara  wrote:

> On Wed, Jul 16, 2014 at 09:07:48AM -0500, Kuldip Madnani wrote:
> > My Linux Distribution is :
> >
> > Red Hat Enterprise Linux Server release 6.3 (Santiago)
>
> HAProxy is not included in RHEL 6.3. You will need RHEL 6.4 with Load
> Balancer AddOn or RHEL7.
>
> Ryan
>
> > On Wed, Jul 16, 2014 at 9:03 AM, Mathew Levett 
> > wrote:
> >
> > > Hi Kuldip,
> > >
> > > I think you may need to provide a little more information, it may be
> that
> > > your Linux distribution may already have haproxy in their repository.
> > > However the information supplied does not really show what your
> running.
> > > Do you know the distribution name?
> > >
> > > If its Debian then something like 'apt-get install haproxy' may be all
> you
> > > need, RedHat based distros may use yum so 'yum install haproxy'.
>  however
> > > its also not that hard to compile the latest version from source and is
> > > well documented in the download file.
> > >
> > > Usually on a list like this you need to supply as much information as
> > > possible so the people here can help.
> > >
> > > Kind Regards,
> > >
> > > Mathew
> > >
> > >
> > > On 16 July 2014 14:50, Kuldip Madnani  wrote:
> > >
> > >> Hi,
> > >>
> > >> Where can i find the compiled binaries for haproxy.My system
> > >> configuration is this :
> > >>
> > >> $ uname -a
> > >> Linux  2.6.32-279.22.1.el6.x86_64 #1 SMP Sun Jan 13 09:21:40 EST 2013
> > >> x86_64 x86_64 x86_64 GNU/Linux
> > >>
> > >> Thanks & Regards,
> > >> Kuldip
> > >>
> > >>
> > >
>
>

Error Logs in Haproxy

[Peter M Souter]

Hello All!

I'm fairly new to Haproxy and I'm configurting it with puppet as a reverse
proxy for several web apps.

An issue I'm running into right now is that we're not getting error logs in
a file, they're just sent to stdout like so:

Message from syslogd@localhost at Jul 16 11:17:06 ...
HAPROXY[8271]: backend foo has no server available!

Right now the haproxy.cfg looks something like this:

global
maxconn 4096
user haproxy
group haproxy
daemon
log 127.0.0.1 local0 debug
log-tag HAPROXY

And my rsyslog config looks like this:

$ModLoad imudp
$UDPServerRun 514

local0.* -/var/log/capd/haproxy.log

I read through the documentation and saw the log-separate-errors flag, if I
turn this on will those errors that normally go to stdout go to the file?
Or will it go to a seperate file that I need to configure in Rsyslog?
Thanks Regards

ENOTCONN from recv() on illumos

[Joshua M. Clulow]

Hi folks,

I was testing haproxy-1.5-dev22 on SmartOS (an illumos-based system)
and ran into a problem.  There's a small window after non-blocking
connect() is called, but before the TCP connection is established,
where recv() may return ENOTCONN.  On Linux, the behaviour here seems
to be always to return EAGAIN.  The fix is relatively trivial, and
appears to make haproxy work reliably on current SmartOS (see patch
below).  It's possible that other UNIX platforms exhibit this
behaviour as well.

Does this fix appear to be acceptable?


--- haproxy-1.5-dev22/src/raw_sock.c2014-02-02 23:41:29.0 +
+++ haproxy-1.5-dev22-PATCHED/src/raw_sock.c2014-03-03
21:38:45.23282 +
@@ -309,7 +309,7 @@
else if (ret == 0) {
goto read0;
}
-   else if (errno == EAGAIN) {
+   else if (errno == EAGAIN || errno == ENOTCONN) {
fd_cant_recv(conn->t.sock.fd);
break;
}


Cheers.

-- 
Joshua M. Clulow
UNIX Admin/Developer
http://blog.sysmgr.org

HAProxy for Solaris 10 X86

[Vinoth M]

Hi,

1) I am using Solaris 10 x86.Could you please let me know if there 
a pre compiled package available for it.


2) Also let me know if HAproxy is supported for Solaris 10 x86.
3) My requirement is to load balance FTP(not http) .Let me know if 
i can use HAProxy for the same.


Regards,
Vinoth

Re: Delays from HAProxy

[Andy M.]

I am trying to get closer to the switch, unfortunately my boxes are in a
data center I do not have access too.  I am working with them to try and
debug this, so far it seems like in general the servers are re transmitting
packets under the load I am testing at, not just packets through HAProxy,
so I don't think this is a problem with HAProxy, but something in their
network.  If anything comes up I will update, but as of right now, I think
the proxy is not that cause.  Thanks again, you've all been a huge help.


On Fri, Oct 11, 2013 at 11:36 AM, Willy Tarreau  wrote:

> On Fri, Oct 11, 2013 at 10:59:51AM -0400, Andy M. wrote:
> > I looked at my pcap file again.  It looks really weird.  My HAProxy gets
> > the GET request, and sends the response.  The the client resends the GET
> > request, and there seems to be a lot of tcp_retransmission and dup ack
> > packets.  Here is a picture of one request to my haproxy:
> >
> > http://i.imgur.com/r3oz6lz.png
> >
> > Any clue what would cause that problem?
>
> Yes, a typical packet loss between you and the client.
>
> > I tried to change the max_syn_backlog, and somaxconn values to both
> > 10240/20480 and 262144/262144, neither seemes to have solved the problem.
>
> Here it's not the SYN backlog since it's the HTTP request that is
> retransmitted.
> It is possible that your network interface has a defect. It happened to me
> once,
> in a batch of 10 NICs, 3 had defective RAM chips which would randomly
> corrupt
> outgoing packets. Try to use another interface or another switch port.
>
> > Conntrack is not loaded, I checked this a while ago, and I am not using
> > anything that would load it.  Here are the commands below.  It also
> doesn't
> > look like anything is being dropped.  The interface I am using is bond1.
>
> Great, since you're using bonding, it's easy to switch to the other NIC and
> see if it works better.
>
> From your stats, I'm assuming you're not running with both NICs attached to
> the same bond in round robin. I was just checking, because doing so would
> expose you to a high probability of disordering packets, which some
> firewalls
> generally don't accept and will block, causing the client to retransmit.
>
> I think now you need to sniff closer to the client to see where the packets
> are lost. If you can make a span on your switch to check if it correctly
> receives them, that will help you.
>
> Willy
>
>

Re: Delays from HAProxy

[Andy M.]

I looked at my pcap file again.  It looks really weird.  My HAProxy gets
the GET request, and sends the response.  The the client resends the GET
request, and there seems to be a lot of tcp_retransmission and dup ack
packets.  Here is a picture of one request to my haproxy:

http://i.imgur.com/r3oz6lz.png

Any clue what would cause that problem?

I tried to change the max_syn_backlog, and somaxconn values to both
10240/20480 and 262144/262144, neither seemes to have solved the problem.

Conntrack is not loaded, I checked this a while ago, and I am not using
anything that would load it.  Here are the commands below.  It also doesn't
look like anything is being dropped.  The interface I am using is bond1.

root@haproxy2:~# dmesg | grep -i full
[1.834160] usb 1-1.2: new full-speed USB device number 3 using ehci_hcd
[   69.515103] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: None
[   69.605840] bonding: bond0: link status definitely up for interface
eth0, 1000 Mbps full duplex.
[  130.088812] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: None
[  130.123654] bonding: bond1: link status definitely up for interface
eth1, 1000 Mbps full duplex.

root@haproxy2:~# netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVRTX-OK TX-ERR TX-DRP TX-OVR
Flg
bond0  1500 0271295  0  0 0639434  0  0
 0 BMmRU
bond1  1500 0503194  0  0 0393965  0  0
 0 BMmRU
eth0   1500 0271295  0  0 0639434  0  0
 0 BMsRU
eth1   1500 0503195  0  0 0393966  0  0
 0 BMsRU
eth2   1500 0 0  0  0 0 0  0  0
 0 BMsU
eth3   1500 0 0  0  0 0 0  0  0
 0 BMsU
lo16436 0   722  0  0 0   722  0  0
 0 LRU


On Fri, Oct 11, 2013 at 1:35 AM, Willy Tarreau  wrote:

> On Thu, Oct 10, 2013 at 03:57:37PM -0400, Andy M. wrote:
> > Thanks for the response pelle,
> >
> > The haproxy box is running at about 10-15% cpu.
> >
> > Looking at the TCP Dump, it seems the client is doing 1 or more
> > TCP_Retransmissions of the incoming request when there is a delay.
>  Anyone
> > know why this would happen?
>
> When you say "the incoming request", you mean the SYN packet, right ?
> If so, that means the backlog is full.
>
> > Here is the requested information:
> > root@haproxy:~# sysctl net.core.somaxconn
> > net.core.somaxconn = 128
>
> This is far too low or 20k CPS. You fill it in 6 ms at 20k CPS.
> Try increasing it to 1024 to see how things go.
>
> > root@haproxy:~# sysctl net.ipv4.tcp_max_syn_backlog
> > net.ipv4.tcp_max_syn_backlog = 2048
>
> In practice, I'm used to increase somaxconn to ~1 and max_syn_backlog
> to ~2, but it's a bit more tricky as you don't want somaxconn to be
> larger than haproxy's listen queue, otherwise the system will disable SYN
> cookies. In practice, 1024 should already be OK (about 51ms to fill it).
>
> > I removed the tcp_tw_recycle, but this did not change anything
>
> OK. If things do not improve with the settings above, you'll have to
> check if you're running netfilter and to see if your contrack table is
> never full :
>
>   # dmesg | grep -i full
>
> You also need to check if your network card or driver is not dropping
> packets :
>
>   # netstat -i
>
> Regards,
> Willy
>
>

Re: Delays from HAProxy

[Andy M.]

Thanks for the response pelle,

The haproxy box is running at about 10-15% cpu.

Looking at the TCP Dump, it seems the client is doing 1 or more
TCP_Retransmissions of the incoming request when there is a delay.  Anyone
know why this would happen?

Here is the requested information:
root@haproxy:~# sysctl net.core.somaxconn
net.core.somaxconn = 128
root@haproxy:~# sysctl net.ipv4.tcp_max_syn_backlog
net.ipv4.tcp_max_syn_backlog = 2048

I removed the tcp_tw_recycle, but this did not change anything

Thank you,

Andy


On Thu, Oct 10, 2013 at 3:44 PM, Andy M.  wrote:

> Thanks for the response pelle,
>
> The haproxy box is running at about 10-15% cpu.
>
> Looking at the TCP Dump, it seems the client is doing 1 or more
> TCP_Retransmissions of the incoming request when there is a delay.  Anyone
> know why this would happen?
>
> Here is the requested information:
> root@haproxy:~# sysctl net.core.somaxconn
> net.core.somaxconn = 128
> root@haproxy:~# sysctl net.ipv4.tcp_max_syn_backlog
> net.ipv4.tcp_max_syn_backlog = 2048
>
> I removed the tcp_tw_recycle, but this did not change anything
>
> Thank you,
>
> Andy
>
>
>
> On Thu, Oct 10, 2013 at 1:58 PM, Pär Åslund  wrote:
>
>> Hi Andy,
>>
>> Can't see anything wrong with your configuration.
>> How about the server running haproxy? anything about that regarding cpu
>> etc?
>>
>> I wouldn't be using net.ipv4.tcp_tw_recycle = 1
>> tw_recycle is a bit dodgy and can give some unwanted side-effects.
>> tw_reuse = 1 should be sufficient.
>>
>> How does a tcpdump look? Since you get this issue on the stats get a dump
>> on that.
>>
>>  What setting are there on the box regarding SYN backlog?
>> net.core.somaxconn & net.ipv4.tcp_max_syn_backlog
>>
>> Best regards,
>> pelle
>>
>>
>> On Thu, Oct 10, 2013 at 7:16 PM, Andy M.  wrote:
>>
>>> Hello,
>>>
>>> I am trying to configure an HAProxy And seem to be running into a
>>> problem where the HAProxy will spike.  I have a high throughput server
>>> cluster, which will need to handle about 10-20k QPS.  I believe that
>>> HAProxy should be able to handle that quite easily from what I have read.
>>>  I am running HAProxy on Ubuntu 12.04LTS.  I am running an a Xeon 1270,
>>> with 8GB of RAM(Which i believe is all overkill).  I am doing a simple load
>>> balance between two or three servers(I have two server clusters, one with
>>> two servers, one with three).  I am running 1.4.24.
>>>
>>> The reason I believe something is wrong with my haproxy is because I can
>>> refresh my webpage a few times, and every 4-5 times I will be get a spike
>>> around 800ms-3sec to resolve.  This also happens when not hitting my
>>> backend server, and instead hitting the HAProxy stats page.  None of my
>>> servers are anywhere close to using up their system resources(all below 20%
>>> in both memory and CPU).  I don't see much in the syslog that seems wrong
>>> other than this message:
>>>
>>> Oct 10 16:48:28 haproxy rsyslogd-2177: imuxsock begins to drop message
>>> 3912 due to rate-limiting
>>>
>>> I believe that has to do with the logging not being able to keep up,
>>> rather than haproxy dropping packets.
>>>
>>> I was wondering if  there were any suggestions for me to look into to
>>> try and fix my problem, or suggest how to configure a HAProxy correctly to
>>> be able to handle the load I need(preferably higher than the load I need,
>>> as this may increase over time).  I am open to changing operating systems
>>> if need be as well.
>>>
>>> Here are my two config files:
>>>
>>> global
>>>log /dev/log   local0 info
>>>log /dev/log   local0 notice
>>>maxconn 5
>>>user y
>>>group y
>>>#debug
>>>
>>> defaults
>>>log global
>>>modehttp
>>>option  httplog
>>>option  dontlognull
>>>option  forwardfor
>>>retries 3
>>>option redispatch
>>>option http-server-close
>>>maxconn 3
>>>contimeout  1
>>>clitimeout  5
>>>srvtimeout  5
>>>balance leastconn
>>>
>>> listen  c_cluster 255.255.255.146:80
>>> maxconn 3
>>> server  c1 10.101.13.74:80 maxconn 1
>>> server  c2 10.101.13.78

Fwd: Delays from HAProxy

[Andy M.]

Thanks for the response pelle,

The haproxy box is running at about 10-15% cpu.

Looking at the TCP Dump, it seems the client is doing 1 or more
TCP_Retransmissions of the incoming request when there is a delay.  Anyone
know why this would happen?

Here is the requested information:
root@haproxy:~# sysctl net.core.somaxconn
net.core.somaxconn = 128
root@haproxy:~# sysctl net.ipv4.tcp_max_syn_backlog
net.ipv4.tcp_max_syn_backlog = 2048

I removed the tcp_tw_recycle, but this did not change anything

Thank you,

Andy



On Thu, Oct 10, 2013 at 1:58 PM, Pär Åslund  wrote:

> Hi Andy,
>
> Can't see anything wrong with your configuration.
> How about the server running haproxy? anything about that regarding cpu
> etc?
>
> I wouldn't be using net.ipv4.tcp_tw_recycle = 1
> tw_recycle is a bit dodgy and can give some unwanted side-effects.
> tw_reuse = 1 should be sufficient.
>
> How does a tcpdump look? Since you get this issue on the stats get a dump
> on that.
>
>  What setting are there on the box regarding SYN backlog?
> net.core.somaxconn & net.ipv4.tcp_max_syn_backlog
>
> Best regards,
> pelle
>
>
> On Thu, Oct 10, 2013 at 7:16 PM, Andy M.  wrote:
>
>> Hello,
>>
>> I am trying to configure an HAProxy And seem to be running into a problem
>> where the HAProxy will spike.  I have a high throughput server cluster,
>> which will need to handle about 10-20k QPS.  I believe that HAProxy should
>> be able to handle that quite easily from what I have read.  I am running
>> HAProxy on Ubuntu 12.04LTS.  I am running an a Xeon 1270, with 8GB of
>> RAM(Which i believe is all overkill).  I am doing a simple load balance
>> between two or three servers(I have two server clusters, one with two
>> servers, one with three).  I am running 1.4.24.
>>
>> The reason I believe something is wrong with my haproxy is because I can
>> refresh my webpage a few times, and every 4-5 times I will be get a spike
>> around 800ms-3sec to resolve.  This also happens when not hitting my
>> backend server, and instead hitting the HAProxy stats page.  None of my
>> servers are anywhere close to using up their system resources(all below 20%
>> in both memory and CPU).  I don't see much in the syslog that seems wrong
>> other than this message:
>>
>> Oct 10 16:48:28 haproxy rsyslogd-2177: imuxsock begins to drop message
>> 3912 due to rate-limiting
>>
>> I believe that has to do with the logging not being able to keep up,
>> rather than haproxy dropping packets.
>>
>> I was wondering if  there were any suggestions for me to look into to try
>> and fix my problem, or suggest how to configure a HAProxy correctly to be
>> able to handle the load I need(preferably higher than the load I need, as
>> this may increase over time).  I am open to changing operating systems if
>> need be as well.
>>
>> Here are my two config files:
>>
>> global
>>log /dev/log   local0 info
>>log /dev/log   local0 notice
>>maxconn 5
>>user y
>>group y
>>#debug
>>
>> defaults
>>log global
>>modehttp
>>option  httplog
>>option  dontlognull
>>option  forwardfor
>>retries 3
>>option redispatch
>>option http-server-close
>>maxconn 3
>>contimeout  1
>>clitimeout  5
>>srvtimeout  5
>>balance leastconn
>>
>> listen  c_cluster 255.255.255.146:80
>> maxconn 3
>> server  c1 10.101.13.74:80 maxconn 1
>> server  c2 10.101.13.78:80 maxconn 1
>> server  c3 10.101.13.82:80 maxconn 1
>> listen  stats :
>> mode http
>> stats enable
>> stats hide-version
>> stats uri /
>>
>>
>> and
>>
>> # this config needs haproxy-1.1.28 or haproxy-1.2.1
>> global
>>log /dev/log   local0 info
>>log /dev/log   local0 notice
>>maxconn 2
>>user y
>>group y
>>#debug
>>
>>  defaults
>>log global
>>modehttp
>>option  httplog
>>option  dontlognull
>>option  forwardfor
>>retries 3
>>option redispatch
>>option http-server-close
>>maxconn 2
>>contimeout  1
>>clitimeout  5
>>srvtimeout  5
>>balance leastconn
>>
>> listen  a_cluster 255.255.255.151:80
>> maxconn 2
>> server  a1 10.101.13.68:80 maxconn 1
>> server  a2 10.101.13.66:80 maxconn 1
>> listen  stats :
>> mode http
>> stats enable
>> stats hide-version
>> stats uri /
>>
>>
>> My sysctl.conf has the following changes:
>>
>> fs.file-max = 100
>> net.ipv4.tcp_tw_reuse = 1
>> net.ipv4.tcp_tw_recycle = 1
>>
>> Thank you,
>>
>> Andy
>>
>
>

Delays from HAProxy

[Andy M.]

Hello,

I am trying to configure an HAProxy And seem to be running into a problem
where the HAProxy will spike.  I have a high throughput server cluster,
which will need to handle about 10-20k QPS.  I believe that HAProxy should
be able to handle that quite easily from what I have read.  I am running
HAProxy on Ubuntu 12.04LTS.  I am running an a Xeon 1270, with 8GB of
RAM(Which i believe is all overkill).  I am doing a simple load balance
between two or three servers(I have two server clusters, one with two
servers, one with three).  I am running 1.4.24.

The reason I believe something is wrong with my haproxy is because I can
refresh my webpage a few times, and every 4-5 times I will be get a spike
around 800ms-3sec to resolve.  This also happens when not hitting my
backend server, and instead hitting the HAProxy stats page.  None of my
servers are anywhere close to using up their system resources(all below 20%
in both memory and CPU).  I don't see much in the syslog that seems wrong
other than this message:

Oct 10 16:48:28 haproxy rsyslogd-2177: imuxsock begins to drop message 3912
due to rate-limiting

I believe that has to do with the logging not being able to keep up, rather
than haproxy dropping packets.

I was wondering if  there were any suggestions for me to look into to try
and fix my problem, or suggest how to configure a HAProxy correctly to be
able to handle the load I need(preferably higher than the load I need, as
this may increase over time).  I am open to changing operating systems if
need be as well.

Here are my two config files:

global
   log /dev/log   local0 info
   log /dev/log   local0 notice
   maxconn 5
   user y
   group y
   #debug

defaults
   log global
   modehttp
   option  httplog
   option  dontlognull
   option  forwardfor
   retries 3
   option redispatch
   option http-server-close
   maxconn 3
   contimeout  1
   clitimeout  5
   srvtimeout  5
   balance leastconn

listen  c_cluster 255.255.255.146:80
maxconn 3
server  c1 10.101.13.74:80 maxconn 1
server  c2 10.101.13.78:80 maxconn 1
server  c3 10.101.13.82:80 maxconn 1
listen  stats :
mode http
stats enable
stats hide-version
stats uri /


and

# this config needs haproxy-1.1.28 or haproxy-1.2.1
global
   log /dev/log   local0 info
   log /dev/log   local0 notice
   maxconn 2
   user y
   group y
   #debug

defaults
   log global
   modehttp
   option  httplog
   option  dontlognull
   option  forwardfor
   retries 3
   option redispatch
   option http-server-close
   maxconn 2
   contimeout  1
   clitimeout  5
   srvtimeout  5
   balance leastconn

listen  a_cluster 255.255.255.151:80
maxconn 2
server  a1 10.101.13.68:80 maxconn 1
server  a2 10.101.13.66:80 maxconn 1
listen  stats :
mode http
stats enable
stats hide-version
stats uri /


My sysctl.conf has the following changes:

fs.file-max = 100
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

Thank you,

Andy

Re: Prevent HAProxy from toggeling back from fallback to primary

[Claudio M.]

In data martedì 23 luglio 2013 17:23:07, hushmeh...@hushmail.com ha scritto:
> On Tue, 23 Jul 2013 14:48:22 +0200 "Claudio M."
> 
>  wrote:
> >server web1.bk  192.168.0.43:80 cookie web1.bk  check port
> >
> >80 maxconn 100 backup
> >I need that when both primary servers go down and web1.bk go
> >online, when web1b.mi and/or web1d.mi go up haproxy not switch to
> >these
> 
> I am not completly sure what you want. If web1b/d.mi is UP again
> after a failure, you still see sessions send to web1.bk and you
> dont want that?
> If so, then use the "non-stick" option and dont add a cookie, like:
> server web1.bk 192.168.0.43:80 check port 80 maxconn 100 backup non-
> stick

web1b adnd web1d are in a cluster and are on sync by drbd, web1.bk is out the 
cluster and is on sync by rsync every 60 minutes. If the DRBD cluster fails, i 
need to resync data from web1.bk to the cluster before haproxy starts to send 
connections to the cluster

Best regards
Claudio

Prevent HAProxy from toggeling back from fallback to primary

[Claudio M.]

Hi, ive the folowing configuration 

backend web1.mi.ext
mode http
option httpchk
balance roundrobin
option httplog
option http-server-close
cookie CLUSTID insert
source 0.0.0.0 usesrc clientip
server web1b.mi 192.168.0.29:80 cookie web1b.mi check port 80 maxconn 
100
server web1d.mi 192.168.0.21:80 cookie web1d.mi check port 80 maxconn 
100
server web1.bk  192.168.0.43:80 cookie web1.bk  check port 80 maxconn 
100 backup

where web1.bk is obviously the backup server

I need that when both primary servers go down and web1.bk go online, when 
web1b.mi and/or web1d.mi go up haproxy not switch to these

Is this possible?

Best regards
Claudio

RE: HAProxy

[Jayadevan M]

Hi,
Got it, timeout check.
Regards,
Jayadevan

From: Jayadevan M
Sent: Thursday, June 20, 2013 4:09 PM
To: 'haproxy@formilux.org'
Subject: HAProxy

Hi,
HAProxy's mysql-check works with a couple of MySQL servers on Windows machines, 
not with another pair. If I try to connect to MySQL on those machines from 
mysql client (on the machine where HAProxy is running), the only noticeable 
difference is that it takes a long time to connect to MySQL on machines where I 
have problems with mysql-check. Telnetting gives different output.


1)  HAProxy's mysql-check gives a timeout for this m/c.
[root@master haproxy-1.4.24]#
[root@master haproxy-1.4.24]# telnet 192.168.210.106 3307
Trying 192.168.210.106...
Connected to 192.168.210.106 (192.168.210.106).
Escape character is '^]'.
B
5.1.44-community-logy)GljP1B:!<^CvGANCH|[2Connection closed by foreign host.



2)  HAProxy's mysql-check works with this one.

[root@master haproxy-1.4.24]# telnet 192.168.210.90 3306
Trying 192.168.210.90...
Connected to 192.168.210.90 (192.168.210.90).
Escape character is '^]'.
L
5.6.4-m71qi;*]vOQ*yJAJ.ESPxmysql_native_passwordConnection closed by foreign 
host.

Is it possible to change HAProxy's timeout for mysql-check? Any other ideas?

Message from HAProxy is
[WARNING] 170/213121 (606) : Server request_mysql/svr2 is DOWN, reason: Layer7 
timeout, check duration: 2000ms. 1 active and 1 backup servers left. 0 sessions 
 
ctive, 0 requeued, 0 remaining in queue.
[WARNING] 170/213123 (606) : Backup Server request_mysql/svr1 is DOWN, reason: 
Layer7 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 
ses 
sions active, 0 requeued, 0 remaining in queue.


Regards,
Jayadevan



DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

HAProxy

[Jayadevan M]

Hi,
HAProxy's mysql-check works with a couple of MySQL servers on Windows machines, 
not with another pair. If I try to connect to MySQL on those machines from 
mysql client (on the machine where HAProxy is running), the only noticeable 
difference is that it takes a long time to connect to MySQL on machines where I 
have problems with mysql-check. Telnetting gives different output.


1)  HAProxy's mysql-check gives a timeout for this m/c.
[root@master haproxy-1.4.24]#
[root@master haproxy-1.4.24]# telnet 192.168.210.106 3307
Trying 192.168.210.106...
Connected to 192.168.210.106 (192.168.210.106).
Escape character is '^]'.
B
5.1.44-community-logy)GljP1B:!<^CvGANCH|[2Connection closed by foreign host.



2)  HAProxy's mysql-check works with this one.

[root@master haproxy-1.4.24]# telnet 192.168.210.90 3306
Trying 192.168.210.90...
Connected to 192.168.210.90 (192.168.210.90).
Escape character is '^]'.
L
5.6.4-m71qi;*]vOQ*yJAJ.ESPxmysql_native_passwordConnection closed by foreign 
host.

Is it possible to change HAProxy's timeout for mysql-check? Any other ideas?

Message from HAProxy is
[WARNING] 170/213121 (606) : Server request_mysql/svr2 is DOWN, reason: Layer7 
timeout, check duration: 2000ms. 1 active and 1 backup servers left. 0 sessions 
 
ctive, 0 requeued, 0 remaining in queue.
[WARNING] 170/213123 (606) : Backup Server request_mysql/svr1 is DOWN, reason: 
Layer7 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 
ses 
sions active, 0 requeued, 0 remaining in queue.


Regards,
Jayadevan



DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

Hi,
>Jayadevan, can you tell us what version you are running (which works) and
>what release you where using before? Just double checking that commit
>212f778d6 fixed that problem ...

Initial - HA-Proxy version 1.4.9 2010/10/28

Now - HA-Proxy version 1.4.24 2013/06/17

>I guess you had to do both things to make it work; or did you revert the the
>plugin change?
>
>Can you post:
>select user,HOST,plugin from mysql.user;

mysql> select user,HOST,plugin from mysql.user ;
+-+---++
| user| HOST  | plugin |
+-+---++
| root| localhost ||
| root| 127.0.0.1 ||
| root| ::1   ||
| | localhost ||
| root| % ||
| galaxy  | % ||
| haproxy | % ||
+-+---++

Regards,
Jayadevan



DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

HAProxy for failover

[Jayadevan M]

Hello all,

I am using HAProxy for MYSQL failover. It is a MySQL master-slave replication 
environment. When master is UP, all reads and writes go to master. In case the 
master is down, reads and writes will go to the slave. Once the master is down 
and HAProxy redirects all reads/writes to the slave, we do not want any 
writes/reads to go to the master even if it comes up automatically. We want to 
ensure that there will be some kind of manual intervention before a master, 
once marked down, gets requests again. Is this possible?


Regards,
Jayadevan



DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

>
>Did you change authentication plugin to make it work ? If this is due to auth
>plugin, we definitely need to update the documentation.
>
I did. But that did not help. So I used latest version of HAProxy. That worked. 
This is the status now -

mysql> select distinct user,HOST,plugin from mysql.user ;
+-+---++
| user| HOST  | plugin |
+-+---++
| root| localhost ||
| root| 127.0.0.1 ||
| root| ::1   ||
| | localhost ||
| root| % ||
| galaxy  | % ||
| haproxy | % ||
+-+---++
Regards,
Jayadevan



DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

Hi,
>
>Verify this for the user you specified in option mysql-check:
>
>select plugin from mysql.user where user='monitor' \G
>*** 1. row ***
>plugin: sha256_password
>1 row in set (0.00 sec)
>
>If you see sha256_password, it won't work.
>
>Since password authentication is not used in health checks by HAproxy, you
>can just create a new user that is not using this kind of authentication to be
>used by the HAproxy:
>
>CREATE USER monitor@ IDENTIFIED WITH
>'mysql_native_password';
>
>Verify it is working by doing:
>
>select plugin from mysql.user where user='monitor' \G
>*** 1. row ***
>plugin: mysql_native_password
>1 row in set (0.00 sec)
>
>
I moved to the latets version of HAProxy and now it is working. Need to do some 
more testing. Thanks for all the input/directions.
Regards,
Jayadevan


DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

Hi,

> listen stats :1936
> mode http
> stats enable
> stats realm Haproxy\ Statistics
> stats uri /
>
> frontend  mysql_proxy *:3309
> mode tcp
> default_backend request_mysql
One error to another - How can I make HAProxy use a new MySQL client/libraries? 
I am getting the error -

[WARNING] 167/224803 (20008) : Server request_mysql/svr1 is DOWN, reason: 
Layer7 wrong status, code: 0, info: "Client does not support authentication 
protocol requested by server; consider upgrading MySQL client", check duration: 
2ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 
remaining in queue.

I have MySQL 5.6.10

Regards,
Jayadevan


DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

HI,

>It should not because as you can see in the trace, the whole check happens in
>only 2 milliseconds, which is quite fast. What are the other parameters you
>changed ? Could you also please share all your timeouts ? Maybe some of
>them are wrong ?
>
I am getting a layer 4 timeout in the environment where I should get this 
running. Here is my file -
global
pidfile /var/run/haproxy1.pid
stats socket /tmp/haproxy
defaults
log global
timeout connect 5 # default 10 second time out if a backend is not found
timeout client  3
timeout server  3

listen stats :1936
mode http
stats enable
stats realm Haproxy\ Statistics
stats uri /

frontend  mysql_proxy *:3309
mode tcp
default_backend request_mysql

backend request_mysql
mode tcp
option mysql-check user haproxy
server svr2 192.168.8.37:3406  weight 1 check port 3406 inter 5000 rise 
3 fall 3
server svr1 192.168.2.27:3306  weight 1 check port 3306 inter 5000 rise 
3 fall 3

Please note that the IPs have changed and now they are both windows machines.
Do we need MySQL client to be installed on the m/c for this to work?


Regards,
Jayadevan



DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

Hi,

>OK. That's really strange. I'll recheck here by copy-pasting your data to a 
>fake
>responder, just in case we missed something.

Does the following parameter have anything to do with this? I tried changing a 
few parameters and now it seems to be working. Will test some more. In the real 
scenario, both MySQL servers will be on Windows.

timeout connect 5

Regards,
Jayadevan


DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

HI,
>OK so linux part here :
>
>> 10:25:57.474629 connect(5, {sa_family=AF_INET, sin_port=htons(3306),
>> sin_addr=inet_addr("192.168.2.27")}, 16) = -1 EINPROGRESS (Operation
>> now in progress) <0.000202>
>> 10:25:57.475000 sendto(5,
>> "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
>> MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 <0.99>
>> 10:25:57.475259 epoll_wait(3, {}, 6, 0) = 0 <0.84>
>> 10:25:57.475488 recvfrom(5,
>> "4\0\0\0\n5.1.52\0\253\27\0\0lUC0d,Jn\0\377\367!\2\0\0\0\0\0\0\0\0\0\0
>> \0\0\0\0mWKTA,VKVNe3\0\3\0\0\2\0\0\0", 16384, 0, NULL, NULL) = 63
>> <0.71>
>> 10:25:57.475691 recvfrom(5, "", 16321, 0, NULL, NULL) = 0 <0.50>
>> 10:25:57.475824 shutdown(5, 2 /* send and receive */) = 0 <0.63>
>> 10:25:57.475943 epoll_wait(3, {}, 6, 0) = 0 <0.27>
>> 10:25:57.476021 close(5)= 0 <0.19>
>
>And windows part here :
>
>> 10:25:59.985133 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
>> <0.000101>
>> 10:25:59.985375 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 <0.83>
>> 10:25:59.985609 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
>> <0.83>
>> 10:25:59.985838 connect(5, {sa_family=AF_INET, sin_port=htons(3406),
>> sin_addr=inet_addr("192.168.8.37")}, 16) = -1 EINPROGRESS (Operation
>> now in progress) <0.000102>
>> 10:25:59.986031 sendto(5,
>> "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
>> MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource
>temporarily
>> unavailable) <0.14>
>> 10:25:59.986110 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLOUT, {u32=5,
>> u64=5}}) = 0 <0.63>
>> 10:25:59.986250 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 6, 1000) =
>> 1 <0.000173>
>> 10:25:59.986487 sendto(5,
>> "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
>> MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 <0.38>
>> 10:25:59.986609 recvfrom(5, 0x1e92770, 16384, 0, 0, 0) = -1 EAGAIN
>> (Resource temporarily unavailable) <0.66>
>> 10:25:59.986778 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5,
>> u64=5}}) = 0 <0.64>
>> 10:25:59.986929 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 6, 1000) =
>> 1 <0.000397>
>> 10:25:59.987373 recvfrom(5,
>> "B\0\0\0\n5.1.34-community-log\0\20\0\0\0?)L&t9]I\0\377\367\10\2\0\0\0
>> \0\0\0\0\0\0\0\0\0\0\0007uj4an%h`*M%\0\3\0\0\2\0\0\0", 16384, 0, NULL,
>> NULL) = 77 <0.15>
>> 10:25:59.987440 recvfrom(5, "", 16307, 0, NULL, NULL) = 0 <0.29>
>> 10:25:59.987508 shutdown(5, 2 /* send and receive */) = 0 <0.16>
>> 10:25:59.987560 close(5)= 0 <0.30>
>
>So the response was received, including the shutdown. It seems like it's the
>response contents which are not correctly parsed then.
>
>I have compared the received data to what the check does and both checks
>seem to parse correctly (both contain two response packets) and both should
>return the major version. And I've double checked, both haproxy 1.4 and 1.5
>use the same parser.
>
>What version of haproxy is this ?
>
It is haproxy-1.4.22
What does " MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource 
temporarily unavailable) <0.14> " mean?

Regards,
Jayadevan




DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

Hi,

>-Original Message-
>From: Willy Tarreau [mailto:w...@1wt.eu]
>Sent: Friday, June 14, 2013 5:53 PM
>To: Jayadevan M
>Cc: Jayadevan M; haproxy@formilux.org
>Subject: Re: haproxy mysql-check
>
>On Fri, Jun 14, 2013 at 10:30:14AM +, Jayadevan M wrote:
>> Telnet works -
>> -bash-3.00# telnet 192.168.8.37 3406
>> Trying 192.168.8.37...
>> Connected to 192.168.8.37.
>> Escape character is '^]'.
>> B
>> 5.1.34-community-logÂBjon[]Ax<`OI_&<;Qg3^CConnection to 192.168.8.37
>closed by foreign host.
>
>OK. Could you show us the complete strace output from haproxy, from the
>connect() call to the close() call ? Ideally if you could take it with strace 
>-ttT, it
>would be nice as we'd also have the timings.
>It is possible that we're simply facing a bug in the timeout handling for
>example.

Here is the output. 192.168.2.27 (MySQL on linux) works. 192.168.8.37 (MySQL on 
Windows) does not.
10:25:57.474629 connect(5, {sa_family=AF_INET, sin_port=htons(3306), 
sin_addr=inet_addr("192.168.2.27")}, 16) = -1 EINPROGRESS (Operation now in 
progress) <0.000202>
10:25:57.475000 sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 <0.99>
10:25:57.475259 epoll_wait(3, {}, 6, 0) = 0 <0.84>
10:25:57.475488 recvfrom(5, 
"4\0\0\0\n5.1.52\0\253\27\0\0lUC0d,Jn\0\377\367!\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0mWKTA,VKVNe3\0\3\0\0\2\0\0\0",
 16384, 0, NULL, NULL) = 63 <0.71>
10:25:57.475691 recvfrom(5, "", 16321, 0, NULL, NULL) = 0 <0.50>
10:25:57.475824 shutdown(5, 2 /* send and receive */) = 0 <0.63>
10:25:57.475943 epoll_wait(3, {}, 6, 0) = 0 <0.27>
10:25:57.476021 close(5)= 0 <0.19>
10:25:57.476075 epoll_wait(3, {}, 5, 1000) = 0 <0.44>
10:25:58.476230 epoll_wait(3, {}, 5, 1000) = 0 <0.999819>
10:25:59.476230 epoll_wait(3, {}, 5, 507) = 0 <0.506766>
10:25:59.983158 epoll_wait(3, {}, 5, 2) = 0 <0.001827>
10:25:59.985133 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 <0.000101>
10:25:59.985375 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 <0.83>
10:25:59.985609 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.83>
10:25:59.985838 connect(5, {sa_family=AF_INET, sin_port=htons(3406), 
sin_addr=inet_addr("192.168.8.37")}, 16) = -1 EINPROGRESS (Operation now in 
progress) <0.000102>
10:25:59.986031 sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily 
unavailable) <0.14>
10:25:59.986110 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLOUT, {u32=5, u64=5}}) = 0 
<0.63>
10:25:59.986250 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 6, 1000) = 1 
<0.000173>
10:25:59.986487 sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 <0.38>
10:25:59.986609 recvfrom(5, 0x1e92770, 16384, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable) <0.000066>
10:25:59.986778 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5, u64=5}}) = 0 
<0.64>
10:25:59.986929 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 6, 1000) = 1 
<0.000397>
10:25:59.987373 recvfrom(5, 
"B\0\0\0\n5.1.34-community-log\0\20\0\0\0?)L&t9]I\0\377\367\10\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0007uj4an%h`*M%\0\3\0\0\2\0\0\0",
 16384, 0, NULL, NULL) = 77 <0.15>
10:25:59.987440 recvfrom(5, "", 16307, 0, NULL, NULL) = 0 <0.29>
10:25:59.987508 shutdown(5, 2 /* send and receive */) = 0 <0.16>
10:25:59.987560 close(5)= 0 <0.30>
10:25:59.987625 epoll_wait(3, {}, 5, 1000) = 0 <0.999548>
10:26:00.987263 epoll_wait(3, {}, 5, 1000) = 0 <0.999853>
10:26:01.987288 epoll_wait(3, {}, 5, 488) = 0 <0.488027>
10:26:02.476277 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 <0.96>
10:26:02.476452 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 <0.11>
10:26:02.476511 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.12>
10:26:02.476563 connect(5, {sa_family=AF_INET, sin_port=htons(3306), 
sin_addr=inet_addr("192.168.2.27")}, 16) = -1 EINPROGRESS (Operation now in 
progress) <0.000166>

Connect from command line -

[root@n3170 ~]# mysql -h 192.168.8.37 --port=3406 -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 74
Server version: 5.1.34-community-log MySQL Community Server (GPL)

Regards,
Jayadevan


DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

RE: haproxy mysql-check

[Jayadevan M]

Hi,


>-Original Message-
>From: Willy Tarreau [mailto:w...@1wt.eu]
>Sent: Friday, June 14, 2013 3:44 PM
>To: Jayadevan M
>Cc: haproxy@formilux.org; Jayadevan M
>Subject: Re: haproxy mysql-check
>
>Hi,
>
>On Fri, Jun 14, 2013 at 03:38:25PM +0530, Jayadevan M wrote:
>> Hi,
>>
>> I was trying to make HAProxy's mysql-check option work with the MySQL
>> Servers on Windows. HAProxy is on linux, MySQL on windows.There are 2
>> MySQL servers - Server1 and Server2. It was not working (HAProxy was
>> always marking the servers down). I tried changing one of the MySQL
>> server entries point to MySQL on a linux box and it was working. Won't
>> HAProxy's mysql-check work if the target MySQL is on windows? Output
>> from a strace of HAProxy -
>>
>> connect(5, {sa_family=AF_INET, sin_port=htons(3406),
>> sin_addr=inet_addr("192.168.8.37")}, 16) = -1 EINPROGRESS (Operation
>> now in
>> progress)
>> sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
>> MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource
>temporarily
>> unavailable)
>>
>> connect(5, {sa_family=AF_INET, sin_port=htons(3306),
>> sin_addr=inet_addr("192.168.2.27")}, 16) = -1 EINPROGRESS (Operation
>> now in
>> progress)
>> sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
>> MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23
>>
>> 192.168.2.27 is linux and 192.168.8.37 windows. I am able to connect
>> to the MySQL server on windows, from mysql command line client on
>> 192.168.2.27. So it is not a port blocked issue.
>
>But from what I understand, it means that the windows server is not
>responding to the connection request. I don't know if this is intentional or a
>mistake, but the port is different on your windows config (3406). Could you
>check with telnet from the haproxy host that the connection establishes on
>this port :
>
>  # telnet 192.168.8.37 3406

Telnet works -
-bash-3.00# telnet 192.168.8.37 3406
Trying 192.168.8.37...
Connected to 192.168.8.37.
Escape character is '^]'.
B
5.1.34-community-logÂBjon[]Ax<`OI_&<;Qg3^CConnection to 192.168.8.37 closed by 
foreign host.

Regards,
Jayadevan


DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

Re: haproxy mysql-check

[Jayadevan M]

Hi,
The port is correct. I am running MySQL on 3406. Will try to take tcpdum.
Regards,
Jayadevan


On Fri, Jun 14, 2013 at 3:43 PM, Willy Tarreau  wrote:

> Hi,
>
> On Fri, Jun 14, 2013 at 03:38:25PM +0530, Jayadevan M wrote:
> > Hi,
> >
> > I was trying to make HAProxy's mysql-check option work with the MySQL
> > Servers on Windows. HAProxy is on linux, MySQL on windows.There are 2
> MySQL
> > servers - Server1 and Server2. It was not working (HAProxy was always
> > marking the servers down). I tried changing one of the MySQL server
> entries
> > point to MySQL on a linux box and it was working. Won't HAProxy's
> > mysql-check work if the target MySQL is on windows? Output from a strace
> of
> > HAProxy -
> >
> > connect(5, {sa_family=AF_INET, sin_port=htons(3406),
> > sin_addr=inet_addr("192.168.8.37")}, 16) = -1 EINPROGRESS (Operation now
> in
> > progress)
> > sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
> > MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily
> > unavailable)
> >
> > connect(5, {sa_family=AF_INET, sin_port=htons(3306),
> > sin_addr=inet_addr("192.168.2.27")}, 16) = -1 EINPROGRESS (Operation now
> in
> > progress)
> > sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
> > MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23
> >
> > 192.168.2.27 is linux and 192.168.8.37 windows. I am able to connect to
> the
> > MySQL server on windows, from mysql command line client on 192.168.2.27.
> So
> > it is not a port blocked issue.
>
> But from what I understand, it means that the windows server is not
> responding
> to the connection request. I don't know if this is intentional or a
> mistake,
> but the port is different on your windows config (3406). Could you check
> with telnet from the haproxy host that the connection establishes on this
> port :
>
>   # telnet 192.168.8.37 3406
>
> That way we'll be sure that the same parameters are used. Also, it would
> be nice
> to take a tcpdump capture from the haproxy machine, maybe it will reveal
> why
> this does not work.
>
> Regards,
> Willy
>
>

haproxy mysql-check

[Jayadevan M]

Hi,

I was trying to make HAProxy's mysql-check option work with the MySQL
Servers on Windows. HAProxy is on linux, MySQL on windows.There are 2 MySQL
servers - Server1 and Server2. It was not working (HAProxy was always
marking the servers down). I tried changing one of the MySQL server entries
point to MySQL on a linux box and it was working. Won't HAProxy's
mysql-check work if the target MySQL is on windows? Output from a strace of
HAProxy -

connect(5, {sa_family=AF_INET, sin_port=htons(3406),
sin_addr=inet_addr("192.168.8.37")}, 16) = -1 EINPROGRESS (Operation now in
progress)
sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily
unavailable)

connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("192.168.2.27")}, 16) = -1 EINPROGRESS (Operation now in
progress)
sendto(5, "\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1", 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23

192.168.2.27 is linux and 192.168.8.37 windows. I am able to connect to the
MySQL server on windows, from mysql command line client on 192.168.2.27. So
it is not a port blocked issue.

Regards,
Jayadevan

HAProxy and MySQL failover

[Jayadevan M]

Hi,
We are using HAProxy to check for MySQL availability and failing over to a 
slave. Is it possible to stop HAProxy going back to the master once it is up? 
We want that to happen after manual intervention.
Regards,
Jayadevan


DISCLAIMER: "The information in this e-mail and any attachment is intended only 
for the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly contact 
the sender and destroy all copies of the original communication. IBS makes no 
warranty, express or implied, nor guarantees the accuracy, adequacy or 
completeness of the information contained in this email or any attachment and 
is not liable for any errors, defects, omissions, viruses or for resultant loss 
or damage, if any, direct or indirect."

Re: issues with very long URLs.

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Thanks so much for your time/response.
We're trying adjusted settings this morning in our DEV/Test environment.
- Kevin

On Sep 20, 2012, at 1:05 AM, Willy Tarreau wrote:

> Hi Kevin,
> 
> On Wed, Sep 19, 2012 at 10:02:07PM -0500, Lange, Kevin M. 
> (GSFC-423.0)[RAYTHEON COMPANY] wrote:
>> Hi,
>> Got a few cycles to do some testing/verification.  
>> Sure enough when we pass a long URL through haproxy/lighttpd, we see this in 
>> the haproxy log:
>> 
>> Sep 19 22:45:14 127.0.0.1 haproxy[4195]: 172.28.xx.xx:55500 
>> [19/Sep/2012:22:45:14.428] kernel_partner_ft kernel_partner_ft/ 
>> -1/-1/-1/-1/0 400 187 - - PR-- 0/0/0/0/0 0/0 ""
>> 
>> From your message, you say that haproxy will be marked with PR if blocked by
>> haproxy, so this looks like haproxy is confirmed to be the cause of our
>> problem.  Correct?
> 
> Exactly !
> 
>> We're going to draft and test a  configuration that increases maxrewrite, and
>> perhaps bufsize.
> 
> You should do the opposite. A buffer is sized to contain a full request plus
> a small space needed to add some headers or rewrite them. So haproxy refrains
> from filling a buffer. The maximum request length it accepts is then
> (bufsize-maxrewrite).
> 
> The default values are 16384 for bufsize, and 8192 for maxrewrite, which 
> results
> in 8192 bytes max for a received request. Start by simply reducing maxrewrite 
> to
> 1024 and you'll automatically get 15kB for an incoming request. And if that's
> not enough, then you'll have to increase bufsize. But be careful. Experience
> tells us that quite often when you hit any component's limit along a path,
> you're close to getting trouble with other components.
> 
>> Any thoughts to modifying haproxy to limit allocating such large buffer areas
>> for only certain frontends, instead of being only a global tunable?  Tuning
>> haproxy globally to solve a problem with processing only one frontend/backend
>> seems rough.
> 
> That's planned but not with high priority. In fact I'd like to have 3 buffer
> sizes, the default ones, the small ones (for idle connections) and the large
> ones (for large requests and for high speed transfers). But we need to go with
> 1.6 first which goal will be to let a task sleep waiting for resources.
> 
> Regards,
> Willy
> 

Kevin Lange
kevin.m.la...@nasa.gov
kla...@raytheon.com
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.nasa.gov  | https://www.raytheon.com



smime.p7s
Description: S/MIME cryptographic signature

Re: issues with very long URLs.

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Hi,
Got a few cycles to do some testing/verification.  
Sure enough when we pass a long URL through haproxy/lighttpd, we see this in 
the haproxy log:

Sep 19 22:45:14 127.0.0.1 haproxy[4195]: 172.28.xx.xx:55500 
[19/Sep/2012:22:45:14.428] kernel_partner_ft kernel_partner_ft/ 
-1/-1/-1/-1/0 400 187 - - PR-- 0/0/0/0/0 0/0 ""

From your message, you say that haproxy will be marked with PR if blocked by 
haproxy, so this looks like haproxy is confirmed to be the cause of our 
problem.  Correct?

We're going to draft and test a  configuration that increases maxrewrite, and 
perhaps bufsize.

Any thoughts to modifying haproxy to limit allocating such large buffer areas 
for only certain frontends, instead of being only a global tunable?  Tuning 
haproxy globally to solve a problem with processing only one frontend/backend 
seems rough.
- Kevin


On Sep 10, 2012, at 4:11 PM, Willy Tarreau wrote:

> Hi,
> 
> On Mon, Sep 10, 2012 at 08:20:02PM +0200, Baptiste wrote:
>> On Mon, Sep 10, 2012 at 7:24 PM, Lange, Kevin M. (GSFC-423.0)[RAYTHEON
>> COMPANY]  wrote:
>>> Hi,
>>> Our public-facing service provides a REST api to search for products
>>> (geospatial science data), which requires in many cases very long URLs to
>>> craft the search.  We seem to be hitting a limit of around 8K before we
>>> receive a 400 Bad request from lighttpd. We're trying to determine if
>>> lighttpd is causing this, or haproxy.  We have a dev/test stack
>>> (lighttpd/haproxy on Linux) and an OPS stack of the same.  Our OPS stack we
>>> thought we had test results of the URL length maximum, but after we upgraded
>>> haproxy to the latest on our OPS stack, we noticed that people began
>>> complaining about the URL length issue (maximum 8k).  Is there a
>>> configurable item in haproxy which would limit a URL length to ~8k?
>>> Suggestions from searches show that tune.bufsize might control this.  We'd
>>> like to offer our customers a 10k length for REST api calls.
>>> - Kevin
>>> 
>>> Kevin Lange
>>> kevin.m.la...@nasa.gov
>>> kla...@raytheon.com
>>> W: +1 (301) 851-8450
>>> Raytheon  | NASA  | ECS Evolution Development Program
>>> https://www.echo.nasa.gov  | https://www.raytheon.com
>>> 
>> 
>> Hi,
>> 
>> The 400 error may be issued by HAProxy.
>> To know it, you should enable logs and share them here, so we'll know
>> the real reason of the error.
> 
> I confirm that, a request blocked by haproxy will be marked "PR" in the
> logs while it will be normal "--" if it's emitted by the server.
> 
>> I would have say like Richard: give a try to a bigger tune.bufsize .
> 
> I'd say that if you need 10k, set your bufsize to the default 16k and
> set tune.maxrewrite to 1k, you'll end up with 15kB for a complete
> request which will be enough to store the large URL.
> 
> Regards,
> Willy
> 

Kevin Lange
kevin.m.la...@nasa.gov
kla...@raytheon.com
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.nasa.gov  | https://www.raytheon.com



smime.p7s
Description: S/MIME cryptographic signature

Re: issues with very long URLs.

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

The documentation recommends against changing the default value for concern of 
memory consumption.  I will assume that if we have sufficient memory, this 
should not be an issue.  We would have to test in our DEV/TEST stack first, 
which we can do this wednesday.

Qs: 
If we're hitting a limit of ~8k, why does the 16k tune.bufsize pose a problem 
for us?

If this parameter does govern maximum URI/URL length, then I would certainly 
think clearer documentation of this parameters effect, as REST APIs are 
becoming commonplace, with lengthy arguments passed.  In this case, long 
cookies/session-ids are not as much an issue as REST is consuming the bulk of 
the URL.  

Comments:
It would be nice to have a more aptly named parameter/construct that allocates 
a larger buffer only for particular front-ends, not a global buffer for every 
connection to every service.  My point here is that HAPROXY should be more 
capable when handling RESTful services (long GETs/PUTs, etc).


tune.bufsize 
  Sets the buffer size to this size (in bytes). Lower values allow more
  sessions to coexist in the same amount of RAM, and higher values allow some
  applications with very large cookies to work. The default value is 16384 and
  can be changed at build time. It is strongly recommended not to change this
  from the default value, as very low values will break some services such as
  statistics, and values larger than default size will increase memory usage,
  possibly causing the system to run out of memory. At least the global maxconn
  parameter should be decreased by the same factor as this one is increased.
On Sep 10, 2012, at 1:41 PM, Richard Stanford wrote:

> We regularly get individual REST GET requests significantly over that length; 
> the only tuning parameter we've done in that regard is:
> 
> tune.bufsize 128000
> 
> I don't actually recall if this was mandatory to address the issue, but I'm 
> thinking that it was.
> 
> -Richard
> 
> Richard Stanford
> CTO | KIMBIA
> 
> 512-474-4447 x777
> 
> On Sep 10, 2012, at 12:24 PM, Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY] 
> wrote:
> 
>> Hi, 
>> Our public-facing service provides a REST api to search for products 
>> (geospatial science data), which requires in many cases very long URLs to 
>> craft the search.  We seem to be hitting a limit of around 8K before we 
>> receive a 400 Bad request from lighttpd. We're trying to determine if 
>> lighttpd is causing this, or haproxy.  We have a dev/test stack 
>> (lighttpd/haproxy on Linux) and an OPS stack of the same.  Our OPS stack we 
>> thought we had test results of the URL length maximum, but after we upgraded 
>> haproxy to the latest on our OPS stack, we noticed that people began 
>> complaining about the URL length issue (maximum 8k).  Is there a 
>> configurable item in haproxy which would limit a URL length to ~8k?  
>> Suggestions from searches show that tune.bufsize might control this.  We'd 
>> like to offer our customers a 10k length for REST api calls.
>> - Kevin
>> 
>> Kevin Lange
>> kevin.m.la...@nasa.gov
>> kla...@raytheon.com
>> W: +1 (301) 851-8450
>> Raytheon  | NASA  | ECS Evolution Development Program
>> https://www.echo.nasa.gov  | https://www.raytheon.com
>> 
> 

Kevin Lange
kevin.m.la...@nasa.gov
kla...@raytheon.com
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.nasa.gov  | https://www.raytheon.com



smime.p7s
Description: S/MIME cryptographic signature

issues with very long URLs.

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Hi, 
Our public-facing service provides a REST api to search for products 
(geospatial science data), which requires in many cases very long URLs to craft 
the search.  We seem to be hitting a limit of around 8K before we receive a 400 
Bad request from lighttpd. We're trying to determine if lighttpd is causing 
this, or haproxy.  We have a dev/test stack (lighttpd/haproxy on Linux) and an 
OPS stack of the same.  Our OPS stack we thought we had test results of the URL 
length maximum, but after we upgraded haproxy to the latest on our OPS stack, 
we noticed that people began complaining about the URL length issue (maximum 
8k).  Is there a configurable item in haproxy which would limit a URL length to 
~8k?  Suggestions from searches show that tune.bufsize might control this.  
We'd like to offer our customers a 10k length for REST api calls.
- Kevin

Kevin Lange
kevin.m.la...@nasa.gov
kla...@raytheon.com
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.nasa.gov  | https://www.raytheon.com



smime.p7s
Description: S/MIME cryptographic signature

How many Twitter Followers do you want? (Sep 5-8)

[Leo M. Gibson]

Hello,

We are sending this e-mail to let our users know that we have launched a new 
Twitter promotion  
campaign where you can buy (up to 25,000) Twitter followers; delivered within 
days.

Simply visit:
http://www.FollowersForTwitter.com

This campaign has a priority to our former users who receive this email and
will not be offered to public until Sep 8th, 2012.

-- 
Best Regards,
Leo | FollowersForTwitter (FFT)
8 Chingford Road, Walthamstow,
E17 4PJ London, U.K.
1.800.750.8374
-- 

You may automatically unsubscribe from this list at any time by visiting the 
following URL:
 
http://notifications.followersfortwitter.com/mail.cgi/u/twa/haproxy/formilux.org/

If the above URL is inoperable, make sure that you have copied the entire 
address. 
Some mail readers will wrap a long URL and thus break this automatic 
unsubscribe mechanism.=

How many Twitter Followers do you want? (Aug 28-31)

[Leo M. Gibson]

Hello,

We are sending this e-mail to let our users know that we have launched a new 
Twitter promotion  
campaign where you can buy (up to 25,000) Twitter followers; delivered within 
days.

Simply visit:
http://www.FollowersForTwitter.com

This campaign has a priority to our former users who receive this email and
will not be offered to public until Aug 31st, 2012.

Thank you
Leo, on behalf of;
---
FollowersForTwitter.com
Toll FREE (U.S.): 1(800)750.8374
---
© 2010 - 2012 FFT 
8 Chingford Road, Walthamstow, 
E17 4PJ London, U.K.
---

You may automatically unsubscribe from this list at any time by visiting the 
following URL:

http://notifications.followersfortwitter.com/mail.cgi/u/twb/haproxy/formilux.org/

If the above URL is inoperable, make sure that you have copied the entire 
address. 
Some mail readers will wrap a long URL and thus break this automatic 
unsubscribe mechanism.=

Re: Problems with layer7 check timeout

[Kevin M Lange]

I've been monitoring our service availability check (http head of a 
resource that truly provides availability status of the application).  
Under normal circumstances, the check takes 2-3 seconds.  We found 
periods of time where the application would take 15+seconds and fail (I 
did not capture HTTP code, but I'm pretty sure it was a 500 series from 
what I've been looking through).  These failure periods match the times 
where haproxy was indicating timeouts of 1002ms.  So, it looks like 
haproxy is doing its job.  Is this then a bug in the logging of the 
timeout value (reporting 1002ms vs 15000+ms)?


We haven't had any problems since 25 May, but we're keeping watch.

- Kevin

On 5/25/12 11:18 AM, Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY] wrote:

Willy,
I'll try the patch, but not until next week because of the holiday 
weekend.  I don't want to make a significant change that I would have 
to support over the long weekend.
I'm capturing tcpdump between SLB and the three backends.  I'd like to 
have a capture during an "outage".  I expect to see something today, 
and I'll send to you.

- Kevin


On May 25, 2012, at 2:12 AM, Willy Tarreau wrote:


Hi again Kevin,

Well, I suspect that there might be a corner case with the bug I fixed
which might have caused what you observed.

The "timeout connect" is computed from the last expire date. Since
"timeout check" was added upon connection establishment but the task
was woken too late, then that after a first check failure reported
too late, you can have the next check timeout shortened.

It's still unclear to me how it is possible that the check timeout is
reported this small, considering that it's updated once the connect
succeeds. But performing computations in the past is never a good way
to have something reliable.

Could you please apply the attached fix for the bug I mentionned in
previous mail, to see if the issue is still present ? After all, I
would not be totally surprized if this bug has nasty side effects
like this.

Thanks,
Willy

<0001-BUG-MINOR-checks-expire-on-timeout.check-if-smaller-.patch>


Kevin Lange
kevin.m.la...@nasa.gov <mailto:kevin.m.la...@nasa.gov>
kla...@raytheon.com <mailto:kla...@raytheon.com>
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.com  | https://www.raytheon.com

Re: Problems with layer7 check timeout

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Willy,
I'll try the patch, but not until next week because of the holiday weekend.  I 
don't want to make a significant change that I would have to support over the 
long weekend.
I'm capturing tcpdump between SLB and the three backends.  I'd like to have a 
capture during an "outage".  I expect to see something today, and I'll send to 
you.
- Kevin


On May 25, 2012, at 2:12 AM, Willy Tarreau wrote:

> Hi again Kevin,
> 
> Well, I suspect that there might be a corner case with the bug I fixed
> which might have caused what you observed.
> 
> The "timeout connect" is computed from the last expire date. Since
> "timeout check" was added upon connection establishment but the task
> was woken too late, then that after a first check failure reported
> too late, you can have the next check timeout shortened.
> 
> It's still unclear to me how it is possible that the check timeout is
> reported this small, considering that it's updated once the connect
> succeeds. But performing computations in the past is never a good way
> to have something reliable.
> 
> Could you please apply the attached fix for the bug I mentionned in
> previous mail, to see if the issue is still present ? After all, I
> would not be totally surprized if this bug has nasty side effects
> like this.
> 
> Thanks,
> Willy
> 
> <0001-BUG-MINOR-checks-expire-on-timeout.check-if-smaller-.patch>

Kevin Lange
kevin.m.la...@nasa.gov
kla...@raytheon.com
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.com  | https://www.raytheon.com



smime.p7s
Description: S/MIME cryptographic signature

Re: Problems with layer7 check timeout

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Willy,
I'll try the patch, but not until next week because of the holiday weekend.  I 
don't want to make a significant change that I would have to support over the 
long weekend.
I'm capturing tcpdump between SLB and the three backends.  I'd like to have a 
capture during an "outage".  I expect to see something today, and I'll send to 
you.
- Kevin


On May 25, 2012, at 2:12 AM, Willy Tarreau wrote:

> Hi again Kevin,
> 
> Well, I suspect that there might be a corner case with the bug I fixed
> which might have caused what you observed.
> 
> The "timeout connect" is computed from the last expire date. Since
> "timeout check" was added upon connection establishment but the task
> was woken too late, then that after a first check failure reported
> too late, you can have the next check timeout shortened.
> 
> It's still unclear to me how it is possible that the check timeout is
> reported this small, considering that it's updated once the connect
> succeeds. But performing computations in the past is never a good way
> to have something reliable.
> 
> Could you please apply the attached fix for the bug I mentionned in
> previous mail, to see if the issue is still present ? After all, I
> would not be totally surprized if this bug has nasty side effects
> like this.
> 
> Thanks,
> Willy
> 
> <0001-BUG-MINOR-checks-expire-on-timeout.check-if-smaller-.patch>

Kevin Lange
kevin.m.la...@nasa.gov
kla...@raytheon.com
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.com  | https://www.raytheon.com



smime.p7s
Description: S/MIME cryptographic signature

Re: Problems with layer7 check timeout

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Willy,
I'll try the patch, but not until next week because of the holiday weekend.  I 
don't want to make a significant change that I would have to support over the 
long weekend.
I'm capturing tcpdump between SLB and the three backends.  I'd like to have a 
capture during an "outage".  I expect to see something today, and I'll send to 
you.
- Kevin


On May 25, 2012, at 2:12 AM, Willy Tarreau wrote:

> Hi again Kevin,
> 
> Well, I suspect that there might be a corner case with the bug I fixed
> which might have caused what you observed.
> 
> The "timeout connect" is computed from the last expire date. Since
> "timeout check" was added upon connection establishment but the task
> was woken too late, then that after a first check failure reported
> too late, you can have the next check timeout shortened.
> 
> It's still unclear to me how it is possible that the check timeout is
> reported this small, considering that it's updated once the connect
> succeeds. But performing computations in the past is never a good way
> to have something reliable.
> 
> Could you please apply the attached fix for the bug I mentionned in
> previous mail, to see if the issue is still present ? After all, I
> would not be totally surprized if this bug has nasty side effects
> like this.
> 
> Thanks,
> Willy
> 
> <0001-BUG-MINOR-checks-expire-on-timeout.check-if-smaller-.patch>

Kevin Lange
kevin.m.la...@nasa.gov
kla...@raytheon.com
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.com  | https://www.raytheon.com



smime.p7s
Description: S/MIME cryptographic signature

Re: Problems with layer7 check timeout

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Monsieur Tarreau,

Actually, we are seeing frontend service availability flapping. This morning 
particularly.  Missing from my snippet is the logic for an unplanned outage 
landing page, that our customers were seeing this morning, so it haproxy truly 
is "timing out" and marking each backend as down until there are no backend 
servers available, throwing up the unplanned outage landing page.

I'll send more logs and details when I analyze later.

Regards,
Kevin Lange


Kevin M Lange
Mission Operations and Services
NASA EOSDIS Evolution and Development
Intelligence and Information Systems
Raytheon Company

+1 (301) 851-8450 (office)
+1 (301) 807-2457 (cell)
kevin.m.la...@nasa.gov
kla...@raytheon.com

5700 Rivertech Court
Riverdale, Maryland 20737

- Reply message -
From: "Willy Tarreau" 
Date: Thu, May 24, 2012 5:18 pm
Subject: Problems with layer7 check timeout
To: "Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]" 
Cc: "haproxy@formilux.org" 

Hi Kevin,

On Thu, May 24, 2012 at 04:04:03PM -0500, Lange, Kevin M. (GSFC-423.0)[RAYTHEON 
COMPANY] wrote:
> Hi,
> We're having odd behavior (apparently have always but didn't realize it), 
> where our backend httpchks "time out":
>
> May 24 04:03:33 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
> May 24 04:41:55 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
> servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
> May 24 08:38:10 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
> May 24 08:53:37 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
> May 24 09:32:20 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
> servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
> May 24 09:35:01 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops3 is 
> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
> May 24 09:41:37 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
> May 24 09:56:41 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops3 is 
> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
> May 24 10:01:45 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>
>
> We've been playing with the timeout values, and we don't know what is 
> controlling the "Layer7 timeout, check duration: 1002ms".  The backend 
> service availability check (by hand) typically takes 2-3 seconds on average.
> Here is the relevant haproxy setup.
>
> #-
> # Global settings
> #-
> global
> log-send-hostname opsslb1
> log 127.0.0.1 local1 info
> #chroot  /var/lib/haproxy
> pidfile /var/run/haproxy.pid
> maxconn 1024
> userhaproxy
> group   haproxy
> daemon
>
> #-
> # common defaults that all the 'listen' and 'backend' sections will
> # use if not designated in their block
> #-
> defaults
> modehttp
> log global
> option  dontlognull
> option  httpclose
> option  httplog
> option  forwardfor
> option  redispatch
> timeout connect 500 # default 10 second time out if a backend is not found
> timeout client 5
> timeout server 360
> maxconn 6
> retries 3
>
> frontend webapp_ops_ft
>
> bind 10.0.40.209:80
> default_backend webapp_ops_bk
>
> bac

Re: Problems with layer7 check timeout

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

I've already put an upgrade to haproxy in place.


Kevin M Lange
Mission Operations and Services
NASA EOSDIS Evolution and Development
Intelligence and Information Systems
Raytheon Company

+1 (301) 851-8450 (office)
+1 (301) 807-2457 (cell)
kevin.m.la...@nasa.gov
kla...@raytheon.com

5700 Rivertech Court
Riverdale, Maryland 20737

- Reply message -
From: "Willy Tarreau" 
Date: Thu, May 24, 2012 5:59 pm
Subject: Problems with layer7 check timeout
To: "Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]" 
Cc: "haproxy@formilux.org" 

On Thu, May 24, 2012 at 04:31:39PM -0500, Lange, Kevin M. (GSFC-423.0)[RAYTHEON 
COMPANY] wrote:
> I thought it was a bug in the reporting, considering we've played with 
> numerous values for the various timeouts as an experiment, but wanted your 
> thoughts.
> This is v1.4.15.
>
>  [root@opsslb1 log]# haproxy -v
> HA-Proxy version 1.4.15 2011/04/08
> Copyright 2000-2010 Willy Tarreau 

OK, I'll try to reproduce. There have been a number of fixes since 1.4.15
BTW, but none of them look like what you observe. Still it would be
reasonable to consider an upgrade to 1.4.21.

Regards,
Willy

Re: Problems with layer7 check timeout

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Err...more precisely...
HA-Proxy version 1.4.15 2011/04/08
Copyright 2000-2010 Willy Tarreau 

Build options :
  TARGET  = linux26
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing
  OPTIONS = USE_REGPARM=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Available polling systems :
 sepoll : pref=400,  test result OK
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.

On May 24, 2012, at 5:31 PM, Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY] 
wrote:

> 
> 
> I thought it was a bug in the reporting, considering we've played with 
> numerous values for the various timeouts as an experiment, but wanted your 
> thoughts.
> This is v1.4.15.
> 
> [root@opsslb1 log]# haproxy -v
> HA-Proxy version 1.4.15 2011/04/08
> Copyright 2000-2010 Willy Tarreau 
> 
> On May 24, 2012, at 5:17 PM, Willy Tarreau wrote:
> 
>> Hi Kevin,
>> 
>> On Thu, May 24, 2012 at 04:04:03PM -0500, Lange, Kevin M. 
>> (GSFC-423.0)[RAYTHEON COMPANY] wrote:
>>> Hi,
>>> We're having odd behavior (apparently have always but didn't realize it), 
>>> where our backend httpchks "time out":
>>> 
>>> May 24 04:03:33 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
>>> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
>>> May 24 04:41:55 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>>> servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
>>> May 24 08:38:10 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
>>> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
>>> May 24 08:53:37 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>>> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>>> May 24 09:32:20 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
>>> servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
>>> May 24 09:35:01 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops3 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>>> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>>> May 24 09:41:37 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>>> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
>>> May 24 09:56:41 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops3 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
>>> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>>> May 24 10:01:45 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
>>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>>> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>>> 
>>> 
>>> We've been playing with the timeout values, and we don't know what is 
>>> controlling the "Layer7 timeout, check duration: 1002ms".  The backend 
>>> service availability check (by hand) typically takes 2-3 seconds on average.
>>> Here is the relevant haproxy setup.
>>> 
>>> #-
>>> # Global settings
>>> #-
>>> global
>>>   log-send-hostname opsslb1
>>>   log 127.0.0.1 local1 info
>>> #chroot  /var/lib/haproxy
>>>   pidfile /var/run/haproxy.pid
>>>   maxconn 1024
>>>   userhaproxy
>>>   group   haproxy
>>>   daemon
>>> 
>>> #-
>>> # common defaults that all the 'listen' and 'backend' sections will
>>> # use if not designated in their block
>>> #

Re: Problems with layer7 check timeout

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

I thought it was a bug in the reporting, considering we've played with numerous 
values for the various timeouts as an experiment, but wanted your thoughts.
This is v1.4.15.

 [root@opsslb1 log]# haproxy -v
HA-Proxy version 1.4.15 2011/04/08
Copyright 2000-2010 Willy Tarreau 

On May 24, 2012, at 5:17 PM, Willy Tarreau wrote:

> Hi Kevin,
> 
> On Thu, May 24, 2012 at 04:04:03PM -0500, Lange, Kevin M. 
> (GSFC-423.0)[RAYTHEON COMPANY] wrote:
>> Hi,
>> We're having odd behavior (apparently have always but didn't realize it), 
>> where our backend httpchks "time out":
>> 
>> May 24 04:03:33 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
>> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
>> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
>> May 24 04:41:55 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>> servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
>> May 24 08:38:10 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
>> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
>> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
>> May 24 08:53:37 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>> May 24 09:32:20 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
>> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
>> servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
>> May 24 09:35:01 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops3 is 
>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>> May 24 09:41:37 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>> servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
>> May 24 09:56:41 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops3 is 
>> DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
>> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>> May 24 10:01:45 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
>> DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
>> servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
>> 
>> 
>> We've been playing with the timeout values, and we don't know what is 
>> controlling the "Layer7 timeout, check duration: 1002ms".  The backend 
>> service availability check (by hand) typically takes 2-3 seconds on average.
>> Here is the relevant haproxy setup.
>> 
>> #-
>> # Global settings
>> #-
>> global
>>log-send-hostname opsslb1
>>log 127.0.0.1 local1 info
>> #chroot  /var/lib/haproxy
>>pidfile /var/run/haproxy.pid
>>maxconn 1024
>>userhaproxy
>>group   haproxy
>>daemon
>> 
>> #-
>> # common defaults that all the 'listen' and 'backend' sections will
>> # use if not designated in their block
>> #-
>> defaults
>>modehttp
>>log global
>>option  dontlognull
>>option  httpclose
>>option  httplog
>>option  forwardfor
>>option  redispatch
>>timeout connect 500 # default 10 second time out if a backend is not found
>>timeout client 5
>>timeout server 360
>>maxconn 6
>>retries 3
>> 
>> frontend webapp_ops_ft
>> 
>>bind 10.0.40.209:80
>>default_backend webapp_ops_bk
>> 
>> backend webapp_ops_bk
>>balance roundrobin
>>option httpchk HEAD /app/availability
>>reqrep ^Host:.* Host:\ webapp.example.com
>>server webapp_ops1 opsapp1.ops.example.com:41000 check inter 3
>>server webapp_ops2 opsapp2.ops.example.com:41000 check inter 3
>>server webapp_ops3 opsapp3

Problems with layer7 check timeout

[Lange, Kevin M. (GSFC-423.0)[RAYTHEON COMPANY]]

Hi,
We're having odd behavior (apparently have always but didn't realize it), where 
our backend httpchks "time out":

May 24 04:03:33 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
May 24 04:41:55 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
May 24 08:38:10 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
May 24 08:53:37 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 24 09:32:20 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
May 24 09:35:01 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops3 is 
DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 24 09:41:37 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops2 is 
DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
May 24 09:56:41 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops3 is 
DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup 
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 24 10:01:45 opsslb1 haproxy[4594]: Server webapp_ops_bk/webapp_ops1 is 
DOWN, reason: Layer7 timeout, check duration: 1001ms. 0 active and 0 backup 
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.


We've been playing with the timeout values, and we don't know what is 
controlling the "Layer7 timeout, check duration: 1002ms".  The backend service 
availability check (by hand) typically takes 2-3 seconds on average.
Here is the relevant haproxy setup.

#-
# Global settings
#-
global
log-send-hostname opsslb1
log 127.0.0.1 local1 info
#chroot  /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 1024
userhaproxy
group   haproxy
daemon

#-
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#-
defaults
modehttp
log global
option  dontlognull
option  httpclose
option  httplog
option  forwardfor
option  redispatch
timeout connect 500 # default 10 second time out if a backend is not found
timeout client 5
timeout server 360
maxconn 6
retries 3

frontend webapp_ops_ft

bind 10.0.40.209:80
default_backend webapp_ops_bk

backend webapp_ops_bk
balance roundrobin
option httpchk HEAD /app/availability
reqrep ^Host:.* Host:\ webapp.example.com
server webapp_ops1 opsapp1.ops.example.com:41000 check inter 3
server webapp_ops2 opsapp2.ops.example.com:41000 check inter 3
server webapp_ops3 opsapp3.ops.example.com:41000 check inter 3
timeout check 15000
timeout connect 15000

Kevin Lange
kevin.m.la...@nasa.gov
kla...@raytheon.com
W: +1 (301) 851-8450
Raytheon  | NASA  | ECS Evolution Development Program
https://www.echo.com  | https://www.raytheon.com



smime.p7s
Description: S/MIME cryptographic signature

total session number in the Statistics Report page

[Alon M]

Hi , 
i was wondering is some one could explain why there is always a small difference
between the 
total session on our single listener called - "pm" -  600885
to the back end server of that listener  called -"localPM"600845

below is the config :

global
log 127.0.0.1 local0
chroot  /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
stats socket/tmp/haproxy level admin
userhaproxy
group   haproxy
daemon
defaults
modehttp
log global
timeout connect 1000 
timeout client 2
timeout server 2000
maxconn 6
retries 3
option httpclose # Disable Keepalive
option  httplog
listen pm 192.168.181.43:8080
timeout http-request 3000
stats uri /haproxy
balance roundrobin # Load Balancing algorithm
option forwardfor header pm-forwarded 
server localPM 192.168.181.43:9090



thanks 

Alon 

Re: lots and lots of request erros - cR 408

[Alon M]

Baptiste  writes:

> 
> hey,
> 
> When a packet is lost, there is a retransmit 3s later (TCP protocol).
> Which is under the 10s of the timeout currently configured.
> So this can't be the reason of too many 408.
> 
> Either you're under attack (somebody trying to take all the resources
> of your website using slowloris-like scripts).
> Or you have a high rate of packet loss.
> 
> Cheers


thanks for the quick replay . 

since we only route traffic to 1 out of our 10 servers cluster , i was wondering
why there are no exception / errors on the tomcat servers that get their traffic
directly from the layer 4 load balancing , when i look at the tomcat access logs
i dont see any 408 error requests. 

is there a recommended  sniffer tool / unix command i can use to check if those
408 errors also occur on server that do not get their traffic from haproxy ? 

that way i can be sure haproxy has just expose a problem that was always there
and we were not a ware of it . 


thanks in advanced .

Alon 

Re: lots and lots of request erros - cR 408

[Alon M]

sorry forgot to add our config file :

global
log 127.0.0.1 local0
chroot  /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
stats socket/tmp/haproxy level admin
userhaproxy
group   haproxy
daemon

defaults
modehttp
log global
timeout connect 1000 # default 10 second time out if a backend is not found
timeout client 2
timeout server 2000
maxconn 6
retries 3
option httpclose # Disable Keepalive
option  httplog
listen pm 192.168.181.43:8080
#stats uri /haproxy
timeout http-request 1
stats uri /haproxy
balance roundrobin # Load Balancing algorithm
option forwardfor header pm-forwarded # This sets X-Forwarded-For
server localPM 192.168.181.43:9090

thanks again. 

lots and lots of request erros - cR 408

[Alon M]

Hello , 
we started to use haproxy on our production environment.
the request are coming to a layer 4 load balancing - ldirector which forwards
them to a haproxy which forwards it to a tomcat on the same server .

our connection handling inside tomcat is fast - ~10 mili seconds, and we want to
delegate the connection handling with the WAN to haproxy instead of a tomcat
thread thus getting batter throughput .

we are getting a lot of http request error, about 10% of the requests are timed
out on the client side, increasing the "timeout http-request", does not help ,
the connection just terminates after the max time has passed.

echo "show errors"  | socat unix-connect:/tmp/haproxy stdio is clear .

below are a few error records from the log : 

Apr  4 14:01:38 localhost.localdomain haproxy[22451]: 95.206.61.53:62766
[04/Apr/2012:14:01:28.855] pm pm/ -1/-1/-1/-1/10001 408 212 - - cR--
13/13/2/0/0 0/0 ""
Apr  4 14:01:39 localhost.localdomain haproxy[22451]: 67.197.4.199:55973
[04/Apr/2012:14:01:29.025] pm pm/ -1/-1/-1/-1/1 408 212 - - cR--
12/12/3/0/0 0/0 ""

here are some details - 

Linux tapp4.ny 2.6.18-274.3.1.el5 #1 SMP Tue Sep 6 20:13:52 EDT 2011 x86_64
x86_64 x86_64 GNU/Linux


[root@tapp4 haproxy-1.4.20]# more /etc/issue
CentOS release 5.5 (Final)
Kernel \r on an \m


[root@tapp4 haproxy-1.4.20]# /usr/sbin/haproxy  -vv
HA-Proxy version 1.4.20 2012/03/10
Copyright 2000-2012 Willy Tarreau 

Build options :
  TARGET  = linux26
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing
  OPTIONS =

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Available polling systems :
 sepoll : pref=400,  test result OK
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.

any help would be much appreciated . 

Alon 

chaining front ends?

[M B]

I have a backend with lots of frontends.

I want one of those frontends to be a listener on port 80.

The port 80 listener needs to route traffic based on the Host http request
header.

After traffic is split based on the Host header, I want to do further header
manipulation and rate limiting specific to each portion of the split traffic
before sending to the big shared backend.

Is there any way to do this?  In the past I tried putting the port of a
frontend in the server section of a backend to chain things, but I observed
problems with -sf graceful port handover functionality.

-Matt

how to associate front and back ends?

[M B]

I have 3 back end environments.  qa (q), uat (u), and live (l).  I want to 
provide many listeners on different ports with different connection limits that 
I will point different clients at.

I used to do this by using "listen" everywhere.  I had listen statements for 
the q, u and t server pools.  I also had listen statements for each client 
listener that did connection limiting, and pointed at localhost:port to get to 
the actual load balancing listeners.

I think the preferred way to do this would be to define 3 backends, and many 
front ends.  

How do I associate a front end with a back end?  "use_backend" appears to want 
a conditional.  I just want to say, "for this frontend, always use this 
backend".

-Matt
  
_
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1