Re: strange behavior with version 1.4.22
Hey Lukas, Indeed this is an AWS instance, its a custom AMI, the interesting thing about this is that the image has been cloned many times for different deployments and none of the other ones have experienced this, some are actually running with a ton of traffic through them. Today I prepared a fresh instance (2.6.18-308.16.1.el5.centos.plusxen #1 SMP Tue Oct 2 23:25:27 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux) and voila, problem solved, replicating this image to the rest of the environments. I'm curious as to why this happen on those boxes though... Thank you for all your help, can't wait for the next release willy! On Mon, Mar 11, 2013 at 8:02 PM, Lukas Tribus luky...@hotmail.com wrote: Linux ip-x.x.x.x 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 x86_64 x86_64 GNU/Linux [...] CentOS release 5.4 (Final) This seems to be a weird config, you have CentOS 5.4 (released 2009 with a 2.6.18 kernel), but you are running a xenified Fedora kernel compiled in 2008. Is this some kind of Amazon instance, like EC2 or something? Did you cloned this from a community provided AMI, like @ [1]? A fresh, updated and vendor supported (!) OS would certainly not hurt. Just think of what kernel exploits you have in a kernel from 5 years ago, a part from the problems you are seeing with haproxy. [1] https://aws.amazon.com/amis/64bit-fedora-8-xenified-kernel-2-6-21-7-2-fc8xen
Re: strange behavior with version 1.4.22
Absolutley, here is the config: global log 127.0.0.1 local0 crit pidfile /var/run/haproxy.pid stats socket /var/run/haproxy.stat mode 666 nbproc 2 maxconn 65000 tune.maxrewrite 1024 tune.bufsize 32768 userhaproxy group haproxy daemon defaults modehttp log global option dontlognull option httplog option http-server-close #option forwardfor except 127.0.0.0/8 option forwardfor header X-Real-IP option redispatch stats enable stats auth change:me stats uri /stav1 timeout connect 5000 # default 5 second time out if a backend is not found timeout client 300s timeout server 300s #timeout http-request10s #timeout queue 1m #timeout http-keep-alive 10s timeout check 5s maxconn 65000 retries 3 frontend sinatra *:5000 default_backendsinatra_backend backend sinatra_backend mode http balance roundrobin option httpchk HEAD /ha.txt HTTP/1.0 On Mon, Mar 11, 2013 at 4:36 PM, Thomas Heil h...@terminal-consulting.dewrote: Hi, On 11.03.2013 21:23, Saul Waizer wrote: Hello List, I am experiencing some erratic behavior on 2 fresh installs (V 1.4.22) that I've never seen before. Basically the haproxy is taking 100% of cpu, there is nothing running on the box and there is no traffic going through it, yet the load average is 8 and and cpu is constantly hitting 100%. I have the same installation/configuration running on 20+ servers and never experienced this before, nothing on the logs either...here is the relevant information: Crazy. Could we take a look at your config? I had a similar problem years ago, when using the option transparent. HA-Proxy version 1.4.22 2012/08/09 Copyright 2000-2012 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. Linux ip-x.x.x.x 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 x86_64 x86_64 GNU/Linux CentOS release 5.4 (Final) Any suggestions are greatly appreciated. Thank You cheers thomas
Re: strange behavior with version 1.4.22
Thanks Willy,
Here is the strace (constantly showing the same output very fast!) I will
try the snapshot and post my findings, let me know your thoughts on the
strace
strace -tt -p 13659
Process 13659 attached - interrupt to quit
17:56:26.908085 epoll_wait(0, {{EPOLLHUP, {u32=4, u64=4}}}, 7, 1000) = 1
17:56:26.908300 gettimeofday({1363038986, 908332}, NULL) = 0
17:56:26.908378 accept(4, 0x7fff0e2118c0, [5854207880927903872]) = -1
EINVAL (Invalid argument)
17:56:26.908527 epoll_wait(0, {{EPOLLHUP, {u32=4, u64=4}}}, 7, 1000) = 1
17:56:26.908596 gettimeofday({1363038986, 908623}, NULL) = 0
17:56:26.908657 accept(4, 0x7fff0e2118c0, [5854207880927903872]) = -1
EINVAL (Invalid argument)
17:56:26.908732 epoll_wait(0, {{EPOLLHUP, {u32=4, u64=4}}}, 7, 1000) = 1
17:56:26.908794 gettimeofday({1363038986, 908821}, NULL) = 0
17:56:26.908855 accept(4, 0x7fff0e2118c0, [5854207880927903872]) = -1
EINVAL (Invalid argument)
17:56:26.908928 epoll_wait(0, {{EPOLLHUP, {u32=4, u64=4}}}, 7, 1000) = 1
17:56:26.908990 gettimeofday({1363038986, 909017}, NULL) = 0
17:56:26.909051 accept(4, 0x7fff0e2118c0, [5854207880927903872]) = -1
EINVAL (Invalid argument)
17:56:26.909122 epoll_wait(0, {{EPOLLHUP, {u32=4, u64=4}}}, 7, 1000) = 1
17:56:26.909184 gettimeofday({1363038986, 909210}, NULL) = 0
17:56:26.909244 accept(4, 0x7fff0e2118c0, [5854207880927903872]) = -1
EINVAL (Invalid argument)
17:56:26.909317 epoll_wait(0, {{EPOLLHUP, {u32=4, u64=4}}}, 7, 1000) = 1
17:56:26.909378 gettimeofday({1363038986, 909404}, NULL) = 0
17:56:26.909443 accept(4, 0x7fff0e2118c0, [5854207880927903872]) = -1
EINVAL (Invalid argument)
17:56:26.909517 epoll_wait(0, {{EPOLLHUP, {u32=4, u64=4}}}, 7, 1000) = 1
17:56:26.909579 gettimeofday({1363038986, 909605}, NULL) = 0
17:56:26.909639 accept(4, 0x7fff0e2118c0, [5854207880927903872]) = -1
EINVAL (Invalid argument)
17:56:26.909711 epoll_wait(0, {{EPOLLHUP, {u32=4, u64=4}}}, 7, 1000) = 1
On Mon, Mar 11, 2013 at 5:53 PM, Willy Tarreau w...@1wt.eu wrote:
Hi Saul,
On Mon, Mar 11, 2013 at 05:29:10PM -0400, Saul Waizer wrote:
Absolutley, here is the config:
global
log 127.0.0.1 local0 crit
pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy.stat mode 666
nbproc 2
maxconn 65000
tune.maxrewrite 1024
tune.bufsize 32768
userhaproxy
group haproxy
daemon
defaults
modehttp
log global
option dontlognull
option httplog
option http-server-close
#option forwardfor except 127.0.0.0/8
option forwardfor header X-Real-IP
option redispatch
stats enable
stats auth change:me
stats uri /stav1
timeout connect 5000 # default 5 second time out if a backend is not
found
timeout client 300s
timeout server 300s
#timeout http-request10s
#timeout queue 1m
#timeout http-keep-alive 10s
timeout check 5s
maxconn 65000
retries 3
frontend sinatra *:5000
default_backendsinatra_backend
backend sinatra_backend
mode http
balance roundrobin
option httpchk HEAD /ha.txt HTTP/1.0
Could you please run strace -tt -p $PID while it's doing
this ?
Also, I suspect you're having a few servers in the backend.
Could you please try to :
- comment out the timeout check
- comment out nbproc 2
Neither of these should cause any issue, but you never know.
Also, a number of bugs were fixed since 1.4.22 in the latest snapshot,
waiting for more important ones to do a release (though I think I could
do one anyway). One of them concerns the pollers where an event might
remain present. I diagnosed that it should not cause such issues but it
is possible that I have overlooked it. Could you then test this latest
snapshot from there to verify if you get any difference :
http://haproxy.1wt.eu/download/1.4/src/snapshot/
Thanks!
Willy
path_beg routing question
Hello list, I am working on routing an upload module that my application uses to a new backend, it seems simple enough but I would like to get some feedback. The application uri looks like this: My.app.com/upload?id=123morestuff=haproxy+is+awesome This is the config I have in mind: acl uploads_capture path_beg /upload use_backend upload if uploads_capture The question is, would this suffice to capture the upload URI and route to the proper backend? The upload POST contains a number of params, not sure if I should add a check to make sure it's a POST as well Any feedback is appreciated! Thank you
Re: How to update haproxy?
Here is a simple script i use to update my HAproxy servers, the logic is simple and I always keep the previous versions in case i need to roll back, basically build from source with the latest stable release and replace the current haproxy, all your init scripts will continue to work. cd /tmp/ wget http://haproxy.1wt.eu/download/latest/version.tar.gz tar -xvzf haproxy-*cd haproxy-(version) make TARGET=linux26 USE_PCRE=1 replace the make with whatever fits your needs mv /usr/sbin/haproxy /usr/sbin/haproxy_v.X.X keeps an old copy of the version cp haproxy /usr/sbin/haproxy your distro may be different /etc/init.d/haproxy restart or whatever init script you have.. Hope this helps. On Tue, Sep 18, 2012 at 1:07 PM, Baptiste bed...@gmail.com wrote: Hi, 2 ways: - The dirty one: exactly the same way you installed haproxy 1.4.20, in the same place - the nice one: install haproxy in a different directory, let say /opt/haproxy-1.4.22, then update your startup script For both ways: then reload haproxy. New process 1.4.22 will replace old 1.4.20 one cheers On Tue, Sep 18, 2012 at 6:57 PM, Odalinda Morales Rojas odalim...@hotmail.com wrote: Hi I have already installed the version 1.4.20 of haproxy, but I need to upgrade to version 1.4.22 I would greatly appreciate being told how I do.
Re: major performance decrease in total throughput with HAproxy 1.4.20
Hey Willy, thanks for your response, answers bellow: - Is this load stable or does it vary a lot during the test ? The load is pretty stable, it doesn't seem to go up more than .70 max - Do you have conntrack loaded on the LB ? Its installed, any specific command you want me to try? - Is the LB a real or virtual machine ? Virtual, the entire environment is virtual including the openAM servers - Are you observing a high CPU or network usage anywhere in the chain ? There is an initial spike on CPU when Jmeter starts but thats normal if you don't use a ramp up period - If you remove one of your servers, does the throughput remain the same or does it drop by half ? Stays exactly the same, probably because of the sticky session? The only thing I'm seeing that is wrong in your config is that you should remove the option httpclose statement in the defaults section and in the backend section, but I'm pretty sure that at such a low load, it won't make any difference. I have removed it and tested with and without it, makes no difference. The strangest thing is that it seems like you reach a limit and it wont go over 80-100 req/sec. One last thing I forgot to mention, I am testing on a hot standby HAproxy that is configured exactly as the first one and I use Keepalived for high availability, so Keepalived is the only other process running on the box. Any ideas? Thanks On Thu, Aug 16, 2012 at 1:16 AM, Willy Tarreau w...@1wt.eu wrote: Hi Saul, On Wed, Aug 15, 2012 at 02:43:57PM -0400, Saul Waizer wrote: Hey list, I am having a strange issue with my latest implementation of HAproxy. I have 2 openAM servers (tomcat) behind my haproxy box running version 1.4.20 on Ubuntu 10 X_86, all properly configured to be behind a load balancer. I used Jmeter to test the openAM servers individually and both give consistent results of ~1600-1800 req/sec, however, when I run the same exact test through the HAproxy I can barely get 100 req/sec! This setup in theory should allow me to double my throughput. Wow, 100 req/s is pretty low. Is this load stable or does it vary a lot during the test ? Do you have conntrack loaded on the LB ? Is the LB a real or virtual machine ? Are you observing a high CPU or network usage anywhere in the chain ? If you remove one of your servers, does the throughput remain the same or does it drop by half ? The only thing I'm seeing that is wrong in your config is that you should remove the option httpclose statement in the defaults section and in the backend section, but I'm pretty sure that at such a low load, it won't make any difference. Regards, Willy
Re: major performance decrease in total throughput with HAproxy 1.4.20
Well, it turns out it was the option httpclose that was set on the defaults I commented out both httpclose and http-server-close and I got the desired throughput, 2k+ req/sec, then I enabled http-server-close and ran the test again and still got the desired throughput, enabling httpclose made it go down to 100 req/sec. Why would this cause such behavior though? Thanks On Thu, Aug 16, 2012 at 1:11 PM, Baptiste bed...@gmail.com wrote: Any ideas? Thanks Hi, Could be interesting to have a look at HAProxy logs :) They may provide useful information about network and application response time (enable the http-server-close option). cheers
major performance decrease in total throughput with HAproxy 1.4.20
Hey list, I am having a strange issue with my latest implementation of HAproxy. I have 2 openAM servers (tomcat) behind my haproxy box running version 1.4.20 on Ubuntu 10 X_86, all properly configured to be behind a load balancer. I used Jmeter to test the openAM servers individually and both give consistent results of ~1600-1800 req/sec, however, when I run the same exact test through the HAproxy I can barely get 100 req/sec! This setup in theory should allow me to double my throughput. Note: This is a pretty decent server, 4gb of ram and 4 procs with nothing else other than HAproxy running. My relevant HAproxy config bellow: #- # Global settings Main #- global log 127.0.0.1 local0 info pidfile /var/run/haproxy.pid # stats socket /var/run/haproxy.stat mode 666 maxconn 65000 userhaproxy group haproxy daemon defaults modehttp log global option dontlognull option httplog option httpclose option http-server-close option forwardfor except 127.0.0.0/8 option redispatch stats enable stats uri /st timeout connect 5000 # default 5 second time out if a backend is not found timeout client 300s timeout server 300s #timeout http-request10s #timeout queue 1m #timeout http-keep-alive 10s timeout check 5s maxconn 65000 retries 3 frontend sso *:8080 default_backend sso acl sso1 hdr_dom(Host) -i auth.mydomain.lan use_backend sso if sso1 backend sso mode http stats enable option httpclose cookie SERVERID insert nocache #appsession amlbcookie len 20 timeout 3h request-learn option httpchk HEAD /opensso/isAlive.jsp HTTP/1.0 balance roundrobin server openam 10.1.1.5:8080 cookie 01 id 1001 check weight 100 server openam2 10.1.1.6:8080 cookie 02 id 1002 check weight 100 Thank you in advance for any assistance in this matter.
Re: HAProxy in High Availability
Thomas, Check out this tutorial: http://www.howtoforge.com/setting-up-a-high-availability-load-balancer-with-haproxy-keepalived-on-debian-lenny I have a much more complex setup running and I have tested the failover one too many times, works like a charm Good Luck On Thu, Jun 28, 2012 at 7:29 AM, Richard Stanford rich...@kimbia.comwrote: With this approach you really want 1 fewer public IP than you have public facing servers. With 2 servers this means 1 IP. DNS is used to distribute the load around, and keepalived is used to move traffic when a server fails. But you always want at least 1 servers worth of spare capacity in your HA environment, otherwise after you fail over the server getting 2X traffic will also die. Actually that's an oversimplified example, since to distribute the load correctly you'd need n-1 public addresses on each of n servers (with diminishing returns). Thankfully few LB scenarios require more than 1X1 machines. -Richard On Jun 28, 2012, at 6:17 AM, David Coulson da...@davidcoulson.net wrote: Multiple IP addresses are used, and managed by keepalived. On 6/28/12 7:11 AM, Thomas Manson wrote: Ok, but then, I don't get where is used DNS Round Robin, if only one IP is used. (it may be obvious, sorry ;);) Regards, Thomas. On Thu, Jun 28, 2012 at 1:08 PM, Türker Sezer turkerse...@tsdesign.infowrote: On Thu, Jun 28, 2012 at 11:59 AM, Manson Thomas m...@mansonthomas.com wrote: usually a client will cache the IP served by the DNS server, in order to not query each time the DNS system. So how can the client switch to another server once it has resolved one. Clients dont switch ip address. They connect same ip address. But we move ip address to backup or another active instance using keepalived so they connect another server using same ip address. -- Türker Sezer TS Design Informatics LTD. http://www.tsdesign.info/
Re: ha proxy Nagios plugin
Hi, I cant seem to get this plugin to work properly, here is the output: ./check_haproxy -u http://my.ha.ip/stats;csv -U myuser -P mypass HAPROXY CRITICAL - HTTP error: 401 Unauthorized (Unauthorized) | t=0.042485s;2;10;0; All credentials are valid and I am able to curl my stats url with no issues On Thu, May 31, 2012 at 5:48 AM, David BERARD cont...@davidberard.frwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, | Also, it seems to rely only on the HTTP socket. Do you think | it can easily be adapted to also support the unix socket, which | is global and does not require opening a TCP port ? | | The plugin works with Nagios which is not installed on the same host. So a | remote access in a way or other is mandatory. | | hey, that obviously makes sense ! If the nagios NRPE plugin is used, the check script can be on the same host. The use of unix socket is more simple in this case. I've made a patch to support unix socket : https://github.com/polymorf/check_haproxy Regards, - -- David BERARD contact(at)davidberard.fr GPG|PGP KeyId 0xC8533354 GPG|PGP Key http://davidberard.fr/C8533354.gpgkey * No electrons were harmed in the transmission of this email * -BEGIN PGP SIGNATURE- iEYEARECAAYFAk/HPmcACgkQOL7fhchTM1S7GgCfYjZqPvugnKv3g79TH9cj6IYj YWcAoKh+QFcndetSBta1Dwbp5APiFuFw =ZpSc -END PGP SIGNATURE-
cannot bind socket Multiple backends tcp mode
Hello List,
I hope someone can shed some light with the following situation:
Setup:
HAproxy frontend proxy and apache SSL backends. I didn't want to use
haproxy+stunnel or apache mod_ssl so I use straight TCP mode and
redirects, it works fine with one backend. The only problem is when I
try to add a second backend for a different farm of servers I get the
following:
Starting frontend https-services-in: cannot bind socket
My understanding was that multiple backends could use the same
interface, perhaps I was wrong, if that is the case, any suggestions
on how to be able to have multiple backends running tcp mode on port
443 so I can match the url and redirect to the appropriate backend
from my HAproxy?
Thank You Very much in advance.
Relevant configuration:
##--
## HTTP FRONTEND
##
frontend www
mode http
bind :80
redirect prefix https://secure.mydomain.com if { hdr_dom(Host) -i
secure.mydomain.com }
redirect prefix https://services.mydomain.com if { hdr_dom(Host) -i
services.mydomain.com }
backend www
mode http
balance leastconn
stats enable
option httpclose
option forwardfor
option httpchk HEAD /ha.txt HTTP/1.0
server nginx_1 10.10.1.1:80 weight 100 check
##--
## HTTPS FRONTEND
##
frontend https-in
mode tcp
bind :443
default_backend https-secure-portal
##--
## HEADER ACL'S
##
acl secure1 hdr_dom(Host) -i secure.mydomain.com
use_backend https-secure-portal if secure1
backend https-secure-portal
mode tcp
balance leastconn
option ssl-hello-chk
server ssl_1 10.10.1.1:443 weight 100 check
##--
## SERVICES FRONTEND
##
frontend https-services-in
mode tcp
bind :443
default_backend https-services
acl services1 hdr_dom(Host) -i services.mydomain.com
use_backend https-services if services1
backend https-services
mode tcp
balance leastconn
option ssl-hello-chk
#option httpclose
#option forwardfor
server nginx2_ssl 10.10.1.110:443 weight 100 check
Problem with rewrites + SSL
Hello List, I am having an issue trying to translate some urls with my haproxy setup and Im hoping someone can shed some light. Information: 4 apache servers need a reliable LB such as HA. These apache servers are listening on 80,443 however all traffic gets rewritten (with apache re-writes) to https if the request comes on port 80, currently there is just a firewall with dnat. The apaches are not serving content directly from disk but rather proxy passing to backend servers based on the request, this information is only relevant because of the different hostnames that a client will be hitting when connecting to the site. The problem: I want to be able to re-write the url at the HA level but I am having some issues trying to do this accurately. I have a front end listening on 80 and a front end listening on 443 https, the latter is set to TCP mode so it will transparently forward requests to the apaches on 443. So what i've done is try to force a redirect to https if the requests comes via 80 to a url, the problem is that because there are many hostnames and calls associated with every requests, I can't simply send all traffic to one URL, I need to be able to just replace the protocol and keep the request intact. Config: ##-- ## HTTP FRONTEND ## frontend www 10.1.1.1:80 mode http acl no_ssl dst_port 80 redirect prefix https://sub1.mydomain.com if no_ssl backend www mode http balance roundrobin stats enable option httpclose option forwardfor option httpchk HEAD /ha.txt HTTP/1.0 server Apache1 10.1.1.13:80 weight 100 check server Apache2 10.1.1.14:80 weight 100 check server Apache3 10.1.1.15:80 weight 100 check server Apache4 10.1.1.16:80 weight 100 check ##-- ## HTTPS FRONTEND ## frontend https-in mode tcp bind :443 default_backend bk-https backend bk-https mode tcp balance source option ssl-hello-chk server Apache_ssl1 10.1.1.13:443 weight 100 check server Apache_ssl2 10.1.1.14:443 weight 100 check server Apache_ssl3 10.1.1.15:443 weight 100 check server Apache_ssl4 10.1.1.16:443 weight 100 check Notes: most of the requests users will make will hit https://sub1.mydomain.com but the problem is that once they get there there are assets that load on sub2.mydomain.com sub3.mydomain.com and because traffic is going through HAproxy and we have that rule to re-write everything to https://sub1.mydomain.com half of the stuff won't load. Any help is greatly appreciated it and Thank you in advance. Willy You Rock!
