Re: limiting conn-curs per-ip using x-forwarded-for
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "Jarno Huuskonen" To: "Klavs Klavsen" Cc: haproxy@formilux.org Sent: Tuesday, March 24, 2015 9:23:04 AM Subject: Re: limiting conn-curs per-ip using x-forwarded-for Hi, On Tue, Mar 24, Klavs Klavsen wrote: > #create a stick-table of 30 IPs for storing active IPs, 5 minute timeout > stick-table type ip size 30 expire 5m store gpc0 > #populate the table with the X-Forwarded-For header > stick store-request hdr(X-Forwarded-For) You've created type ip stick table, but I think hdr(X-Forwarded-For) returns a string. Do you see any entries in the stick table: show table nocache -> to stats socket Maybe it's possible to use one of the converters to map str -> ip (http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#7.3.1-map) or try with a string stick table: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-stick-table -Jarno -- Jarno Huuskonen
Re: How can i get haproxy reload its configuration
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "jeff saremi" To: haproxy@formilux.org Sent: Tuesday, March 24, 2015 8:52:20 AM Subject: RE: How can i get haproxy reload its configuration > Subject: Re: How can i get haproxy reload its configuration > From: Vincent > Bernat > Date: 2015-03-24 12:07:34 > Message-ID: 87619q61kp.fsf () zoro ! > exoscale ! ch > � 24 mars 2015 07:45 -0400, jeff saremi  : > > #!/bin/sh > > > pidfile=/data/haproxy.pidhaproxy -db \ > > -f /haproxy-1.5.8/haproxy.cfg -p > $pidfile \ > > -sf $(cat $pidfile) > The shell does variable substitution > first, then execute the > line. Hence, $pidfile is expanded to "", not to > "/data/haproxy.pid". Put > pidfile=... on its own line. > -- > I'll burn my > books. > -- Christopher Marlowe Sorry that was just how my email got through. > Those are separate lines. So here it is again: #!/bin/sh > pidfile=/data/haproxy.pid haproxy -db \ -f /haproxy-1.5.8/haproxy.cfg -p > $pidfile \ -sf $(cat $pidfile) What i'm wondering is how this could have ever > worked? since the "$(cat $pidfile) and the creation of the $pidfile are both > on the same line. Is it possible that when the pidfile is concatenated for > the sake of "-sf" the file does not exist or contains old pids?
Re: timeout values for redis?
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "Jim Gronowski" To: "haproxy@formilux.org" Sent: Tuesday, March 24, 2015 1:25:33 PM Subject: timeout values for redis? Does anyone have any feedback on sane timeout values for load balancing redis? The testing config I was using had ‘timeout client 5’ and I was getting consistent client disconnects in the logs. I increased it to two minutes and things have improved significantly, though I do see client disconnects every few hours (but the application is behaving normally). Client is StackExchange.Redis if that helps. Google wasn’t much use. HA-Proxy version 1.5.10. Full config: global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon defaults log global mode tcp option tcplog option dontlognull timeout connect 5000 timeout client 2m timeout server 12 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http frontend redisFE bind *:6379 mode tcp maxconn 10240 default_backend redisBE backend redisBE mode tcp option tcplog balance source option tcp-check tcp-check send PING\r\n tcp-check expect string +PONG tcp-check send info\ replication\r\n tcp-check expect string role:master tcp-check send QUIT\r\n tcp-check expect string +OK server A-redis-01 X:6379 maxconn 1024 check inter 1s server A-redis-02 X:6379 maxconn 1024 check inter 1s server B-redis-01 X:6379 maxconn 1024 check inter 1s server B-redis-02 X:6379 maxconn 1024 check inter 1s Jim Gronowski Network Administrator DiTronics, LLC. Ditronics, LLC email disclaimer: This communication, including attachments, is intended only for the exclusive use of addressee and may contain proprietary, confidential, or privileged information. Any use, review, duplication, disclosure, dissemination, or distribution is strictly prohibited. If you were not the intended recipient, you have received this communication in error. Please notify sender immediately by return e-mail, delete this communication, and destroy any copies.
Re: limiting conn-curs per-ip using x-forwarded-for
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "Klavs Klavsen" To: "Mark S." , haproxy@formilux.org Sent: Tuesday, March 24, 2015 10:56:03 AM Subject: Re: limiting conn-curs per-ip using x-forwarded-for Mark S. wrote on 03/24/2015 03:36 PM: > Hi Klavs, > > Try using this in the backend section for the track-sc2 statement to use > the previous instance of the Forwarded-For header. > > I have a version of this working in a similar fashion, but I am using a > "stick-table type binary" instead of string - it probably works both > ways, but I haven't tested type string. I'm using http_req_rate as I'm > concerned about the rate rather than the number of active connections. > > stick-table type binary len 32 size 5k expire 5m store > http_req_rate(10s),gpc0 > tcp-request content track-sc2 hdr(X-Forwarded-For,-1) > Thank you Mark, It did help.. somewhat.. now first connection is allowed :) I now have: stick-table type string size 100k store conn_cur,gpc0 stick store-request hdr(X-Forwarded-For,-1) tcp-request content track-sc2 hdr(X-Forwarded-For) acl allowed sc2_conn_cur lt 2 block unless allowed But everyone gets blocked - when one connection is active.. (ie. not just the source of that one connection). show table shows this: table: kk.dk-ds-backend, type: string, size:102400, used:2 0x1304dfc: key=10.27.174.10 use=1 exp=0 server_id=1 gpc0=0 conn_cur=1 shouldn't the key - be the x-forwarded-for header? [CUT] -- Regards, Klavs Klavsen, GSEC - k...@vsen.dk - http://www.vsen.dk - Tlf. 61281200 "Those who do not understand Unix are condemned to reinvent it, poorly." --Henry Spencer
Re: HAProxy signal queue not working correctly
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "Alan Fitton" To: "Willy Tarreau" Cc: haproxy@formilux.org Sent: Tuesday, March 24, 2015 2:01:59 PM Subject: RE: HAProxy signal queue not working correctly Hi, I've been trying out this logging, and a few variations of my own, on one of the RHEL5 (2.6.18-371.9.1.el5) systems that was exhibiting the problem more frequently. I am seeing what you saw, signals queued and processed without any issue.. It's strange, I can't figure out why the problem isn't reproducible from a large amount of signals on the command line, yet would be happening every few days through occasional use of the reload command. I will keep looking. Probably by deploying a patched version which logs the below and some more to syslog, so I can look out for the problem in a more "realistic" setting. Since deploying my workaround patch which traverses the signal_queue instead of checking signal_state[sig].count, the problem hasn't been seen in our test environment. Agree it's probably a bug which needs fixing from a signal handling point of view, and I'm definitely up for trying to find the cause. My thoughts about queueing reloads on the automation side were because it seems unnecessary and not optimal (for us anyway) for the automation to potentially reload HAProxy more than once every few seconds. while true; do kill -QUIT 28292; kill -TTOU 28292; kill -QUIT 28292; kill -TTIN 28292; done leave loop: signal_queue_len=3, count[SIGTTOU]=0, count[SIGTTIN]=0, count[SIGQUIT]=0 signal handler: signal_queue_len=0, count[SIGTTOU]=0, count[SIGTTIN]=0, count[SIGQUIT]=0 signal handler: signal_queue_len=0, count[SIGTTOU]=0, count[SIGTTIN]=0, count[SIGQUIT]=0 signal handler: signal_queue_len=0, count[SIGTTOU]=0, count[SIGTTIN]=0, count[SIGQUIT]=0 enter loop: signal_queue_len=3, count[SIGTTOU]=1, count[SIGTTIN]=1, count[SIGQUIT]=1 leave loop: signal_queue_len=3, count[SIGTTOU]=0, count[SIGTTIN]=0, count[SIGQUIT]=0 Regards, Alan -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: 19 March 2015 15:17 To: Alan Fitton Cc: haproxy@formilux.org Subject: Re: HAProxy signal queue not working correctly Hi Alan, On Thu, Mar 19, 2015 at 10:56:35AM +, Alan Fitton wrote: > Hi Willy, > > Thank you for your reply and your work on HAProxy. I will add some > instrumentation and hopefully be able to demonstrate your theory. I agree, > it's the one I had arrived at too :) It seemed unlikely at first since the > signals are masked inside __signal_process_queue, but it's still possible if > timing goes against you. I don't think we're sensitive to timing here. I've done this to try to reproduce the issue : diff --git a/src/signal.c b/src/signal.c index e9301ed..241feac 100644 --- a/src/signal.c +++ b/src/signal.c @@ -37,6 +37,7 @@ int signal_pending = 0; /* non-zero if t least one signal remains unprocessed */ */ void signal_handler(int sig) { + fprintf(stderr, "received signal %d\n", sig); if (sig < 0 || sig >= MAX_SIGNAL) { /* unhandled signal */ signal(sig, SIG_IGN); @@ -77,6 +78,7 @@ void __signal_process_queue() * handler. That allows real signal handlers to redistribute signals * to tasks subscribed to signal zero. */ + fprintf(stderr, "enter loop: signal_queue_len=%d, count[%d]=%d\n", signal_queue_len, SIGUSR1, signal_state[SIGUSR1].count); for (cur_pos = 0; cur_pos < signal_queue_len; cur_pos++) { sig = signal_queue[cur_pos]; desc = &signal_state[sig]; @@ -90,7 +92,9 @@ void __signal_process_queue() } desc->count = 0; } + sleep(1); } + fprintf(stderr, "leave loop: signal_queue_len=%d, count[%d]=%d\n", signal_queue_len, SIGUSR1, signal_state[SIGUSR1].count); signal_queue_len = 0; /* restore signal delivery */ Then I'm doing that : $ killall -QUIT haproxy;killall -1 haproxy; kill -USR1 $(pidof haproxy) $(pidof haproxy) $(pidof haproxy) $(pidof haproxy) $(pidof haproxy) $(pidof haproxy) $(pidof haproxy) $(pidof haproxy); (SIGQUIT send a large response, and SIGHUP talks as well). The sleep(1) is here to ensure that signals get delivered and queued before we re-enable signals. I'm clearly seeing my messages being queued : received signal 3 enter loop: signal_queue_len=1, count[10]=0 Dumping pools usage. Use SIGQUIT to flush them. - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 4 users [SHARED] - Pool hlua_com (48 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool task (112 bytes) : 2 allocated (224 bytes), 2 used, 1 users [SHARED] - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool connection (336 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1
Re: How can i get haproxy reload its configuration
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "jeff saremi" To: haproxy@formilux.org Sent: Tuesday, March 24, 2015 3:22:09 PM Subject: RE: How can i get haproxy reload its configuration From: jeffsar...@hotmail.com To: haproxy@formilux.org Subject: RE: How can i get haproxy reload its configuration Date: Tue, 24 Mar 2015 08:52:20 -0400 > Subject: Re: How can i get haproxy reload its configuration> From: Vincent Bernat> Date: 2015-03-24 12:07:34> Message-ID: 87619q61kp.fsf () zoro ! exoscale ! ch> � 24 mars 2015 07:45 -0400, jeff saremi  :> > #!/bin/sh> > pidfile=/data/haproxy.pidhaproxy -db \> > -f /haproxy-1.5.8/haproxy.cfg -p $pidfile \> > -sf $(cat $pidfile)> The shell does variable substitution first, then execute the> line. Hence, $pidfile is expanded to "", not to "/data/haproxy.pid". Put> pidfile=... on its own line.> --> I'll burn my books.> -- Christopher Marlowe Sorry that was just how my email got through. Those are separate lines. So here it is again: #!/bin/sh pidfile=/data/haproxy.pid haproxy -db \ -f /haproxy-1.5.8/haproxy.cfg -p $pidfile \ -sf $(cat $pidfile) What i'm wondering is how this could have ever worked? since the "$(cat $pidfile) and the creation of the $pidfile are both on the same line. Is it possible that when the pidfile is concatenated for the sake of "-sf" the file does not exist or contains old pids? So my suspicion around the -sf was correct. Each time i run the haproxy command I get the following printed on the console: cat: /data/haproxy.pid: No such file or directory which seems to be a result of "-sf $(cat $pidfile)" in the command line. Has anyone successfully done this in practice? is there possibly other ways of achieving this with having to restart the haproxy process? thanks
Re: limiting conn-curs per-ip using x-forwarded-for
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "Jarno Huuskonen" To: "Klavs Klavsen" Cc: haproxy@formilux.org Sent: Tuesday, March 24, 2015 12:58:21 PM Subject: Re: limiting conn-curs per-ip using x-forwarded-for Hi, On Tue, Mar 24, Klavs Klavsen wrote: > I now have: > stick-table type string size 100k store conn_cur,gpc0 > stick store-request hdr(X-Forwarded-For,-1) > tcp-request content track-sc2 hdr(X-Forwarded-For) > acl allowed sc2_conn_cur lt 2 > block unless allowed tcp-request inspect-delay ? Most of the examples seem to use inspect-delay: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4.2-tcp-request%20content > shouldn't the key - be the x-forwarded-for header? Have you checked that the requests have (one) x-forwarded-for header ? hdr(X-Forwarded-For) = first header, and hdr(X-Forwarded-For,-1) = last header. (http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#7.3.6-req.hdr) And is the haproxy ip the only one thats in the stick table ? -Jarno -- Jarno Huuskonen
Re: using backend node details in acls/response manipulation
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "Martin Nikolov" To: haproxy@formilux.org Sent: Tuesday, March 24, 2015 3:36:36 PM Subject: using backend node details in acls/response manipulation Hi guys, I'm wondering if it is possible to use things like selected backend node's ip, name or port as variables. My goal is to set a header in the http response with the selected backend's details to a certain set of source ip addresses (hence the acl, which is the easy part). I searched in the documentation, but was not able to find a solution. Thanks in advance. Regards.
Re: Retrying a request on other servers
Thanks, I sent request previously to you but I have done it. Ha. - Original Message - From: "jeff saremi" To: haproxy@formilux.org Sent: Tuesday, March 24, 2015 4:12:32 PM Subject: Retrying a request on other servers Is this a feature of HaProxy? and if so what is the corresponding option to enable it. Basically, i want Harpoxy to resend the request which as already received a 503 from one server, to another server in the same backend. And i'd like to control how many tries to do. I looked at that options: "retries" and "redispatch" but they don't seem to be doing this. thanks Jeff
Re: OCSP stappling full support
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Pierre Ozoux" To: haproxy@formilux.org Sent: Friday, March 20, 2015 11:59:16 AM Subject: OCSP stappling full support Hi, HAProxy provides OCSP stapling since v1.5 and this is nice! Although, it is still difficult to setup compared to Apache or Nginx. I had to do this little script: https://wiki.mozilla.org/Talk:Security/Server_Side_TLS#OCSP_in_HAProxy It would be really nice if it was as easy as Apache and Nginx. Is it planned in the roadmap? Thanks a lot for your work, Pierre -- I use PGP to protect our privacy, if you want to know more, you can follow this https://emailselfdefense.fsf.org/en/ If you have further questions, please do not hesitate to ask. You can verify my public key here: https://keybase.io/pierreozoux
Re: sticky sessions with tcp mode
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Richard Bassler" To: haproxy@formilux.org Sent: Friday, March 20, 2015 10:27:33 PM Subject: sticky sessions with tcp mode I am having some issues with sticky sessions. The sessions are not sticking. I am using tcp mode with ssl. I have implemented the example out of the manual. The user is getting bounced back and forth between the two servers, which is causing issues in the application . Am I missing something? HA-Proxy version 1.5.2 2014/07/12 frontend pc bind 192.168.10.72:443 mode tcp option tcplog default_backend pc backend pc mode tcp balance roundrobin stick-table type binary len 32 size 30k expire 30m acl clienthello req_ssl_hello_type 1 acl serverhello rep_ssl_hello_type 2 tcp-request inspect-delay 5s tcp-request content accept if clienthello tcp-response content accept if serverhello stick on payload_lv(43,1) if clienthello stick store-response payload_lv(43,1) if serverhello server r15-21 192.168.10.21:443 check server r16-22 192.168.10.22:443 check “CONFIDENTIALITY AND PROPRIETARY INFORMATION NOTICE: This email, including attachments, is covered by the Electronic Communications Privacy Act (18 U.S.C. 2510-2521) and contains confidential information belonging to the sender which may be legally privileged. The information is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance of the contents of this information is strictly prohibited. If you have received this electronic transmission in error, please immediately notify the sender by return e-mail and delete this message from your computer or arrange for the return of any transmitted information.”
Re: HAProxy and site failover
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Brendan Kearney" To: haproxy@formilux.org Sent: Friday, March 20, 2015 8:29:07 PM Subject: HAProxy and site failover hi, first time / long time... i am wondering if the ability exists in HAProxy to reply to a HTTP proxy request with a reset (RST) if no backend server is available. the scenario goes as such: i have a proxy pac file that assigns multiple proxies to all clients, and through the logic tree in the pac file, the proxies are assigned in a specific order. i have multiple sites with load balanced proxies, and the intention is to provide site failover, should a larger event occur like ISP issues that breaks internet access. with the pac file assigning all proxy VIPs to the client, should the default VIP not have an available backend server to fulfill the request, i would want to configure HAProxy to send a reset to the client, indicating that the next assigned proxy should be used. with site failover happening transparently, a user who would normally browse through the proxy/proxies at site1 would be automatically failed over and browse through the proxy/proxies at site2. if no servers were available in site2, then the next assigned proxy would be used and failures with RST replies would result in failovers until all assigned proxies are exhausted. the intention is not to provide / assign hundreds of proxies in the pac file, but to provide resiliency with a couple of sites serving as backups to each other, should an event warrant it. thank you, brendan kearney
Re: 503 on alive backends, hanging processes on reload
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Jeff Mitchell" To: haproxy@formilux.org Sent: Friday, March 20, 2015 6:11:14 PM Subject: 503 on alive backends, hanging processes on reload I'm running haproxy 1.5.11-1ppa1~trusty from https://launchpad.net/~vbernat/+archive/ubuntu/haproxy-1.5 on Trusty (Ubuntu 14.04). It is a fairly basic configuration that mostly comes straight from the defaults: global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL ssl-default-bind-options no-sslv3 defaults log global mode http option httplog option dontlognull maxconn 1024 timeout queue 5000 timeout connect 5000 timeout client 5 timeout server 5 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http frontend ft_poml_vip bind :80 acl host_apibrowse hdr_beg(host) -i apibrowse use_backend be_apibrowse if host_apibrowse backend be_apibrowse server registry 10.88.24.3:49163 I also have several more ACLs and backends that are not shown, but follow the exact same pattern as above (with different host header matching). The main differences from the default are maxconn/timeout queue, both of which I set to try to solve this problem, and my simple frontend/backend. After a time, calls from a web browser to haproxy are sometimes, but not always, being given 503 errors. When I see this happening, if I sit on a very simple page and refresh rapidly, I will sometimes get 503s and sometimes not. I turned off health checks to ensure that failing health checks were not the source of the 503s. What I have noticed is some oddness with the haproxy processes. Here is "date" and "ps -ef" output when I am seeing this behavior: Fri Mar 20 21:55:38 GMT 2015 haproxy 19621 1 0 17:35 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 19599 haproxy 20075 1 0 20:50 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 20063 haproxy 20121 1 0 20:50 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 20112 "service haproxy reload" has been called at various times when the backends have come and gone and the config file has been rewritten, including at 17:35 and 20:50. When haproxy is in this state, "service haproxy stop" does not stop all processes: # service haproxy stop * Stopping haproxy haproxy [ OK ] # ps -ef | grep haproxy haproxy 19621 1 0 17:35 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 19599 haproxy 20075 1 0 20:50 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 20063 If I then start the service again, those same processes run, but with a new one: # ps -ef | grep haproxy haproxy 19621 1 0 17:35 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 19599 haproxy 20075 1 0 20:50 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 20063 haproxy 20395 1 0 22:04 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid When I run "service haproxy stop" and then manually kill any remaining processes, and then run "service haproxy start", I get just the one process: # ps -ef | grep haproxy haproxy 20443 1 0 22:05 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid At this point I do *not* get the 503 errors. Everything runs great until the cycle repeats itself. It feels like this is some issue with haproxy reloading. It is possible that reload was called multiple times rapidly when being performed by the automated system, but in my testing if I call it very rapidly from the command line I haven't been able to replicate the issue. Any help would be much appreciated. Thanks! --Jeff
Re: Attached: May you support me for this issue logs of HAProxy
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Ramin K" To: "Ha Quan Le" , haproxy@formilux.org Sent: Friday, March 20, 2015 6:56:59 PM Subject: Re: Attached: May you support me for this issue logs of HAProxy Loads simpler to do the following: ## /etc/haproxy/haproxy.cfg global log /dev/log local0 notice ## /etc/rsyslog.d/49-haproxy.conf # Create an additional socket in haproxy's chroot in order # to allow logging via /dev/log to chroot'ed HAProxy processes $AddUnixListenSocket /var/lib/haproxy/dev/log # Send HAProxy messages to a dedicated logfile if $programname startswith 'haproxy' then /var/log/haproxy.log &~ I believe most of the above is included in Vincent Bernat's very nice Ubuntu/Debian packages, https://launchpad.net/~vbernat Ramin
Re: HAProxy and site failover
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Brendan Kearney" To: "Lukas Tribus" Cc: haproxy@formilux.org Sent: Saturday, March 21, 2015 9:11:57 AM Subject: Re: HAProxy and site failover On Sat, 2015-03-21 at 14:03 +0100, Lukas Tribus wrote: > > haproxy is a tcp (layer 3/4) proxy, that can perform application (layer > > 7) functions. i am already doing service checks against my proxies to > > validate their availability. when no pool member is available, haproxy > > knows it. there are no external helpers needed to make this > > determination. the layer 7 capabilities make this possible. > > > > the injection of a RST is part-and-parcel to the tcp proxy > > functionality. i can understand if the functionality in not in haproxy, > > but it is not outside the realm of capability for a t. > > The 3 way TCP handshake happens before the application (haproxy) is even > aware of the session, therefor this is only possible if the kernel handles > it (iptables), which is why I said its only possible with external helpers. > > Or is what you are requesting to send a RST in the middle of an already > established TCP session? > > > Please CC the mailing list. > > > Lukas > > sorry, thought i did cc the list. i will have to test out the behavior, as this is an implemented solution where i work, using other products. i can test a couple of different scenarios that come to mind. 1, new browser session comes in to the load balancer, and no backend servers are available. where / when is the RST sent? 2, a session to the load balanced exists, and the backend servers become unavailable. where / when is the RST sent? i'll run these scenarios and let you know what i find in a packet capture.
Re: HAProxy with multiple certificates, one of which being wild card, and the other being sub of that wildcard
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Peter BUtler" To: haproxy@formilux.org Sent: Sunday, March 22, 2015 1:15:07 AM Subject: Re: HAProxy with multiple certificates, one of which being wild card, and the other being sub of that wildcard forgot to CC the mailing list. thanks Cryil. > Date: Fri, 20 Mar 2015 00:47:53 +0100 > From: cyril.bo...@free.fr > To: peter_butler1...@outlook.com; haproxy@formilux.org > Subject: Re: HAProxy with multiple certificates, one of which being wild > card, and the other being sub of that wildcard > > Hi Peter, > > Le 20/03/2015 00:32, Peter Butler a écrit : > > I logged this on stackoverflow > > (http://stackoverflow.com/questions/29133477/haproxy-with-multiple-certificates-one-of-which-being-wild-card-and-the-other), > > > > but HAProxy usage there is pretty low. > > > > THis is my first mailing list email in years, please let me know if I > > have broken any rules. > > > > > > I have HAProxy working pretty well, I have port 80 and 443 open, TCP > > over 443 over a internal proxy. HTTPS end to end. I am serving > > certificates: *.mycompany.com.au www.secure.mycompany.com.au > > SNI does not seem to be working, as when I go to > > secure.mycompany.com.au, I keep getting the www.secure.mycompany.com.au > > certificate (and weirdly *most* browsers do not throw a error). But when > > I go to mail.mycompany.com.au I get the *.mycompany.com.au certificate > > correctly. > > Indeed, keep in mind that a wildcard certificate works only for > subdomains, not sub-subdomains. this is fine, and as mentioned its working great for my other wild card certs, but the difference is they do not have a specific.sub- which-is-the-same-as-a-wildcard .mycompany.com.au (am I believe the blod part is my issue here. > > So www.secure.mycompany.com.au won't match the "*.mycompany.com.au" > wildcard certificate. As no certificate matches the hostname, the first > one loaded is applied. I guess that in your SSL directory, the > certificate file for "secure.mycompany.com.au" is listed first. I have tried this change already, by renaming them alphabetically. Didn't make any difference. > > You can read the RFC 2818, chapter "3.1. Server Identity" for more details : > http://tools.ietf.org/html/rfc2818#section-3.1 > I think my issue is here, from your link: E.g., *.a.com matches foo.a.com but not bar.foo.a.com In my case I have a cert for both: *.a.com.au and bar.foo.a.com.au > > > If I remove the www.secure.mycompany.com.au cert from the SSL directory, > > and I browse to secure.mycompany.com.au I get the correct certificate, > > but of course when I go to www.secure.mycompany.com.au, I get a error on > > the browser, as it tries to give me *.mycompany.com.au certificate. > > It tends to confirm this : the wildcard still doesn't match, and the > first certificate is applied (which in this case is by chance the > wildcard). I you insert a fake certificate before the wildcard, I'm sure > it will be this one that will match. > > > Can anyone see what I am doing wrong please. > > > > |By the way, I also have a *.abc.def.mycompany.com.au and > > *.def.mycompany.com.au in this same config (removed from below, but in same > > syntax). and they are working perfect.| > > > > || > > > > |haproxy -version > > HA-Proxy version 1.5.8 2014/10/31 > > Copyright 2000-2014 Willy Tarreau > > > > > > > > > > ##| > > > > > > |##| > > > > > > |## > > > > global > > daemon > > user haproxy > > group haproxy > > log /dev/log local0 info > > log /dev/log local0 notice > > ulimit-n 2 > > pidfile /var/run/haproxy.pid > > tune.ssl.default-dh-param 2048 > > > > ## > > > > > > ## Port80 is open only to forward all requests to port 443. > > frontend unsecure 123.123.123.155:80 #Prod > > bind 192.168.14.155:80 #Prod > > mode http > > redirect scheme https code 301 if !{ ssl_fc } > > > > ## > > > > > > ##Listen on 443, and forward to internal proxy on 88. Needed for SSL end to > > end. > > listen ssl-proxy > > ##Only accept mycompany host headers > > http-request deny if !{ hdr_end(Host) -i mycompany.com.au } !{ > > hdr_end(Host) -i www.secure.mycompany.com.au } > > > > # Get Certificates from SSL directory. > > bind 123.123.123.155:443 ssl crt /etc/haproxy/ssl npn http/1.1 ciphers > > ECDHE-RSA-AES256-SHA:RC4-SHA:!RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM no-sslv3 > > bind 192.168.14.155:443 ssl crt /etc/haproxy/ssl npn http/1.1 ciphers > > ECDHE-RS
Re: HAProxy and site failover
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Lukas Tribus" To: "Brendan Kearney" , haproxy@formilux.org Sent: Saturday, March 21, 2015 9:03:20 AM Subject: RE: HAProxy and site failover > haproxy is a tcp (layer 3/4) proxy, that can perform application (layer > 7) functions. i am already doing service checks against my proxies to > validate their availability. when no pool member is available, haproxy > knows it. there are no external helpers needed to make this > determination. the layer 7 capabilities make this possible. > > the injection of a RST is part-and-parcel to the tcp proxy > functionality. i can understand if the functionality in not in haproxy, > but it is not outside the realm of capability for a t. The 3 way TCP handshake happens before the application (haproxy) is even aware of the session, therefor this is only possible if the kernel handles it (iptables), which is why I said its only possible with external helpers. Or is what you are requesting to send a RST in the middle of an already established TCP session? Please CC the mailing list. Lukas
Re: HAProxy with multiple certificates, one of which being wild card, and the other being sub of that wildcard
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Lukas Tribus" To: "Peter BUtler" , haproxy@formilux.org Sent: Sunday, March 22, 2015 6:27:15 AM Subject: RE: HAProxy with multiple certificates, one of which being wild card, and the other being sub of that wildcard > I have tried this change already, by renaming them alphabetically. > Didn't make any difference. It won't in 1.5.8. Only 1.5.11 respects alphabetical ordering of the certificates in a folder. Please specify them manually: crt /etc/haproxy/ssl/wildcard.mycompany.com.au.crt crt /etc/haproxy/ssl/www.secure.mycompany.com.au.crt (or vice versa, if I haven't got the problem right). > > You can read the RFC 2818, chapter "3.1. Server Identity" for more > details : > > http://tools.ietf.org/html/rfc2818#section-3.1 > > > > I think my issue is here, from your link: > E.g., *.a.com matches foo.a.com but not bar.foo.a.com > > In my case I have a cert for both: > *.a.com.au > and > bar.foo.a.com.au If those informations are correct, there is simply a bug here. The wildcard certificate should be served for secure.mycompany.com.au, not the www.secure.mycompany.com.au certificate. You are making sure that all Browsers support SNI that you test, correct (meaning no test with Internet Explorer on Windows XP)? Lukas
Re: Setting compression for specific request paths
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Sachin Shetty" To: haproxy@formilux.org Sent: Monday, March 23, 2015 3:38:53 AM Subject: Setting compression for specific request paths Hi, I see that we can set "compression type" on a frontend or backend. Due to some application level complication we want haproxy to not compress specific request path for example /api and compress the rest as usual. Any idea on how this can be done? One way would be to route the requests through a different backend and disable compression there, but that would be a ugly config to maintain. Thanks Sachin
Re: sticky sessions with tcp mode
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Richard Bassler" To: haproxy@formilux.org, "Lukas Tribus" Sent: Monday, March 23, 2015 9:41:01 AM Subject: RE: sticky sessions with tcp mode Thanks for the response. I changed to source ip. Application seems to like it better. >>> Lukas Tribus 3/21/2015 4:44 AM >>> > I am having some issues with sticky sessions. The sessions are not sticking. > > I am using tcp mode with ssl. I have implemented the example out of the > manual. > The user is getting bounced back and forth between the two servers, > which is causing issues in the application. > Am I missing something? Probably because TLS tickets are used instead of the SSL session ID. Either disable TLS tickets on your backend or use some other stickiness methods, like source ip stickiness. Lukas -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. “CONFIDENTIALITY AND PROPRIETARY INFORMATION NOTICE: This email, including attachments, is covered by the Electronic Communications Privacy Act (18 U.S.C. 2510-2521) and contains confidential information belonging to the sender which may be legally privileged. The information is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance of the contents of this information is strictly prohibited. If you have received this electronic transmission in error, please immediately notify the sender by return e-mail and delete this message from your computer or arrange for the return of any transmitted information.”
Re[4]: HAProxy and site failover
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "brendan kearney" To: "Lukas Tribus" Cc: haproxy@formilux.org Sent: Monday, March 23, 2015 2:40:57 PM Subject: Re: HAProxy and site failover I have confirmed the behavior. In both cases all new connections receive a RST when a backend server is not available to service the request. The behavior is Syn -> RST in both cases. Any existing connections timeout. On Mar 21, 2015 9:11 AM, "Brendan Kearney" < bpk...@gmail.com > wrote: On Sat, 2015-03-21 at 14:03 +0100, Lukas Tribus wrote: > > haproxy is a tcp (layer 3/4) proxy, that can perform application (layer > > 7) functions. i am already doing service checks against my proxies to > > validate their availability. when no pool member is available, haproxy > > knows it. there are no external helpers needed to make this > > determination. the layer 7 capabilities make this possible. > > > > the injection of a RST is part-and-parcel to the tcp proxy > > functionality. i can understand if the functionality in not in haproxy, > > but it is not outside the realm of capability for a t. > > The 3 way TCP handshake happens before the application (haproxy) is even > aware of the session, therefor this is only possible if the kernel handles > it (iptables), which is why I said its only possible with external helpers. > > Or is what you are requesting to send a RST in the middle of an already > established TCP session? > > > Please CC the mailing list. > > > Lukas > > sorry, thought i did cc the list. i will have to test out the behavior, as this is an implemented solution where i work, using other products. i can test a couple of different scenarios that come to mind. 1, new browser session comes in to the load balancer, and no backend servers are available. where / when is the RST sent? 2, a session to the load balanced exists, and the backend servers become unavailable. where / when is the RST sent? i'll run these scenarios and let you know what i find in a packet capture.
Re: HAProxy and site failover
Thank you for support, we have fixed our issues. Ha. - Original Message - From: "Lukas Tribus" To: "brendan kearney" Cc: haproxy@formilux.org Sent: Monday, March 23, 2015 5:52:48 PM Subject: RE: HAProxy and site failover > I have confirmed the behavior. In both cases all new connections > receive a RST when a backend server is not available to service the > request. The behavior is Syn -> RST in both cases. Any existing > connections timeout. That doesn't change the fact that an application can't do this, the TCP stack needs to do this, and HAProxy doesn't implement its own TCP stack, it leverages the TCP stack of the OS. As I said, you need external helpers to install iptables rules or something like that. The only thing an application could theoretically do, would be to close the listening socket and reopen it once the services are available again, but this introduces so much issues (for example we can't drop privileges, we can't monitor the proxy at all, we need to call the bind logic again and again, etc), that I don't think its worth the trouble. I understand that your existing (probably proprietary) load-balancing solution can do this (and it probably only can because some user presented a huge business case to that vendor), but that doesn't mean it makes sense to implement it in an a layer 4/7 reverse proxy application. Lukas
Re: Checked: send/accept-proxy over unix socket not working
Good, local@PGSYNCTEST:~$ sudo /etc/init.d/haproxy restart [sudo] password for local: * Restarting haproxy haproxy [ALERT] 078/181429 (1600) : Starting frontend pgdbplatform_frontend_cluster01: cannot bind socket [0.0.0.0:5432] [fail] This is because Postgres occupied port 5432 Hence, we try again local@PGSYNCTEST:~$ sudo service postgresql stop * Stopping PostgreSQL 9.3 database server [ OK ] local@PGSYNCTEST:~$ sudo /etc/init.d/haproxy restart * Restarting haproxy haproxy [ OK ] local@PGSYNCTEST:~$ sudo service postgresql start * Starting PostgreSQL 9.3 database server But my questions are not about how to start HA-Proxy, my questions are that we received no logs of RSYSLOG from HA-Proxy, even log files of HA-Proxy are not created. May I send you the attachment, you will see all of our settings inside, our errors are that NO LOGS OF HA-PROXY CREATION. May you support, Ha. - Original Message - From: "Jarno Huuskonen" To: "Ha Quan Le" Cc: haproxy@formilux.org Sent: Friday, March 20, 2015 1:42:24 PM Subject: Re: Checked: send/accept-proxy over unix socket not working Hi, On Fri, Mar 20, Ha Quan Le wrote: > local@PGSYNCTEST:~$ sudo -u haproxy /etc/init.d/haproxy restart > * Restarting haproxy haproxy > [WARNING] 078/171401 (1267) : [/usr/local/sbin/haproxy.main()] Cannot raise > FD limit to 90034. You're trying to start/restart haproxy as haproxy user (sudo -u). start/restart as root and let haproxy drop privs (user / group in haproxy.cfg). > [ALERT] 078/171401 (1267) : Starting frontend GLOBAL: error when trying to > preserve previous UNIX socket [/var/run/haproxy.sock] > [ALERT] 078/171401 (1267) : Starting frontend > pgdbplatform_frontend_cluster01: cannot bind socket [0.0.0.0:5432] Try restarting as root. If it still doesn't work then make sure that there's no haproxy process left from earlier tests. And if it still complains about cannot bind [0.0.0.0:5432] then you have something else already listening to that socket. -Jarno -- Jarno Huuskonen GOOD HAProxy 1.5 -- 1) Removal of old HA-Proxy1.4 $ sudo apt-get remove haproxy 2) Preparation $ cd /usr/local/src/ local@PGSYNCTEST:/usr/local/src$ sudo apt-get install build-essential local@PGSYNCTEST:/usr/local/src$ sudo apt-get build-dep -y haproxy local@PGSYNCTEST:/usr/local/src$ sudo apt-get install -y libssl-dev local@PGSYNCTEST:/usr/local/src$ sudo apt-get install libpcre3-dev 3) Download HA-Proxy 1.5 local@PGSYNCTEST:/usr/local/src$ sudo wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.11.tar.gz local@PGSYNCTEST:/usr/local/src$ sudo tar xvf haproxy-1.5.11.tar.gz local@PGSYNCTEST:/usr/local/src$ cd haproxy-1.5.11 4) Installation local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY -DCONFIG_HAP_CRYPT -DUSE_ZLIB -DENABLE_POLL -DENABLE_EPOLL -DUSE_CPU_AFFINITY -DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX -DUSE_PCRE -I/usr/include -DCONFIG_HAPROXY_VERSION=\"1.5.11\" -DCONFIG_HAPROXY_DATE=\"2015/01/31\" \ -DBUILD_TARGET='"linux2628"' \ -DBUILD_ARCH='""' \ -DBUILD_CPU='"generic"' \ -DBUILD_CC='"gcc"' \ -DBUILD_CFLAGS='"-O2 -g -fno-strict-aliasing"' \ -DBUILD_OPTIONS='"USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1"' \ -c -o src/haproxy.o src/haproxy.c gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY -DCONFIG_HAP_CRYPT -DUSE_ZLIB -DENABLE_POLL -DENABLE_EPOLL -DUSE_CPU_AFFINITY -DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX -DUSE_PCRE -I/usr/include -DCONFIG_HAPROXY_VERSION=\"1.5.11\" -DCONFIG_HAPROXY_DATE=\"2015/01/31\" -c -o src/sessionhash.o src/sessionhash.c gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY -DCONFIG_HAP_CRYPT -DUSE_ZLIB -DENABLE_POLL -DENABLE_EPOLL -DUSE_CPU_AFFINITY -DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX -DUSE_PCRE -I/usr/include -DCONFIG_HAPROXY_VERSION=\"1.5.11\" -DCONFIG_HAPROXY_DATE=\"2015/01/31\" -c -o src/base64.o src/base64.c local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo make install install -d "/usr/local/sbin" install haproxy "/usr/local/sbin" install haproxy-systemd-wrapper "/usr/local/sbi
Re: Attached: May you support me for this issue logs of HAProxy
Dear Sir, I did create /var/lib/haproxy/var/log, and the same issue still remaining, Ha. - Original Message - From: "Thrawn" To: haproxy@formilux.org Sent: Thursday, March 19, 2015 11:59:55 PM Subject: Re: Attached: May you support me for this issue logs of HAProxy Perhaps the trouble is that you're putting HAProxy inside a chroot? Does /var/lib/haproxy/var/log exist?
Re[4]: Checked: send/accept-proxy over unix socket not working
Dear Mr Huuskonen, I would like to return the haproxy.cfg as it was, the error now is local@PGSYNCTEST:~$ sudo -u haproxy /etc/init.d/haproxy restart * Restarting haproxy haproxy [WARNING] 078/171401 (1267) : [/usr/local/sbin/haproxy.main()] Cannot raise FD limit to 90034. [ALERT] 078/171401 (1267) : Starting frontend GLOBAL: error when trying to preserve previous UNIX socket [/var/run/haproxy.sock] [ALERT] 078/171401 (1267) : Starting frontend pgdbplatform_frontend_cluster01: cannot bind socket [0.0.0.0:5432] Other tests local@PGSYNCTEST:~$ sudo service rsyslog status rsyslog start/running Hence rsyslog is running well But local@PGSYNCTEST:~$ sudo netstat -aun | grep 514 (nothing output) local@PGSYNCTEST:~$ sudo netstat -napt | grep 514 (nothing output) local@PGSYNCTEST:~$ logger -d -n 127.0.0.1 −p local0.info "Test message" I open /var/log/syslog and we see it here Mar 20 17:06:50 PGSYNCTEST logger: −p local0.info Test message I would like to send you the haproxy.cfg as global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 45000 # Total Max Connections. This is dependent on your limit user haproxy group haproxy daemon chroot /var/lib/haproxy pidfile /var/run/haproxy.pid tune.ssl.default-dh-param 2048 stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m defaults log global mode tcp option tcplog #option dontlognull option redispatch retries 3 timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout check 10s maxconn 2000 frontend pgdbplatform_frontend_cluster01 bind *:5432 mode tcp timeout client 168h option tcplog option logasap default_backend pgdbplatform_backend_cluster01 backend pgdbplatform_backend_cluster01 mode tcp option tcplog balance roundrobin # Load Balancing algorithm timeout server 168h server Primary 10.236.49.12:5432 check server HotStandby 172.24.22.68:5432 check server AsyncStandby 10.236.134.187:5432 check --- Please support, Quan Ha Le. - Original Message - From: "Jarno Huuskonen" To: "Ha Quan Le" Cc: haproxy@formilux.org Sent: Friday, March 20, 2015 12:57:46 PM Subject: Re: Checked: send/accept-proxy over unix socket not working Hi, On Fri, Mar 20, Ha Quan Le wrote: > global > log /var/log local0 > log /var/log local1 notice Are you sure that /var/log is a syslog socket ? Usually it's a directory so that's not going to work. http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-log > frontend pgdbplatform_frontend_cluster01 > #bind *:5432 > bind /var/lib/haproxy/test accept-proxy user haproxy group haproxy > mode tcp > timeout client 168h > option tcplog > option logasap > default_backend pgdbplatform_backend_cluster01 > > backend pgdbplatform_backend_cluster01 > mode tcp > option tcplog > balance roundrobin # Load Balancing algorithm > timeout server 168h > #server Primary 10.236.49.12:5432 check > #server HotStandby 172.24.22.68:5432 check > #server AsyncStandby 10.236.134.187:5432 check > server clear /var/lib/haproxy/test send-proxy user haproxy group haproxy What are you trying to do ? Now you've tried to create a loop: frontend pgdbplatform_frontend_cluster01 listens on the unix socket and backend pgdbplatform_backend_cluster01 tries to send the traffic back over the same socket --> loop ? In another thread (http://marc.info/?l=haproxy&m=142681568229626&w=2) your problem was that rsyslog didn't create haproxy logfiles ? Is your rsyslog listening for udp/port 514 (netstat -aun | grep 514) ? Does your logger command support -n option to send udp packets ? (man logger) if it does then try to send log messages logger -d -n 127.0.0.1 −p local0.info "Test message" and see if you can get rsyslog to accept the log messages. -Jarno -- Jarno Huuskonen
Re: Checked: send/accept-proxy over unix socket not working
> And/or do you have chroot in haproxy.cfg ? Yes, may I send you our haproxy.cfg now global log /var/log local0 log /var/log local1 notice maxconn 45000 # Total Max Connections. This is dependent on your limit user haproxy group haproxy daemon #nbproc 4 # Number of processors, quad-core by default chroot /var/lib/haproxy pidfile /var/run/haproxy.pid tune.ssl.default-dh-param 2048 stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m defaults log global mode tcp option tcplog #option dontlognull option redispatch retries 3 timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout check 10s maxconn 2000 frontend pgdbplatform_frontend_cluster01 #bind *:5432 bind /var/lib/haproxy/test accept-proxy user haproxy group haproxy mode tcp timeout client 168h option tcplog option logasap default_backend pgdbplatform_backend_cluster01 backend pgdbplatform_backend_cluster01 mode tcp option tcplog balance roundrobin # Load Balancing algorithm timeout server 168h #server Primary 10.236.49.12:5432 check #server HotStandby 172.24.22.68:5432 check #server AsyncStandby 10.236.134.187:5432 check server clear /var/lib/haproxy/test send-proxy user haproxy group haproxy Thank you, Ha. - Original Message - From: "Ha Quan Le" To: denni...@conversis.de Cc: haproxy@formilux.org Sent: Friday, March 20, 2015 11:37:38 AM Subject: Checked: send/accept-proxy over unix socket not working Dear Mr Jacobfeuerborn, May I reply to you, may you check > What version of haproxy are you using ? (And what OS) ? HA-Proxy 1.5.11 and Ubuntu 14.04 with Postgres 9.3 >>>>> Are you able to connect to the /var/lib/haproxy/test socket with >>>>> netcat or socat ? And/or do you have chroot in haproxy.cfg ? local@PGSYNCTEST:~$ sudo netcat localhost < /var/lib/haproxy/test -bash: /var/lib/haproxy/test: Permission denied local@PGSYNCTEST:~$ sudo -u haproxy netcat localhost < /var/lib/haproxy/test -bash: /var/lib/haproxy/test: Permission denied > > To fix your issue, simply update your bind line: >> bind > > /var/lib/haproxy/test accept-proxy user haproxy group haproxy >> >> Same on server line: >> server clear /var/lib/haproxy/test send-proxy user haproxy group haproxy local@PGSYNCTEST:~$ sudo -u haproxy /etc/init.d/haproxy restart * Restarting haproxy haproxy [ALERT] 078/153401 (25721) : parsing [/etc/haproxy/haproxy.cfg:46] : 'server clear' unknown keyword 'user'. Registered keywords : [ ALL] id [ SSL] ca-file [ SSL] check-ssl [ SSL] ciphers [ SSL] crl-file [ SSL] crt [ SSL] force-sslv3 [ SSL] force-tlsv10 [ SSL] force-tlsv11 [ SSL] force-tlsv12 [ SSL] no-sslv3 [ SSL] no-tlsv10 [ SSL] no-tlsv11 [ SSL] no-tlsv12 [ SSL] no-tls-tickets [ SSL] send-proxy-v2-ssl [ SSL] send-proxy-v2-ssl-cn [ SSL] ssl [ SSL] verify [ SSL] verifyhost [ALERT] 078/153401 (25721) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 078/153401 (25721) : Fatal errors found in configuration. After that I removed the line server clear /var/lib/haproxy/test send-proxy user haproxy group haproxy, then returning back to my own server lines Then like this local@PGSYNCTEST:~$ sudo -u haproxy /etc/init.d/haproxy restart * Restarting haproxy haproxy [WARNING] 078/152556 (25673) : [/usr/local/sbin/haproxy.main()] Cannot raise FD limit to 90034. [ALERT] 078/152556 (25673) : Starting frontend GLOBAL: error when trying to preserve previous UNIX socket [/var/run/haproxy.sock] [ALERT] 078/152556 (25673) : Starting frontend pgdbplatform_frontend_cluster01: cannot change UNIX socket ownership [/var/lib/haproxy/test] [ALERT] 078/152556 (25673) : sendto logger #1 failed: Permission denied (errno=13) [ALERT] 078/152556 (25673) : sendto logger #2 failed: Permission denied (errno=13) Therefore may you observe and advise me, Ha. IT Manager of Canwrx and Anvy Digital Companies. - Original Message - From: "Dennis Jacobfeuerborn" To: haproxy@formilux.org Sent: Friday, March 20, 2015 8:23:50 AM Subject: Re: send/accept-proxy over unix socket not working On 18.03.2015 13:53, Baptiste wrote: > On Wed, Mar 18, 2015 at 1:07 PM, Lukas Tribus wrote: >> >> >> >>> Date: Wed, 18 Mar 2015 01:49:47 +0100 >>> From: denni...@conversis.de >>> To: luky...@hotmail.com; jarno.huusko...@uef.fi >>> CC: haproxy@formilux.org >>> Subject: Re: send/accept-proxy over unix socket not working >>> >>> On 13.03.2015 18:44, Lukas Tribus wrote: >>>>> What version of haproxy are you using ? (And what OS) ? >>>&g
Checked: send/accept-proxy over unix socket not working
Dear Mr Jacobfeuerborn, May I reply to you, may you check > What version of haproxy are you using ? (And what OS) ? HA-Proxy 1.5.11 and Ubuntu 14.04 with Postgres 9.3 > Are you able to connect to the /var/lib/haproxy/test socket with > netcat or socat ? And/or do you have chroot in haproxy.cfg ? local@PGSYNCTEST:~$ sudo netcat localhost < /var/lib/haproxy/test -bash: /var/lib/haproxy/test: Permission denied local@PGSYNCTEST:~$ sudo -u haproxy netcat localhost < /var/lib/haproxy/test -bash: /var/lib/haproxy/test: Permission denied > > To fix your issue, simply update your bind line: >> bind > > /var/lib/haproxy/test accept-proxy user haproxy group haproxy >> >> Same on server line: >> server clear /var/lib/haproxy/test send-proxy user haproxy group haproxy local@PGSYNCTEST:~$ sudo -u haproxy /etc/init.d/haproxy restart * Restarting haproxy haproxy [ALERT] 078/153401 (25721) : parsing [/etc/haproxy/haproxy.cfg:46] : 'server clear' unknown keyword 'user'. Registered keywords : [ ALL] id [ SSL] ca-file [ SSL] check-ssl [ SSL] ciphers [ SSL] crl-file [ SSL] crt [ SSL] force-sslv3 [ SSL] force-tlsv10 [ SSL] force-tlsv11 [ SSL] force-tlsv12 [ SSL] no-sslv3 [ SSL] no-tlsv10 [ SSL] no-tlsv11 [ SSL] no-tlsv12 [ SSL] no-tls-tickets [ SSL] send-proxy-v2-ssl [ SSL] send-proxy-v2-ssl-cn [ SSL] ssl [ SSL] verify [ SSL] verifyhost [ALERT] 078/153401 (25721) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 078/153401 (25721) : Fatal errors found in configuration. After that I removed the line server clear /var/lib/haproxy/test send-proxy user haproxy group haproxy, then returning back to my own server lines Then like this local@PGSYNCTEST:~$ sudo -u haproxy /etc/init.d/haproxy restart * Restarting haproxy haproxy [WARNING] 078/152556 (25673) : [/usr/local/sbin/haproxy.main()] Cannot raise FD limit to 90034. [ALERT] 078/152556 (25673) : Starting frontend GLOBAL: error when trying to preserve previous UNIX socket [/var/run/haproxy.sock] [ALERT] 078/152556 (25673) : Starting frontend pgdbplatform_frontend_cluster01: cannot change UNIX socket ownership [/var/lib/haproxy/test] [ALERT] 078/152556 (25673) : sendto logger #1 failed: Permission denied (errno=13) [ALERT] 078/152556 (25673) : sendto logger #2 failed: Permission denied (errno=13) Therefore may you observe and advise me, Ha. IT Manager of Canwrx and Anvy Digital Companies. - Original Message - From: "Dennis Jacobfeuerborn" To: haproxy@formilux.org Sent: Friday, March 20, 2015 8:23:50 AM Subject: Re: send/accept-proxy over unix socket not working On 18.03.2015 13:53, Baptiste wrote: > On Wed, Mar 18, 2015 at 1:07 PM, Lukas Tribus wrote: >> >> >> >>> Date: Wed, 18 Mar 2015 01:49:47 +0100 >>> From: denni...@conversis.de >>> To: luky...@hotmail.com; jarno.huusko...@uef.fi >>> CC: haproxy@formilux.org >>> Subject: Re: send/accept-proxy over unix socket not working >>> >>> On 13.03.2015 18:44, Lukas Tribus wrote: > What version of haproxy are you using ? (And what OS) ? > >> In the first frontend I set: >> server clear /var/lib/haproxy/test send-proxy >> >> In the second frontend I set: >> bind /var/lib/haproxy/test accept-proxy > > Are you able to connect to the /var/lib/haproxy/test socket with > netcat or socat ? And/or do you have chroot in haproxy.cfg ? Also if you drop privileges, check permission with the haproxy user. If supported by your kernel, you could use abstract namespaces instead. >>> >>> According to the documentation abstract namespaces are not recommended >>> when using nbproc> 1. The reason I'm dealing with unix sockets at all >>> is that I want to get around the problem of losing the stick table >>> content on reload I posted about in another mail. The idea is to run two >>> instances. One with nbproc> 1 for ssl offloading and that forwards the >>> requests to the second instance that is using nbproc = 1 and contains >>> the http frontend and a backend. In theory this should allow me to >>> reload the config of the backend instance without losing the stick table >>> content. >>> >>> I'm using chroot /var/lib/haproxy but the behavior is the same without >>> this directive. Either way a socket gets created as >>> /var/lib/haproxy/test as intended but for some reason I keep getting 503 >>> when using a unix socket but everything works fine when using abstract >>> namespaces or an ip address. >>> >>> I've attached the configuration and the debug output in case that helps >>> to pinpoint the issue. >> >> Comment user and group and run haproxy as root. If thats works, it means >> you have a permission problem. >> >> >> Lukas >> >> >> > > Hi > > He has a permission problem! > > That's what I mentionned with the user parameter on the bind line.. > > Actual
Re: send/accept-proxy over unix socket not working
Dear Mr Jacobfeuerborn , I would like to appreciate, I am trying to follow your instructions our my system and will do the tests right away. > What version of haproxy are you using ? (And what OS) ? HA-Proxy 1.5.11 and Ubuntu 14.04 with Postgres 9.3 You bet, Ha. - Original Message - From: "Dennis Jacobfeuerborn" To: haproxy@formilux.org Sent: Friday, March 20, 2015 8:23:50 AM Subject: Re: send/accept-proxy over unix socket not working On 18.03.2015 13:53, Baptiste wrote: > On Wed, Mar 18, 2015 at 1:07 PM, Lukas Tribus wrote: >> >> >> >>> Date: Wed, 18 Mar 2015 01:49:47 +0100 >>> From: denni...@conversis.de >>> To: luky...@hotmail.com; jarno.huusko...@uef.fi >>> CC: haproxy@formilux.org >>> Subject: Re: send/accept-proxy over unix socket not working >>> >>> On 13.03.2015 18:44, Lukas Tribus wrote: > What version of haproxy are you using ? (And what OS) ? > >> In the first frontend I set: >> server clear /var/lib/haproxy/test send-proxy >> >> In the second frontend I set: >> bind /var/lib/haproxy/test accept-proxy > > Are you able to connect to the /var/lib/haproxy/test socket with > netcat or socat ? And/or do you have chroot in haproxy.cfg ? Also if you drop privileges, check permission with the haproxy user. If supported by your kernel, you could use abstract namespaces instead. >>> >>> According to the documentation abstract namespaces are not recommended >>> when using nbproc> 1. The reason I'm dealing with unix sockets at all >>> is that I want to get around the problem of losing the stick table >>> content on reload I posted about in another mail. The idea is to run two >>> instances. One with nbproc> 1 for ssl offloading and that forwards the >>> requests to the second instance that is using nbproc = 1 and contains >>> the http frontend and a backend. In theory this should allow me to >>> reload the config of the backend instance without losing the stick table >>> content. >>> >>> I'm using chroot /var/lib/haproxy but the behavior is the same without >>> this directive. Either way a socket gets created as >>> /var/lib/haproxy/test as intended but for some reason I keep getting 503 >>> when using a unix socket but everything works fine when using abstract >>> namespaces or an ip address. >>> >>> I've attached the configuration and the debug output in case that helps >>> to pinpoint the issue. >> >> Comment user and group and run haproxy as root. If thats works, it means >> you have a permission problem. >> >> >> Lukas >> >> >> > > Hi > > He has a permission problem! > > That's what I mentionned with the user parameter on the bind line.. > > Actually, HAProxy starts up as root and create the socket with root > user, then it drops it switches to user haproxy, group haproxy > (according to your conf). > This user is not allowed to access the socket, since there is no write > allowed for "others". > > To fix your issue, simply update your bind line: > bind /var/lib/haproxy/test accept-proxy user haproxy group haproxy > > Same on server line: > server clear /var/lib/haproxy/test send-proxy user haproxy group haproxy > > > Hope this helps. Yes, that did the trick, thanks! Now I'm wondering if I can stick with the abstract namespace after all though. The Documentation has this to say: "Note: since abstract sockets are not "rebindable", they do not cope well with multi-process mode during soft-restart, so it is better to avoid them if nbproc is greater than 1. The effect is that if the new process fails to start, only one of the old ones will be able to rebind to the socket." Since the SSL offloading instance with nbproc > 1 only addresses the abstract namespace in the server directive and the second instance that does the bind in the abstract namespace is running with nbproc = 1 would it be correct to assume that in this specific case the rebinding is not an issue since it is only the nbproc=1 instance that is doing the actual "bind" call? If that is correct then I could keep using the abstract namespace and have no need for the traditional unix sockets. Regards, Dennis
Attached: May you support me for this issue logs of HAProxy
Attached! Dear HA-Proxy support, I am installing HA-Proxy 1.5 to make load balancing for my 3 Ubuntu servers (version 14.04) using with Postgresql database Primary 10.236.49.12:5432 HotStandby 172.24.22.68:5432 AsyncStandby 10.236.134.187:5432 The HA-Proxy is installed on the Primary only, and not on the HotStandby neither on the AsyncStandby. After I have done all of the steps below, (you can also download the scripting file from this email attachment); But however, I still do not have the log files /var/log/haproxy_0.log and /var/log/haproxy_1.log (the files are not created), and also not logs of haproxy found inside /var/log/syslog May you check, here is all the steps that I did. My Ha-Proxy does not have any logs. I would like to look forward to hearing from you, Sincerely, Quan Ha Le, IT Manager of Canwrx IT Manager of Anvy Digital, Calgary, Alberta, Canada. PS: May you make it urgently, because I tried to fix unsuccessfully for two days already. GOOD HAProxy 1.5 -- 1) Removal of old HA-Proxy1.4 $ sudo apt-get remove haproxy 2) Preparation $ cd /usr/local/src/ local@PGSYNCTEST:/usr/local/src$ sudo apt-get install build-essential local@PGSYNCTEST:/usr/local/src$ sudo apt-get build-dep -y haproxy local@PGSYNCTEST:/usr/local/src$ sudo apt-get install -y libssl-dev local@PGSYNCTEST:/usr/local/src$ sudo apt-get install libpcre3-dev 3) Download HA-Proxy 1.5 local@PGSYNCTEST:/usr/local/src$ sudo wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.11.tar.gz local@PGSYNCTEST:/usr/local/src$ sudo tar xvf haproxy-1.5.11.tar.gz local@PGSYNCTEST:/usr/local/src$ cd haproxy-1.5.11 4) Installation local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY -DCONFIG_HAP_CRYPT -DUSE_ZLIB -DENABLE_POLL -DENABLE_EPOLL -DUSE_CPU_AFFINITY -DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX -DUSE_PCRE -I/usr/include -DCONFIG_HAPROXY_VERSION=\"1.5.11\" -DCONFIG_HAPROXY_DATE=\"2015/01/31\" \ -DBUILD_TARGET='"linux2628"' \ -DBUILD_ARCH='""' \ -DBUILD_CPU='"generic"' \ -DBUILD_CC='"gcc"' \ -DBUILD_CFLAGS='"-O2 -g -fno-strict-aliasing"' \ -DBUILD_OPTIONS='"USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1"' \ -c -o src/haproxy.o src/haproxy.c gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY -DCONFIG_HAP_CRYPT -DUSE_ZLIB -DENABLE_POLL -DENABLE_EPOLL -DUSE_CPU_AFFINITY -DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX -DUSE_PCRE -I/usr/include -DCONFIG_HAPROXY_VERSION=\"1.5.11\" -DCONFIG_HAPROXY_DATE=\"2015/01/31\" -c -o src/sessionhash.o src/sessionhash.c gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY -DCONFIG_HAP_CRYPT -DUSE_ZLIB -DENABLE_POLL -DENABLE_EPOLL -DUSE_CPU_AFFINITY -DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX -DUSE_PCRE -I/usr/include -DCONFIG_HAPROXY_VERSION=\"1.5.11\" -DCONFIG_HAPROXY_DATE=\"2015/01/31\" -c -o src/base64.o src/base64.c local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo make install install -d "/usr/local/sbin" install haproxy "/usr/local/sbin" install haproxy-systemd-wrapper "/usr/local/sbin" install -d "/usr/local/share/man"/man1 install -m 644 doc/haproxy.1 "/usr/local/share/man"/man1 install -d "/usr/local/doc/haproxy" for x in configuration architecture haproxy-en haproxy-fr; do \ install -m 644 doc/$x.txt "/usr/local/doc/haproxy" ; \ done 5) Service start local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo vi /etc/init.d/haproxy HAPROXY=/usr/local/sbin/haproxy local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo mkdir /var/lib/haproxy local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo chown -R haproxy:haproxy /var/lib/haproxy local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo /etc/init.d/haproxy start * Starting haproxy haproxy [ OK ] 6) Checking if HA-Proxy 1.5 is running local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo /etc/init.d/haproxy status haproxy is running. 7) Re-configuration of HA-Proxy local@PGSYNCTEST:/usr/local/src/haproxy-1.5.11$ sudo vi /etc/haproxy/haproxy.cfg global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 45000 # Total Max Connections. This is dependent on your limit user haproxy group haproxy daemon nbproc 4 # Number of processors, quad-core by default chroot /var/lib/haproxy pidfile /var/run/haproxy.pid tune.ssl.default-dh-param 2048 defaults log global mode tcp option tcplog #option dontlognull option redispatch retries 3 timeout queue 1m timeout connect 10s timeout client 1m timeou