unsubscribe
unsubscribe
Bytes in / out counters for TCP Keepalive Sessions
Hello! We have just placed haproxy (1.5) in front of our RabbitMQ servers. The statistics show no change in bytes in / bytes out until a connection is closed — which in Rabbit’s case should be virtually never. The configuration looks like this: frontend fe_rabbitmq bind 192.168.205.7:5672 timeout client 3h mode tcp option clitcpka default_backend be_rabbitmq backend be_rabbitmq mode tcp timeout server 3h server app-m-03 app-m-03:5672 check on-marked-down shutdown-sessions server app-m-02 app-m-02:5672 check on-marked-down shutdown-sessions server app-m-01 app-m-01:5672 check on-marked-down shutdown-sessions Is this the expected behavior? If so, is there any configuration option we can change to show “live” stats of bytes flowing through the persistent connections? Thanks! Daniel -- Daniel Schneller Principal Cloud Engineer CenterDevice GmbH | Merscheider Straße 1 | 42699 Solingen tel: +49 1754155711| Deutschland daniel.schnel...@centerdevice.de | www.centerdevice.de
Haproxy 1.6.9 failed to compile regex
Hi, I tried to upgrade from 1.6.8 to 1.6.9, but found strange errors printed by haproxy 1.6.9. Any ideas, why? [ALERT] 250/112901 (12026) : parsing [/etc/haproxy/haproxy.cfg:57] : 'reqirep' : regular expression '^([^ :]*) /(.*)' : failed to compile regex '^([^ :]*) /(.*)' (error=unknown or incorrect option bit(s) set) [ALERT] 250/112901 (12026) : parsing [/etc/haproxy/haproxy.cfg:205] : 'reqidel' : regular expression '^If-Match:.*' : failed to compile regex '^If-Match:.*' (error=unknown or incorrect option bit(s) set) [ALERT] 250/112901 (12026) : parsing [/etc/haproxy/haproxy.cfg:279] : 'rspidel' : regular expression '^Content-Location' : failed to compile regex '^Content-Location' (error=unknown or incorrect option bit(s) set) Downgrading to 1.6.8 solves this error. # haproxy -vv HA-Proxy version 1.6.9 2016/08/30 Copyright 2000-2016 Willy Tarreau Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.7 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.0-fips 29 Mar 2010 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built without Lua support Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. # haproxy -vv HA-Proxy version 1.6.8 2016/08/14 Copyright 2000-2016 Willy Tarreau Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built without Lua support Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Veiko
Incorrect counters in stats interface
Hello, (sorry for reposting, but I do not see my e-mail in ML archive, so I assume it was blocked due to screenshots in attachments. I replace them with links now). I am using haproxy-1.6.9. In web stats interface, I mouse-over backend's Total Sessions counter (1728 in my case), and I see HTTP 5xx responses=46 (see screenshot: https://people.freebsd.org/~demon/scr1.png) Then I mouse-over each server's Total sessions counter and none has positive number of HTTP 5xx responses (see second screenshot: https://people.freebsd.org/~demon/scr2.png). Is it a bug or I misunderstand these counters? Thanks!
Re: Haproxy 1.6.9 failed to compile regex
On 07/09/16 14:37, Veiko Kukk wrote: I tried to upgrade from 1.6.8 to 1.6.9, but found strange errors printed by haproxy 1.6.9. Any ideas, why? Another strange issue is that 1.6.9 shows: Running on OpenSSL version : OpenSSL 1.0.0-fips 29 Mar 2010 System does have openssl 1.0.1e-48.el6_8.1 installed and nothing else. So how is it possible that it's using different version than system has? On the other hand - 1.6.8 reports proper openssl version: Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Veiko
how can i get pass original ip in tcp mode
HI haproxy: My haproxy version is 1.6. And I use haproxy before my game_server on tcp mode Client on A(172.16.77.32) HaProxy and game_server on B (172.16.77.37) Config file is: [cid:image001.png@01D20959.A98C01B0] When I use client connect to game_server through haproxy , everything is ok, but in my game_server I find the connection remote ip is 172.16.77.37 not 172.16.77.32 The question is how can I get original ip though haproxy on tcp mode Best Regards Long Ma = 百纳(武汉)信息技术有限公司 游戏后台开发 马龙 TEL:18616693720 QQ:363419410 ADD:武汉市光谷大道77号光谷金融港A2栋2楼
Re: Haproxy 1.6.9 failed to compile regex
Am 07.09.2016 um 16:42 schrieb Veiko Kukk: On 07/09/16 14:37, Veiko Kukk wrote: I tried to upgrade from 1.6.8 to 1.6.9, but found strange errors printed by haproxy 1.6.9. Any ideas, why? Another strange issue is that 1.6.9 shows: Running on OpenSSL version : OpenSSL 1.0.0-fips 29 Mar 2010 System does have openssl 1.0.1e-48.el6_8.1 installed and nothing else. So how is it possible that it's using different version than system has? Do you compile on the same box were the executable runs? Looks like you have a mess with your system libraries. Compare "ldd haproxy" (working and non-working executable). You will probably see that it points to a different path (both pcre and openssl). Lukas
Re: Haproxy 1.6.9 failed to compile regex
❦ 7 septembre 2016 16:42 CEST, Veiko Kukk : >> I tried to upgrade from 1.6.8 to 1.6.9, but found strange errors printed >> by haproxy 1.6.9. Any ideas, why? > > Another strange issue is that 1.6.9 shows: > Running on OpenSSL version : OpenSSL 1.0.0-fips 29 Mar 2010 > > System does have openssl 1.0.1e-48.el6_8.1 installed and nothing > else. So how is it possible that it's using different version than > system has? It could be statically compiled. Check with ldd. -- Too much is just enough. -- Mark Twain, on whiskey
Re: Need help to configure ha proxy
Hi, Will you please help me in configuration on HAPROXY. Example - api.example.com server api01 10.0.0.10:80 check server api02 10.0.0.11:80 check Requirement - 10 Backend server and every backend with host name and 2 server under backend with roundrobin. Now issue is if we deploy on Prod, with jenkins, once deploy tomcat/apache will restart and use another server. During restart request fails those send on server 01 because this server goes under restart. How to handel that. AWS we do with ELB, before deployment server take out from elb then deploy then attach and make the inservice. Regard's Harish Chander 8529142143 * From: Jeff Palmer Sent: Tuesday, August 30, 2016 7:05 PM To: Harish Chander Cc: haproxy@formilux.org Subject: Re: Need help to configure ha proxy This config appears to be a decent start. and looks to meet your requirements for http. Now you just need another frontend configured for 443, it would match the :80 frontend, aside from port, using SSL, and a path to the certificates. On Tue, Aug 30, 2016 at 8:47 AM, Harish Chander wrote: > Hi, > > > I shall be really thankful you if you help in configure haproxy or its > possible or not. > > > External ELB - In external AWS ELB i have 2 Ha proxy server > > > HA Proxy > > connect > > haproxy > beta.example.com > > beta.example.com > api-example.com > > > beta.example.com server work's on 80 and 443 both, If i add A Name in DNS of > direct server IP then work everything. > > > Requirement - beta.example.com should work on both 443 and 80. now its > working for 80 only. Please help me out. you can call me +918529142143 any > time. > > > Current haproxy conf under below > > > > haproxy.conf > > > global > > log /dev/log local0 > > log /dev/log local1 notice > > chroot /var/lib/haproxy > > stats socket /run/haproxy/admin.sock mode 660 level admin > > stats timeout 30s > > user haproxy > > group haproxy > > daemon > > > # Default SSL material locations > > ca-base /etc/ssl/certs > > crt-base /etc/ssl/private > > > # Default ciphers to use on SSL-enabled listening sockets. > > # For more information, see ciphers(1SSL). This list is from: > > # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ > > ssl-default-bind-ciphers > ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS > > ssl-default-bind-options no-sslv3 > > > defaults > > log global > > mode http > > option httplog > > option dontlognull > > timeout connect 5000 > > timeout client 5 > > timeout server 5 > > > frontend haproxy > >bind *:80 > >stats uri /stats > >stats realm Strictly\ Private > >stats auth pass:word > > > # Define hosts > > #urls > > acl beta.example hdr(host) -i beta.example.com > > > > acl api.example hdr(host) -i api-example.com > > > > #cluster > > use_backend b.example if beta.example > > > use_backend z.api if api.example > > > #Frontend Server > > > backend b.example > > mode http > > balance roundrobin > > option forwardfor > >server server01 10.0.0.1:80 check > > > ##API > > backend z.api > > mode http > > balance roundrobin > > option forwardfor > > server api01 192.168.1.1:80 check > > > > Regard's > Harish Chander > 8529142143 > > -- Jeff Palmer https://PalmerIT.net
Re: Bytes in / out counters for TCP Keepalive Sessions
Hi Daniel, Op 7-9-2016 om 12:31 schreef Daniel Schneller: We have just placed haproxy (1.5) in front of our RabbitMQ servers. Why not use 1.6 ? its been released for a while.. The statistics show no change in bytes in / bytes out until a connection is closed — which in Rabbit’s case should be virtually never. Is this the expected behavior? If so, is there any configuration option we can change to show “live” stats of bytes flowing through the persistent connections? Thanks! Daniel http://cbonte.github.io/haproxy-dconv/1.7/snapshot/configuration.html#4.2-option%20contstats Above option sounds like you want it :) Regards, PiBa-NL
Re: Incorrect counters in stats interface
Hi Dmitry, Op 7-9-2016 om 15:54 schreef Dmitry Sivachenko: Hello, (sorry for reposting, but I do not see my e-mail in ML archive, so I assume it was blocked due to screenshots in attachments. I replace them with links now). I am using haproxy-1.6.9. In web stats interface, I mouse-over backend's Total Sessions counter (1728 in my case), and I see HTTP 5xx responses=46 (see screenshot: https://people.freebsd.org/~demon/scr1.png) Then I mouse-over each server's Total sessions counter and none has positive number of HTTP 5xx responses (see second screenshot: https://people.freebsd.org/~demon/scr2.png). Is it a bug or I misunderstand these counters? Thanks! In a case if all servers are down (or very busy). A request could be queued and then timeout, so haproxy itself will return for example a 503, while none of the servers ever returned anything for that specific request. I'm not saying this is the exact scenario you see, but it might explain it.. Regards, PiBa-NL
Re: how can i get pass original ip in tcp mode
Greetings, On 09/07/2016 10:48 AM, Long Ma wrote: HI haproxy: My haproxy version is 1.6. And I use haproxy before my game_server on tcp mode Client on A(172.16.77.32) HaProxy and game_server on B (172.16.77.37) Config file is: When I use client connect to game_server through haproxy , everything is ok, but in my game_server I find the connection remote ip is 172.16.77.37 not 172.16.77.32 The question is how can I get original ip though haproxy on tcp mode I see your using send-proxy there; does the software in question understand the PROXY protocol (http://www.haproxy.org/download/1.6/doc/proxy-protocol.txt)? If so, it should be able to use the client IP address it gets from that. If you don't want to go that route, you could use full transparent proxying mode (see https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4.2-source); but for that to work properly you will need to ensure your network setup works with that. - Chad Best Regards Long Ma = 百纳(武汉)信息技术有限公司 游戏后台开发马龙 TEL:18616693720 QQ:363419410 ADD:武汉市光谷大道77号光谷金融港A2栋2楼
Experimental patch: Consistent Hashing with Bounded Loads
Hello all, === Background material, skip this if you just want to know what I've done === For some time I've been wishing for a way to combine the best aspects of consistent hashing and "balance leastconn". I use haproxy to distribute traffic across a cluster of servers doing dynamic video delivery. Without going into too much detail, there's a substantial cache effect — there's a bunch of information that needs to be calculated for each video, and if that information is cached, then the request is quite a bit faster and uses less resources. Also, the server cluster runs in the cloud and auto-scales based on load. Initially our approach was to cache the information locally on each server, and use uri-based balancing with consistent hashing to distribute the traffic among servers according to video ID. This gave a satisfying number of cache hits, but had the problem that a single server could become overloaded by receiving the traffic for one or two exceptionally popular videos, resulting in bad performance. Our second whack was to change to "balance leastconn" (which destroys the cache hit ratio) and add a second layer of shared cache. This gives more consistent performance, but the bandwidth used for cache coherency is substantial, and the cache servers create a single point of failure. What I really wanted was a policy that uses consistent hashing most of the time, but at the same time, avoids overloading any given server, by diverting traffic from an overloaded server to a less-loaded server, in a consistent order based on the consistent hash ring. I did some experimenting with algorithms, but didn't find a satisfactory one. === End of background material, here's the paper and the patch === Recently I came upon the paper "Consistent Hashing with Bounded Loads", http://arxiv.org/abs/1608.01350 . It describes a very simple algorithm for doing exactly what I wanted. It defines a parameter c > 1, and enforces that no server receives more than ceil(c * avg(conns)) connections. If a server is already at the limit, it goes around the hash ring until it finds one that is below the limit, and assigns the connection to that server. A large "c" provides stricter hashing (a greater chance of requests with the same hash key going to the same server), while a small "c" provides stricter balancing (less difference between the smallest and largest number of connections to a server). I simulated this algorithm, found that it would work better than the ones I tested before, and then implemented it in HAProxy. On testing in production I saw reduced 95th-percentile response times, and a great reduction in cache-coherency traffic. I'm attaching the patch that I tested. However, I don't expect that it is in any way mergeable. I'm not very familiar with the HAProxy codebase and I did a number of things that are almost certainly wrong. I replaced the existing chash_get_server_hash instead of attempting to add a new method or flag to the configuration. The "c" parameter is hard-coded instead of configurable, as haproxy currently has no awareness of non-integer numbers, and I wasn't sure how to deal with that. My code for enumerating servers and assigning capacities is probably suboptimal, although it didn't cause a noticeable bump in CPU usage on my production systems. There are probably any number of coding-style gaffes. And there is at least one bug that causes haproxy to become unresponsive when servers are marked down. That being said, I would be thrilled to work with someone to repair all of these problems, if there is interest in having this feature in HAProxy. I think that it's an exciting feature and it would be really great to add to HAProxy's already excellent set of load-balancing tools. I just need a little guidance to make it happen. My changes are on Github at https://github.com/arodland/haproxy and they're up-to-date against haproxy master. I'd encourage anyone to review the code, and to contact me by email, on IRC (I'm on #haproxy as hobbs), or on GitHub. Thank you, Andrew
Re: Incorrect counters in stats interface
> On 07 Sep 2016, at 21:10, PiBa-NL wrote: > > Hi Dmitry, > Op 7-9-2016 om 15:54 schreef Dmitry Sivachenko: >> Hello, >> >> (sorry for reposting, but I do not see my e-mail in ML archive, so I assume >> it was blocked due to screenshots in attachments. I replace them with links >> now). >> >> I am using haproxy-1.6.9. >> >> In web stats interface, I mouse-over backend's Total Sessions counter (1728 >> in my case), and I see HTTP 5xx responses=46 >> (see screenshot: https://people.freebsd.org/~demon/scr1.png) >> >> Then I mouse-over each server's Total sessions counter and none has positive >> number of HTTP 5xx responses (see second screenshot: >> https://people.freebsd.org/~demon/scr2.png). >> >> Is it a bug or I misunderstand these counters? >> >> Thanks! > > In a case if all servers are down (or very busy). > > A request could be queued and then timeout, so haproxy itself will return for > example a 503, while none of the servers ever returned anything for that > specific request. > > I'm not saying this is the exact scenario you see, but it might explain it.. > In "Queue" section I have all zeroes in Cur and Max.
Re: Incorrect counters in stats interface
Queue Cur is a gauge and so not representative of historical values. Queue Max of zero is telling though. In addition to queue timeout, there are other ways haproxy can synthesize an http response on behalf of the backend server. Check for connection errors. On Wed, Sep 7, 2016 at 12:15 PM Dmitry Sivachenko wrote: > > > On 07 Sep 2016, at 21:10, PiBa-NL wrote: > > > > Hi Dmitry, > > Op 7-9-2016 om 15:54 schreef Dmitry Sivachenko: > >> Hello, > >> > >> (sorry for reposting, but I do not see my e-mail in ML archive, so I > assume it was blocked due to screenshots in attachments. I replace them > with links now). > >> > >> I am using haproxy-1.6.9. > >> > >> In web stats interface, I mouse-over backend's Total Sessions counter > (1728 in my case), and I see HTTP 5xx responses=46 > >> (see screenshot: https://people.freebsd.org/~demon/scr1.png) > >> > >> Then I mouse-over each server's Total sessions counter and none has > positive number of HTTP 5xx responses (see second screenshot: > https://people.freebsd.org/~demon/scr2.png). > >> > >> Is it a bug or I misunderstand these counters? > >> > >> Thanks! > > > > In a case if all servers are down (or very busy). > > > > A request could be queued and then timeout, so haproxy itself will > return for example a 503, while none of the servers ever returned anything > for that specific request. > > > > I'm not saying this is the exact scenario you see, but it might explain > it.. > > > > > In "Queue" section I have all zeroes in Cur and Max. > > >
actual state of http/2
Hi, Can somebody tell me whats the actual implementation state of http/2. I read that h2 is useable in tcp mode with alpn. Is sni in this combination possible, because we need to save public ip adresses? Is there any comparison in case of speed for http/1.1 and http/2 ? thanks for the help, cheers thomas