Re: Haproxy CPU 100%, after running about two weeks
Thanks! I follow your advise, and upgrade my haproxy. And I will observe if there is any problem. Regards Jinge On 2013-5-2, at 下午3:49, Lukas Tribus wrote: > Hi Jinge! > > > I believe you are facing 2 different issues here. > > > >> Today, our haproxy CPU grow to 100%. And the machine become terribly slow. > > Please upgrade to recent 1.4 code, you are missing a a few fixes, including > one a security fix. I suggest the snapshot 20130427 which also includes a > loop fix (causing 100% load from haproxy). Download at [1]. > > > >> [1297314.773541] cleanup rbuf bug: copied DBE7B6DA seq DBE7B3C8 rcvnxt >> DBE7B6DA >> [...] >> [1297314.773625] [] ? warn_slowpath_common+0x78/0x8c > > This is a kernel issue with tcp splicing and has probably been fixed. > Please see [2]. Not sure if Debian is backporting this fix though. > > You could just disable tcp splicing as a intermediate workaround. > > > > Cheers, > Lukas > > [1] http://haproxy.1wt.eu/download/1.4/src/snapshot/ > [2] http://comments.gmane.org/gmane.linux.network/231555 >
Re: Transparent TCP LoadBalancing on FreeBSD
Hi PiBa, i will try those patch to day, i will let you know when it done.. TIA On 5/3/13 12:16 AM, PiBa-NL wrote: Hi ZeN & Willy, To use transparent proxying on FreeBSD you currently need to compile with "USE_LINUX_TPROXY=yes". And make a few changes to the source code (else it wont compile). As a "quick and dirty fix" you could (manually?) apply this patch [1]: http://marc.info/?l=haproxy&m=136700170314757&w=2 For the better/cleaner fix this one should be usable [2]: http://marc.info/?l=haproxy&m=136707895800761&w=2 , which is what i would like to get committed to the main HAProxy source tree. @Willy could you take a look at the patch attached to that mail [2] ? Greets, PiBa-NL Op 2-5-2013 5:13, ZeN schreef: Dear Users, sorry if i open new thread, but i really want to solve this problem.. i manage to compile haproxy via port using TPROXY : haproxy -vv HA-Proxy version 1.5-dev18 2013/04/03 Copyright 2000-2013 Willy Tarreau Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -O2 -pipe -fno-strict-aliasing -DFREEBSD_PORTS OPTIONS = USE_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.7 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 0.9.8y 5 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes but when i started the service with the "source 0.0.0.0 usesrc clientip" option, the haproxy wont start with this messages: parsing [/usr/local/etc/haproxy.conf:28] : 'usesrc' not allowed here because support for TPROXY was not compiled in. what i should i do to make haproxy compile with transparent option? Rgds ZeN
redirects from haproxy
Configuration: haproxy 1.5 latest loadbalancing 2 backends. no stickiness. There are just too many 307 redirects from haproxy between the 2 backends even when the response is quick from the first backend. Why?
Re: Monitor always returns HTTP 200
On Thu, May 2, 2013 at 8:55 AM, James Bensley wrote: > acl backend_down nbsrv(http--servers) lt 2 # HAProxy can see > lee than 2 backend servers > monitor-uri /checkuri > monitor-net 172.22.0.0/24 What's the address of the computer making the requests? If it's in the 172.22.0.0/24 network, all responses for any URI will be 200 as long as "monitor fail" is false. -Bryan
Re: HAProxy on EC2 for high burst traffic
We use ELB on top of our haproxy clusters since in virtualized env. since you cant predict behavior of your instances. ELB will detect any failure in virutalized haproxy and manage them. The bigger instances you use, the less issues you will have, this is AWS support answer. 2013/5/2 Architecture Mercenary > I am running HAProxy 1.4.18 on a c1.medium EC2 ubuntu 12.10 instance and > have had very good success with it so far. However, I need to make it > capable of handling short-term bursts of 5,000 simultaneous requests. > These burst will typically last for 5-10 minutes max. I have been unable > to drive this much load through it so far. I've read mixed reviews about > running HAProxy on virtual instances and the performance penalties, but > does anyone have any suggestions for maximum performance tuning of haproxy > on EC2? Thanks. >
Re: HAProxy on EC2 for high burst traffic
Found this the other week, you might want to take a read: https://github.com/eucalyptus/architecture/blob/master/features/elb/3.3/elb-benchmark.wiki On Thu, May 2, 2013 at 2:08 PM, Architecture Mercenary < itarchm...@archmerc.com> wrote: > I am running HAProxy 1.4.18 on a c1.medium EC2 ubuntu 12.10 instance and > have had very good success with it so far. However, I need to make it > capable of handling short-term bursts of 5,000 simultaneous requests. > These burst will typically last for 5-10 minutes max. I have been unable > to drive this much load through it so far. I've read mixed reviews about > running HAProxy on virtual instances and the performance penalties, but > does anyone have any suggestions for maximum performance tuning of haproxy > on EC2? Thanks. > -- www.ariens.ca
HAProxy on EC2 for high burst traffic
I am running HAProxy 1.4.18 on a c1.medium EC2 ubuntu 12.10 instance and have had very good success with it so far. However, I need to make it capable of handling short-term bursts of 5,000 simultaneous requests. These burst will typically last for 5-10 minutes max. I have been unable to drive this much load through it so far. I've read mixed reviews about running HAProxy on virtual instances and the performance penalties, but does anyone have any suggestions for maximum performance tuning of haproxy on EC2? Thanks.
log and no log directives
Hi everyone, Not sure I'm getting the way that log works in HAProxy. Is there a way of telling a specific backend to not log stuff? I currently have a log global in the defaults section. I tried putting no log in that backend, but didn't work (still don't understand exactly no log). Is the only way to go removing log global from the defaults and have each backend specify it, and just skip it for the backend I don't want to log? Thanks, Pedro.
Re: Transparent TCP LoadBalancing on FreeBSD
Hi ZeN & Willy, To use transparent proxying on FreeBSD you currently need to compile with "USE_LINUX_TPROXY=yes". And make a few changes to the source code (else it wont compile). As a "quick and dirty fix" you could (manually?) apply this patch [1]: http://marc.info/?l=haproxy&m=136700170314757&w=2 For the better/cleaner fix this one should be usable [2]: http://marc.info/?l=haproxy&m=136707895800761&w=2 , which is what i would like to get committed to the main HAProxy source tree. @Willy could you take a look at the patch attached to that mail [2] ? Greets, PiBa-NL Op 2-5-2013 5:13, ZeN schreef: Dear Users, sorry if i open new thread, but i really want to solve this problem.. i manage to compile haproxy via port using TPROXY : haproxy -vv HA-Proxy version 1.5-dev18 2013/04/03 Copyright 2000-2013 Willy Tarreau Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -O2 -pipe -fno-strict-aliasing -DFREEBSD_PORTS OPTIONS = USE_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.7 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 0.9.8y 5 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes but when i started the service with the "source 0.0.0.0 usesrc clientip" option, the haproxy wont start with this messages: parsing [/usr/local/etc/haproxy.conf:28] : 'usesrc' not allowed here because support for TPROXY was not compiled in. what i should i do to make haproxy compile with transparent option? Rgds ZeN
RE: Monitor always returns HTTP 200
Hi James! (sorry for repost, first mail was accidentally html) > I always receive a HTTP 200 response to my browser How do you know that? Do you have a browser extension or are you using tcpdump/wireshark? Can you show this with a curl request, so that we understand whats going on excatly? Something like this should do the job: curl -vv http://1.2.3.4:80/nonexistingfile-bla-bla >/dev/null In what condition does this happen (when you have less than 2 backends alive or even with 2 or more backends alive?) > sends back HTTP 200 when at least 2 back end Nope, afaik haproxy should never generated 200 OK, expect on the stats page. When you see 200 OK, than that response is usually coming directly from the backend. > default_backend http--servers > [...] > backend http-servers ^ The config doesn't seem to match, there are 2 hyphens in the frontend configuration when referring to the backend, but the backend is named with a single hyphen. Can you please post the complete configuration (obfuscate confidential data, but leave everything else as is). Please post the output of haproxy -vv. In the end, either capturing the backend traffic with tcpdump or enabling the debug mode will show us what happens. Cheers, Lukas
RE: Monitor always returns HTTP 200
Hi James! > I always receive a HTTP 200 response to my browser How do you know that? Do you have a browser extension or are you using tcpdump/wireshark? Can you show this with a curl request, so that we understand whats going on excatly? Something like this should do the job: curl -vv http://1.2.3.4:80/nonexistingfile-bla-bla >/dev/null In what condition does this happen (when you have less than 2 backends alive or even with 2 or more backends alive?) > sends back HTTP 200 when at least 2 back end Nope, afaik haproxy should never generated 200 OK, expect on the stats page. When you see 200 OK, than that response is usually coming directly from the backend. > default_backend http--servers > [...] > backend http-servers ^ The config doesn't seem to match, there are 2 hyphens in the frontend configuration when referring to the backend, but the backend is named with a single hyphen. Can you please post the complete configuration (obfuscate confidential data, but leave everything else as is). Please post the output of haproxy -vv. In the end, either capturing the backend traffic with tcpdump or enabling the debug mode will show us what happens. Cheers, Lukas
Monitor always returns HTTP 200
Hi all, I have configured haproxy using the below configuration. No matter what URL I browser to I always receive a HTTP 200 response to my browser. If I comment out the ACL and three monitor lines from the frontend configuration, normal behaviour is resumed. I have that gut feeling that I have done something obviously wrong, but I can't spot it :) Where am I going wrong with this? I assume that haproxy captures requests to /checkuri and sends back HTTP 200 when at least 2 back end servers are up (which they are, I can see the requests coming into them) and 503 when one or more is down. Otherwise, all other requests are passed directly to the back end servers. It seems to be intercepting whatever URI I request via GET and returns HTTP 200 OK, nothing reaches the back end servers when the monitor URI is configured. frontend monitor-http-servers bind 1.2.3.4:80 acl backend_down nbsrv(http--servers) lt 2 # HAProxy can see lee than 2 backend servers monitor-uri /checkuri monitor-net 172.22.0.0/24 monitor fail if backend_down default_backend http--servers backend http-servers cookie cook insert option persist option redispatch server cook1 192.168.0.1:80 cookie c1 check inter 2000 rise 3 fall 3 server cook2 192.168.0.2:80 cookie c2 check inter 2000 rise 3 fall 3 balance roundrobin Cheers, James.
Re: Limit frontend bandwidth rate?
Hi, Baptiste, you may misunderstand, it's limit speed like at rate 1Mbps :) Bests, -Igor On Thu, May 2, 2013 at 2:10 PM, Baptiste wrote: > Hi, > > What you can do with 1.5 currently is using a stick table and monitor > bandwith per Host header for example. > Then if you go over a limit, you can redirect requests to an > overloaded explanation page. > > Baptiste > > > On Wed, May 1, 2013 at 8:40 PM, Igor wrote: > > Limit frontend bandwidth speed would be handy for some product > environment, > > is this still planned in 1.5 dev? > > > > Bests, > > -Igor >
RE: Haproxy CPU 100%, after running about two weeks
Hi Jinge! I believe you are facing 2 different issues here. > Today, our haproxy CPU grow to 100%. And the machine become terribly slow. Please upgrade to recent 1.4 code, you are missing a a few fixes, including one a security fix. I suggest the snapshot 20130427 which also includes a loop fix (causing 100% load from haproxy). Download at [1]. > [1297314.773541] cleanup rbuf bug: copied DBE7B6DA seq DBE7B3C8 rcvnxt > DBE7B6DA > [...] > [1297314.773625] [] ? warn_slowpath_common+0x78/0x8c This is a kernel issue with tcp splicing and has probably been fixed. Please see [2]. Not sure if Debian is backporting this fix though. You could just disable tcp splicing as a intermediate workaround. Cheers, Lukas [1] http://haproxy.1wt.eu/download/1.4/src/snapshot/ [2] http://comments.gmane.org/gmane.linux.network/231555
Re: Haproxy CPU 100%, after running about two weeks
On Thu, May 02, 2013 at 03:03:01PM +0800, ??? ??? wrote: > Hi! > Today, our haproxy CPU grow to 100%. And the machine become terribly slow. Please check if that's 100% user or 100% system (or even softirq). From your traces, it looks like it's system because the kernel complains about some processing to take a long time. It could be a faulty network driver which has trouble with splice() for example, you see : > [1297314.773614] Call Trace: > [1297314.773625] [] ? warn_slowpath_common+0x78/0x8c > [1297314.773629] [] ? warn_slowpath_fmt+0x45/0x4a > [1297314.773632] [] ? tcp_cleanup_rbuf+0x4a/0xfb > [1297314.773635] [] ? tcp_read_sock+0x127/0x138 > [1297314.773637] [] ? tcp_splice_read+0xd1/0x21f > [1297314.773643] [] ? sys_splice+0x389/0x404 > [1297314.773649] [] ? system_call_fastpath+0x16/0x1b > [1297314.773651] ---[ end trace 54eae6935f54c0f5 ]--- You seem to be using an igb driver. From my experience, splice() is useless with most gigabit drivers (including e1000e/igb) before kernel 3.5 where GRO really became efficient. And anyway, on such a machine, you don't need splicing to forward at gigabit rate ! Best regards, Willy
Re: Limit frontend bandwidth rate?
On Thu, May 02, 2013 at 03:22:33PM +0800, Delta Yeh wrote: > Is server side keepalive planned in 1.6? it's planned for 1.5 and is the condition to release 1.5. Willy
Re: Limit frontend bandwidth rate?
Is server side keepalive planned in 1.6? 2013/5/2 Willy Tarreau > Hi Igor, > > On Thu, May 02, 2013 at 02:40:05AM +0800, Igor wrote: > > Limit frontend bandwidth speed would be handy for some product > environment, > > is this still planned in 1.5 dev? > > no, it's marked for 1.6, not 1.5. > > Willy > > >
Haproxy CPU 100%, after running about two weeks
Hi! Today, our haproxy CPU grow to 100%. And the machine become terribly slow. There are some messages below. model name : Quad-Core AMD Opteron(tm) Processor 2384*2 memory: 8GB NIC: Intel Corporation 82576 Gigabit Network Linux haproxy 3.2.0-4-amd64 #1 SMP Debian 3.2.32-1 x86_64 GNU/Linux root@haproxybackup:/usr/local/etc# haproxy -vv HA-Proxy version 1.4.22 2012/08/09 Copyright 2000-2012 Willy Tarreau Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_EPOLL=1 USE_REGPARM=1 USE_PCRE=1 USE_STATIC_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. and the coredump messages [1297314.773522] [ cut here ] [1297314.773536] WARNING: at /build/buildd-linux_3.2.32-1-amd64-bkoeca/linux-3.2.32/net/ipv4/tcp.c:1201 tcp_cleanup_rbuf+0x4a/0xfb() [1297314.773539] Hardware name: PowerEdge SC1435 [1297314.773541] cleanup rbuf bug: copied DBE7B6DA seq DBE7B3C8 rcvnxt DBE7B6DA [1297314.773542] Modules linked in: ipt_REDIRECT xt_TPROXY nf_tproxy_core xt_set xt_mark xt_socket nf_defrag_ipv6 ip6_tables xt_tcpudp ip_set_hash_net ip_set nfnetlink iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ip_tables x_tables ip_vs nf_conntrack crc32c libcrc32c nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc 8021q garp stp bonding tcp_htcp ext4 crc16 jbd2 loop radeon ttm drm_kms_helper drm power_supply i2c_algo_bit k10temp mperf i2c_piix4 i2c_core processor shpchp amd64_edac_mod edac_mce_amd edac_core snd_pcm snd_page_alloc snd_timer snd soundcore psmouse dcdbas serio_raw pcspkr evdev button thermal_sys ext2 mbcache microcode sg sr_mod cdrom usbhid hid sd_mod crc_t10dif ata_generic ohci_hcd igb(O) pata_serverworks dca sata_svw libata ehci_hcd tg3 usbcore libphy scsi_mod usb_common [last unloaded: scsi_wait_scan] [1297314.773612] Pid: 23579, comm: haproxy Tainted: G O 3.2.0-4-amd64 #1 Debian 3.2.32-1 [1297314.773614] Call Trace: [1297314.773625] [] ? warn_slowpath_common+0x78/0x8c [1297314.773629] [] ? warn_slowpath_fmt+0x45/0x4a [1297314.773632] [] ? tcp_cleanup_rbuf+0x4a/0xfb [1297314.773635] [] ? tcp_read_sock+0x127/0x138 [1297314.773637] [] ? tcp_splice_read+0xd1/0x21f [1297314.773643] [] ? sys_splice+0x389/0x404 [1297314.773649] [] ? system_call_fastpath+0x16/0x1b [1297314.773651] ---[ end trace 54eae6935f54c0f5 ]--- [1297315.006259] [ cut here ] [1297315.006270] WARNING: at /build/buildd-linux_3.2.32-1-amd64-bkoeca/linux-3.2.32/net/ipv4/tcp.c:1495 tcp_recvmsg+0x24e/0x8f7() [1297315.006273] Hardware name: PowerEdge SC1435 [1297315.006276] recvmsg bug 2: copied DBE7B6DA seq DBE7A5BE rcvnxt DBE7B6DA fl 0 [1297315.006277] Modules linked in: ipt_REDIRECT xt_TPROXY nf_tproxy_core xt_set xt_mark xt_socket nf_defrag_ipv6 ip6_tables xt_tcpudp ip_set_hash_net ip_set nfnetlink iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ip_tables x_tables ip_vs nf_conntrack crc32c libcrc32c nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc 8021q garp stp bonding tcp_htcp ext4 crc16 jbd2 loop radeon ttm drm_kms_helper drm power_supply i2c_algo_bit k10temp mperf i2c_piix4 i2c_core processor shpchp amd64_edac_mod edac_mce_amd edac_core snd_pcm snd_page_alloc snd_timer snd soundcore psmouse dcdbas serio_raw pcspkr evdev button thermal_sys ext2 mbcache microcode sg sr_mod cdrom usbhid hid sd_mod crc_t10dif ata_generic ohci_hcd igb(O) pata_serverworks dca sata_svw libata ehci_hcd tg3 usbcore libphy scsi_mod usb_common [last unloaded: scsi_wait_scan] [1297315.006357] Pid: 23579, comm: haproxy Tainted: GW O 3.2.0-4-amd64 #1 Debian 3.2.32-1 [1297315.006359] Call Trace: [1297315.006369] [] ? warn_slowpath_common+0x78/0x8c [1297315.006372] [] ? warn_slowpath_fmt+0x45/0x4a [1297315.006377] [] ? _raw_spin_lock_bh+0xe/0x1c [1297315.006381] [] ? should_resched+0x5/0x23 [1297315.006383] [] ? tcp_recvmsg+0x24e/0x8f7 [1297315.006386] [] ? _raw_spin_lock_bh+0xe/0x1c [1297315.006388] [] ? should_resched+0x5/0x23 [1297315.006393] [] ? inet_recvmsg+0x5b/0x6f [1297315.006397] [] ? sock_recvmsg+0xcd/0xec [1297315.006402] [] ? dev_queue_xmit+0x448/0x45b [1297315.006407] [] ? virt_to_head_page+0x6/0x29 [1297315.006410] [] ? virt_to_slab+0x6/0x16 [1297315.006417] [] ? __cache_free.isra.41+0x7d/0x198 [1297315.006420] [] ? sock_pipe_buf_release+0x8/0x8 [1297315.006425] [] ? fget_light+0x2c/0x73 [1297315.006428] [] ? sys_recvfrom+0xbf/0x121 [1297315.006434] [] ? sys_splice+0x389/0x404 [1297315.006438] [] ? system_call_fa
