I'm experimenting with haproxy on a centos6 VM here. I found that when I specified a health check page (option httpchk GET /url), and that page didn't exist, we have a large 404 page returned, and that causes haproxy to quickly segfault (seems like on the second try GET'ing and parsing the page). I couldn't figure out from the website where to submit a bug, so I figure I'll try here first.
Steps to reproduce: - setup http backend, with option httpchk and httpcheck expect string x. Make option httpchk point to a non-existent page - On backend server, set it up to serve large 404 response (in my case, the 404 page is 186kB, as it has an inline graphic and inline css) - Start haproxy, and wait for it to segfault I wasn't sure exactly what was causing this at first, so I did some work to narrow it down with GDB. The variable values from gdb led me to the cause on my side, and hopefully can help you fix the issue. I could not make this work with simply a large page for the http response - in that case, it seems to work as advertised, only inspecting the response up to tune.chksize (default 16384 as i've left it). But if I do this with a 404, it seems to kill it. Let me know what additional information you need if any. Thanks and kudos for the great bit of software! *#haproxy config:* #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # #--------------------------------------------------------------------- # Help in developing config here: # https://www.twilio.com/engineering/2013/10/16/haproxy #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 info chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon #enable stats stats socket /tmp/haproxy.sock listen ha_stats :8088 balance source mode http timeout client 30000ms stats enable stats auth haproxystats:foobar stats uri /haproxy?stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull #keep persisten client connection open option http-server-close option forwardfor except 127.0.0.0/8 option redispatch # Limit number of retries - total time trying to connect = connect timeout * (#retries + 1) retries 2 timeout http-request 10s timeout queue 1m #timeout opening a tcp connection to server - should be shorter than timeout client and server timeout connect 3100 timeout client 30s timeout server 30s timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend https_frontend bind :80 redirect scheme https if !{ ssl_fc } #config help: https://github.com/observing/balancerbattle/blob/master/haproxy.cfg bind *:443 ssl crt /etc/certs/mycert.pem ciphers RC4-SHA:AES128-SHA:AES:!ADH:!aNULL:!DH:!EDH:!eNULL mode http default_backend webapp #--------------------------------------------------------------------- # Main backend for web application servers #--------------------------------------------------------------------- backend webapp balance roundrobin #Insert cookie SERVERID to pin it to one leg cookie SERVERID insert nocache indirect #http check should pull url below option httpchk GET /cp/testcheck.html HTTP/1.0 #option httpchk GET /cp/testcheck.php HTTP/1.0 #http check should find string below in response to be considered up http-check expect string good #Define servers - inter=interval of 5s, rise 2=become avail after 2 successful checks, fall 3=take out after 3 fails server app1 app1.myco.com:80 weight 5 cookie app1 check inter 5000 rise 2 fall 3 server app2 app2.myco.com:80 weight 5 cookie app1 check inter 5000 rise 2 fall 3 #Example sorry server #server backup host.name:80 backup #--------------------------------------------------------------------- # Balancing for postfix / smtp #--------------------------------------------------------------------- listen smtp 10.40.60.25:25 option tcpka mode tcp option tcplog balance roundrobin server app1 app1.myco.com:25 check inter 20000 server app2 app2.myco.com:25 check inter 20000 *GDB STack trace:* Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7fea7c0 (LWP 6083)] 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-18.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 openssl-1.0.1e-16.el6_5.1.x86_64 pcre-7.8-6.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 #1 0x0000000000423b42 in bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:388 #2 0x0000000000425963 in process_chk (t=0x70fd80) at src/checks.c:1536 #3 0x000000000040e7e0 in process_runnable_tasks (next=0x7fffffffe45c) at src/task.c:240 #4 0x00000000004051d0 in run_poll_loop () at src/haproxy.c:1278 #5 0x00000000004075f1 in main (argc=<value optimized out>, argv=<value optimized out>) at src/haproxy.c:1609 (gdb) *Full GDB output:* [root@proxy1 ~]# gdb --args /usr/sbin/haproxy -Ds -f /etc/haproxy/haproxy.cfg GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/haproxy...Reading symbols from /usr/lib/debug/usr/sbin/haproxy.debug...done. done. (gdb) set follow-fork-mode child (gdb) break src/checks.c:1536 Breakpoint 1 at 0x425949: file src/checks.c, line 1536. (gdb) run Starting program: /usr/sbin/haproxy -Ds -f /etc/haproxy/haproxy.cfg [Thread debugging using libthread_db enabled] [WARNING] 009/124757 (7470) : config : 'option forwardfor' ignored for proxy 'smtp' as it requires HTTP mode. [WARNING] 009/124757 (7470) : config : 'option http-server-close' ignored for proxy 'smtp' as it requires HTTP mode. [New process 7473] [Thread debugging using libthread_db enabled] [Switching to Thread 0x7ffff7fea7c0 (LWP 7473)] Breakpoint 1, process_chk (t=0x70fd00) at src/checks.c:1536 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-18.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 openssl-1.0.1e-16.el6_5.1.x86_64 pcre-7.8-6.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) continue Continuing. Breakpoint 1, process_chk (t=0x70fd80) at src/checks.c:1536 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); (gdb) continue Continuing. Breakpoint 1, process_chk (t=0x70fd00) at src/checks.c:1536 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); (gdb) continue Continuing. Breakpoint 1, process_chk (t=0x70fd80) at src/checks.c:1536 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); (gdb) step bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:375 375 { (gdb) step 379 if (len > b->size - cur_len) (gdb) step 376 int cur_len = buffer_len(b); (gdb) step buffer_len (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:151 151 return buf->i + buf->o; (gdb) step bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:375 375 { (gdb) step 376 int cur_len = buffer_len(b); (gdb) step buffer_len (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:151 151 return buf->i + buf->o; (gdb) step 379 if (len > b->size - cur_len) (gdb) step bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:380 380 len = (b->size - cur_len); (gdb) step 381 if (!len) (gdb) step 384 half = buffer_contig_space(b); (gdb) step buffer_contig_space (b=<value optimized out>, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:222 222 if (buf->data + buf->o <= buf->p) (gdb) step 223 right = buf->data + buf->size; (gdb) step 222 if (buf->data + buf->o <= buf->p) (gdb) step 223 right = buf->data + buf->size; (gdb) step 222 if (buf->data + buf->o <= buf->p) (gdb) step 227 left = buffer_wrap_add(buf, buf->p + buf->i); (gdb) step buffer_wrap_add (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:191 191 if (ptr - buf->size >= buf->data) (gdb) step bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:388 388 memcpy(b->p, blk, half); (gdb) print b->p $1 = 0x6effa4 "play:table-column;float:none}table td[class*=\"col-\"],table th[class*=\"col-\"]{display:table-cell;float:none}.table>thead>tr>td.active,.table>tbody>tr>td.active,.table>tfoot>tr>td.active,.table>thead>tr"... (gdb) print blk $2 = 0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n" (gdb) print half $3 = <value optimized out> (gdb) $4 = <value optimized out> (gdb) step 384 half = buffer_contig_space(b); (gdb) step buffer_contig_space (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:227 227 left = buffer_wrap_add(buf, buf->p + buf->i); (gdb) step buffer_wrap_add (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:191 191 if (ptr - buf->size >= buf->data) (gdb) step buffer_contig_space (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:228 228 return right - left; (gdb) step bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", len=32) at include/common/buffer.h:388 388 memcpy(b->p, blk, half); (gdb) print b->p $5 = 0x6effa4 "play:table-column;float:none}table td[class*=\"col-\"],table th[class*=\"col-\"]{display:table-cell;float:none}.table>thead>tr>td.active,.table>tbody>tr>td.active,.table>tfoot>tr>td.active,.table>thead>tr"... (gdb) print blk $6 = 0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n" (gdb) print half $7 = -2119952626 (gdb) $8 = -2,119,952,626 (gdb) step Program received signal SIGSEGV, Segmentation fault. 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 Steve Ruiz Manager - Hosting Operations Mirth ste...@mirth.com <ste...@mirthcorp.com> -- CONFIDENTIALITY NOTICE: The information contained in this electronic transmission may be confidential. If you are not an intended recipient, be aware that any disclosure, copying, distribution or use of the information contained in this transmission is prohibited and may be unlawful. If you have received this transmission in error, please notify us by email reply and then erase it from your computer system.
