Hi, We’re running into segmentation faults on a new haproxy system we’re
developing. We’ve been building haproxy 1.6.7 on ubuntu 14.04.5 with
openssl,pcre, and zlib. The problem doesn’t manifest when running a single
process. Load testing is approximately 1gbps of ssl traffic from four test
servers on the internet, there are two backend servers handling it. It seems
that only processes assigned to handle the traffic die and only under load. We
have tried with Pthreads and Mutex off, but the problem remained. In the
config listed below I have omitted some other front/backends for brevity, they
are unused at present and are very simple (no ssl, no process assignments).
Segmentation fault is as:
[592869.807299] haproxy[31045]: segfault at 7f02cfca88e8 ip 00007f02cf971eee sp
00007ffe1380d4a8 error 4 in libc-2.19.so[7f02cf8da000+1ba000]
Kernel is: "Linux hap01 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux”, Cpu is a single E5-2650 v3, nic is an
Intel X710 with i40e driver version 1.5.16
We are running nbproc, relevant config sections:
**************
global
daemon
ssl-server-verify none
log /dev/log local0 info
# log /dev/log local1 debug
# user haproxy
# group haproxy
spread-checks 50
#maxpipes 64000
tune.idletimer 0
#tune.maxpollevents 1
#tune.comp.maxlevel 9
#tune.zlib.memlevel 9
stats socket /run/haproxy/stats1 uid 0 gid 0 mode 0777 level user
process 1
stats socket /run/haproxy/stats2 uid 0 gid 0 mode 0777 level user
process 2
stats socket /run/haproxy/stats3 uid 0 gid 0 mode 0777 level user
process 3
stats socket /run/haproxy/stats4 uid 0 gid 0 mode 0777 level user
process 4
stats socket /run/haproxy/stats5 uid 0 gid 0 mode 0777 level user
process 5
stats socket /run/haproxy/stats6 uid 0 gid 0 mode 0777 level user
process 6
stats socket /run/haproxy/stats7 uid 0 gid 0 mode 0777 level user
process 7
stats socket /run/haproxy/stats8 uid 0 gid 0 mode 0777 level user
process 8
#stats socket /run/haproxy/stats9 uid 0 gid 0 mode 0777 level user
process 9
stats bind-process all
nbproc 8
cpu-map 1 1
cpu-map 2 2
cpu-map 3 3
cpu-map 4 4
cpu-map 5 5
cpu-map 6 6
cpu-map 7 7
cpu-map 8 8
#cpu-map 9 9
maxconn 100000
defaults
log global
timeout server 5s
timeout connect 5s
timeout client 5s
option accept-invalid-http-request
# option http-ignore-probes
# option dontlognull
mode http
option dontlognull
option splice-request
option splice-response
default-server inter 100s
timeout connect 5000
timeout client 50000
timeout server 50000
compression algo gzip
frontend app-http
bind public.ip:80 interface p2p1 process 1-3
bind public.ip:443 ssl crt /haproxy/ssl/_wildcard_.pem interface p2p1 process
4-7
option httplog
log global
mode http
acl white_list src someiprange/24 someip
tcp-request content accept if white_list
tcp-request content reject
default_backend app-http-backend
backend app-http-backend
bind-process 8
mode http
option httplog
log global
option httpchk
balance static-rr
server server1-8080 internal.ip:8082 check port 8082
server server2-8080 internal.ip:8082 check port 8082
listen stats
bind internal.ip:1901 process 1
bind internal.ip:1902 process 2
bind internal.ip:1903 process 3
bind internal.ip:1904 process 4
bind internal.ip:1905 process 5
bind internal.ip:1906 process 6
bind internal.ip:1907 process 7
bind internal.ip:1908 process 8
mode http
stats enable
stats uri /
stats show-node
stats show-legends
*************************
Compile Line:
make TARGET=linux2628 USE_OPENSSL=1 SSL_INC=$STATICLIBSSL/include
SSL_LIB=$STATICLIBSSL/lib ADDLIB=-ldl USE_ZLIB=1
ZLIB_INC=/opt/zlib-$ZLIB_VERSION/ ZLIB_LIB=/opt/zlib-$ZLIB_VERSION/
USE_STATIC_PCRE=1 PCRE_LIB=$PCRESTUFFS/lib/ PCRE_INC=$PCRESTUFFS/include/
Output of haproxy -vv
**********************
/opt/haproxy-1.6.7# ./haproxy -vv
HA-Proxy version 1.6.7 2016/07/13
Copyright 2000-2016 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1t 3 May 2016
Running on OpenSSL version : OpenSSL 1.0.1t 3 May 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
*******************************
While I did notice a newer PCRE I haven’t built with it yet, but that doesn’t
seem to be the area of problem. I also noticed when using the
USE_PTHREAD_PSHARED=1 or USE_FUTEX= options on the make command the “OPTION=“
output of haproxy -vv doesn’t change, though a close examination of the output
of the make indicates they are being respected. I do the static compile of
pcre, zlib, and openssl once. For each compile of haproxy I do make clean
first.
p.s. In the Atomic Operations section of the readme: "you willy have to either
use”
