On 18/05/16 15:42, Willy Tarreau wrote:
Hi Sebastian,
On Thu, May 12, 2016 at 09:58:22AM +0200, Sebastian Heid wrote:
Hi Lukas,
starting from around 200mbit/s in&out, haproxy processes (nbproc 6) are
hitting 100% cpu regularly (noticed up to 3 processes at the same time with
100%), but recover again on its own after some time.
stracing such a process yesterday showed the following:
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
Unfortunately I can't do any more debugging in this setup. HAproxy 1.5.14 is
never near to 10% cpu usage with way higher bandwidth.
So far I've got good reports from people having experienced similar issues
with recent versions, thus I'm thinking about something, are you certain
that you did a make clean after upgrading and before rebuilding ? Sometimes
we tend to forget it, especially after a simple "git pull". It is very
possible that some old .o files were not properly rebuilt and still contain
these bugs. If in doubt, you can simply keep a copy of your latest haproxy
binary, make clean, build again and run cmp between them. It should not
report any difference otherwise it means there was an issue (which would be
a great news).
I can confirm that on CentOS 6 with HAproxy 1.6.5 this 100% CPU load
still happens. Exactly the same:
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, {}, 200, 0) = 0
epoll_wait(0, ^CProcess 6200 detached
<detached ...>
# haproxy -vv
HA-Proxy version 1.6.5 2016/05/10
Copyright 2000-2016 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
-Wdeclaration-after-statement
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Veiko
