Hi Thierry,
Op 6-10-2015 om 9:47 schreef Thierry FOURNIER:
On Mon, 5 Oct 2015 21:04:08 +0200
PiBa-NL <piba.nl....@gmail.com> wrote:
Hi Thierry,
Hi Pieter,
With or without "option http-server-close" does not seem to make any
difference.
Sure, it is only an answer to the Cyril keep alive problem. I encounter
again the keepalive problem :(
The HAProxy applet (services) can't directly uses the keepalive. The
service send its response with an "internal" connection: close. If you
activate the debug, you will see the header "connection: close".
You must configure HAProxy to use keepalive between the frontend and
the client.
Ok. whell without further specific configuration it is keeping
connections alive, but as that is the default thats ok.
Adding a empty backend does seem to resolve the problem, stats also show
the backend handling connections and tracking its 2xx http result
session totals when configured like this.:
frontend http_frt
mode http
bind :801
http-request use-service lua.hello-world
default_backend http-lua-service
backend http-lua-service
mode http
I can't reproduce the problem with the last dev version. But, I
regognize the backtrace, I already encounter the same. I'm believe that
is fixed in the dev6 :(
Using dev7 i can still reproduce it..
I try to bench with my http injector, and I try with ab with and
without keep alive. I try also to stress the admin page, and I can't
reproduce pthe problem.
Argg, I see a major difference: to use freebsd. I don't have the
environment for testing it. I must install a VM.
Op 5-10-2015 om 16:06 schreef Thierry FOURNIER:
Hi,
I process this email later. For waiting, I propose to you to set the
"option http-server-close". Actually, the "services" doesn't support
itself the keepalive, but HAProxy does this job.
The "option http-server-close" expectes a server-close from the service
stream. The front of HAProxy maintains the keep-alive between the
client and the haproxy.
This method embbed a limitation: if some servers are declared in the
backend, the "option http-server-close" forbid the keepalive between
haproxy and the serveur.
Can you test with this option ?
Thierry
On Thu, 1 Oct 2015 23:00:45 +0200
Cyril Bonté <cyril.bo...@free.fr> wrote:
Hi,
Le 01/10/2015 20:52, PiBa-NL a écrit :
Hi List,
With the config below while running 'siege' i get a core dump within a
few hundreds of requests.. Viewing the stats page from a chrome browser
while siege is running seems to crash it sooner..
Is below enough to find the cause? Anything else i should try?
This is embarrassing because with your configuration, I currently can't
reproduce a segfault but I can reproduce another issue with HTTP
keep-alive requests !
(details below)
Using the haproxy snapshot from: 1.6-dev6 ss-20150930
Or perhaps i just did compile it wrong?..
make NO_CHECKSUM=yes clean debug=1 reinstall WITH_DEBUG=yes
global
stats socket /tmp/hap.socket level admin
maxconn 60000
lua-load /haproxy/brute/hello.lua
defaults
timeout client 10000
timeout connect 10000
timeout server 10000
frontend HAProxyLocalStats
bind :2300 name localstats
mode http
stats enable
stats refresh 1000
stats admin if TRUE
stats uri /
frontend http_frt
bind :801
mode http
http-request use-service lua.hello-world
Here, if I use :
$ ab -n100 -c1 -k http://127.0.0.1:801/
There will be a 10000ms delay after the first request.
Or with another test case :
echo -ne "GET / HTTP/1.1\r\nHost: localhost\r\n\r\nGET /
HTTP/1.1\r\nHost: localhost\r\n\r\n"| nc localhost 801
HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked
d
Hello World !
0
=> only 1 response
Now, if I change "frontend http_frt" to "listen http_frt", I get the
awaited behaviour.
The second test case with "listen" :
echo -ne "GET / HTTP/1.1\r\nHost: localhost\r\n\r\nGET /
HTTP/1.1\r\nHost: localhost\r\n\r\n"| nc localhost 801
HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked
d
Hello World !
0
HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked
d
Hello World !
0
=> the 2 responses are returned
core.register_service("hello-world", "http", function(applet)
local response = "Hello World !"
applet:set_status(200)
applet:add_header("content-type", "text/plain")
applet:start_response()
applet:send(response)
end )
(gdb) bt full
#0 0x0000000801a2da75 in memcpy () from /lib/libc.so.7
No symbol table info available.
#1 0x0000000000417388 in buffer_slow_realign (buf=0x7d3c90) at
src/buffer.c:166
block1 = -3306
block2 = 0
#2 0x0000000000480c42 in http_wait_for_request (s=0x80247d600,
req=0x80247d610, an_bit=4)
at src/proto_http.c:2686
cur_idx = -6336
sess = (struct session *) 0x80241e400
txn = (struct http_txn *) 0x802bb2140
msg = (struct http_msg *) 0x802bb21a0
ctx = {line = 0x2711079 <Address 0x2711079 out of bounds>, idx
= 3, val = 0, vlen = 7, tws = 0,
del = 33, prev = 0}
#3 0x00000000004d55b1 in process_stream (t=0x80244b390) at
src/stream.c:1759
max_loops = 199
ana_list = 52
ana_back = 52
flags = 4227584
srv = (struct server *) 0x0
s = (struct stream *) 0x80247d600
sess = (struct session *) 0x80241e400
rqf_last = 8397312
rpf_last = 2248179715
rq_prod_last = 7
rq_cons_last = 9
rp_cons_last = 7
rp_prod_last = 0
req_ana_back = 8192
req = (struct channel *) 0x80247d610
res = (struct channel *) 0x80247d650
si_f = (struct stream_interface *) 0x80247d7f8
si_b = (struct stream_interface *) 0x80247d818
#4 0x000000000041fe78 in process_runnable_tasks () at src/task.c:238
t = (struct task *) 0x80244b390
max_processed = 0
#5 0x000000000040cc4e in run_poll_loop () at src/haproxy.c:1539
next = 549107027
#6 0x000000000040daee in main (argc=4, argv=0x7fffffffeaf0) at
src/haproxy.c:1892
err = 0
retry = 200
limit = {rlim_cur = 120032, rlim_max = 120032}
errmsg =
"\000êÿÿÿ\177\000\000\030ëÿÿÿ\177\000\000ðêÿÿÿ\177\000\000\004\000\000\000\000\000\000\000Ðêÿÿÿ\177\000\000]A}\000\b\000\000\000pêÿÿÿ\177\000\000\000\000\000\000\000\000\000\000èêÿÿÿ\177\000\000\030ëÿÿÿ\177\000\000ðêÿÿÿ\177\000\000\004\000\000\000\000\000\000\000\220êÿÿ"
pidfd = -1
# haproxy -vv
[ALERT] 273/021153 (10691) : SSLv3 support requested but unavailable.
HA-Proxy version 1.6-dev6-10770fa 2015/09/29
Copyright 2000-2015 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing
-DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1
USE_STATIC_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2d 9 Jul 2015
Running on OpenSSL version : OpenSSL 1.0.2d 9 Jul 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.37 2015-04-28
PCRE library supports JIT : yes
Built with Lua version : Lua 5.3.0
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
--
Cyril Bonté
Seems that buffer_slow_realign() isn't used regularly during normal
haproxy operation, and it crashes first time that specific function gets
called.
Reproduction is pretty consistent with chrome browser refreshing stats
every second.
Then starting: wrk -c 200 -t 2 -d 10 http://127.0.0.1:801/
I tried adding some Alert(); items in the code to see what parameters
are set at what step, but am not understanding the exact internals of
that code..
This negative bh=-7800 is not supposed to be there i think? It is from
one of the dprintf statements, how are those supposed generate output?..
[891069718] http_wait_for_request: stream=0x80247d600 b=0x80247d610,
exp(r,w)=0,0 bf=00c08200 bh=-7800 analysers=34
Anything else i can check or provide to help get this fixed?
Best regards,
PiBa-NL
