Re: SEGFAULT in in buffer_insert_line2
On Mon, Dec 21, 2015 at 08:47:58AM +0100, Bernd Helm wrote: > Nice, thank you! By the way it's already fixed in 1.7-dev1, you may want to give it a try before we issue 1.6.3. Willy
Re: SEGFAULT in in buffer_insert_line2
Nice, thank you! On 12/21/2015 08:37 AM, thierry.fourn...@arpalert.org wrote: Hi, The new release of the 1.6 will fix this problem. It was due to an HTTP analyer called and use after the flush of some http data by the AppletHTTP Lua things. Thierry On Sat, 5 Dec 2015 02:00:50 +0100 Willy Tarreau wrote: On Fri, Dec 04, 2015 at 09:18:38AM +0100, Bernd Helm wrote: On 12/03/2015 06:53 PM, Willy Tarreau wrote: Maybe we're facing a bug where the buffer wraps at the end or something like this. Bernd, if you still have the core, could you please issue "print *b" while in buffer_insert_line2() ? yes, i still have the core. (gdb) frame 1 #1 0x00413349 in buffer_insert_line2 (b=0x1e47a70, pos=0x1e47acb "ntent-Type: text/html; charset=UTF-8\r\nTransfer-Encoding: chunked\r\n\r\n1b9\r\nhttp://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\";>\n The buffer was corrupted :-( I guess it was corrupted just before doing this buffer_insert_line2(), causing it to crash. Buffers are manipulated a lot, so it must have been still good a few nanoseconds before calling this. I'll review the code in this area in case I can find any hint relevant to your config. Thanks, Willy -- Mit freundlichen Grüßen / Best regards, B.Sc. Bernd Michael Helm -- Helm & Walter IT-Solutions GbR Müglitztalstr. 63 D-01809 Dohna Tel.: +49 3529 529129 Fax.: +49 3529 2348140 Mobile: +49 151 21240208 Email: bernd.h...@helmundwalter.de WWW: https://www.helmundwalter.de
Re: SEGFAULT in in buffer_insert_line2
Hi,
The new release of the 1.6 will fix this problem. It was due to an HTTP
analyer called and use after the flush of some http data by the
AppletHTTP Lua things.
Thierry
On Sat, 5 Dec 2015 02:00:50 +0100
Willy Tarreau wrote:
> On Fri, Dec 04, 2015 at 09:18:38AM +0100, Bernd Helm wrote:
> > On 12/03/2015 06:53 PM, Willy Tarreau wrote:
> > >Maybe we're facing a bug where the buffer wraps at the end or something
> > >like this. Bernd, if you still have the core, could you please issue
> > >"print *b" while in buffer_insert_line2() ?
> > yes, i still have the core.
> >
> > (gdb) frame 1
> > #1 0x00413349 in buffer_insert_line2 (b=0x1e47a70,
> > pos=0x1e47acb "ntent-Type: text/html;
> > charset=UTF-8\r\nTransfer-Encoding: chunked\r\n\r\n1b9\r\n > html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"
> >
> > \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\";>\n > str=0x513979 "Connection: close",
> > len=17) at src/buffer.c:126
> > (gdb) print *b
> > $1 = {p = 0x0, size = 822083584, i = 64, o = 2818572288, data = 0x1e47a84
> > "z\344\001"}
> >
> > if you need more information, let me know. thank you.
>
> The buffer was corrupted :-(
> I guess it was corrupted just before doing this buffer_insert_line2(),
> causing it to crash. Buffers are manipulated a lot, so it must have
> been still good a few nanoseconds before calling this. I'll review the
> code in this area in case I can find any hint relevant to your config.
>
> Thanks,
> Willy
>
>
Re: SEGFAULT in in buffer_insert_line2
On Fri, Dec 04, 2015 at 09:18:38AM +0100, Bernd Helm wrote:
> On 12/03/2015 06:53 PM, Willy Tarreau wrote:
> >Maybe we're facing a bug where the buffer wraps at the end or something
> >like this. Bernd, if you still have the core, could you please issue
> >"print *b" while in buffer_insert_line2() ?
> yes, i still have the core.
>
> (gdb) frame 1
> #1 0x00413349 in buffer_insert_line2 (b=0x1e47a70,
> pos=0x1e47acb "ntent-Type: text/html;
> charset=UTF-8\r\nTransfer-Encoding: chunked\r\n\r\n1b9\r\n html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"
> \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\";>\n str=0x513979 "Connection: close",
> len=17) at src/buffer.c:126
> (gdb) print *b
> $1 = {p = 0x0, size = 822083584, i = 64, o = 2818572288, data = 0x1e47a84
> "z\344\001"}
>
> if you need more information, let me know. thank you.
The buffer was corrupted :-(
I guess it was corrupted just before doing this buffer_insert_line2(),
causing it to crash. Buffers are manipulated a lot, so it must have
been still good a few nanoseconds before calling this. I'll review the
code in this area in case I can find any hint relevant to your config.
Thanks,
Willy
Re: SEGFAULT in in buffer_insert_line2
On 12/03/2015 06:53 PM, Willy Tarreau wrote: Maybe we're facing a bug where the buffer wraps at the end or something like this. Bernd, if you still have the core, could you please issue "print *b" while in buffer_insert_line2() ? yes, i still have the core. (gdb) frame 1 #1 0x00413349 in buffer_insert_line2 (b=0x1e47a70, pos=0x1e47acb "ntent-Type: text/html; charset=UTF-8\r\nTransfer-Encoding: chunked\r\n\r\n1b9\r\nhttp://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\";>\nhttps://www.helmundwalter.de
Re: SEGFAULT in in buffer_insert_line2
Hi Lukas, On Thu, Dec 03, 2015 at 06:22:12PM +0100, Lukas Tribus wrote: > Hi Bernd, Willy, > > > > Hello, > > > > im getting segfault, it happens on 1 of ~500 million requests that are > > processed on haproxy 1.6.2-2 on debian wheezy and jessie (systems > > updated, crash stayed). > > > > If you need more informations, let me know. > > > > Thank You. > > > > Trace: > > (gdb) thread apply all bt full > > > > Thread 1 (Thread 0x7fd811254700 (LWP 19002)): > > #0 __memmove_ssse3_back () at > > ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1652 > > No locals. > > #1 0x00413349 in buffer_insert_line2 (b=0x1e47a70, > > pos=0x1e47acb "ntent-Type: text/html; > > charset=UTF-8\r\nTransfer-Encoding: chunked\r\n\r\n1b9\r\n > PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" > > \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\";>\n > str=0x513979 "Connection: close", > > len=17) at src/buffer.c:126 > > delta = 19 > > > Could this be: > https://sourceware.org/bugzilla/show_bug.cgi?id=12518 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=627818 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=625521 > > > Short description: > > Starting with version 2.13, eglibc provides an SSSE3 optimized version > > of memcpy() on the amd64 architecture. This version might copy memory > > backward in some conditions, which causes issues if the source and > > destination overlap. memmove() should be used in such cases, but some > > programs still wrongly use memcpy(). > > > Although this seems like an old change (2011), so its probably something > else. Thanks for these pointers. I don't think it's the same, because we do have memmove() here, not memcpy(). The fact that glibc decided to use memcpy in the end is probably only related to the fact that it implements half of the memmove() using memcpy(). That said I really don't understand here. The new length is short (19 bytes inserted), we're not crossing a page boundary, everything looks pretty clean. Maybe we're facing a bug where the buffer wraps at the end or something like this. Bernd, if you still have the core, could you please issue "print *b" while in buffer_insert_line2() ? Thanks, Willy
RE: SEGFAULT in in buffer_insert_line2
Hi Bernd, Willy, > Hello, > > im getting segfault, it happens on 1 of ~500 million requests that are > processed on haproxy 1.6.2-2 on debian wheezy and jessie (systems > updated, crash stayed). > > If you need more informations, let me know. > > Thank You. > > Trace: > (gdb) thread apply all bt full > > Thread 1 (Thread 0x7fd811254700 (LWP 19002)): > #0 __memmove_ssse3_back () at > ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1652 > No locals. > #1 0x00413349 in buffer_insert_line2 (b=0x1e47a70, > pos=0x1e47acb "ntent-Type: text/html; > charset=UTF-8\r\nTransfer-Encoding: chunked\r\n\r\n1b9\r\n PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" > \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\";>\n str=0x513979 "Connection: close", > len=17) at src/buffer.c:126 > delta = 19 Could this be: https://sourceware.org/bugzilla/show_bug.cgi?id=12518 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=627818 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=625521 Short description: > Starting with version 2.13, eglibc provides an SSSE3 optimized version > of memcpy() on the amd64 architecture. This version might copy memory > backward in some conditions, which causes issues if the source and > destination overlap. memmove() should be used in such cases, but some > programs still wrongly use memcpy(). Although this seems like an old change (2011), so its probably something else. Regards, Lukas
SEGFAULT in in buffer_insert_line2
Hello,
im getting segfault, it happens on 1 of ~500 million requests that are
processed on haproxy 1.6.2-2 on debian wheezy and jessie (systems
updated, crash stayed).
If you need more informations, let me know.
Thank You.
Trace:
(gdb) thread apply all bt full
Thread 1 (Thread 0x7fd811254700 (LWP 19002)):
#0 __memmove_ssse3_back () at
../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1652
No locals.
#1 0x00413349 in buffer_insert_line2 (b=0x1e47a70,
pos=0x1e47acb "ntent-Type: text/html;
charset=UTF-8\r\nTransfer-Encoding: chunked\r\n\r\n1b9\r\nPUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"
\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\";>\nstr=0x513979 "Connection: close",
len=17) at src/buffer.c:126
delta = 19
#2 0x0046f6b9 in http_header_add_tail2 (msg=0x2978910,
hdr_idx=0x29788b0, text=0x513979 "Connection: close", len=17) at
src/proto_http.c:595
bytes = 19706248
#3 0x00472a36 in http_change_connection_header (txn=0x29788b0,
msg=0x2978910, wanted=4194304) at src/proto_http.c:2068
ctx = {line = 0x0, idx = 0, val = 0, vlen = 26108128, tws = 0,
del = -57605088, prev = 32765}
hdr_val = 0x513979 "Connection: close"
hdr_len = 17
#4 0x0047a179 in http_process_request (s=0x2978540,
req=0x2978550, an_bit=512) at src/proto_http.c:4800
want_flags = 4194304
sess = 0x158a4d0
txn = 0x29788b0
msg = 0x2978910
cli_conn = 0x18e60e0
#5 0x004bf1dc in process_stream (t=0x158a560) at src/stream.c:1807
max_loops = 199
ana_list = 8704
ana_back = 8704
flags = 8388641
srv = 0x0
s = 0x2978540
sess = 0x158a4d0
rqf_last = 8388640
rpf_last = 2151677952
rq_prod_last = 7
rq_cons_last = 7
rp_cons_last = 7
rp_prod_last = 7
req_ana_back = 4237362608
req = 0x2978550
res = 0x2978590
si_f = 0x2978738
si_b = 0x2978758
#6 0x0041a9bf in process_runnable_tasks () at src/task.c:238
t = 0x158a560
max_processed = 0
#7 0x0040c023 in run_poll_loop () at src/haproxy.c:1559
next = 1736300278
#8 0x0040ccd9 in main (argc=8, argv=0x7ffdfc910858) at
src/haproxy.c:1912
err = 0
retry = 200
limit = {rlim_cur = 80049, rlim_max = 80049}
errmsg = "\000\026%\001", '\000' ,
"p\026%\001\000\000\000\000\003", '\000' ,
"\205%\021\330\177\000\000=\005\000\000\000\000\000\000\025\210`\020\330\177\000\000\320\325t",
'\000' ,
"\002\000\000\000\026\000\000\000\000\000\000\000\060\a\221",
pidfd = 10
(gdb)
Config:
global
log /dev/loglocal0
log /dev/loglocal1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
maxconn 4
ssl-default-bind-ciphers
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
tune.ssl.default-dh-param 2048
#custom lua script for clicks clounting
lua-load /etc/haproxy/lua/click-stats.lua
defaults
log global
modehttp
option dontlognull
option dontlog-normal
log-format %ci:%cp\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\
%CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r
timeout connect 5000
timeout client 5
timeout server 5
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
maxconn 4
frontend incoming
bind *:80
bind *:443 ssl crt /etc/haproxy/ssl/cert.comb
http-request set-header S-Forwarded-Proto https if { ssl_fc }
option http-server-close
option forwardfor except 172.16.0.0/12 header S-Forwarded-For
capture request headerHost len 500
acl is_favicon path_reg ^/favicon.ico
use_backend notfound if is_favicon
default_backend delivery
backend notfound
errorfile 503 /etc/haproxy/errors/404.http
http-request set-log-level silent
backend delivery
balance leastconn
mode http
option httpchk HEAD / HTTP/1.1\r\nHost:\ sdfkjsndhebsdf.com
option http-server-close
option allbackups
http-request lua.track-clicks
http-response lua.track-responses
server localhost localhost:88 check inter 5000
