Hi Willy, Great thanks for your suggestion. I will try it now nad give you feedback later.
Best regards, Hogan From hogan's iPhone On 2010-11-19, at 19:02, Willy Tarreau <[email protected]> wrote: > Hi Hogan, > > I have good news. I found two bugs that could cause what you describe in > case of out of memory conditions, and I could trigger them with your config > by limiting my amount of memory. > > One of them is triggered by the cookie capture. It it failed to allocate > memory to store the cookie, it would emit an alert but still use it. That > was the case only for the response cookies, the request cookies has the > correct fix. Not sure how that slipped through fixes, but it's fixed now. > > The other one was caused by a risk of freeing a non-allocated pool in the > appsession in case of memory shortage : some fields were released while > not yet allocated, resulting in a segfault too during the call to the libc's > free(). > > However, I found a third bug with my glibc : the call to regexec() spins > forever when it fails to allocate memory. It is very possible that with > your libc, instead of looping it crashes. And the only way to have this > code called with your config is precisely by adding the reqirep rule : > > 0x6f755d7d in mmap () from /lib/libc.so.6 > (gdb) bt > #0 0x6f755d7d in mmap () from /lib/libc.so.6 > #1 0x6f703bb4 in sYSMALLOc () from /lib/libc.so.6 > #2 0x6f702835 in _int_realloc () from /lib/libc.so.6 > #3 0x6f701100 in realloc () from /lib/libc.so.6 > #4 0x6f731d6f in re_string_realloc_buffers () from /lib/libc.so.6 > #5 0x6f73dfb3 in extend_buffers () from /lib/libc.so.6 > #6 0x6f73b222 in transit_state () from /lib/libc.so.6 > #7 0x6f739a6f in check_matching () from /lib/libc.so.6 > #8 0x6f7392a5 in re_search_internal () from /lib/libc.so.6 > #9 0x6f73874d in regexec@@GLIBC_2.3.4 () from /lib/libc.so.6 > #10 0x0806b8da in apply_filter_to_req_line (t=0x0, req=0x8ba5da8, > exp=0x8b07b08) at src/proto_http.c:5534 > #11 0x0806badc in apply_filters_to_request (s=0x8ba5650, req=0x8ba5da8, > px=0x8afec80) at src/proto_http.c:5649 > #12 0x080682fe in http_process_req_common (s=0x8ba5650, req=0x8ba5da8, > an_bit=-12, px=0x8afec80) > at src/proto_http.c:3013 > #13 0x0807e776 in process_session (t=0x8ba5a08) at src/session.c:1068 > #14 0x0804ec8d in process_runnable_tasks (next=0x77bcd29c) at src/task.c:234 > #15 0x0804b005 in run_poll_loop () at src/haproxy.c:974 > #16 0x0804b389 in main (argc=6, argv=0x77bcd334) at src/haproxy.c:1255 > > I'm used to build with libpcre which is much faster than libc and which I > never got to fail. > > So what I'm suggesting for your case : > > 1) I'm certain that you're running out of memory, maybe because of > too many learned cookies, or maybe because your machine is under-sized > for the amount of traffic that passes through. However, since you're > saying it takes one hour to die, I suspect that it's a combination of > both. You need much memory, and the appsession that slowly piles up > progressively reduces the amount of memory available until it dies. > So you must find why you have so little memory (maybe a ulimit -m or > something like this). > > 2) rebuild with support for libpcre, which handles out of memory conditions > much better than libc. An out of memory should in theory never happen, > but since it does in your case, let's be careful. > > 3) apply the two patches below to your current version to fix the two bugs : > > http://git.1wt.eu/web?p=haproxy-1.4.git;a=commitdiff_plain;h=62e360 > http://git.1wt.eu/web?p=haproxy-1.4.git;a=commitdiff_plain;h=75eae4 > > Best regards, > Willy >
