Hello Jean,
On Fri, May 26, 2017 at 01:00:17PM +0000, Jean LUBATTI wrote:
> Hello,
>
> When using a vulnerability scanner on haproxy 1.7.5, we discovered a scenario
> under which the haproxy segfaults.
>
> Unfortunately, this is a "bundled" scanner whith no access to the exact
> requests, and the haproxy terminates the SSL for https, so not easy to
> capture the actual traffic, but we managed to gather a core of haproxy.
You could run haproxy with "-d", it will dump to stdout/stderr each header
line it receives. I even think we could get a complete capture of the whole
request like this :
tcp-request content capture req.hdrs_bin len 2000
It will then appear in the core in s->req_cap.
> This happens only when using the cookie SERVERID for session stickiness:
We've had similar issues a very long time ago, I hoped they were fixed :-/
> The backtrace is as follow:
>
> I believe the scanner injects a screwed up header and/or cookie. The result
> appears to be the incorrect memmove of the last header line :
>
> (gdb) bt full
> #0 _wordcopy_fwd_dest_aligned (dstp=14237664, srcp=14237696,
> len=2305843009213653720) at wordcopy.c:196
> a0 = <value optimized out>
> a1 = <value optimized out>
> a2 = 0
> a3 = 0
> sh_1 = 40
> sh_2 = 24
> #1 0x0000003727a838be in memmove (dest=0xd4575d, src=<value optimized out>,
> len=18446744073709551439) at memmove.c:73
> dstp = <value optimized out>
> srcp = <value optimized out>
> #2 0x0000000000411217 in buffer_replace2 (b=0xd456b0, pos=0xd4575d
> "Accept-Encoding: gzip\r\n\r\n.32.31\r\nConnection: close\r\n\r\nre\r\n\r\n",
> end=<value optimized out>, str=0x0, len=0) at src/buffer.c:90
> delta = -21
> #3 0x000000000045de02 in manage_client_side_cookies (s=0xcfa800,
> req=0xcfa810) at src/proto_http.c:7976
> delta = <value optimized out>
> cur_hdr = 0xcfac6c
> val = <value optimized out>
> txn = <value optimized out>
> sess = 0xc1c930
> preserve_hdr = 0
> cur_idx = 3
> old_idx = 2
> hdr_beg = 0xd4575d "Accept-Encoding:
> gzip\r\n\r\n.32.31\r\nConnection: close\r\n\r\nre\r\n\r\n"
> hdr_end = 0xd45770 "ip\r\n\r\n.32.31\r\nConnection:
> close\r\n\r\nre\r\n\r\n"
> hdr_next = 0xd45772 "\r\n\r\n.32.31\r\nConnection:
> close\r\n\r\nre\r\n\r\n"
> del_from = 0xd45763 "-Encoding: gzip\r\n\r\n.32.31\r\nConnection:
> close\r\n\r\nre\r\n\r\n"
> prev = <value optimized out>
> att_beg = <value optimized out>
> att_end = <value optimized out>
> equal = <value optimized out>
> val_beg = <value optimized out>
> val_end = <value optimized out>
> next = <value optimized out>
> #4 0x0000000000460a86 in http_process_req_common (s=0xcfa800, req=0xcfa810,
> an_bit=256, px=0x86a650) at src/proto_http.c:4474
> sess = 0xc1c930
> txn = 0xcfab50
> msg = 0xcfabb0
> rule = <value optimized out>
> wl = <value optimized out>
> verdict = <value optimized out>
> deny_status = 2
> #5 0x0000000000486d0e in process_stream (t=0xb8fa70) at src/stream.c:1798
> max_loops = 199
> ana_list = 2304
> ana_back = 2304
> flags = 2
> srv = <value optimized out>
> s = 0xcfa800
> sess = 0xc1c930
> rqf_last = <value optimized out>
> rpf_last = 2147745792
> rq_prod_last = <value optimized out>
> rq_cons_last = <value optimized out>
> rp_cons_last = 7
> rp_prod_last = 0
> req_ana_back = <value optimized out>
> req = 0xcfa810
> res = 0xcfa850
> si_f = 0xcfaa38
> si_b = 0xcfaa60
> #6 0x0000000000415ac0 in process_runnable_tasks () at src/task.c:238
> t = <value optimized out>
> max_processed = <value optimized out>
> #7 0x0000000000407028 in run_poll_loop () at src/haproxy.c:1724
> next = <value optimized out>
> #8 0x000000000040a308 in main (argc=<value optimized out>, argv=<value
> optimized out>) at src/haproxy.c:2105
> err = <value optimized out>
> retry = <value optimized out>
> limit = {rlim_cur = 4091, rlim_max = 4091}
> errmsg =
> "\000\347K\000\000\000\000\000\b\300n\000\000\000\000\000\020\340\377\377\377\177\000\000\000\300n\000\000\000\000\000\006\000\000\000\000\000\000\000H\341\377\377\377\177\000\000\200\341\377\377\377\177\000\000XpA\000\000\000\000\000\000\300n\000\000\000\000\000\226\070K\000\000\000\000\000\340\070\370/7\000\000\000\340\067K\000\000\000\000\000\000\000\000"
>
>
> Basically, the memove in #1 is called with len=18446744073709551439 from
> buffer_replace2, which is a negative value.
>
> I am not sure exactly in which case this is possible (last line of the header
> incorrect or something), but bi_end(b) is < to end, so the unsigned size_t
> expected by memmove is incorrect.
For sure there's a bug there. However I'd be very interested in understanding
in which case it can happen, as it will reveal the root cause of this bug.
We know that this part is tricky and I'd rather be sure we don't miss a
single case.
> I recompiled haproxy with the patch below and now it survives the
> vulnerability scanner, but it might not be at the proper place of the code
> (maybe the logic fault is better addressed above in
> manage_client_side_cookies... ):
This obviously confirms that it's where it dies, but it doesn't explain
what the problem really is. Would you happen to have an exploitable core
with the associated executable ? I suspect it would already miss the
important information due to the memmove() though.
Thanks,
Willy
