Re: haproxy hangs on old linux kernels (2.6.24)

[Willy Tarreau]

On Tue, Apr 12, 2016 at 12:53:47PM +0530, Krishna Kumar (Engineering) wrote:
> This is actually a kernel Oops as it is accessing an invalid memory
> location: fff4 (bug in code). Only kernel upgrade can fix that.

Yes and quite frankly with my kernel maintainer hat on, I'm having a hard
time believing 2.6.24 to be stable in field. 2.6.22 and 2.6.25 used to be
maintained for quite some extra time and appeared reasonably stable after
several months of fixes, but it's from an era where nobody was yet sensibilized
to tag fixes for stable backport and most important fixes were missing. Talk
about 2.6.32.x, 3.2.x, 3.4.x or even 3.10.x as stable, but not 2.6.24. The
fact that it happens to achieve years of uptime doing nothing is not a sign
of stability. A stable kernel achieves years of uptime under stress and here
apparently a simple recvform() syscall is enough to crash it under certain
conditions :-/

Regards,
Willy

Re: haproxy hangs on old linux kernels (2.6.24)

[Krishna Kumar (Engineering)]

This is actually a kernel Oops as it is accessing an invalid memory
location: fff4 (bug in code). Only kernel upgrade can fix that.

On Tue, Apr 12, 2016 at 12:43 PM, Alexey Vlasov  wrote:

> Hi,
>
> I have some linux boxes with very old kernels. Unfortunately, I cannot
> upgrade them due to the fact that they work very stable. for
> example,their uptime is already some
> years, which is not true speaking about modern kernels.
> But there is one problem: HAPproxy hangs when I turn on SSL options.
>
> # haproxy -v
> HA-Proxy version 1.5.4 2014/09/02
>
> My config:
> global
> tune.ssl.default-dh-param 2048
> ssl-default-bind-ciphers
> ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
>
> frontend https-in
> bind111.222.111.222:443 ssl strict-sni no-sslv3 crt-list
> /etc/haproxy_aux2_pools/crt.list
> errorfile   408 /dev/null
> option  http-keep-alive
> option  http-server-close
> http-requestadd-header X-Forwarded-Port %[dst_port]
> http-requestadd-header X-Forwarded-Proto https
> use_backend apache_aux2_workers
>
> # ps -o s,pid,start,comm -C haproxy_aux2_pools
> S   PID  STARTED COMMAND
> D   472   Apr 07 haproxy_aux2_po
> D   725   Apr 07 haproxy_aux2_po
> D  1185   Apr 07 haproxy_aux2_po
> D  1706   Apr 07 haproxy_aux2_po
> D  2168   Apr 07 haproxy_aux2_po
> D  2749   Apr 07 haproxy_aux2_po
> D  2996   Apr 07 haproxy_aux2_po
> D  3620   Apr 07 haproxy_aux2_po
> D  3960   Apr 07 haproxy_aux2_po
>
> and kernel trace:
> Apr  7 17:40:23 l4 kernel: Unable to handle kernel paging request at
> fff4 RIP:
> Apr  7 17:40:23 l4 kernel: []
> dma_unpin_iovec_pages+0x10/0x80
> Apr  7 17:40:23 l4 kernel: PGD 203067 PUD 204067 PMD 0
> Apr  7 17:40:23 l4 kernel: Oops:  [1] SMP
> Apr  7 17:40:23 l4 kernel: CPU 0
> Apr  7 17:40:23 l4 kernel: Pid: 17747, comm: haproxy_aux2_po Not tainted
> 2.6.24-1gb-1 #4
> Apr  7 17:40:23 l4 kernel: RIP: 0010:[]
> [] dma_unpin_iovec_pages+0x10/0x80
> Apr  7 17:40:23 l4 kernel: RSP: 0018:8101164dbbb8  EFLAGS: 00010282
> Apr  7 17:40:23 l4 kernel: RAX: 0001 RBX: 
> RCX: 
> Apr  7 17:40:23 l4 kernel: RDX:  RSI: 
> RDI: fff4
> Apr  7 17:40:23 l4 kernel: RBP: 8102acf5c6b0 R08: 0040
> R09: 
> Apr  7 17:40:23 l4 kernel: R10: 80629900 R11: 80398920
> R12: 8102acf5c600
> Apr  7 17:40:23 l4 kernel: R13: 8102acf5c6b0 R14: fff4
> R15: 7fff
> Apr  7 17:40:23 l4 kernel: FS:  2b5d03469b20()
> GS:8062f000() knlGS:
> Apr  7 17:40:23 l4 kernel: CS:  0010 DS:  ES:  CR0:
> 80050033
> Apr  7 17:40:23 l4 kernel: CR2: fff4 CR3: 0001c50f2000
> CR4: 06e0
> Apr  7 17:40:23 l4 kernel: DR0:  DR1: 
> DR2: 
> Apr  7 17:40:23 l4 kernel: DR3:  DR6: 0ff0
> DR7: 0400
> Apr  7 17:40:23 l4 kernel: Process haproxy_aux2_po (pid: 17747, threadinfo
> 8101164da000, task 8101b733a000)
> Apr  7 17:40:23 l4 kernel: Stack:   8102acf5c6b0
> 8102acf5c600 8102acf5c6b0
> Apr  7 17:40:23 l4 kernel: 8102acf5c9dc 804e2f11
> 810010535900 804e1a53
> Apr  7 17:40:23 l4 kernel:  4020
> 8101164dbee8 07524a80
> Apr  7 17:40:23 l4 kernel: Call Trace:
> Apr  7 17:40:23 l4 kernel: Call Trace:
> Apr  7 17:40:23 l4 kernel: [] tcp_recvmsg+0x581/0xcd0
> Apr  7 17:40:23 l4 kernel: [] tcp_sendmsg+0x593/0xc30
> Apr  7 17:40:23 l4 kernel: [] _spin_lock_bh+0x9/0x20
> Apr  7 17:40:23 l4 kernel: [] release_sock+0x13/0xb0
> Apr  7 17:40:23 l4 kernel: []
> sock_common_recvmsg+0x30/0x50
> Apr  7 17:40:23 l4 kernel: [] sock_recvmsg+0x14a/0x160
> Apr  7 17:40:23 l4 kernel: [] filemap_fault+0x21e/0x420
> Apr  7 17:40:23 l4 kernel: []
> autoremove_wake_function+0x0/0x30
> Apr  7 17:40:23 l4 kernel: [] __do_fault+0x1e5/0x460
> Apr  7 17:40:23 l4 kernel: [] handle_mm_fault+0x1af/0x7c0
> Apr  7 17:40:23 l4 kernel: [] sys_recvfrom+0xfe/0x1a0
> Apr  7 17:40:23 l4 kernel: [] do_page_fault+0x1e0/0x830
> Apr  7 17:40:23 l4 kernel: [] vma_merge+0x161/0x1f0
> Apr  7 17:40:23 l4 kernel: [] system_call+0x7e/0x83
> Apr  7 17:40:23 l4 kernel:
> Apr  7 17:40:23 l4 kernel:
> Apr  7 17:40:23 l4 kernel: Code: 8b 37 85 f6 7e 51 48 8d 6f 08 45 31 ed