Re: haproxy hangs on old linux kernels (2.6.24)
On Tue, Apr 12, 2016 at 12:53:47PM +0530, Krishna Kumar (Engineering) wrote: > This is actually a kernel Oops as it is accessing an invalid memory > location: fff4 (bug in code). Only kernel upgrade can fix that. Yes and quite frankly with my kernel maintainer hat on, I'm having a hard time believing 2.6.24 to be stable in field. 2.6.22 and 2.6.25 used to be maintained for quite some extra time and appeared reasonably stable after several months of fixes, but it's from an era where nobody was yet sensibilized to tag fixes for stable backport and most important fixes were missing. Talk about 2.6.32.x, 3.2.x, 3.4.x or even 3.10.x as stable, but not 2.6.24. The fact that it happens to achieve years of uptime doing nothing is not a sign of stability. A stable kernel achieves years of uptime under stress and here apparently a simple recvform() syscall is enough to crash it under certain conditions :-/ Regards, Willy
Re: haproxy hangs on old linux kernels (2.6.24)
This is actually a kernel Oops as it is accessing an invalid memory location: fff4 (bug in code). Only kernel upgrade can fix that. On Tue, Apr 12, 2016 at 12:43 PM, Alexey Vlasov wrote: > Hi, > > I have some linux boxes with very old kernels. Unfortunately, I cannot > upgrade them due to the fact that they work very stable. for > example,their uptime is already some > years, which is not true speaking about modern kernels. > But there is one problem: HAPproxy hangs when I turn on SSL options. > > # haproxy -v > HA-Proxy version 1.5.4 2014/09/02 > > My config: > global > tune.ssl.default-dh-param 2048 > ssl-default-bind-ciphers > ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK > > frontend https-in > bind111.222.111.222:443 ssl strict-sni no-sslv3 crt-list > /etc/haproxy_aux2_pools/crt.list > errorfile 408 /dev/null > option http-keep-alive > option http-server-close > http-requestadd-header X-Forwarded-Port %[dst_port] > http-requestadd-header X-Forwarded-Proto https > use_backend apache_aux2_workers > > # ps -o s,pid,start,comm -C haproxy_aux2_pools > S PID STARTED COMMAND > D 472 Apr 07 haproxy_aux2_po > D 725 Apr 07 haproxy_aux2_po > D 1185 Apr 07 haproxy_aux2_po > D 1706 Apr 07 haproxy_aux2_po > D 2168 Apr 07 haproxy_aux2_po > D 2749 Apr 07 haproxy_aux2_po > D 2996 Apr 07 haproxy_aux2_po > D 3620 Apr 07 haproxy_aux2_po > D 3960 Apr 07 haproxy_aux2_po > > and kernel trace: > Apr 7 17:40:23 l4 kernel: Unable to handle kernel paging request at > fff4 RIP: > Apr 7 17:40:23 l4 kernel: [] > dma_unpin_iovec_pages+0x10/0x80 > Apr 7 17:40:23 l4 kernel: PGD 203067 PUD 204067 PMD 0 > Apr 7 17:40:23 l4 kernel: Oops: [1] SMP > Apr 7 17:40:23 l4 kernel: CPU 0 > Apr 7 17:40:23 l4 kernel: Pid: 17747, comm: haproxy_aux2_po Not tainted > 2.6.24-1gb-1 #4 > Apr 7 17:40:23 l4 kernel: RIP: 0010:[] > [] dma_unpin_iovec_pages+0x10/0x80 > Apr 7 17:40:23 l4 kernel: RSP: 0018:8101164dbbb8 EFLAGS: 00010282 > Apr 7 17:40:23 l4 kernel: RAX: 0001 RBX: > RCX: > Apr 7 17:40:23 l4 kernel: RDX: RSI: > RDI: fff4 > Apr 7 17:40:23 l4 kernel: RBP: 8102acf5c6b0 R08: 0040 > R09: > Apr 7 17:40:23 l4 kernel: R10: 80629900 R11: 80398920 > R12: 8102acf5c600 > Apr 7 17:40:23 l4 kernel: R13: 8102acf5c6b0 R14: fff4 > R15: 7fff > Apr 7 17:40:23 l4 kernel: FS: 2b5d03469b20() > GS:8062f000() knlGS: > Apr 7 17:40:23 l4 kernel: CS: 0010 DS: ES: CR0: > 80050033 > Apr 7 17:40:23 l4 kernel: CR2: fff4 CR3: 0001c50f2000 > CR4: 06e0 > Apr 7 17:40:23 l4 kernel: DR0: DR1: > DR2: > Apr 7 17:40:23 l4 kernel: DR3: DR6: 0ff0 > DR7: 0400 > Apr 7 17:40:23 l4 kernel: Process haproxy_aux2_po (pid: 17747, threadinfo > 8101164da000, task 8101b733a000) > Apr 7 17:40:23 l4 kernel: Stack: 8102acf5c6b0 > 8102acf5c600 8102acf5c6b0 > Apr 7 17:40:23 l4 kernel: 8102acf5c9dc 804e2f11 > 810010535900 804e1a53 > Apr 7 17:40:23 l4 kernel: 4020 > 8101164dbee8 07524a80 > Apr 7 17:40:23 l4 kernel: Call Trace: > Apr 7 17:40:23 l4 kernel: Call Trace: > Apr 7 17:40:23 l4 kernel: [] tcp_recvmsg+0x581/0xcd0 > Apr 7 17:40:23 l4 kernel: [] tcp_sendmsg+0x593/0xc30 > Apr 7 17:40:23 l4 kernel: [] _spin_lock_bh+0x9/0x20 > Apr 7 17:40:23 l4 kernel: [] release_sock+0x13/0xb0 > Apr 7 17:40:23 l4 kernel: [] > sock_common_recvmsg+0x30/0x50 > Apr 7 17:40:23 l4 kernel: [] sock_recvmsg+0x14a/0x160 > Apr 7 17:40:23 l4 kernel: [] filemap_fault+0x21e/0x420 > Apr 7 17:40:23 l4 kernel: [] > autoremove_wake_function+0x0/0x30 > Apr 7 17:40:23 l4 kernel: [] __do_fault+0x1e5/0x460 > Apr 7 17:40:23 l4 kernel: [] handle_mm_fault+0x1af/0x7c0 > Apr 7 17:40:23 l4 kernel: [] sys_recvfrom+0xfe/0x1a0 > Apr 7 17:40:23 l4 kernel: [] do_page_fault+0x1e0/0x830 > Apr 7 17:40:23 l4 kernel: [] vma_merge+0x161/0x1f0 > Apr 7 17:40:23 l4 kernel: [] system_call+0x7e/0x83 > Apr 7 17:40:23 l4 kernel: > Apr 7 17:40:23 l4 kernel: > Apr 7 17:40:23 l4 kernel: Code: 8b 37 85 f6 7e 51 48 8d 6f 08 45 31 ed 0f > 1f 00 8b 4d 08
haproxy hangs on old linux kernels (2.6.24)
Hi, I have some linux boxes with very old kernels. Unfortunately, I cannot upgrade them due to the fact that they work very stable. for example,their uptime is already some years, which is not true speaking about modern kernels. But there is one problem: HAPproxy hangs when I turn on SSL options. # haproxy -v HA-Proxy version 1.5.4 2014/09/02 My config: global tune.ssl.default-dh-param 2048 ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK frontend https-in bind111.222.111.222:443 ssl strict-sni no-sslv3 crt-list /etc/haproxy_aux2_pools/crt.list errorfile 408 /dev/null option http-keep-alive option http-server-close http-requestadd-header X-Forwarded-Port %[dst_port] http-requestadd-header X-Forwarded-Proto https use_backend apache_aux2_workers # ps -o s,pid,start,comm -C haproxy_aux2_pools S PID STARTED COMMAND D 472 Apr 07 haproxy_aux2_po D 725 Apr 07 haproxy_aux2_po D 1185 Apr 07 haproxy_aux2_po D 1706 Apr 07 haproxy_aux2_po D 2168 Apr 07 haproxy_aux2_po D 2749 Apr 07 haproxy_aux2_po D 2996 Apr 07 haproxy_aux2_po D 3620 Apr 07 haproxy_aux2_po D 3960 Apr 07 haproxy_aux2_po and kernel trace: Apr 7 17:40:23 l4 kernel: Unable to handle kernel paging request at fff4 RIP: Apr 7 17:40:23 l4 kernel: [] dma_unpin_iovec_pages+0x10/0x80 Apr 7 17:40:23 l4 kernel: PGD 203067 PUD 204067 PMD 0 Apr 7 17:40:23 l4 kernel: Oops: [1] SMP Apr 7 17:40:23 l4 kernel: CPU 0 Apr 7 17:40:23 l4 kernel: Pid: 17747, comm: haproxy_aux2_po Not tainted 2.6.24-1gb-1 #4 Apr 7 17:40:23 l4 kernel: RIP: 0010:[] [] dma_unpin_iovec_pages+0x10/0x80 Apr 7 17:40:23 l4 kernel: RSP: 0018:8101164dbbb8 EFLAGS: 00010282 Apr 7 17:40:23 l4 kernel: RAX: 0001 RBX: RCX: Apr 7 17:40:23 l4 kernel: RDX: RSI: RDI: fff4 Apr 7 17:40:23 l4 kernel: RBP: 8102acf5c6b0 R08: 0040 R09: Apr 7 17:40:23 l4 kernel: R10: 80629900 R11: 80398920 R12: 8102acf5c600 Apr 7 17:40:23 l4 kernel: R13: 8102acf5c6b0 R14: fff4 R15: 7fff Apr 7 17:40:23 l4 kernel: FS: 2b5d03469b20() GS:8062f000() knlGS: Apr 7 17:40:23 l4 kernel: CS: 0010 DS: ES: CR0: 80050033 Apr 7 17:40:23 l4 kernel: CR2: fff4 CR3: 0001c50f2000 CR4: 06e0 Apr 7 17:40:23 l4 kernel: DR0: DR1: DR2: Apr 7 17:40:23 l4 kernel: DR3: DR6: 0ff0 DR7: 0400 Apr 7 17:40:23 l4 kernel: Process haproxy_aux2_po (pid: 17747, threadinfo 8101164da000, task 8101b733a000) Apr 7 17:40:23 l4 kernel: Stack: 8102acf5c6b0 8102acf5c600 8102acf5c6b0 Apr 7 17:40:23 l4 kernel: 8102acf5c9dc 804e2f11 810010535900 804e1a53 Apr 7 17:40:23 l4 kernel: 4020 8101164dbee8 07524a80 Apr 7 17:40:23 l4 kernel: Call Trace: Apr 7 17:40:23 l4 kernel: Call Trace: Apr 7 17:40:23 l4 kernel: [] tcp_recvmsg+0x581/0xcd0 Apr 7 17:40:23 l4 kernel: [] tcp_sendmsg+0x593/0xc30 Apr 7 17:40:23 l4 kernel: [] _spin_lock_bh+0x9/0x20 Apr 7 17:40:23 l4 kernel: [] release_sock+0x13/0xb0 Apr 7 17:40:23 l4 kernel: [] sock_common_recvmsg+0x30/0x50 Apr 7 17:40:23 l4 kernel: [] sock_recvmsg+0x14a/0x160 Apr 7 17:40:23 l4 kernel: [] filemap_fault+0x21e/0x420 Apr 7 17:40:23 l4 kernel: [] autoremove_wake_function+0x0/0x30 Apr 7 17:40:23 l4 kernel: [] __do_fault+0x1e5/0x460 Apr 7 17:40:23 l4 kernel: [] handle_mm_fault+0x1af/0x7c0 Apr 7 17:40:23 l4 kernel: [] sys_recvfrom+0xfe/0x1a0 Apr 7 17:40:23 l4 kernel: [] do_page_fault+0x1e0/0x830 Apr 7 17:40:23 l4 kernel: [] vma_merge+0x161/0x1f0 Apr 7 17:40:23 l4 kernel: [] system_call+0x7e/0x83 Apr 7 17:40:23 l4 kernel: Apr 7 17:40:23 l4 kernel: Apr 7 17:40:23 l4 kernel: Code: 8b 37 85 f6 7e 51 48 8d 6f 08 45 31 ed 0f 1f 00 8b 4d 08 85 Apr 7 17:40:23 l4 kernel: RIP [] dma_unpin_iovec_pages+0x10/0x80 Apr 7 17:40:23 l4 kernel: RSP Apr 7 17:40:23 l4 kernel: CR2: fff4 Apr 7 17:40:23 l4 kernel: ---[ end trace e1ec26f01a394080 ]--- Can it be fixed in haproxy? Or it can only be solved by kernel updating? Thanks for help.
