Hi Lukas,
We upgraded to 1.6, went back to nbproc 1 from 12 and the problem showed
up again. Haproxy hitting 90-100% and monitors reported download speed
drop from 100MBPS to 10MBPS immediately.
I ran strace as you said, output it huge, have attached a small subset of
it in the email. Please let me know if you need more of strace output.
Thanks
Sachin
On 4/7/16, 5:51 PM, "Lukas Tribus" wrote:
>Hi,
>
>Am 05.04.2016 um 09:38 schrieb Sachin Shetty:
>> Hi Lukas, Pavlos,
>>
>> Thanks for your response, more info as requested.
>>
>> 1. Attached conf with some obfuscation
>> 2. Haproxy -vv
>> HA-Proxy version 1.5.4 2014/09/02
>> Copyright 2000-2014 Willy Tarreau
>>
>
>I would upgrade to something more recent, the number of bugfixes
>since 1.5.4 amount to more than 100!
>
>That said, I've not stumbled upon a particular bug explaining what
>you are seeing.
>
>My suggestion would be to go back to nbproc 1 (its easier to
>troubleshoot), and run the 100% spinning process through
>strace -tt -p and post the output.
>
>
>
>
>Thanks,
>
>Lukas
23:30:41.257757 sendto(120, "...", 16384, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_MORE,
NULL, 0) = 16384
23:30:41.258001 sendto(87, "...", 919, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 919
23:30:41.258077 read(33, "\27\3\3\0020", 5) = 5
23:30:41.258134 read(33, "...", 560) = 560
23:30:41.258201 read(3, "\26\3\3\0F", 5) = 5
23:30:41.258244 read(3, "...", 70) = 70
23:30:41.259294 read(3, "\24\3\3\0\1", 5) = 5
23:30:41.259347 read(3, "\1", 1)= 1
23:30:41.259514 read(3, "\26\3\3\0@", 5) = 5
23:30:41.259559 read(3, "...", 64) = 64
23:30:41.259668 write(3, "...", 75) = 75
23:30:41.259748 read(3, 0x7feeaed21343, 5) = -1 EAGAIN (Resource temporarily
unavailable)
23:30:41.259818 read(71, "\26\3\1\2\6", 5) = 5
23:30:41.259863 read(71, "...", 518) = 518
23:30:41.280711 read(71, "\24\3\1\0\1", 5) = 5
23:30:41.280790 read(71, "\1", 1) = 1
23:30:41.280967 read(71, "\26\3\1\", 5) = 5
23:30:41.281012 read(71, "...", 48) = 48
23:30:41.281121 write(71, "...", 59) = 59
23:30:41.281199 read(71, 0x7feeaed21343, 5) = -1 EAGAIN (Resource temporarily
unavailable)
23:30:41.281246 read(51, "...", 14977) = 14977
23:30:41.281405 sendto(56, "...", 16384, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_MORE,
NULL, 0) = 16384
23:30:41.281472 read(38, 0x7feeaeb15183, 5) = -1 EAGAIN (Resource temporarily
unavailable)
23:30:41.281517 read(140, "...", 7677) = 5840
23:30:41.281562 read(140, 0x7feeaec87a2b, 1837) = -1 EAGAIN (Resource
temporarily unavailable)
23:30:41.281605 read(45, "\27\3\3\2\240", 5) = 5
23:30:41.281647 read(45, "...", 672) = 672
23:30:41.281699 read(31, "...", 48) = 48
23:30:41.281811 sendto(272, "...", 16384, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_MORE,
NULL, 0) = 16384
23:30:41.281948 write(167, "...", 15525) = 15525
23:30:41.282025 read(72, "...", 15923) = 8184
23:30:41.282076 read(72, "...", 7739) = 1364
23:30:41.282119 read(72, 0x7feeaebf89c1, 6375) = -1 EAGAIN (Resource
temporarily unavailable)
23:30:41.282162 read(24, "...", 1837) = 1837
23:30:41.282278 sendto(107, "...", 16384, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_MORE,
NULL, 0) = 16384
23:30:41.282328 recvfrom(41, "...", 16384, 0, NULL, NULL) = 16384
23:30:41.282382 recvfrom(81, "...", 15360, 0, NULL, NULL) = 214
23:30:41.282438 write(21, "...", 389) = 389
23:30:41.282497 write(25, "...", 389) = 389
23:30:41.282563 write(25, "...", 53) = 53
23:30:41.282613 shutdown(25, SHUT_WR) = 0
23:30:41.282660 read(18, 0x7feeae813be3, 5) = -1 EAGAIN (Resource temporarily
unavailable)
23:30:41.282704 sendto(92, "...", 818, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 818
23:30:41.282753 read(39, 0x7feeae813be3, 5) = -1 EAGAIN (Resource temporarily
unavailable)
23:30:41.282796 sendto(88, "...", 2062, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) =
2062
23:30:41.282944 getsockname(33, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("Some-IP")}, [16]) = 0
23:30:41.283008 getsockopt(33, SOL_IP, 0x50 /* IP_??? */,
"\2\0\1\273\n\31\220\17\0\0\0\0\0\0\0\0", [16]) = 0
23:30:41.283082 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 77
23:30:41.283132 fcntl(77, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
23:30:41.283188 setsockopt(77, SOL_TCP, TCP_NODELAY, [1], 4) = 0
23:30:41.283233 connect(77, {sa_family=AF_INET, sin_port=htons(7300),
sin_addr=inet_addr("Some-IP")}, 16) = -1 EINPROGRESS (Operation now in progress)
23:30:41.283415 getsockname(45, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("Some-IP")}, [16]) = 0
23:30:41.283467 getsockopt(45, SOL_IP, 0x50 /* IP_??? */,
"\2\0\1\273\n\31\220\17\0\0\0\0\0\0\0\0", [16]) = 0
23:30:41.283521 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 79
23:30:41.283565 fcntl(79, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
23:30:41.283605 setsockopt(79, SOL_TCP, TCP_NODELAY, [1], 4) = 0
23:30:41.283647 connect(79, {sa_family=AF_INET, sin_port=htons(9930),
sin_addr=inet_addr("Some-IP")}, 16) = -1 EINPROGRESS (Operation now in progress)
23:30:41.283723 setsockopt(81, SOL_SOCKET, SO_LINGER, {onoff=1, linger=0}, 8) = 0
23:30:41.283772 close(81)