Re: 100% cpu , epoll_wait()

2016-06-10 Thread Willy Tarreau
Hi Holger, On Fri, Jun 10, 2016 at 04:32:55PM +0200, Holger Just wrote: > Hi Willy et al., > > > Thank you for this report, it helps. How often does it happen, and/or after > > how long on average after you start it ? What's your workload ? Do you use > > SSL, compression, TCP and/or HTTP mode,

Re: 100% cpu , epoll_wait()

2016-06-10 Thread Holger Just
Hi Willy et al., > Thank you for this report, it helps. How often does it happen, and/or after > how long on average after you start it ? What's your workload ? Do you use > SSL, compression, TCP and/or HTTP mode, peers synchronization, etc ? Yesterday, we upgraded from 1.5.14 to 1.5.18 and now

Re: 100% cpu , epoll_wait()

2016-05-23 Thread Willy Tarreau
Hi Veiko, On Mon, May 23, 2016 at 11:39:02AM +0300, Veiko Kukk wrote: > I can confirm that on CentOS 6 with HAproxy 1.6.5 this 100% CPU load still > happens. Exactly the same: > epoll_wait(0, {}, 200, 0) = 0 > epoll_wait(0, {}, 200, 0) = 0 > epoll_wait(0, {}, 200, 0)

Re: 100% cpu , epoll_wait()

2016-05-23 Thread Veiko Kukk
On 18/05/16 15:42, Willy Tarreau wrote: Hi Sebastian, On Thu, May 12, 2016 at 09:58:22AM +0200, Sebastian Heid wrote: Hi Lukas, starting from around 200mbit/s in, haproxy processes (nbproc 6) are hitting 100% cpu regularly (noticed up to 3 processes at the same time with 100%), but recover

Re: 100% cpu , epoll_wait()

2016-05-18 Thread Willy Tarreau
On Wed, May 18, 2016 at 03:39:34PM +0200, Sebastian Heid wrote: > Hi Willy, > > thanks for the feedback. I'm using OBS (Open Build Service) for creating > CentOS/RHEL RPMs, so it should be as clean as it can get. OK thanks for checking. > When I find some time, I'll try to reproduce it and will

Re: 100% cpu , epoll_wait()

2016-05-18 Thread Sebastian Heid
<w...@1wt.eu> > Gesendet: Mit 18 Mai 2016 14:44 > An: Sebastian Heid <m...@sebastian-heid.de> > CC: Lukas Tribus <lu...@gmx.net>; HAProxy <haproxy@formilux.org> > Betreff: Re: 100% cpu , epoll_wait() > > Hi Sebastian, > > On Thu, May 12, 2016 at 0

Re: 100% cpu , epoll_wait()

2016-05-18 Thread Willy Tarreau
Hi Sebastian, On Thu, May 12, 2016 at 09:58:22AM +0200, Sebastian Heid wrote: > Hi Lukas, > > starting from around 200mbit/s in, haproxy processes (nbproc 6) are > hitting 100% cpu regularly (noticed up to 3 processes at the same time with > 100%), but recover again on its own after some time.

Re: 100% cpu , epoll_wait()

2016-05-13 Thread Willy Tarreau
Hi Lukas, On Fri, May 13, 2016 at 06:36:38PM +0200, Lukas Tribus wrote: > Not sure if that's what you meant by the other issue, but if there are still > buffer issues it may also caused the reported crash in zlib (since 1.6.4 but > also affects 1.6.5), that would be thread "Crash with kernel

Re: 100% cpu , epoll_wait()

2016-05-13 Thread Lukas Tribus
Hi Willy, Am 13.05.2016 um 17:01 schrieb Willy Tarreau: Hi Sebastian, On Thu, May 12, 2016 at 09:58:22AM +0200, Sebastian Heid wrote: Hi Lukas, starting from around 200mbit/s in, haproxy processes (nbproc 6) are hitting 100% cpu regularly (noticed up to 3 processes at the same time with

Re: 100% cpu , epoll_wait()

2016-05-13 Thread Willy Tarreau
Hi Sebastian, On Thu, May 12, 2016 at 09:58:22AM +0200, Sebastian Heid wrote: > Hi Lukas, > > starting from around 200mbit/s in, haproxy processes (nbproc 6) are > hitting 100% cpu regularly (noticed up to 3 processes at the same time with > 100%), but recover again on its own after some time.

AW: 100% cpu , epoll_wait()

2016-05-12 Thread Sebastian Heid
with way higher bandwidth. Bye, Sebastian -Ursprüngliche Nachricht- > Von:Lukas Tribus <lu...@gmx.net> > Gesendet: Mit 11 Mai 2016 22:55 > An: Sebastian Heid <m...@sebastian-heid.de>; HAProxy <haproxy@formilux.org> > Betreff: Re: 100% cpu , epoll_wait()

Re: 100% cpu , epoll_wait()

2016-05-11 Thread Lukas Tribus
Hi Sebastian, Am 11.05.2016 um 16:07 schrieb Sebastian Heid: Hi, I updated from 1.5.17 to 1.5.18 today, but sadly this issue still exits in the latest version in our environment. However downgrading to 1.5.14 "fixed" the issue for us. Seems like a different issue then. Can you elaborate

Re: 100% cpu , epoll_wait()

2016-05-11 Thread Sebastian Heid
result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Bye, Sebastian -Ursprüngliche Nachricht- > Von:Jim Freeman <sovr...@gmail.com> > Gesendet: Fre 22 April 2016 01:21 > An: HAProxy <haproxy@formilux.

Re: 100% cpu , epoll_wait()

2016-04-21 Thread Jim Freeman
[ Apologies for consuming yet more vertical space ] With this in .cfg : log-format

Re: 100% cpu , epoll_wait()

2016-04-21 Thread Jim Freeman
Another alert+followup : Cpu pegged again - connected to host and ran : == # netstat -pantu | egrep "(^Proto|:5)" Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 0.0.0.0:5 0.0.0.0:* LISTEN 7944/haproxy tcp

Re: 100% cpu , epoll_wait()

2016-04-20 Thread Willy Tarreau
On Wed, Apr 20, 2016 at 11:50:20AM +0300, Veiko Kukk wrote: > >is not good unfortunately. I'm assuming this is with 1.5.17, that's it ? If > >so we still have an issue :-/ > > It is 1.6.3 > > # haproxy -vv > HA-Proxy version 1.6.3 2015/12/25 > Copyright 2000-2015 Willy Tarreau

Re: 100% cpu , epoll_wait()

2016-04-20 Thread Veiko Kukk
On 20/04/16 11:43, Willy Tarreau wrote: On Tue, Apr 19, 2016 at 09:53:36PM +0300, Veiko Kukk wrote: On 19/04/16 18:52, Willy Tarreau wrote: On Tue, Apr 19, 2016 at 04:15:08PM +0200, Willy Tarreau wrote: OK in fact it's different. Above we have a busy polling loop, which may very be caused by

Re: 100% cpu , epoll_wait()

2016-04-19 Thread Veiko Kukk
On 19/04/16 18:52, Willy Tarreau wrote: On Tue, Apr 19, 2016 at 04:15:08PM +0200, Willy Tarreau wrote: OK in fact it's different. Above we have a busy polling loop, which may very be caused by the buffer space miscalculation bug and which results in a process not completing its job until a

Re: 100% cpu , epoll_wait()

2016-04-19 Thread Willy Tarreau
On Tue, Apr 19, 2016 at 04:15:08PM +0200, Willy Tarreau wrote: > On Tue, Apr 19, 2016 at 02:54:35PM +0200, Lukas Tribus wrote: > > >We use haproxy 1.6.3 (latest CentOS 6.7) and experience similar situation > > >after some reloads (-sf). The old haproxy process does not exit and uses > > >100% cpu,

Re: 100% cpu , epoll_wait()

2016-04-19 Thread Willy Tarreau
Hi guys, On Tue, Apr 19, 2016 at 02:54:35PM +0200, Lukas Tribus wrote: > >We use haproxy 1.6.3 (latest CentOS 6.7) and experience similar situation > >after some reloads (-sf). The old haproxy process does not exit and uses > >100% cpu, strace showing: > >epoll_wait(0, {}, 200, 0) =

Re: 100% cpu , epoll_wait()

2016-04-19 Thread Lukas Tribus
Hi, Am 19.04.2016 um 09:39 schrieb Veiko Kukk: We use haproxy 1.6.3 (latest CentOS 6.7) and experience similar situation after some reloads (-sf). The old haproxy process does not exit and uses 100% cpu, strace showing: epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0)

Re: 100% cpu , epoll_wait()

2016-04-19 Thread Veiko Kukk
On 16/04/16 01:53, Jim Freeman wrote: I'm suspecting that a connection to the stats port goes wonky with a '-sf' reload, but I'll have to wait for it to re-appear to poke further. I'll look first for a stats port connection handled by the pegged process, then use 'tcpkill' to kill just that

Re: 100% cpu , epoll_wait()

2016-04-15 Thread Jim Freeman
Did a bit more digging on the most recent instance, and found that the haproxy pid doing the hogging was handling a connection to the stats port : listen haproxy_stats :5 stats enable stats uri / no log , with this 'netstat -pantlu' entry : tcp0 99756

Re: 100% cpu , epoll_wait()

2016-04-15 Thread Cyril Bonté
Hi Jim, Le 15/04/2016 23:20, Jim Freeman a écrit : I have haproxy slaved to 2d cpu (CPU1), with frequent config changes and a '-sf' soft-stop with the now-old non-listening process nannying old connections. Sometimes CPU1 goes to %100, and then a few minutes later request latencies suffer

100% cpu , epoll_wait()

2016-04-15 Thread Jim Freeman
I have haproxy slaved to 2d cpu (CPU1), with frequent config changes and a '-sf' soft-stop with the now-old non-listening process nannying old connections. Sometimes CPU1 goes to %100, and then a few minutes later request latencies suffer across multiple haproxy peers. An strace of the nanny