Re: Haproxy running on 100% CPU and slow downloads

To close this thread out: we found the issue to be in 1.6.4-20160426 patch that I was using. The issue is fixed in 1.6.5. Thanks Willy and Lukas. Thanks Sachin On 5/13/16, 8:14 PM, "Willy Tarreau" wrote: >On Fri, May 13, 2016 at 07:32:36PM +0530, Sachin Shetty wrote: >> In 24

Re: Haproxy running on 100% CPU and slow downloads

On Fri, May 13, 2016 at 07:32:36PM +0530, Sachin Shetty wrote: > In 24 hours all servers had connections growing, we have reverted the > patch for now. > > I have the show sess all output if you would like to see. Interestingly in the "show sess all" from yesterday I'm seeing only negative

Re: Haproxy running on 100% CPU and slow downloads

Hi Sachin, On Fri, May 13, 2016 at 07:32:36PM +0530, Sachin Shetty wrote: > In 24 hours all servers had connections growing, we have reverted the > patch for now. > > I have the show sess all output if you would like to see. Thank you very much, that's extremely useful. I'll probably get back

Re: Haproxy running on 100% CPU and slow downloads

In 24 hours all servers had connections growing, we have reverted the patch for now. I have the show sess all output if you would like to see. Thanks Sachin On 5/12/16, 10:08 PM, "Sachin Shetty" wrote: >Hi Lukas, > >Attached output. > >Thanks >Sachin > >On 5/12/16, 7:41

Re: Haproxy running on 100% CPU and slow downloads

Hi, Am 12.05.2016 um 14:37 schrieb Sachin Shetty: Hi Willy, We are seeing a strange problem on the patched server. We have several haproxy servers running but only one with the latest patch, and this haproxy has frozen twice in last two days, basically it hits max open connections 2000 on

Re: Haproxy running on 100% CPU and slow downloads

Hi Willy, We are seeing a strange problem on the patched server. We have several haproxy servers running but only one with the latest patch, and this haproxy has frozen twice in last two days, basically it hits max open connections 2000 on frontend and then stalls. From the logs it has 1999

Re: Haproxy running on 100% CPU and slow downloads

On Tue, May 10, 2016 at 11:10:14AM +0530, Sachin Shetty wrote: > We deployed the latest and we saw throughput still dropped around peak > hours a bit, then we swithed to nbproc 4 which is holding up ok. So probably you were reaching the processing limits for a single process, that can easily

Re: Haproxy running on 100% CPU and slow downloads

We deployed the latest and we saw throughput still dropped around peak hours a bit, then we swithed to nbproc 4 which is holding up ok. Note that 4 Cpus was not sufficient earlier, so I believe the latest version is scaling better. Thanks Lukas and Willy. On 4/29/16, 11:09 AM, "Willy Tarreau"

Re: Haproxy running on 100% CPU and slow downloads

Thanks Lukas and Willy. I am in the process of getting 1.6.4-20160426 deployed in our QA, I will keep you guys posted. On 4/29/16, 11:09 AM, "Willy Tarreau" wrote: >Hi guys, > >On Tue, Apr 26, 2016 at 08:46:37AM +0200, Lukas Tribus wrote: >> Hi Sachin, >> >> >> there is another

Re: Haproxy running on 100% CPU and slow downloads

Hi guys, On Tue, Apr 26, 2016 at 08:46:37AM +0200, Lukas Tribus wrote: > Hi Sachin, > > > there is another fix Willy recently committed, its ff9c7e24fb [1] > and its in the snapshots [2] since 1.6.4-20160426. > > This is supposed to fix the issue altogether. > > Please let us know if this

Re: Haproxy running on 100% CPU and slow downloads

Hi Sachin, there is another fix Willy recently committed, its ff9c7e24fb [1] and its in the snapshots [2] since 1.6.4-20160426. This is supposed to fix the issue altogether. Please let us know if this works for you. Thanks, Lukas [1]

Re: Haproxy running on 100% CPU and slow downloads

Hi Lukas, We tried the patch, it seems better. As soon as we switched nbproc off, throughput did not drop immediately like it did with earlier version, it started deteriorating slowly as traffic increased to peak hours, but eventually it did crash to the same levels as before. CPU Usage was also

Re: Haproxy running on 100% CPU and slow downloads

Hi, Am 21.04.2016 um 08:11 schrieb Sachin Shetty: Hi, any hints to further isolate this - we have deferred the problem by adding all the cores we had, but I have a feeling that our request rate is not that high (7K per minute a peak) and it will show up again as traffic increases. Thanks

Re: Haproxy running on 100% CPU and slow downloads

Hi, any hints to further isolate this - we have deferred the problem by adding all the cores we had, but I have a feeling that our request rate is not that high (7K per minute a peak) and it will show up again as traffic increases. Thanks Sachin On 4/18/16, 12:22 PM, "Sachin Shetty"

Re: Haproxy running on 100% CPU and slow downloads

Hi Lukas, We upgraded to 1.6, went back to nbproc 1 from 12 and the problem showed up again. Haproxy hitting 90-100% and monitors reported download speed drop from 100MBPS to 10MBPS immediately. I ran strace as you said, output it huge, have attached a small subset of it in the email. Please let

Re: Haproxy running on 100% CPU and slow downloads

agree to both the points. Thanks Sachin On 4/7/16, 11:24 PM, "Willy Tarreau" wrote: >On Thu, Apr 07, 2016 at 10:59:24PM +0530, Sachin Shetty wrote: >> Hi Willy, >> >> Sorry for the confusion. I wrote to you much before in my >>investigation. I >> will take care going forward. >

Re: Haproxy running on 100% CPU and slow downloads

On Thu, Apr 07, 2016 at 10:59:24PM +0530, Sachin Shetty wrote: > Hi Willy, > > Sorry for the confusion. I wrote to you much before in my investigation. I > will take care going forward. OK but in general the point remains, and it's not just for you but for everyone in general, the mailing list

Re: Haproxy running on 100% CPU and slow downloads

Hi Willy, Sorry for the confusion. I wrote to you much before in my investigation. I will take care going forward. Only now I realized that I messed up the version numbers because it seems we have different versions in our cluster. We are now testing with 1.6.4 and trying to fast track it.

Re: Haproxy running on 100% CPU and slow downloads

Hi Sachin, On Thu, Apr 07, 2016 at 02:21:16PM +0200, Lukas Tribus wrote: > Hi, > > Am 05.04.2016 um 09:38 schrieb Sachin Shetty: > >Hi Lukas, Pavlos, > > > >Thanks for your response, more info as requested. > > > >1. Attached conf with some obfuscation > >2. Haproxy -vv > >HA-Proxy version 1.5.4

Re: Haproxy running on 100% CPU and slow downloads

Hi, Am 05.04.2016 um 09:38 schrieb Sachin Shetty: Hi Lukas, Pavlos, Thanks for your response, more info as requested. 1. Attached conf with some obfuscation 2. Haproxy -vv HA-Proxy version 1.5.4 2014/09/02 Copyright 2000-2014 Willy Tarreau I would upgrade to something more

Re: Haproxy running on 100% CPU and slow downloads

Hi Lukas, Pavlos, Thanks for your response, more info as requested. 1. Attached conf with some obfuscation 2. Haproxy -vv HA-Proxy version 1.5.4 2014/09/02 Copyright 2000-2014 Willy Tarreau Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2

Re: Haproxy running on 100% CPU and slow downloads

Hi Sachin, (due to email troubles on my side this may look like a new thread, sorry about that) > We have quite a few regex and acls in our config, is there a way to profile > haproxy and see what could be slowing it down? You can use strace for syscalls or ltrace for library calls to see

Re: Haproxy running on 100% CPU and slow downloads

On 04/04/2016 05:23 μμ, Sachin Shetty wrote: > Hi, > > I am chasing some weird capacity issues in our setup. > > Haproxy which also does SSL is forwarding request to various other > servers upstream. I am seeing a simple 100MB file download from our > upstream components starts to slow down

Haproxy running on 100% CPU and slow downloads

Hi, I am chasing some weird capacity issues in our setup. Haproxy which also does SSL is forwarding request to various other servers upstream. I am seeing a simple 100MB file download from our upstream components starts to slow down time to time like hitting as low as 1MBPS, usually is it