Re: Long Running TCP Connections and Reloads

[Pean, David S.]

We are using a patched version from April that fixed a big memory leak that 
occurred on another project, so it looks like it would have had that fix.

HAPROXY_VERSION=1.7.7a450ca
HAPROXY_MD5=fa9c6d9af600e12ef31567b941fa1dfb

Dave

From: "Krishna Kumar (Engineering)" 
<krishna...@flipkart.com<mailto:krishna...@flipkart.com>>
Date: Friday, September 15, 2017 at 1:23 AM
To: "david.s.p...@espn.com<mailto:david.s.p...@espn.com>" 
<david.s.p...@espn.com<mailto:david.s.p...@espn.com>>
Cc: "haproxy@formilux.org<mailto:haproxy@formilux.org>" 
<haproxy@formilux.org<mailto:haproxy@formilux.org>>
Subject: Re: Long Running TCP Connections and Reloads

Regarding #1, I think this was fixed sometime back. Maybe you are running
a old version of haproxy?

commit e39683c4d4c527d1b561c3ba3983d26cc3e7f42d
Author: Hongbo Long <dragonorlo...@gmail.com<mailto:dragonorlo...@gmail.com>>
Date:   Fri Mar 10 18:41:51 2017 +0100

BUG/MEDIUM: stream: fix client-fin/server-fin handling

A tcp half connection can cause 100% CPU on expiration.



On Thu, Sep 14, 2017 at 6:59 PM, Pean, David S. 
<david.s.p...@espn.com<mailto:david.s.p...@espn.com>> wrote:
Hello!

I am using a TCP front-end that potentially keeps connections open for several 
hours, while also frequently issuing reloads due to an id to server mapping 
that is changing constantly. This causes many processes to be running at any 
given time, which generally works as expected. However, after some time I see 
some strange behavior with the processes and stats that doesn’t appear to have 
any pattern to it.

Here is the setup in general:

Every two minutes, there is a process that checks if HAProxy should be 
reloaded. If that is the case, this command is run:

/usr/local/sbin/haproxy -D –f -sf PID

The PID is the current HAProxy process. If there are TCP connections to that 
process, it will stay running until those connection drop, then generally it 
will get killed.

1. Sometimes a process will appear to not get killed, and have no connections. 
It will be running for several hours and have 99 CPU. When straced, it doesn't 
appear to be actually doing anything -- just clock and polls very frequently. 
Is there some sort of timeout for the graceful shutdown of the old processes?

2. Is it possible for the old processes to accept new connections? Even though 
a pid has been sent the shutdown signal, I have seen requests reference old 
server mappings that would have been in an earlier process.

3. Often the stats page will become out of whack over time. The number of 
requests per second will become drastically different from what is actually 
occuring. It looks like the old stuck processes might be sending more data that 
is maybe not getting cleared?

Are there any considerations for starting up or reloading when dealing with 
long running connections?

Thanks!

David Pean

Re: Long Running TCP Connections and Reloads

[Krishna Kumar (Engineering)]

Regarding #1, I think this was fixed sometime back. Maybe you are running
a old version of haproxy?

commit e39683c4d4c527d1b561c3ba3983d26cc3e7f42d
Author: Hongbo Long 
Date:   Fri Mar 10 18:41:51 2017 +0100

BUG/MEDIUM: stream: fix client-fin/server-fin handling

A tcp half connection can cause 100% CPU on expiration.



On Thu, Sep 14, 2017 at 6:59 PM, Pean, David S. 
wrote:

> Hello!
>
> I am using a TCP front-end that potentially keeps connections open for
> several hours, while also frequently issuing reloads due to an id to server
> mapping that is changing constantly. This causes many processes to be
> running at any given time, which generally works as expected. However,
> after some time I see some strange behavior with the processes and stats
> that doesn’t appear to have any pattern to it.
>
> Here is the setup in general:
>
> Every two minutes, there is a process that checks if HAProxy should be
> reloaded. If that is the case, this command is run:
>
> /usr/local/sbin/haproxy -D –f -sf PID
>
> The PID is the current HAProxy process. If there are TCP connections to
> that process, it will stay running until those connection drop, then
> generally it will get killed.
>
> 1. Sometimes a process will appear to not get killed, and have no
> connections. It will be running for several hours and have 99 CPU. When
> straced, it doesn't appear to be actually doing anything -- just clock and
> polls very frequently. Is there some sort of timeout for the graceful
> shutdown of the old processes?
>
> 2. Is it possible for the old processes to accept new connections? Even
> though a pid has been sent the shutdown signal, I have seen requests
> reference old server mappings that would have been in an earlier process.
>
> 3. Often the stats page will become out of whack over time. The number of
> requests per second will become drastically different from what is actually
> occuring. It looks like the old stuck processes might be sending more data
> that is maybe not getting cleared?
>
> Are there any considerations for starting up or reloading when dealing
> with long running connections?
>
> Thanks!
>
> David Pean
>
>
>

Long Running TCP Connections and Reloads

[Pean, David S.]

Hello!

I am using a TCP front-end that potentially keeps connections open for several 
hours, while also frequently issuing reloads due to an id to server mapping 
that is changing constantly. This causes many processes to be running at any 
given time, which generally works as expected. However, after some time I see 
some strange behavior with the processes and stats that doesn’t appear to have 
any pattern to it.

Here is the setup in general:

Every two minutes, there is a process that checks if HAProxy should be 
reloaded. If that is the case, this command is run:

/usr/local/sbin/haproxy -D –f -sf PID

The PID is the current HAProxy process. If there are TCP connections to that 
process, it will stay running until those connection drop, then generally it 
will get killed.

1. Sometimes a process will appear to not get killed, and have no connections. 
It will be running for several hours and have 99 CPU. When straced, it doesn't 
appear to be actually doing anything -- just clock and polls very frequently. 
Is there some sort of timeout for the graceful shutdown of the old processes?

2. Is it possible for the old processes to accept new connections? Even though 
a pid has been sent the shutdown signal, I have seen requests reference old 
server mappings that would have been in an earlier process.

3. Often the stats page will become out of whack over time. The number of 
requests per second will become drastically different from what is actually 
occuring. It looks like the old stuck processes might be sending more data that 
is maybe not getting cleared?

Are there any considerations for starting up or reloading when dealing with 
long running connections?

Thanks!

David Pean