AW: Firewall TCP Timeouts

2009-11-18 Thread Conny Martin
We had the same issue. We solved this by using libkeepalive 
(http://libkeepalive.sourceforge.net/) on the arserver machine. But it would 
work only if you are using Linux.

HTH 

Kind Regards Conny 

-Ursprüngliche Nachricht-
Von: Action Request System discussion list(ARSList) 
[mailto:arsl...@arslist.org] Im Auftrag von Leihkauff, Kenneth
Gesendet: Mittwoch, 18. November 2009 18:18
An: arslist@ARSLIST.ORG
Betreff: Re: Firewall TCP Timeouts

Thanks for your time, Axton.  I spoke with our network engineer and he studied 
the firewall logs and sees quit a few denials and some resets, but is confident 
things should be set up correctly on the firewall.  He only sees these 
denials/resets for the Midtier application, not some of our other apps.  

Here is what we're observing on the Linux Midtier server.  Below is an example 
of 3 existing tcp connections that remain intact indefinitely or until midtier 
is restarted.  The firewall knows about these 3 tcp connections, but if the 
connections are idle for more than 60 minutes (default firewall setting), the 
firewall will drop these idle tcp connections.  Then, when a user gets on 
midtier (after the idle period), midtier will attempt to use one or more of 
these tcp connections and the firewall responds with a Deny (since it has aged 
off the inactive connection).  This is normal behavior for a firewall as I 
understand things, but the application (midtier in this case) should ideally 
make use of the tcp keepalive or inactivetly timeout supported by Linux.  
Apparently, midtier does not take advantage of this.

I suspect other customers are having this problem IF their topology utilizes a 
firewall between Midtier and ARS and there are long periods of inactivity.  The 
reason, however, they may not be complaining is because once you get the RCP 
failure (arerr 91), then subsequent connections seem to work fine.  For the 
example below, once the arerr 91 timeout finally occurred, the associated tcp 
connection was terminated on the midtier server and a new one was created.  

$netstat -antopp|grep 2031
Proto Recv-Q Send-Q Local Address   Foreign Address 
State   Timer
tcp0  0 :::172.16.0.129:54310   :::10.1.1.141:2031  
ESTABLISHED 24977/java  off (0.00/0/0)
tcp0864 :::172.16.0.129:54309   :::10.1.1.141:2031  
ESTABLISHED 24977/java  on (3.59/12/0)
tcp0   1416 :::172.16.0.129:43717   :::10.1.1.141:2031  
ESTABLISHED 24977/java  on (16.19/11/0)

After a very long wait trying to login in, the user will receive this message:
ARERR [91]
RPC call failed : ONC/RPC call timed out


Ken 


-Original Message-
From: Action Request System discussion list(ARSList) 
[mailto:arsl...@arslist.org] On Behalf Of Axton
Sent: Tuesday, November 17, 2009 3:24 PM
To: arslist@ARSLIST.ORG
Subject: Re: Firewall TCP Timeouts

You need to configure your firewall properly.  With a firewall, you can define 
what types of packets can create a state entry on the firewall.  Typical is 
syn, syn ack.  Even if there is no state entry, a new packet should create a 
new state (not be dropped).  If a new packet does not create a new state, 
change the firewall rules so that it does.  A network dump will tell you what 
type of packet is going out; the firewall logs should tell you what type of 
packet was rejected.

I could see that this would be a problem if the midtier servers were behind a 
NAT.  Is this the case?

Axton Grams

On Tue, Nov 17, 2009 at 12:07 PM, Leihkauff, Kenneth 
kenneth.g.leihka...@saic.com wrote:
 **

 Hello,



 We have a firewall between our MidTier server and ARS system.  The 
 firewall is configured to drop tcp connections after being idle for 60 
 minutes (typical/default firewall setting).  Several MidTier user 
 sessions will make use of a shared tcp connection so you might have 
 100 sessions but significantly fewer tcp connections.  During idle 
 times (like at night), the firewall will discard these idle tcp 
 connections but the MidTier server will still retain these tcp 
 references (this can be seen by using netstat -anto).  So, when 
 users get back on the system, MidTier apparently is trying to utilize 
 one of these defunct tcp connections so you end up with problems like 
 ARERR 91 rpc timeouts because these tcp connections are broken pipes.



 Is there a MidTier/Tomcat or other setting that you have found 
 addresses this problem?



 Thanks.



 Ken



 Background:

 Version 7.5 patch 3

 MidTier - Linux, Tomcat JSP

 ARS - Linux, Oracle 10g

 _Platinum Sponsor: rmisoluti...@verizon.net ARSlist: Where the 
 Answers Are_

___
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum 
Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are

___
UNSUBSCRIBE 

AW: Firewall TCP Timeouts

2009-11-18 Thread Conny Martin
on the arserver! 

-Ursprüngliche Nachricht-
Von: Action Request System discussion list(ARSList) 
[mailto:arsl...@arslist.org] Im Auftrag von Leihkauff, Kenneth
Gesendet: Mittwoch, 18. November 2009 22:51
An: arslist@ARSLIST.ORG
Betreff: Re: Firewall TCP Timeouts

Thanks, Conny. Are you saying the libkeepalive was implemented on the ARserver 
-- not the MidTier linux server? Our midtier and arserver are linux with a 
firewall in between.  

Unfortunately, the firewall is not an application level firewall so the 
firewall rule changes Axton proposed cannot be implemented according to our 
network engineer.

Ken 

-Original Message-
From: Action Request System discussion list(ARSList) 
[mailto:arsl...@arslist.org] On Behalf Of Conny Martin
Sent: Wednesday, November 18, 2009 3:30 PM
To: arslist@ARSLIST.ORG
Subject: AW: Firewall TCP Timeouts

We had the same issue. We solved this by using libkeepalive 
(http://libkeepalive.sourceforge.net/) on the arserver machine. But it would 
work only if you are using Linux.

HTH 

Kind Regards Conny 

-Ursprüngliche Nachricht-
Von: Action Request System discussion list(ARSList) 
[mailto:arsl...@arslist.org] Im Auftrag von Leihkauff, Kenneth
Gesendet: Mittwoch, 18. November 2009 18:18
An: arslist@ARSLIST.ORG
Betreff: Re: Firewall TCP Timeouts

Thanks for your time, Axton.  I spoke with our network engineer and he studied 
the firewall logs and sees quit a few denials and some resets, but is confident 
things should be set up correctly on the firewall.  He only sees these 
denials/resets for the Midtier application, not some of our other apps.  

Here is what we're observing on the Linux Midtier server.  Below is an example 
of 3 existing tcp connections that remain intact indefinitely or until midtier 
is restarted.  The firewall knows about these 3 tcp connections, but if the 
connections are idle for more than 60 minutes (default firewall setting), the 
firewall will drop these idle tcp connections.  Then, when a user gets on 
midtier (after the idle period), midtier will attempt to use one or more of 
these tcp connections and the firewall responds with a Deny (since it has aged 
off the inactive connection).  This is normal behavior for a firewall as I 
understand things, but the application (midtier in this case) should ideally 
make use of the tcp keepalive or inactivetly timeout supported by Linux.  
Apparently, midtier does not take advantage of this.

I suspect other customers are having this problem IF their topology utilizes a 
firewall between Midtier and ARS and there are long periods of inactivity.  The 
reason, however, they may not be complaining is because once you get the RCP 
failure (arerr 91), then subsequent connections seem to work fine.  For the 
example below, once the arerr 91 timeout finally occurred, the associated tcp 
connection was terminated on the midtier server and a new one was created.  

$netstat -antopp|grep 2031
Proto Recv-Q Send-Q Local Address   Foreign Address 
State   Timer
tcp0  0 :::172.16.0.129:54310   :::10.1.1.141:2031  
ESTABLISHED 24977/java  off (0.00/0/0)
tcp0864 :::172.16.0.129:54309   :::10.1.1.141:2031  
ESTABLISHED 24977/java  on (3.59/12/0)
tcp0   1416 :::172.16.0.129:43717   :::10.1.1.141:2031  
ESTABLISHED 24977/java  on (16.19/11/0)

After a very long wait trying to login in, the user will receive this message:
ARERR [91]
RPC call failed : ONC/RPC call timed out


Ken 


-Original Message-
From: Action Request System discussion list(ARSList) 
[mailto:arsl...@arslist.org] On Behalf Of Axton
Sent: Tuesday, November 17, 2009 3:24 PM
To: arslist@ARSLIST.ORG
Subject: Re: Firewall TCP Timeouts

You need to configure your firewall properly.  With a firewall, you can define 
what types of packets can create a state entry on the firewall.  Typical is 
syn, syn ack.  Even if there is no state entry, a new packet should create a 
new state (not be dropped).  If a new packet does not create a new state, 
change the firewall rules so that it does.  A network dump will tell you what 
type of packet is going out; the firewall logs should tell you what type of 
packet was rejected.

I could see that this would be a problem if the midtier servers were behind a 
NAT.  Is this the case?

Axton Grams

On Tue, Nov 17, 2009 at 12:07 PM, Leihkauff, Kenneth 
kenneth.g.leihka...@saic.com wrote:
 **

 Hello,



 We have a firewall between our MidTier server and ARS system.  The 
 firewall is configured to drop tcp connections after being idle for 60 
 minutes (typical/default firewall setting).  Several MidTier user 
 sessions will make use of a shared tcp connection so you might have 
 100 sessions but significantly fewer tcp connections.  During idle 
 times (like at night), the firewall will discard these idle tcp 
 connections but the MidTier server will still retain these tcp