on the arserver!
-Ursprüngliche Nachricht-
Von: Action Request System discussion list(ARSList)
[mailto:arsl...@arslist.org] Im Auftrag von Leihkauff, Kenneth
Gesendet: Mittwoch, 18. November 2009 22:51
An: arslist@ARSLIST.ORG
Betreff: Re: Firewall TCP Timeouts
Thanks, Conny. Are you saying the libkeepalive was implemented on the ARserver
-- not the MidTier linux server? Our midtier and arserver are linux with a
firewall in between.
Unfortunately, the firewall is not an application level firewall so the
firewall rule changes Axton proposed cannot be implemented according to our
network engineer.
Ken
-Original Message-
From: Action Request System discussion list(ARSList)
[mailto:arsl...@arslist.org] On Behalf Of Conny Martin
Sent: Wednesday, November 18, 2009 3:30 PM
To: arslist@ARSLIST.ORG
Subject: AW: Firewall TCP Timeouts
We had the same issue. We solved this by using libkeepalive
(http://libkeepalive.sourceforge.net/) on the arserver machine. But it would
work only if you are using Linux.
HTH
Kind Regards Conny
-Ursprüngliche Nachricht-
Von: Action Request System discussion list(ARSList)
[mailto:arsl...@arslist.org] Im Auftrag von Leihkauff, Kenneth
Gesendet: Mittwoch, 18. November 2009 18:18
An: arslist@ARSLIST.ORG
Betreff: Re: Firewall TCP Timeouts
Thanks for your time, Axton. I spoke with our network engineer and he studied
the firewall logs and sees quit a few denials and some resets, but is confident
things should be set up correctly on the firewall. He only sees these
denials/resets for the Midtier application, not some of our other apps.
Here is what we're observing on the Linux Midtier server. Below is an example
of 3 existing tcp connections that remain intact indefinitely or until midtier
is restarted. The firewall knows about these 3 tcp connections, but if the
connections are idle for more than 60 minutes (default firewall setting), the
firewall will drop these idle tcp connections. Then, when a user gets on
midtier (after the idle period), midtier will attempt to use one or more of
these tcp connections and the firewall responds with a Deny (since it has aged
off the inactive connection). This is normal behavior for a firewall as I
understand things, but the application (midtier in this case) should ideally
make use of the tcp keepalive or inactivetly timeout supported by Linux.
Apparently, midtier does not take advantage of this.
I suspect other customers are having this problem IF their topology utilizes a
firewall between Midtier and ARS and there are long periods of inactivity. The
reason, however, they may not be complaining is because once you get the RCP
failure (arerr 91), then subsequent connections seem to work fine. For the
example below, once the arerr 91 timeout finally occurred, the associated tcp
connection was terminated on the midtier server and a new one was created.
$netstat -antopp|grep 2031
Proto Recv-Q Send-Q Local Address Foreign Address
State Timer
tcp0 0 :::172.16.0.129:54310 :::10.1.1.141:2031
ESTABLISHED 24977/java off (0.00/0/0)
tcp0864 :::172.16.0.129:54309 :::10.1.1.141:2031
ESTABLISHED 24977/java on (3.59/12/0)
tcp0 1416 :::172.16.0.129:43717 :::10.1.1.141:2031
ESTABLISHED 24977/java on (16.19/11/0)
After a very long wait trying to login in, the user will receive this message:
ARERR [91]
RPC call failed : ONC/RPC call timed out
Ken
-Original Message-
From: Action Request System discussion list(ARSList)
[mailto:arsl...@arslist.org] On Behalf Of Axton
Sent: Tuesday, November 17, 2009 3:24 PM
To: arslist@ARSLIST.ORG
Subject: Re: Firewall TCP Timeouts
You need to configure your firewall properly. With a firewall, you can define
what types of packets can create a state entry on the firewall. Typical is
syn, syn ack. Even if there is no state entry, a new packet should create a
new state (not be dropped). If a new packet does not create a new state,
change the firewall rules so that it does. A network dump will tell you what
type of packet is going out; the firewall logs should tell you what type of
packet was rejected.
I could see that this would be a problem if the midtier servers were behind a
NAT. Is this the case?
Axton Grams
On Tue, Nov 17, 2009 at 12:07 PM, Leihkauff, Kenneth
kenneth.g.leihka...@saic.com wrote:
**
Hello,
We have a firewall between our MidTier server and ARS system. The
firewall is configured to drop tcp connections after being idle for 60
minutes (typical/default firewall setting). Several MidTier user
sessions will make use of a shared tcp connection so you might have
100 sessions but significantly fewer tcp connections. During idle
times (like at night), the firewall will discard these idle tcp
connections but the MidTier server will still retain these tcp