> -----Original Message-----
> From: Rainer Jung [mailto:[email protected]]
> Sent: Friday, May 22, 2009 2:53 AM
> To: Tomcat Users List
> Subject: Re: Running out of tomcat threads - why many threads in
> RUNNABLEstage even with no activity
>
> My point is: persistent connections are good, but connections which are
> idle for a long time are not as good, so close them after some idle
> time, like e.g. 10 minutes. Of course this means you need to create new
> ones once your load goes up again, but that's not a big problem.
[Pantvaidya, Vishwajit] Why are connections idle for a long time not good? I
thought threads when idle take only a little memory and cpu. Are there any
other reasons?
Thanks a lot Rainer, Chuck, Chris, Andre, Pid, Martin and everyone else I
missed. I spent quite some time yesterday chewing on everything I gathered in
the last few days' interactions and the conflicting behavior we are seeing in
our systems - that led to following conclusions and action plan:
Behavior observed in diff production systems:
a. medium-to-large thread count whether firewall exists or not
b. % of runnable threads is much higher where firewall between httpd/tomcat
c. atleast 1 server where firewall exists has run out of threads
d. atleast 1 server where no firewall exists has run out of threads
Conclusions:
1. In general, runnable threads should not be a prob, unless they correspond to
dropped connections. Since on our servers that have firewall between httpd and
tomcat, runnable connections are not being used for new requests and tomcat
keeps on creating new threads (leading to #b/c above), those threads could
correspond to:
i. connections dropped by firewall or
ii. hanging tomcat threads as httpd recycle timeout disconnected the
connection from that side (and there was no connectiontimeout in server.xml so
that tomcat could do the same) or
iii. combination of these "i" and "ii"
2. Runnable threads on servers where no firewall exist (and we do not see
server running out of threads) should not be a point of concerns as they do not
correspond to dropped connections, as seen from netstat o/p at the end of this
email. So #a above could be ignored.
3. Observation #d above is puzzling and currently I have no answers for that
Action:
- check both sides by using "netstat -anop" (Apache side and the Tomcat side
without connectionTimeout, so you can see the problem in the original form).
See whether the number of AJP connections in the various TCP states differs
much between the netstat output on the Apache and on the Tomcat system.
- Bring workers.properties settings in line with Apache recommendations:
- Worker...cachesize=10 - set to 1
- Worker...cache_timeout=600 - remove
- Worker...recycle_timeout=300 - remove
Netstat o/p's: connector running on 21005, no firewall between httpd/tomcat
Httpd Side:
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name Timer
tcp 0 0 129.41.29.241:53777 129.41.29.48:21005
ESTABLISHED - keepalive (2869.65/0/0)
tcp 0 0 129.41.29.241:53943 129.41.29.48:21005
ESTABLISHED - keepalive (3341.39/0/0)
tcp 0 0 129.41.29.241:49950 129.41.29.48:21005
ESTABLISHED - keepalive (6701.51/0/0)
tcp 0 0 129.41.29.241:49927 129.41.29.48:21005
ESTABLISHED - keepalive (6240.25/0/0)
tcp 0 0 129.41.29.241:49926 129.41.29.48:21005
ESTABLISHED - keepalive (6239.47/0/0)
tcp 0 0 129.41.29.241:49971 129.41.29.48:21005
ESTABLISHED - keepalive (6931.40/0/0)
tcp 0 0 129.41.29.241:49868 129.41.29.48:21005
ESTABLISHED - keepalive (5743.83/0/0)
tcp 0 0 129.41.29.241:49865 129.41.29.48:21005
ESTABLISHED - keepalive (5741.65/0/0)
tcp 0 0 129.41.29.241:49867 129.41.29.48:21005
ESTABLISHED - keepalive (5743.16/0/0)
tcp 0 0 129.41.29.241:49901 129.41.29.48:21005
ESTABLISHED - keepalive (5906.92/0/0)
tcp 0 0 129.41.29.241:49795 129.41.29.48:21005
ESTABLISHED - keepalive (4659.11/0/0)
tcp 0 0 129.41.29.241:49558 129.41.29.48:21005
ESTABLISHED - keepalive (1705.06/0/0)
tcp 0 0 129.41.29.241:50796 129.41.29.48:21005
ESTABLISHED - keepalive (4551.79/0/0)
tcp 0 0 129.41.29.241:50784 129.41.29.48:21005
ESTABLISHED - keepalive (4539.53/0/0)
tcp 0 0 129.41.29.241:50711 129.41.29.48:21005
ESTABLISHED - keepalive (4237.02/0/0)
tcp 0 0 129.41.29.241:50841 129.41.29.48:21005
ESTABLISHED - keepalive (4544.81/0/0)
tcp 0 0 129.41.29.241:50526 129.41.29.48:21005
ESTABLISHED - keepalive (3799.53/0/0)
tcp 0 0 129.41.29.241:50558 129.41.29.48:21005
ESTABLISHED - keepalive (3766.94/0/0)
tcp 0 0 129.41.29.241:50642 129.41.29.48:21005
ESTABLISHED - keepalive (4022.29/0/0)
tcp 0 0 129.41.29.241:50387 129.41.29.48:21005
ESTABLISHED - keepalive (3723.14/0/0)
tcp 0 0 129.41.29.241:50404 129.41.29.48:21005
ESTABLISHED - keepalive (3807.13/0/0)
tcp 0 0 129.41.29.241:52042 129.41.29.48:21005
ESTABLISHED - keepalive (6770.05/0/0)
tcp 0 0 129.41.29.241:52092 129.41.29.48:21005
ESTABLISHED - keepalive (542.41/0/0)
tcp 0 0 129.41.29.241:52222 129.41.29.48:21005
ESTABLISHED - keepalive (555.67/0/0)
tcp 0 0 129.41.29.241:52109 129.41.29.48:21005
ESTABLISHED - keepalive (580.33/0/0)
tcp 0 0 129.41.29.241:51907 129.41.29.48:21005
ESTABLISHED - keepalive (6761.11/0/0)
tcp 0 0 129.41.29.241:52379 129.41.29.48:21005
ESTABLISHED - keepalive (864.69/0/0)
tcp 0 0 ::ffff:129.41.29.112:443 ::ffff:198.70.193.2:21005
TIME_WAIT - timewait (32.66/0/0)
$ date
Thu May 21 17:07:17 PDT 2009
Tomcat Side:
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name Timer
tcp 0 0 :::21005 :::*
LISTEN 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50796
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50784
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52092
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50558
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52042
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50526
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49971
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 865 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49926
ESTABLISHED 18131/java on (0.20/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49927
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49950
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53777
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49901
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50404
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52222
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49868
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49865
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49867
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:51907
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50642
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50387
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53943
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53943
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52109
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49795
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50841
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52379
ESTABLISHED 18131/java off (0.00/0/0)
tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49558
ESTABLISHED 18131/java off (0.00/0/0)
$ date
Thu May 21 17:07:05 PDT 2009
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
