Re: AJP connections just stop working
Hi,
How did the results of your monitoring go ? Were you connection/keepalive
timeouts helping ? Are they still set at 10mins ?
Please outline the steps you made if the problem has indeed been resolved.
Thanks
Kris
Guy Cube wrote:
Here are my updates :
Interesting finding is whatever AJP connectors stuck in Keepalive mode for
long time are for requests coming from remote clients. All connections for
requests coming from local (internal network) clients were changed to
CLOSE_WAIT after some time.
Today I tried with keepAliveTimeout and connectionTimeout setting at
server.xml
Connector port=8009 address=${jboss.bind.address} protocol=AJP/1.3
emptySessionPath=true enableLookups=false redirectPort=8443
connectionTimeout=30 keepAliveTimeout=30 /
I monitored from both JBoss JMX console and netstat. Connectors at JMX
console stayed at stage 6 (STAGE_KEEPALIVE) for 5 minutes and changed to
stage 7 (STAGE_ENDED ) after keep alive timeout (5 minutes). But in
netstat it is still shown up as follows:
netstat -aonp | grep 8009
tcp0 0 0.0.0.0:8009 0.0.0.0:*
LISTEN 5229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49771
ESTABLISHED 25229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49768
ESTABLISHED 25229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49767
ESTABLISHED 25229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49766
ESTABLISHED 25229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49765
ESTABLISHED 25229/java off (0.00/0/0)
pid 25229 is process id of JBoss. I am running JBoss on CentOS 5.2, Kernel
2.6.18-.
I will be monitoring the outcomes tonight.
Regards,
GC
kvancamp wrote:
Guy Cube wrote:
Hello,
We are also haveing exact same issue in our environment. We are using
JBoss 4.2.3, Apache 2.2.11 with mod_proxy_ajp using ajp to connect
between Apache and JBoss. It happens intermittently and when i happnes
all AJP's TCP connections are stayed in ESTABLISHED state on JBoss
machine. In thread dump, all AJP threads are in waiting state.
Anyone found solution or work around for this issue?
Thanks,
GC
For now I have set the connectionTimeout to 10 minutes on my AJP
connection, e.g.
!-- Define an AJP 1.3 Connector on port 8009, timeout in 10 mins.
--
Connector port=8009 address=${jboss.bind.address}
protocol=AJP/1.3
emptySessionPath=true enableLookups=false
redirectPort=8443
connectionTimeout=60
/
I am monitoring results, it is too soon to tell if this alone will solve
the problem or if I also have to delve into my firewall (I was incorrect
in my original response above - there IS a firewall between my IIS proxy
and my JBoss server, and it has a 1 hour idle tcp session timeout), or my
IIS proxy (which is configured with a 5-minute socket timeout).
Ken
--
View this message in context:
http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p25251096.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
:2220 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:1443 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2208 ESTABLISHED tcp 516 0 172.17.3.88:8009172.17.5.42:2214 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2235 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2234 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3002 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3513 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3518 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3260 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:4019 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:4789 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2184 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3213 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:1667 ESTABLISHED tcp 516 0 172.17.3.88:8009172.17.5.42:2183 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2182 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:4767 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2207 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:2204 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2195 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2196 ESTABLISHED tcp1 0 172.17.3.88:8009172.17.1.73:4169 CLOSE_WAIT If anyone has any leads on this problem, or suggestions for things to try, it would be appreciated. -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23737460.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
On 27.05.2009 09:31, Guy Cube wrote: Hello, We are also haveing exact same issue in our environment. We are using JBoss 4.2.3, Apache 2.2.11 with mod_proxy_ajp using ajp to connect between Apache and JBoss. It happens intermittently and when i happnes all AJP's TCP connections are stayed in ESTABLISHED state on JBoss machine. In thread dump, all AJP threads are in waiting state. Anyone found solution or work around for this issue? Please read the full discussion thread. There were some solutions proposed. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
Guy Cube wrote:
Hello,
We are also haveing exact same issue in our environment. We are using
JBoss 4.2.3, Apache 2.2.11 with mod_proxy_ajp using ajp to connect between
Apache and JBoss. It happens intermittently and when i happnes all AJP's
TCP connections are stayed in ESTABLISHED state on JBoss machine. In
thread dump, all AJP threads are in waiting state.
Anyone found solution or work around for this issue?
Thanks,
GC
For now I have set the connectionTimeout to 10 minutes on my AJP connection,
e.g.
!-- Define an AJP 1.3 Connector on port 8009, timeout in 10 mins. --
Connector port=8009 address=${jboss.bind.address}
protocol=AJP/1.3
emptySessionPath=true enableLookups=false redirectPort=8443
connectionTimeout=60
/
I am monitoring results, it is too soon to tell if this alone will solve the
problem or if I also have to delve into my firewall (I was incorrect in my
original response above - there IS a firewall between my IIS proxy and my
JBoss server, and it has a 1 hour idle tcp session timeout), or my IIS proxy
(which is configured with a 5-minute socket timeout).
Ken
--
View this message in context:
http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23741629.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
Here are my updates :
Interesting finding is whatever AJP connectors stuck in Keepalive mode for
long time are for requests coming from remote clients. All connections for
requests coming from local (internal network) clients were changed to
CLOSE_WAIT after some time.
Today I tried with keepAliveTimeout and connectionTimeout setting at
server.xml
Connector port=8009 address=${jboss.bind.address} protocol=AJP/1.3
emptySessionPath=true enableLookups=false redirectPort=8443
connectionTimeout=30 keepAliveTimeout=30 /
I monitored from both JBoss JMX console and netstat. Connectors at JMX
console stayed at stage 6 (STAGE_KEEPALIVE) for 5 minutes and changed to
stage 7 (STAGE_ENDED ) after keep alive timeout (5 minutes). But in
netstat it is still shown up as follows:
netstat -aonp | grep 8009
tcp0 0 0.0.0.0:8009 0.0.0.0:*
LISTEN 5229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49771
ESTABLISHED 25229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49768
ESTABLISHED 25229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49767
ESTABLISHED 25229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49766
ESTABLISHED 25229/java off (0.00/0/0)
tcp0 0 xx.xx.xx.xxx:8009yy.yy.yyy.yyy:49765
ESTABLISHED 25229/java off (0.00/0/0)
pid 25229 is process id of JBoss. I am running JBoss on CentOS 5.2, Kernel
2.6.18-.
I will be monitoring the outcomes tonight.
Regards,
GC
kvancamp wrote:
Guy Cube wrote:
Hello,
We are also haveing exact same issue in our environment. We are using
JBoss 4.2.3, Apache 2.2.11 with mod_proxy_ajp using ajp to connect
between Apache and JBoss. It happens intermittently and when i happnes
all AJP's TCP connections are stayed in ESTABLISHED state on JBoss
machine. In thread dump, all AJP threads are in waiting state.
Anyone found solution or work around for this issue?
Thanks,
GC
For now I have set the connectionTimeout to 10 minutes on my AJP
connection, e.g.
!-- Define an AJP 1.3 Connector on port 8009, timeout in 10 mins. --
Connector port=8009 address=${jboss.bind.address}
protocol=AJP/1.3
emptySessionPath=true enableLookups=false redirectPort=8443
connectionTimeout=60
/
I am monitoring results, it is too soon to tell if this alone will solve
the problem or if I also have to delve into my firewall (I was incorrect
in my original response above - there IS a firewall between my IIS proxy
and my JBoss server, and it has a 1 hour idle tcp session timeout), or my
IIS proxy (which is configured with a 5-minute socket timeout).
Ken
--
View this message in context:
http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23755629.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
Rainer Jung-3 wrote: I would say you should: - set connectionTimeout on the AJP connector of JBOSS - ensure you are using a recent version of the IIS plugin (1.2.28) - read the timeouts documentation page of the plugin and set appropriate timeouts. - monitor the use of the ajp threads in order to find out, whether the problem occurs slowly step by step until at the end all threads are bound, or it occurs spontaneously The thread use monitoring would also give you an idea, what a good number of ajp pool threads in your situation would be. Do you have a firewall between IIS and JBOSS? Regards, Rainer Rainer, Thanks for your prompt response! There is no firewall between IIS and JBoss. I will experiment with the connection timeouts, I am thinking I will start with a fairly large number like 10 minutes. One other thing I forgot to mention: When I observed the problem earlier this week, I telnet'd to the AJP port and was able to connect successfully. This was making me think it was not a problem that all the connections were used up. However, I didn't really do anything in the telnet session, because (unlike HTTP) I don't know how to make a simple GET request through telnet, so it's possible it wouldn't have responded. One more question: With my HTTP port, I know I can always easily test it, via my browser or scripted using wget. Anybody know of a simple command-line utility like wget, that works with AJP? I think this would be a good tool to have, to help diagnose AJP problems (and would allow me to easily set up some automated stress tests). Thanks. Ken -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23669241.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
On 22.05.2009 14:09, kvancamp wrote: One more question: With my HTTP port, I know I can always easily test it, via my browser or scripted using wget. Anybody know of a simple command-line utility like wget, that works with AJP? I think this would be a good tool to have, to help diagnose AJP problems (and would allow me to easily set up some automated stress tests). Coud you please open an enhancement request in Bugzilla for this? There is some code floating around, which we might be able to bundle with the mod_jk source release. Not sure, how soon we would provide a windows binary, but at least we should offer what we already have lying around. It would definitely make sense to have an AJP commandline client to test connectivity. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: AJP connections just stop working
Fyi, you should be able to use jmeter to test AJP connections -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Friday, May 22, 2009 6:42 AM To: Tomcat Users List Subject: Re: AJP connections just stop working On 22.05.2009 14:09, kvancamp wrote: One more question: With my HTTP port, I know I can always easily test it, via my browser or scripted using wget. Anybody know of a simple command-line utility like wget, that works with AJP? I think this would be a good tool to have, to help diagnose AJP problems (and would allow me to easily set up some automated stress tests). Coud you please open an enhancement request in Bugzilla for this? There is some code floating around, which we might be able to bundle with the mod_jk source release. Not sure, how soon we would provide a windows binary, but at least we should offer what we already have lying around. It would definitely make sense to have an AJP commandline client to test connectivity. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: AJP connections just stop working
I was about to say the same earlier, but when i verified with the version (2.3.2) I have installed last month AJP is not listed anymore... Rgds - Fred Anthony J. Biacco wrote: Fyi, you should be able to use jmeter to test AJP connections -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Friday, May 22, 2009 6:42 AM To: Tomcat Users List Subject: Re: AJP connections just stop working On 22.05.2009 14:09, kvancamp wrote: One more question: With my HTTP port, I know I can always easily test it, via my browser or scripted using wget. Anybody know of a simple command-line utility like wget, that works with AJP? I think this would be a good tool to have, to help diagnose AJP problems (and would allow me to easily set up some automated stress tests). Coud you please open an enhancement request in Bugzilla for this? There is some code floating around, which we might be able to bundle with the mod_jk source release. Not sure, how soon we would provide a windows binary, but at least we should offer what we already have lying around. It would definitely make sense to have an AJP commandline client to test connectivity. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23674885.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
I'm pretty sure sebb knows very well JMeter, at least after looking at the subversion history :) The AJP sampler ist still listed in the user guide inside 18.1.2 HTTP Request and the class is still existing in trunk. So they might simply have moved it as a sub type of HTTP request sometime in the past. Regards, Rainer On 22.05.2009 19:26, fredk2 wrote: I was about to say the same earlier, but when i verified with the version (2.3.2) I have installed last month AJP is not listed anymore... Rgds - Fred Anthony J. Biacco wrote: Fyi, you should be able to use jmeter to test AJP connections -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Friday, May 22, 2009 6:42 AM To: Tomcat Users List Subject: Re: AJP connections just stop working On 22.05.2009 14:09, kvancamp wrote: One more question: With my HTTP port, I know I can always easily test it, via my browser or scripted using wget. Anybody know of a simple command-line utility like wget, that works with AJP? I think this would be a good tool to have, to help diagnose AJP problems (and would allow me to easily set up some automated stress tests). Coud you please open an enhancement request in Bugzilla for this? There is some code floating around, which we might be able to bundle with the mod_jk source release. Not sure, how soon we would provide a windows binary, but at least we should offer what we already have lying around. It would definitely make sense to have an AJP commandline client to test connectivity. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
I looked at the dumps. Comments inline.
On 18.05.2009 20:33, kvancamp wrote:
My problem seems to be most similar to this post. We are having intermittent
problems with the JBoss/Tomcat AJP 1.3 connector hanging. From searching
the JBoss and Tomcat user forums, other issues that are similar to mine are:
http://marc.info/?l=tomcat-userm=116231271819840w=2
http://www.nabble.com/Problem-with-AJP-connector-td19657959.html#a19657959
neither of which really seems to offer a solution. Here are my specifics:
We are running JBoss 4.2.2 (which uses Tomcat 6) running on Linux (RedHat
5.3) behind an IIS proxy, which is proxying to the JBoss AJP port. I have
left AJP at its default settings in my server.xml:
!-- A AJP 1.3 Connector on port 8009 --
Connector protocol=AJP/1.3 port=8009
address=${jboss.bind.address}
redirectPort=8443 /
The behavior I’m observing is only occurring about once every 2 weeks,
making it difficult to reproduce. From the user’s perspective, the site is
unreachable. The IIS proxy is logging this when the problem occurs:
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1011):
(adastarNode) can't receive the response message from tomcat, network
problems or tomcat (172.17.3.240:8009) is down (errno=54)
54 is winsock error 10054, connection reset by peer.
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1766):
(adastarNode) Tomcat is down or refused connection. No response has been
sent to the client (yet)
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [info] jk_ajp_common.c (2186):
(adastarNode) sending request to tomcat failed (recoverable), (attempt=1)
My JBoss instance is not logging any errors during this timeframe. As far
as how to solve the problem, in one case the server was left like this for
several hours and seemed to recover on its own, only to hang again a couple
of hours later; otherwise the only solution that’s worked is to restart
JBoss.
The main difference I can observe in a thread dump is that the AJP acceptor
thread, which is normally in a RUNNABLE state, is in a WAITING state when
the hang occurs:
ajp-abeitmpr1.andesatpa.com%2F172.17.3.88-8009-Acceptor-0 daemon prio=10
tid=0x2aaad7a70400 nid=0x7dae in Object.wait()
[0x4424..0x44240c10]
java.lang.Thread.State: WAITING (on object monitor)
Good analysis. If you look further down the stack of this thread you see
it sticks in org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread().
Furthermore all you threads available for requests at port 8009 are in
the same state, namely they are connected to the web server waiting for
the next request to come in over the existing connection they handle.
In all dumps there were 40 of those, so I assume your pool size of the
AJP connector (port 8009) is maximum 40, all of those are connected, and
when the next connection came in, the acceptor tries to get another free
thread, none exists and the acceptor is blocked. As a consequence,
neither the new connection is handled, not is your JBOSS able to accept
any more connections.
The root cause is, that all threads in your pool are busy, waiting for
new rquests coming in over existing connections. So there are two
possiblities:
- either you need more threads to handle more connections
- or something is wrong and instead those threads should be freed.
It's not totally trivial to guess now, which of the two cases you are
in, but I would say the second is more likely.
Lately I’ve been trying to also use netstat to look at the problem when a
hang occurs, but I’m not sure I’ve caught it during a true hang. It appears
to me that I have a growing number of ESTABLISHED connections prior to the
hang, plus one CLOSE_WAIT connection:
[it...@abeitmpr1 log]$ netstat -vatn |grep 8009
tcp0 0 172.17.3.88:80090.0.0.0:*
LISTEN
tcp 516 0 172.17.3.88:8009172.17.5.42:2154
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:3690
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2159
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2158
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2144
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:3680
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2171
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2170
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:1395
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:2935
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:4724
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2120
Re: AJP connections just stop working
kvancamp wrote: Rainer Jung-3 wrote: Can you try to get a couple of full thread dump (e.g. 3 dumps, each 3 seconds apart form each other), when the problem happens next time? If the problem lasts for some time, your could e.g. generate those dumps automatically using cron. http://www.nabble.com/file/p23620960/threaddump-20090507.txt threaddump-20090507.txt Rainer, I've attached a file containing 2 thread dumps. I took these the last time I saw the hang - the 2 thread dumps are about 12 minutes apart. Next time I'll get three. Thanks for responding. Ken This problem just happened again this morning, so this time I got 3 thread dumps, each about a minute apart: http://www.nabble.com/file/p23635003/threaddump.20090520.txt threaddump.20090520.txt Ken -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23635003.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
Rainer Jung-3 wrote: Can you try to get a couple of full thread dump (e.g. 3 dumps, each 3 seconds apart form each other), when the problem happens next time? If the problem lasts for some time, your could e.g. generate those dumps automatically using cron. http://www.nabble.com/file/p23620960/threaddump-20090507.txt threaddump-20090507.txt Rainer, I've attached a file containing 2 thread dumps. I took these the last time I saw the hang - the 2 thread dumps are about 12 minutes apart. Next time I'll get three. Thanks for responding. Ken -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23620960.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3518 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3260 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:4019 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:4789 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2184 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3213 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:1667 ESTABLISHED tcp 516 0 172.17.3.88:8009172.17.5.42:2183 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2182 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:4767 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2207 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:2204 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2195 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2196 ESTABLISHED tcp1 0 172.17.3.88:8009172.17.1.73:4169 CLOSE_WAIT If anyone has any leads on this problem, or suggestions for things to try, it would be appreciated. -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23603003.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
On 18.05.2009 20:33, kvancamp wrote:
My problem seems to be most similar to this post. We are having intermittent
problems with the JBoss/Tomcat AJP 1.3 connector hanging. From searching
the JBoss and Tomcat user forums, other issues that are similar to mine are:
http://marc.info/?l=tomcat-userm=116231271819840w=2
http://www.nabble.com/Problem-with-AJP-connector-td19657959.html#a19657959
neither of which really seems to offer a solution. Here are my specifics:
We are running JBoss 4.2.2 (which uses Tomcat 6) running on Linux (RedHat
5.3) behind an IIS proxy, which is proxying to the JBoss AJP port. I have
left AJP at its default settings in my server.xml:
!-- A AJP 1.3 Connector on port 8009 --
Connector protocol=AJP/1.3 port=8009
address=${jboss.bind.address}
redirectPort=8443 /
The behavior I’m observing is only occurring about once every 2 weeks,
making it difficult to reproduce. From the user’s perspective, the site is
unreachable. The IIS proxy is logging this when the problem occurs:
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1011):
(adastarNode) can't receive the response message from tomcat, network
problems or tomcat (172.17.3.240:8009) is down (errno=54)
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1766):
(adastarNode) Tomcat is down or refused connection. No response has been
sent to the client (yet)
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [info] jk_ajp_common.c (2186):
(adastarNode) sending request to tomcat failed (recoverable), (attempt=1)
My JBoss instance is not logging any errors during this timeframe. As far
as how to solve the problem, in one case the server was left like this for
several hours and seemed to recover on its own, only to hang again a couple
of hours later; otherwise the only solution that’s worked is to restart
JBoss.
The main difference I can observe in a thread dump is that the AJP acceptor
thread, which is normally in a RUNNABLE state, is in a WAITING state when
the hang occurs:
ajp-abeitmpr1.andesatpa.com%2F172.17.3.88-8009-Acceptor-0 daemon prio=10
tid=0x2aaad7a70400 nid=0x7dae in Object.wait()
[0x4424..0x44240c10]
java.lang.Thread.State: WAITING (on object monitor)
Lately I’ve been trying to also use netstat to look at the problem when a
hang occurs, but I’m not sure I’ve caught it during a true hang. It appears
to me that I have a growing number of ESTABLISHED connections prior to the
hang, plus one CLOSE_WAIT connection:
[it...@abeitmpr1 log]$ netstat -vatn |grep 8009
tcp0 0 172.17.3.88:80090.0.0.0:*
LISTEN
tcp 516 0 172.17.3.88:8009172.17.5.42:2154
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:3690
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2159
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2158
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2144
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:3680
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2171
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2170
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:1395
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:2935
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:4724
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2120
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:2375
ESTABLISHED
tcp 516 0 172.17.3.88:8009172.17.5.42:2119
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2118
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2372
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:1114
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2143
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:1116
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2131
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:3923
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:2133
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2132
ESTABLISHED
tcp 514 0 172.17.3.88:8009172.17.5.42:2347
ESTABLISHED
tcp0 0 172.17.3.88:8009172.17.5.42:1834
ESTABLISHED
tcp 514 0 172.17.3.88:8009
