Re: AJP connections just stop working
My problem seems to be most similar to this post. We are having intermittent problems with the JBoss/Tomcat AJP 1.3 connector hanging. From searching the JBoss and Tomcat user forums, other issues that are similar to mine are: http://marc.info/?l=tomcat-user&m=116231271819840&w=2 http://www.nabble.com/Problem-with-AJP-connector-td19657959.html#a19657959 neither of which really seems to offer a solution. Here are my specifics: We are running JBoss 4.2.2 (which uses Tomcat 6) running on Linux (RedHat 5.3) behind an IIS proxy, which is proxying to the JBoss AJP port. I have left AJP at its default settings in my server.xml: The behavior I’m observing is only occurring about once every 2 weeks, making it difficult to reproduce. From the user’s perspective, the site is unreachable. The IIS proxy is logging this when the problem occurs: [Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1011): (adastarNode) can't receive the response message from tomcat, network problems or tomcat (172.17.3.240:8009) is down (errno=54) [Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1766): (adastarNode) Tomcat is down or refused connection. No response has been sent to the client (yet) [Tue Apr 21 04:13:14.775 2009] [3192:2500] [info] jk_ajp_common.c (2186): (adastarNode) sending request to tomcat failed (recoverable), (attempt=1) My JBoss instance is not logging any errors during this timeframe. As far as how to solve the problem, in one case the server was left like this for several hours and seemed to recover on its own, only to hang again a couple of hours later; otherwise the only solution that’s worked is to restart JBoss. The main difference I can observe in a thread dump is that the AJP acceptor thread, which is normally in a RUNNABLE state, is in a WAITING state when the hang occurs: "ajp-abeitmpr1.andesatpa.com%2F172.17.3.88-8009-Acceptor-0" daemon prio=10 tid=0x2aaad7a70400 nid=0x7dae in Object.wait() [0x4424..0x44240c10] java.lang.Thread.State: WAITING (on object monitor) Lately I’ve been trying to also use netstat to look at the problem when a hang occurs, but I’m not sure I’ve caught it during a true hang. It appears to me that I have a growing number of ESTABLISHED connections prior to the hang, plus one CLOSE_WAIT connection: [it...@abeitmpr1 log]$ netstat -vatn |grep 8009 tcp0 0 172.17.3.88:80090.0.0.0:* LISTEN tcp 516 0 172.17.3.88:8009172.17.5.42:2154 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3690 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2159 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2158 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2144 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3680 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2171 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2170 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:1395 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:2935 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:4724 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2120 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:2375 ESTABLISHED tcp 516 0 172.17.3.88:8009172.17.5.42:2119 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2118 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2372 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:1114 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2143 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:1116 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2131 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:3923 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:2133 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2132 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2347 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:1834 ESTABLISHED tcp 514 0 172.17.3.88:8009172.17.5.42:2093 ESTABLISHED tcp0 0 172.17.3.88:8009172.17.5.42:1837 ESTABLISHED tcp 516 0 172.17.3.88:8009172.17.5.42:2092 ESTABLISHED tcp 514 0 172.17.3.88:8009172.
Re: AJP connections just stop working
Rainer Jung-3 wrote: > > Can you try to get a couple of full thread dump (e.g. 3 dumps, each 3 > seconds apart form each other), when the problem happens next time? If > the problem lasts for some time, your could e.g. generate those dumps > automatically using cron. > http://www.nabble.com/file/p23620960/threaddump-20090507.txt threaddump-20090507.txt Rainer, I've attached a file containing 2 thread dumps. I took these the last time I saw the hang - the 2 thread dumps are about 12 minutes apart. Next time I'll get three. Thanks for responding. Ken -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23620960.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
kvancamp wrote: > > > > Rainer Jung-3 wrote: >> >> Can you try to get a couple of full thread dump (e.g. 3 dumps, each 3 >> seconds apart form each other), when the problem happens next time? If >> the problem lasts for some time, your could e.g. generate those dumps >> automatically using cron. >> > http://www.nabble.com/file/p23620960/threaddump-20090507.txt > threaddump-20090507.txt > Rainer, > > I've attached a file containing 2 thread dumps. I took these the last > time I saw the hang - the 2 thread dumps are about 12 minutes apart. Next > time I'll get three. > > Thanks for responding. > Ken > This problem just happened again this morning, so this time I got 3 thread dumps, each about a minute apart: http://www.nabble.com/file/p23635003/threaddump.20090520.txt threaddump.20090520.txt Ken -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23635003.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
Rainer Jung-3 wrote: > > I would say you should: > > - set connectionTimeout on the AJP connector of JBOSS > - ensure you are using a recent version of the IIS plugin (1.2.28) > - read the timeouts documentation page of the plugin and set appropriate > timeouts. > - monitor the use of the ajp threads in order to find out, whether the > problem occurs slowly step by step until at the end all threads are > bound, or it occurs spontaneously > > The thread use monitoring would also give you an idea, what a good > number of ajp pool threads in your situation would be. > > Do you have a firewall between IIS and JBOSS? > > Regards, > > Rainer > Rainer, Thanks for your prompt response! There is no firewall between IIS and JBoss. I will experiment with the connection timeouts, I am thinking I will start with a fairly large number like 10 minutes. One other thing I forgot to mention: When I observed the problem earlier this week, I telnet'd to the AJP port and was able to connect successfully. This was making me think it was not a problem that all the connections were used up. However, I didn't really do anything in the telnet session, because (unlike HTTP) I don't know how to make a simple GET request through telnet, so it's possible it wouldn't have responded. One more question: With my HTTP port, I know I can always easily test it, via my browser or scripted using wget. Anybody know of a simple command-line utility like wget, that works with AJP? I think this would be a good tool to have, to help diagnose AJP problems (and would allow me to easily set up some automated stress tests). Thanks. Ken -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23669241.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connections just stop working
Guy Cube wrote: > > Hello, > > We are also haveing exact same issue in our environment. We are using > JBoss 4.2.3, Apache 2.2.11 with mod_proxy_ajp using ajp to connect between > Apache and JBoss. It happens intermittently and when i happnes all AJP's > TCP connections are stayed in ESTABLISHED state on JBoss machine. In > thread dump, all AJP threads are in waiting state. > > Anyone found solution or work around for this issue? > > Thanks, > GC > For now I have set the connectionTimeout to 10 minutes on my AJP connection, e.g. I am monitoring results, it is too soon to tell if this alone will solve the problem or if I also have to delve into my firewall (I was incorrect in my original response above - there IS a firewall between my IIS proxy and my JBoss server, and it has a 1 hour idle tcp session timeout), or my IIS proxy (which is configured with a 5-minute socket timeout). Ken -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23741629.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
