3.2.2b3 mod_jk gets stuck in readFully
I've been testing Tomcat 3.2.2b3 and Apache 1.3 on Solaris, connected with mod_jk. Things are generally working, but there is a serious problem that occurs under load. The problem is that certain Apache children get stuck talking to Tomcat. The children are always requesting JSP pages. Using the Apache status display, a stuck child looks like: 141-0 29388 0/1/1 W 0.00 1092 0 0.0 0.01 0.01 154.x.x.x bingoe01.mysite.com GET /ad/loading-applet.jsp?tabl=1&site=pogo&scrn=motormind&anam It is not uncommon to have half of the Apache children stuck like this. On the tomcat side, the threads appear at this location: "Thread-256" (TID:0xd020a0, sys_thread_t:0xd01fd8, state:R, thread_t: t@263, threadID:0x63cb1dd8, stack_bottom:0x63cb2000, stack_si ze:0x2) prio=5 [1] java.net.SocketInputStream.socketRead(Native Method) [2] java.net.SocketInputStream.read(SocketInputStream.java:85) [3] org.apache.tomcat.service.connector.TcpConnector.receiveFully(TcpConnector.java:148) [4] org.apache.tomcat.service.connector.TcpConnector.receive(TcpConnector.java:118) [5] org.apache.tomcat.service.connector.Ajp13ConnectionHandler.processConnection(Ajp13ConnectionHandler.java:109) [6] org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java:393) [7] org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java:475) [8] java.lang.Thread.run(Thread.java:478) Generally, the number of threads inside receiveFully matches the number of stuck Apache children. Losing Apache children like this eventually starves the server of ability to do useful work, even though there is a lot of idle time on the machine. Is this a known issue? Thanks, Bill Lipa __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
RE: 3.2.2b3 mod_jk gets stuck in readFully
After quite a bit of struggle, I think I found out what is going on. The problem is that the default configuration of Tomcat does not have enough threads in its thread pool for the default configuration of Apache. This issue would only be apparent if many Apache children were in use. The result was that any Apache children over the number of Tomcat threads would hang waiting for Tomcat to respond to requests. Tomcat would not respond until threads became available, which could be quite a long time if Apache children were not dying off (ie, because load was increasing during the day). I was wrong about the threads being stuck in readFully. The real problem is that not enough threads existed at all (ie, the thread handling socket accept would be blocked). The simplest workaround is to change the AJP13 connector to SimpleTcpConnector rather than PoolTcpConnector in server.xml. I strongly suggest that thread pool exhaustion emit a log message, since this was quite difficult to track down. Additionally, it would be better for the default configuration to be more robust. Bill --- Marc Saegesser <[EMAIL PROTECTED]> wrote: > I finally got some time to look at this and I think I can duplicate the > problem your seeing. Hopefully, its the problem your seeing, or else we > have two serious problems. __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
RE: 3.2.2b3 mod_jk gets stuck in readFully
Thanks for your help, Marc. Would it be possible to log a message to tomcat.log if the thread pool gets exhausted? I believe the default Apache installation calls for 256 children, so busy sites are going to run into this. A log message suggesting to increase max_threads could save a lot of aggravation! Bill --- Marc Saegesser <[EMAIL PROTECTED]> wrote: > As for the thread pool stuff. By default, Tomcat 3.2.x thread pools create > 10 threads. This can be changed using the min_spare_threads parameter. The > pools will grow as needed up to the maximum number of threads allowed (100, > by default). You can increase the maximum number of allowed threads using > the max_threads parameter. See if this fixes your problem better than using > SimpleTcpConnector. __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
[PATCH] mod_jk timestamp and process id logging
>2) I suggest adding a timestamp to mod_jk-logging in jk_util.c. Logging >without a timestamp is not very useful. (change 1 line, add 2 lines) Yes, this is a must-have... the other thing that is really useful is the Apache child process id. That way if one process gets stuck, you can get the id from the Apache mod_status, and grep the mod_jk.log for that pid to see where it is. Attached is another version of the patch that adds both, relative to 3.3-m2. This was the only way that I could figure out that Tomcat didn't have enough threads in its thread pool for my Apache config! Bill __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/ diff.out
RE: [PATCH] mod_jk timestamp and process id logging
Regarding logging the Apache child process id in mod_jk.log: >Timestamp is already present in CVS. >Did others modules add the pid ? I am not sure, but the process id makes debugging a problem with an individual Apache child much simpler, because it directs you to the events of interest. Without the process id, you have to use timestamps, which are significantly less accurate. Bill __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
RE: [PATCH] mod_jk timestamp and process id logging
>We could add the getpid() but what about Apache 2.0 >in MPM mode (threaded) ? The getpid() isn't going to be very helpful in that case, but at least it doesn't hurt. Is there some other notion like a thread id that would be applicable to the multithreaded Apache case? Bill __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
RE: [PATCH] mod_jk timestamp and process id logging
This is starting to sound complicated. I'd say go with the getpid(), since it
covers a major case and is pretty portable and simple.
Bill
--- GOMEZ Henri <[EMAIL PROTECTED]> wrote:
> Depend the OS:
>
> AS/400 =>
>
> int GetThreadId()
> {
> pthread_t lSelf = pthread_self();
> pthread_id_np_t lTid;
>
> pthread_getunique_np(&lSelf, &lTid);
> return (lTid.intId.lo);
> }
>
> Linux =>
>
> int GetThreadId()
> {
> return (pthread_self());
> }
>
> What about others platforms like AIX/HPUX/Windows
__
Do You Yahoo!?
Yahoo! Auctions - buy the things you want at great prices
http://auctions.yahoo.com/
