For the sake of record-keeping, I believe we found the fix for this.
The client got in touch with us today stating that the entire site was
now unresponsive. We reviewed the log files and miraculously this time
there WAS an error message (lots of them).
The error was as follows:
-----------
SEVERE: Error allocating socket processor
java.lang.OutOfMemoryError: unable to create new native thread
-----------
Googling this led me to the following:
http://java-monitor.com/forum/showthread.php?t=570
and
http://jroller.com/rreyelts/date/20040909
So, looking at my setenv.sh values, it turns out that Xss256m is
extremely massive. It means each time a new thread was created, 256
*MEGS* would get allocated to that thread (even if it wasn't used). So
we would look at top and see that there was plenty of free memory, but
not realize that it had all been allocated to the threads. I reduced the
value to 512k, and things are now working beautifully.
I can only guess at why there were no logs when this happened previously
(maybe no memory left to spawn a new logging thread? no idea...), but
the page that was hanging now runs very quickly even when I hammer my
refresh button on it. =D
Thanks to those of you who tried to help with this issue. Hopefully this
post can help someone in the future!
Warm regards,
Jordan Michaels
Vivio Technologies
http://www.viviotech.net/
Open BlueDragon Steering Committee
Railo Community Distributions
Jordan Michaels wrote:
We have an interesting situation and I'm wondering if anyone has any
ideas on where we could look for possible solutions.
We have a JSP page that performs a large mysql query of product titles
(just the names). The list is then spit out on the page with drill-down
links. Pretty simple really. The problem is that just after a tomcat
restart, the first and occasionally the second and third requests will
process all the way just fine, but most subsequent requests will "hang"
on a random record. Occasionally one will go all the way through, but
most will hang.
Let me clarify, we restart tomcat, hit the page, and the page loads
fine. We hit refresh once or twice, and the page will fail to load
fully. The page will hang after a random product title and just stop
responding. You can click on any other page on the site, but this one
page with the large product listing query hangs in random spots.
We've studied this intently. The server this is running on is a CentOS
4, 32-bit box with 1.5 GB RAM, Apache 2, and mod_jk.
Our first thought was memory issues. We increased Tomcat's memory using
the setenv.sh file, which now contains the following line:
JAVA_OPTS="-Xms128m -Xmx1024m -Xss256m -XX:MaxPermSize=256m "; export
JAVA_OPTS
We've verified that the settings are in place by using the "ps ax"
command and making sure the java process includes the updated settings.
Unfortunately, this seems to have no effect at all.
We then reviewed the logs: catalina.out, mod_jk.log, and apache's
error_log, among others. We monitored these logs using "tail" so we
could see if any errors were thrown as we hit the page. No errors were
thrown if the request went through good or it failed. The logs were
completely quiet.
We've updated Tomcat, mod_jk, the JVM, and the MySQL driver to the
absolute latest versions available as of this writing.
Does anyone have any clues as to where we might look for what is
happening to make this one page hang? It still feels like a memory
issue, but with nothing in the logs and no indicators other then the
problem itself, we'd appreciate any suggestions on what to look for at
this point.
Thank you in advance!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
