HUGE apologies, Pid! I had no idea that a changed subject would keep my email in a previous thread. I saw no visible
connection to your thread, I didn't know that my email client (Thunderbird) would pass the reference to your thread back
in the email header. I'll be sure to start a fresh
Hi, folks. I've got what I consider to be a fairly significant production implementation of Tomcat with, currently, 89
contexts in 30 hosts on two concurrent servers. Everything works fine ... except when it doesn't. As the number of
contexts and hosts have increased, there have been more and
Caldarale, Charles R wrote:
I'll poke around in the webapp classloader to see if I can find anything
interesting, but in the meantime, is there a /net directory on the problematic
server? If there is and it targets a remote file system, that might explain
the long delay on these stat()
Sam Hokin wrote:
Caldarale, Charles R wrote:
I'll poke around in the webapp classloader to see if I can find
anything interesting, but in the meantime, is there a /net directory
on the problematic server? If there is and it targets a remote file
system, that might explain the long delay
Caldarale, Charles R wrote:
Once you have your test Tomcat set up on the problematic server, try adding
this line to the conf/logging.properties file:
org.apache.jasper.level = ALL
That will display a *lot* of information about what's going on during JSP
compilation. One thing in particular
Caldarale, Charles R wrote:
Do you have a CLASSPATH environment variable set for the
Tomcat process? (You shouldn't.)
Or is there a plain / in the -cp argument in the Tomcat startup script? Or has
conf/catalina.properties been modified to include the file system root?
Any of those are a
Tim Funk wrote:
While I am late to this ... Is this an accurate summary?
- Slow re-load on a server
- Server is a production server
- Other servers are OK so it can only be reproduced on production server
- Initial looks at network seem to be no network activity
- There seems to be a hint at
Pieter Temmerman wrote:
Try this strace command:
strace -f -etrace=\!futex,gettimeofday,clock_gettime -p
PID_OF_TOMCAT_JAVA
where PID_OF_TOMCAT_JAVA is the PID of your Tomcat, duh! :)
Thanks! That certainly did the trick. The first time I ran strace against the Tomcat process, it brought
André Warnier wrote:
Sam, because there is quite a bit of traffic on this issue, and because
the exact description seems to change a bit over time (e.g.
compilation-only or not), could you remake a summary of what you are
sure of so far ?
1. Initial requests to ANY JSP page on a site,
NEW DISCOVERY - using a wildcard package import results in a much longer
response delay!
I've taken Mark's suggestion that I import individual classes rather than importing all with a wildcard to see if a
particular class is slowing down classloading. It turns out that it's the WILDCARD
André Warnier wrote:
What in this directory path is (are) a symlink(s) ?
/home/sites/ims/jcms/webapps/ROOT/WEB-INF/classes/org/apache/jsp/test6_jsp.class
There are no symlinks in that path.
And how is this different from the other 2 systems that work fine ?
It differs in no known way,
Caldarale, Charles R wrote:
The key lines have a 3+ second response time:
2498 15:24:35 stat64(/net/ims/jcms/Object.class, 0x7b6b9c20) = -1 ENOENT (No such
file or directory) 3.020996
Most of the calls return in much less than a millisecond, but there are a few
ENOENT (not all) responses
Mark Thomas wrote:
Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found. I wish I could fix this by simply as swapping out a bad disk
(notwithstanding Andre's desire
Juha Laiho wrote:
One tool that I haven't yet seen suggested is 'strace', the Linux system
call tracer. This will show all the calls your application makes to the
operating system. As you say the application is mostly idle during the
delay, it is, in one way or another, waiting for some OS
János Löbb wrote:
How about just using tcpdump during the long delay and see what the
machine is doing network wise ?
I have, and network timeout was suggested earlier on this thread. There is no TCP activity of particular interest
during the delay. I've hunted for, and not seen, anything
Christopher Schultz wrote:
Sam,
On 2/20/2009 6:44 PM, Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found.
Did you run badblocks on the array, or on an individual disk?
I would
Caldarale, Charles R wrote:
1) What are the servers that don't have a problem?
One is a single AMD Athlon 64 2800+, running the 64-bit Sun JDK (same version).
The other is a dual AMD Athlon MP 2000+, running the 32-bit Sun JDK (same
version).
And the bad server has 8 Intel Xeon E5310 CPUs
Ron McNulty wrote:
Hi Sam
You jar - does it include any JSP tag files written as JSP fragments?
There are known issues in this area. The dependency management seems to
get confused, resulting in lots of unnecessary compilation of the .tag
files
No, it doesn't, Ron. But thanks for the
Christopher Schultz wrote:
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before Christmas on
this one server, is that the initial request of a JSP or servlet takes
AGES to respond, usually exceeding several minutes.
Good... then you'll have
Caldarale, Charles R wrote:
Run jstack several times during the delay, and then let's compare the set of
thread dumps.
What is the CPU utilization during the delay?
OK, done. The CPU utilization during the delay is tiny, 1% or less. I've run jstack before, during, and after I've
done a
Christopher Schultz wrote:
jstack.8.txt is the last thread dump where File.exists was stalled.
http://ims.net/jstack/jstack.9.txt
http://ims.net/jstack/jstack.10.txt
http://ims.net/jstack/jstack.11.txt
The server appears to be idle, here.
It's a little weird that thread 9770 has NO STACK
I'd like to correct an error I made in my earlier report of this problem. It turns out that the response delay in my
simple test JSP IS during compilation! I ran stat on the files in question, as well as looked at the time shown on the
response page (which is all the test JSP does). (I think
Chuck's comment about JSP to Java translation done by Jasper reminded me that I should have included the time on the
Java source file created during JSP compilation. It shows that the Java source file was created immediately, but
accessed again six seconds later (!), presumably by the compiler.
Actually, /usr/local/tomcat is a symlink to /usr/local/apache-tomcat-6.0.18. But that's a pretty important symlink for
me, to make it easy to change Tomcat versions. But I'll investigate that. That's the only symlink involved that I can
think of.
Caldarale, Charles R wrote:
From: Sam Hokin
An update. I've written a simple standalone Java routine that calls File.exists(). It never hangs. In fact, I ran it
against the class file that Tomcat creates from a test JSP; it doesn't hang, returns false for the (long) time it takes
Tomcat to create the class file on the slow system, then
software, either in some service mucking up the JVM
or the JVM itself. But it only manifests itself under Tomcat, and then only when this particular package is imported.
Christopher Schultz wrote:
Sam and David,
On 2/20/2009 1:36 PM, David Smith wrote:
Sam Hokin wrote:
It's a logical RAID
André Warnier wrote:
Sam Hokin wrote:
I'd like to correct an error I made in my earlier report of this
problem. It turns out that the response delay in my simple test JSP
IS during compilation! I ran stat on the files in question, as well
as looked at the time shown on the response page
I've been struggling with a very severe problem on one of my three Tomcat servers since late December. I'm completely
stumped. I'm running 6.0.18 on all three servers, which are all running Fedora Core 9, all currently running Sun's JDK
1.6.0_12. The only difference is hardware. The server
28 matches
Mail list logo
