I'm glad you got it solved Sam.
It was an exciting topic to follow.
I haven't been reading the mailinglist for a long time, but there might
be a good chance you got the longest thread in the mailinglist
history :)
Cheers.
Pieter
On Thu, 2009-02-26 at 10:31 -0600, Sam Hokin wrote:
> Caldarale, Ch
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/26/2009 10:50 AM, Sam Hokin wrote:
> Caldarale, Charles R wrote:
>>
>> I'll poke around in the webapp classloader to see if I can find
>> anything interesting, but in the meantime, is there a /net directory
>> on the problematic server? If
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Mark,
On 2/25/2009 4:42 AM, Mark Thomas wrote:
> Ok. To summarise when you include net.ims.jcms.* in your imports the
> page compiles quickly but takes ages to respond to the first request.
>
> I wonder if this is related to loading a specific class
Sam Hokin wrote:
Caldarale, Charles R wrote:
Once you have your test Tomcat set up on the problematic server, try
adding this line to the conf/logging.properties file:
org.apache.jasper.level = ALL
That will display a *lot* of information about what's going on during
JSP compilation. One th
Caldarale, Charles R wrote:
Once you have your test Tomcat set up on the problematic server, try adding
this line to the conf/logging.properties file:
org.apache.jasper.level = ALL
That will display a *lot* of information about what's going on during JSP
compilation. One thing in particular
> From: Jorge Medina [mailto:jmed...@e-dialog.com]
> Subject: RE: very slow class loading on initial JSP/servlet
> request after restart
>
> Just for curiosity, Is "/" included in your classpath ?
Note that the classpath of interest is the one JDT uses, not any global on
Just for curiosity, Is "/" included in your classpath ?
-Original Message-
From: Sam Hokin [mailto:s...@ims.net]
Sent: Thursday, February 26, 2009 10:59 AM
To: Tomcat Users List
Subject: Re: very slow class loading on initial JSP/servlet request
after restart
Sam H
Sam Hokin wrote:
Caldarale, Charles R wrote:
I'll poke around in the webapp classloader to see if I can find
anything interesting, but in the meantime, is there a /net directory
on the problematic server? If there is and it targets a remote file
system, that might explain the long delay on
Caldarale, Charles R wrote:
I'll poke around in the webapp classloader to see if I can find anything
interesting, but in the meantime, is there a /net directory on the problematic
server? If there is and it targets a remote file system, that might explain
the long delay on these stat() calls
Sam Hokin wrote:
André Warnier wrote:
What in this directory path is (are) a symlink(s) ?
/home/sites/ims/jcms/webapps/ROOT/WEB-INF/classes/org/apache/jsp/test6_jsp.class
There are no symlinks in that path.
Then the question is : why is that thing doing lstat's ?
(I mean in the trace list
Caldarale, Charles R wrote:
The key lines have a 3+ second response time:
2498 15:24:35 stat64("/net/ims/jcms/Object.class", 0x7b6b9c20) = -1 ENOENT (No such
file or directory) <3.020996>
Most of the calls return in much less than a millisecond, but there are a few
ENOENT (not all) response
André Warnier wrote:
What in this directory path is (are) a symlink(s) ?
/home/sites/ims/jcms/webapps/ROOT/WEB-INF/classes/org/apache/jsp/test6_jsp.class
There are no symlinks in that path.
And how is this different from the other 2 systems that work fine ?
It differs in no known way, othe
NEW DISCOVERY - using a wildcard package import results in a much longer
response delay!
I've taken Mark's suggestion that I import individual classes rather than importing all with a wildcard to see if a
particular class is slowing down classloading. It turns out that it's the WILDCARD itself
André Warnier wrote:
Sam, because there is quite a bit of traffic on this issue, and because
the exact description seems to change a bit over time (e.g.
compilation-only or not), could you remake a summary of what you are
sure of so far ?
1. Initial requests to ANY JSP page on a site, precomp
Pieter Temmerman wrote:
Try this strace command:
strace -f -etrace=\!futex,gettimeofday,clock_gettime -p
PID_OF_TOMCAT_JAVA
where PID_OF_TOMCAT_JAVA is the PID of your Tomcat, duh! :)
Thanks! That certainly did the trick. The first time I ran strace against the Tomcat process, it brought Tom
Tim Funk wrote:
While I am late to this ... Is this an accurate summary?
- Slow re-load on a server
- Server is a production server
- Other servers are OK so it can only be reproduced on production server
- Initial looks at network seem to be no network activity
- There seems to be a hint at Fil
Mark Thomas wrote:
Sam Hokin wrote:
Mark Thomas wrote:
Sam Hokin wrote:
[...]
My thinking is if we can reduce the scope of the problem to importing a
single class, we can then separate out that class and reduce the code in
it bit by bit until we have the bare minimum that causes the problem
While I am late to this ... Is this an accurate summary?
- Slow re-load on a server
- Server is a production server
- Other servers are OK so it can only be reproduced on production server
- Initial looks at network seem to be no network activity
- There seems to be a hint at File.exists() causin
> Thanks, Juha. Actually Pieter suggested it a little while ago, and I've been
> trying to get some information out of
> strace. The best I can do is to put strace in front of the java command
> that's inside catalina.sh. That's the command
> that shows with ps -ef when Tomcat is running.
Sam Hokin wrote:
> Mark Thomas wrote:
>> Sam Hokin wrote:
>>> Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
>>> when I tested it earlier. And I just ran badblocks again - 0 bad blocks
>>> found. I wish I could fix this by simply as swapping out a bad disk
>>> (notwithstan
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/24/2009 1:54 PM, Sam Hokin wrote:
> Mark Thomas wrote:
>> Sam Hokin wrote:
>>> Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
>>> when I tested it earlier. And I just ran badblocks again - 0 bad blocks
>>> found. I
János Löbb wrote:
How about just using tcpdump during the long delay and see what the
machine is doing network wise ?
I have, and network timeout was suggested earlier on this thread. There is no TCP activity of particular interest
during the delay. I've hunted for, and not seen, anything
On Feb 24, 2009, at 4:16 PM, Sam Hokin wrote:
Juha Laiho wrote:
One tool that I haven't yet seen suggested is 'strace', the Linux
system
call tracer. This will show all the calls your application makes to
the
operating system. As you say the application is mostly idle during
the
delay, i
On Feb 24, 2009, at 3:57 PM, Juha Laiho wrote:
So, looks like I'm late to the party, but will crash in nonetheless.
Sam Hokin wrote:
Christopher Schultz wrote:
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before
Christmas on
this one server,
Juha Laiho wrote:
One tool that I haven't yet seen suggested is 'strace', the Linux system
call tracer. This will show all the calls your application makes to the
operating system. As you say the application is mostly idle during the
delay, it is, in one way or another, waiting for some OS servi
So, looks like I'm late to the party, but will crash in nonetheless.
Sam Hokin wrote:
> Christopher Schultz wrote:
>> On 2/19/2009 6:23 PM, Sam Hokin wrote:
>>> The problem, which spontaneously appeared a few days before Christmas on
>>> this one server, is that the initial request of a JSP or se
Mark Thomas wrote:
Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found. I wish I could fix this by simply as swapping out a bad disk
(notwithstanding Andre's desire for intellectua
Christopher Schultz wrote:
Sam,
On 2/20/2009 6:44 PM, Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found.
Did you run badblocks on the array, or on an individual disk?
I would
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/20/2009 6:44 PM, Sam Hokin wrote:
> Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
> when I tested it earlier. And I just ran badblocks again - 0 bad blocks
> found.
Did you run badblocks on the array, or on an indi
A summary so far :
- the problem is : on one certain machine, when deploying any JSP which
imports a certain self-made library of classes, that JSP takes an
inordinate amount of time (one minute or more) to compile for a 5-line JSP
- after compilation, the JSP in question responds totally normal
dorse content contained within this transmission.
> Date: Fri, 20 Feb 2009 23:50:21 +
> From: ma...@apache.org
> To: users@tomcat.apache.org
> Subject: Re: very slow class loading on initial JSP/servlet request after
> restart
>
> Sam Hokin wrote:
> > Thanks,
Sam Hokin wrote:
> Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
> when I tested it earlier. And I just ran badblocks again - 0 bad blocks
> found. I wish I could fix this by simply as swapping out a bad disk
> (notwithstanding Andre's desire for intellectual pursuits), b
André Warnier wrote:
Sam Hokin wrote:
I'd like to correct an error I made in my earlier report of this
problem. It turns out that the response delay in my simple test JSP
IS during compilation! I ran stat on the files in question, as well
as looked at the time shown on the response page (whi
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks, when I tested it earlier. And I just ran
badblocks again - 0 bad blocks found. I wish I could fix this by simply as swapping out a bad disk (notwithstanding
Andre's desire for intellectual pursuits), but I really think it's
Sam Hokin wrote:
I'd like to correct an error I made in my earlier report of this
problem. It turns out that the response delay in my simple test JSP IS
during compilation! I ran stat on the files in question, as well as
looked at the time shown on the response page (which is all the test JSP
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam and David,
On 2/20/2009 1:36 PM, David Smith wrote:
Sam Hokin wrote:
It's a logical RAID array on four physical disks. I've
run fsck on it (when this all first happened) and it's fine.
It _could_ be a disk issue.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam and David,
On 2/20/2009 1:36 PM, David Smith wrote:
> Sam Hokin wrote:
>> It's a logical RAID array on four physical disks. I've
>> run fsck on it (when this all first happened) and it's fine.
>
> It _could_ be a disk issue. I've seen disks that
An update. I've written a simple standalone Java routine that calls File.exists(). It never hangs. In fact, I ran it
against the class file that Tomcat creates from a test JSP; it doesn't hang, returns false for the (long) time it takes
Tomcat to create the class file on the slow system, then
Chuck's comment about JSP to Java translation done by Jasper reminded me that I should have included the time on the
Java source file created during JSP compilation. It shows that the Java source file was created immediately, but
accessed again six seconds later (!), presumably by the compiler.
Sam Hokin wrote:
I'd like to correct an error I made in my earlier report of this
problem. It turns out that the response delay in my simple test JSP IS
during compilation!
Ha !
If it may help, I remember seeing an earlier thread this week, I believe
answered by Rainer Jung, providing some
I'd like to correct an error I made in my earlier report of this problem. It turns out that the response delay in my
simple test JSP IS during compilation! I ran stat on the files in question, as well as looked at the time shown on the
response page (which is all the test JSP does). (I think I
Sam Hokin wrote:
> Christopher Schultz wrote:
>>
>> jstack.8.txt is the last thread dump where File.exists was stalled.
>>
>>> http://ims.net/jstack/jstack.9.txt
>>> http://ims.net/jstack/jstack.10.txt
>>> http://ims.net/jstack/jstack.11.txt
>>
>> The server appears to be idle, here.
>>
>> It's a l
Christopher Schultz wrote:
jstack.8.txt is the last thread dump where File.exists was stalled.
http://ims.net/jstack/jstack.9.txt
http://ims.net/jstack/jstack.10.txt
http://ims.net/jstack/jstack.11.txt
The server appears to be idle, here.
It's a little weird that thread 9770 has NO STACK IN
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/20/2009 12:11 PM, Sam Hokin wrote:
> jstack.0.txt ran BEFORE my test request.
Your server is completely idle, here.
> jstack.1-11.txt ran DURING the long wait.
Observing jstack.1.txt, it looks like this is the only running thread
(complet
Caldarale, Charles R wrote:
Run jstack several times during the delay, and then let's compare the set of
thread dumps.
What is the CPU utilization during the delay?
OK, done. The CPU utilization during the delay is tiny, 1% or less. I've run jstack before, during, and after I've
done a fi
Christopher Schultz wrote:
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before Christmas on
this one server, is that the initial request of a JSP or servlet takes
AGES to respond, usually exceeding several minutes.
Good... then you'll have p
Ron McNulty wrote:
Hi Sam
You jar - does it include any JSP tag files written as JSP fragments?
There are known issues in this area. The dependency management seems to
get confused, resulting in lots of unnecessary compilation of the .tag
files
No, it doesn't, Ron. But thanks for the heads
Caldarale, Charles R wrote:
1) What are the servers that don't have a problem?
One is a single AMD Athlon 64 2800+, running the 64-bit Sun JDK (same version).
The other is a dual AMD Athlon MP 2000+, running the 32-bit Sun JDK (same
version).
And the "bad" server has "8" Intel Xeon E5310 CP
André Warnier wrote:
> Christopher Schultz wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Sam,
>>
>> On 2/19/2009 6:23 PM, Sam Hokin wrote:
>>> The problem, which spontaneously appeared a few days before
>>> Christmas on
>>> this one server, is that the initial request of a JSP or
On Feb 20, 2009, at 10:25 AM, André Warnier wrote:
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before
Christmas on
this one server, is that the initial request of a J
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before Christmas on
this one server, is that the initial request of a JSP or servlet takes
AGES to respond, usually exceedin
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
> The problem, which spontaneously appeared a few days before Christmas on
> this one server, is that the initial request of a JSP or servlet takes
> AGES to respond, usually exceeding several minutes.
Good
> From: André Warnier [mailto:a...@ice-sa.com]
> >> From: Sam Hokin [mailto:s...@ims.net]
> >> The server that's being a problem is a Dell PowerEdge
> >> 1950, quad dual core.
> Since we have 4 processors here (do we, really? I am not so familiar
> with the multi-core architecture), can each of th
Caldarale, Charles R wrote:
From: Sam Hokin [mailto:s...@ims.net]
Subject: very slow class loading on initial JSP/servlet
request after restart
The server that's being a problem is a Dell PowerEdge
1950, quad dual core.
Since I could never believe anything bad could be caused by choc
quot;
To:
Sent: Friday, February 20, 2009 12:23 PM
Subject: very slow class loading on initial JSP/servlet request after
restart
I've been struggling with a very severe problem on one of my three Tomcat
servers since late December. I'm completely stumped. I'm running 6.0
> From: Sam Hokin [mailto:s...@ims.net]
> Subject: very slow class loading on initial JSP/servlet
> request after restart
>
> The server that's being a problem is a Dell PowerEdge
> 1950, quad dual core.
Since I could never believe anything bad could be caused by choc
Sam Hokin wrote:
[...]
Not being a Tomcat or Java specialist at all, but this being fairly late
in the day, I am piqued by your nicely told story and just trying to
imagine what could be the cause of the issue you describe. We Belgians,
like Hercule Poirot, have a knack for such things.
Consid
I've been struggling with a very severe problem on one of my three Tomcat servers since late December. I'm completely
stumped. I'm running 6.0.18 on all three servers, which are all running Fedora Core 9, all currently running Sun's JDK
1.6.0_12. The only difference is hardware. The server th
58 matches
Mail list logo
