Jonathon,
I have yet to do any sampling, I have been visually monitoring the servers
with the VisualJVM, the Slackware system monitor, top and vmstat. I was
using JMeter to stress test the application.
The old server is Slackware 12.1 (32 bit), Tomcat 5.5, Java 1.5.0_01.
The new servers are Slackware 13 (64 bit), Tomcat 6.0.24, Java 1.6_18.
The old server has run flawlessly for several years. The new servers are
kicking my butt:
- They will run for a period of time (could be 10 days, could be 15 minutes)
and silently fail, no error logs, no memory dump, nothing in catalina.out,
no tracks that I have been able to find.
- The issue seems load related and cumulative. A week ago was a very heavy
processing day and the server failed repeatedly starting about noon.
Otherwise, it would usually fail late in the day.
- I have tried different heap sizes. My first inclination was a larger
heap but, as Chuck pointed out, that was probably the wrong thing. I am
currently running with a 500MB heap (384MB permGen.) Been up for two days
and looks great.
- I have quite a bit of logging to catalina.out because I was, at one
point, looking for a connection leak. I also started incremental GC and
printing the GC instances using the appropriate java flags. The last entry
for the last failure was a GC.
- I tried provoking the failure by stressing the application using
JMeter... wouldn't fail even when I forced an OOM.
Your idea of sampling is interesting... may try that next.
Thanks,
Carl
----- Original Message -----
From: "Jonathan Mast" <jhmast.develo...@gmail.com>
To: "Tomcat Users List" <users@tomcat.apache.org>
Sent: Friday, February 05, 2010 7:11 PM
Subject: Re: Tomcat dies suddenly
Hi Carl, I've skimmed thru all your posts, please excuse me if these
questions are redundant.
You've mentioned VisualJVM, JMeter, and Slackware monitoring tools, it
seems
to me that you are using these to occasionally to monitor you Tomcat
instance, when things seem to be going badly (thats my perception from
reading your posts)
Have you done any long term sampling (Tomcat threads, memory, etc) of your
system?
I use Slackware 12.2 (32-bit), Httpd 2.2, Java 1.6, Tomcat 6.0.18 and had
Tomcat die in a manner similiar to yours: First it died silently (nothing
in the catalina.out), the next time (about a week later) it became
non-responsive complaining about PermGen, OOMException and so on. I
haven't
determined the cause yet, I rolled over to another server (exact same
hardware/software btw).
I ran Memtest on the problematic server, no problems detected.
Anyways, I've been sampling the httpd process count, tomcat thread count,
and free memory (sans cache and buffers) every minute via a cron job and
writing the results to a file on both machines since this issue arose
about
2 weeks ago. My hope is that the next time it bugs out, i'll be able to
say
"oh the thread count exploded, but the memory use only increased
incrementally" or whatever the cause actually is.
Have you done any monitoring at that fine grained a level on your systems?
It would be especially interesting to see how your old HW setup compares
to
the new ones.
Hope I've helped a little bit :)
On Fri, Feb 5, 2010 at 11:26 AM, Carl <c...@etrak-plus.com> wrote:
Chris,
I agree but this seemed like a low risk change. I also understand that
changing too many things at once sometimes confuses the issue.
Thanks,
Carl
----- Original Message ----- From: "Christopher Schultz" <
ch...@christopherschultz.net>
To: "Tomcat Users List" <users@tomcat.apache.org>
Sent: Friday, February 05, 2010 10:48 AM
Subject: Re: Tomcat dies suddenly
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Carl,
On 2/5/2010 10:16 AM, Carl wrote:
Do you see any harm in just doubling the number (to 2048) just to see
if
it has an impact?
We would expect to see an OOME (strange, I know) if you were running out
of file descriptors.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktsPbYACgkQ9CaO5/Lv0PBWnwCfXyZVJNsd/mMCgJWVamRBVZ+F
AMkAn1H2romyi8eFunc3lF+eDiQa0Hlq
=muVu
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
