We are going to run a test this evening with an application that someone else wrote other than our own developers to see if the problem still exists. When the problem occurs we hit 100% cpu util with jsvc (tomcat deamon) taking all of the cpu. Also we switched back to the standard collector with 128 mb min / 256 mb max which resolves the out of memory problem, but we still have the pauses every so often. We have watched Visualgc and went through the jvmstat logs that showed us that new and old gen had plenty of memory when the problem exibited itself. We didn't have any gc for a period of time before and after the problem occured.
Again thanks for all of the help and suggestions. On Thu, 14 Oct 2004 21:42:52 +0100, Dale, Matt <[EMAIL PROTECTED]> wrote: > > My next line of attack would be at the application itself. Is it possible that the > application is locking somewhere or is poorly threaded in some respect? > > What does top say when the cpu spikes? ie what is the load? what is the top process? > > If you experienced out of memory errors before you used the incremental collector > have you tried switching back to the standard collector but increasing the Xmx > setting to what is required by the app so as not to experience out of memory > exceptions. > > > > > -----Original Message----- > From: Sam Gallant [mailto:[EMAIL PROTECTED] > Sent: 14 October 2004 21:25 > To: Tomcat Users List > Subject: Re: Tomcat 5 pause (free gmail account for fix :-) > > Matt, > We have used jvmstat/visualgc which is how we found out that the > problem happens at times when no gc is taking place. We are actually > logging every second with jvmstat everyday at this point. The problem > we are stuck at now is trying to figure out how to get more infomation > out of the jvm when we can't catch the problem with a profiler and > can't run verbose gc for a long period of time. > > On Thu, 14 Oct 2004 16:06:37 -0400, Shapira, Yoav <[EMAIL PROTECTED]> wrote: > > > > Hi, > > Excellent post, that one from Matt Dale -- good advice. > > > > Yoav Shapira http://www.yoavshapira.com > > > > > > > > > > >-----Original Message----- > > >From: Dale, Matt [mailto:[EMAIL PROTECTED] > > >Sent: Thursday, October 14, 2004 4:04 PM > > >To: Tomcat Users List; Sam Gallant > > >Subject: RE: Tomcat 5 pause (free gmail account for fix :-) > > > > > >I reckon this still might be a garbage collection problem. > > > > > >I would download jvmstat from sun and use visualgc to monitor this. It > > >gives a good graphical representation of when the box is garbage > > collecting > > >and how much stuff is in each of the generations. > > > > > >We had a similar situation and this arose because the app created lots > > of > > >very short lived objects. Increasing the young generation size > > alleviated > > >this problem and also reducing tenuring so medium term objects made it > > to > > >the old generation. > > > > > >There are also a few more garbage collection parameters you can play > > with. > > >If you have multiple processors you could try enabling the parallel > > >collector. There is also the Concurrent Mark Sweep collector which > > locks > > >the JVM only in a small part of it's collection. You could try using > > the > > >aggressive heap settings which will try to use all the memory in the > > >machine that it can. > > > > > >I think more visibility of your garbage collection cycles is the first > > port > > >of call in any event. > > > > > >-----Original Message----- > > >From: Sam Gallant [mailto:[EMAIL PROTECTED] > > >Sent: 14 October 2004 20:17 > > >To: [EMAIL PROTECTED] > > >Subject: Tomcat 5 pause (free gmail account for fix :-) > > > > > > > > >Everyone, > > >Thanks in advance for any help. Also I have a gmail invite for the > > >person who has a fix for this if they are interested. > > > > > >My company has been using Tomcat for several years, but a problem has > > >crept up that we have not been able to solve. Basically, tomcat will > > >stop processing requests for 2-60 second period several times a day. > > > > > >Here is a list of software that we have tried. (Note we have tried > > >changing each key componant to see if we isolate the componant that is > > >the problem, but no luck yet) > > > > > >OS: RedHat 9 & AS3 > > >Threading model: linux threads & nptl > > >JVM: sun 1.4.2_4 & latest ibm > > >Http connector: ajp w/apache 2 and coyote connector > > >JBDC connector 1.0 > > > > > > > > >1. Doesn't always happen durning old gen garbage collection, but does > > >sometimes > > >2. Before switching to incremental gc we received out of memory errors > > >which resulted in Tomcat completly hanging > > >3. After switching to incremental gc the effect changed to 2-60 second > > >periods of time that Tomcat won't process request, but it does resume > > >on its' own. > > >4. Cpu usage for most of the day is less than 20% utiliztion, but when > > >the problem occurs the cpu spikes to 100% utiliztion briefly. > > > > > >--------------------------------------------------------------------- > > >To unsubscribe, e-mail: [EMAIL PROTECTED] > > >For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > This e-mail, including any attachments, is a confidential business communication, > > and may contain information that is confidential, proprietary and/or privileged. > > This e-mail is intended only for the individual(s) to whom it is addressed, and > > may not be saved, copied, printed, disclosed or used by anyone else. If you are > > not the(an) intended recipient, please immediately delete this e-mail from your > > computer system and notify the sender. Thank you. > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
