The tomcat service is managed by the Solaris service management system (I can see references to svcs in their start-up/shutdown logs (it's on a customer's site so I'm not 100% on how it's configured).
I should say there are 2 other tomcat instances all being managed the same way and they don't have any problems at all. We've investigated the shutdown and start-up systems thoroughly and are happy that it's not an external agent that's trying to fire up another tomcat instance. In fact we've manually fired up a second instance and it comes up completely differently to these ghost processes, which as I mentioned before are allocating a whole new chunk of memory but then only running 1 thread and taking no CPU at all. -----Original Message----- From: André Warnier [mailto:a...@ice-sa.com] Sent: 09 September 2011 13:35 To: Tomcat Users List Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional copies of itself with only 1 thread and no CPU Ok, fishing in the dark really, but trying to accumulate more clues. What about the other questions (how is Tomcat started) ? And, looking at your original "ps" list, some vague thing : all these ghost processes seem to a) not be doing much (CPU time 0), even after a long time. b) all be started around the clock hour:00, or the hour:30, give or take a couple of minutes. Got anything particular that would run at approximately 30 minutes interval ? (Like some monitoring process which sends a request every 30 minutes just to check ? And if so : has this been added just before the problem started ? and what request does it send ?) As an alternative to netstat under Solaris, there is the "lsof" utility. It has a million options to sort through, but it certainly can tell you which process owns which TCP socket. The point here is to try to figure out what these ghosts are doing and which configuration settings they use. They seem to be using the same command-line settings as the original, but we only see a small part (there may be other switches on your version of "ps" which list the whole command line of each process.). You cannot have several processes opening the same TCP port in "LISTEN" mode. If all these processes use the same config and try to open the same ports (Tomcat opens several), all but the first one should fail, and that should normally leave a trace in the Tomcat logs. Else, they are not real processes and really are threads, no matter what "ps" is saying. You could also try taking some thread dumps of the Tomcat JVM (see the Tomcat FAQ), to see if some thread-id there match the PID's seen in your ps displays. Dave Stubbs wrote: > Solaris environment is.... > > SunOS ********** 5.10 Generic_127127-11 sun4v sparc > SUNW,SPARC-Enterprise-T5120 > > System = SunOS > Node = ********** > Release = 5.10 > KernelID = Generic_127127-11 > Machine = sun4v > BusType = <unknown> > Serial = <unknown> > Users = <unknown> > OEM# = 0 > Origin# = 1 > NumCPU = 64 > > -----Original Message----- > From: André Warnier [mailto:a...@ice-sa.com] > Sent: 09 September 2011 10:25 > To: Tomcat Users List > Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional > copies of itself with only 1 thread and no CPU > > Intriguing. > > For a complement of information : > - what is the exact platform ? > - is Tomcat started via jsvc ? > - as a matter of fact, *how* is the "main" Tomcat being started ? > > What does the command "netstat -pan | grep LISTEN" show when you have such > multiple copies running ? > (the form of the command will depend a bit on the platform; the command > above, under Linux, shows the ports on which processes are listening, and > the corresponding PID's and maybe program name) > > Dave Stubbs wrote: >> We're experiencing a problem on both production and test systems that >> has only started to occur in the last few weeks (no new version of any >> software have been put live to anyone's knowledge) that has us quite > stumped. >> >> >> We are seeing tomcat starting up additional copies of itself, each new >> copy is allocating a chunk of storage, it only starts 1 thread, >> nothing gets written to any logs and no CPU is being listed as having been > used. >> This keeps happening, and if it's not monitored the system memory all >> gets taken up and we start to get SIGBUS errors. >> >> We've tried turning all logging on, there don't appear to be any >> software related events at the heart of the issue, and we can't see >> any system triggers. >> >> When we list the system tasks we see that the ghost process PPID is >> the same as the PID of the still running main tomcat instance. Here is >> list of what happened when it wasn't picked up for a while. >> >> >> >> tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 4270 21115 0 18:29:29 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 5988 21115 0 18:59:32 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 27209 21115 0 01:01:02 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 13155 21115 0 05:32:09 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 16785 21115 0 06:34:52 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 12948 21115 0 20:58:50 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 20054 21115 0 23:00:21 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat >> 23730 21115 0 00:03:47 ? 0:00 /usr/java/bin/java >> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti >> >> The main process 21115 starts off fine, and is even still running with >> no problems the next day, but we keep getting these ghost processes >> (for want of a better description). >> >> We have multiple tomcat servers running on the same system, but this >> is the only one experiencing the problem, all of the others are > unaffected. >> The web apps themselves are pretty standard, servlets, JSPs, web >> services, Oracle DB connection and a JMX link to an Apache MQ server >> for monitoring purposes. This isn't even a heavily used application. >> >> >> >> The web has come up empty for us, we can't find any information on >> this fault at all, so would really appreciate any help you may be able >> to give us. >> >> >> >> Regards >> >> Dave >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
