RE: Simple Tomcat monitoring ?
Try to analyze your java process with http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html and http://visualvm.java.net/ -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: martedì 12 aprile 2011 15:43 To: Tomcat Users List Subject: Re: Simple Tomcat monitoring ? Caldarale, Charles R wrote: >> From: André Warnier [mailto:a...@ice-sa.com] >> Subject: Simple Tomcat monitoring ? > >> 5396 root 20 0 832m 610m 9224 S 100 5.1 12:32.43 java > >> the second task visible above is the interface to the search engine). > > Which seems to be single-thread and thus a possible bottleneck. The 100% CPU > usage is very suspicious. I thought so too, but it does not seem to be. When I use the command ps -p 5396 -Lf I see multiple "instances", all running the same command. They all have the same PID, but different numbers in the "LWP" column (Light-Weight-Process ?). > >> which would be the easiest/quickest way of starting to figure out >> what this tomcat is doing, and where the system/application >> bottleneck might be ? > > This has proven to be a useful tool to look inside a running Tomcat: > http://moskito.anotheria.net/ > I had a (very) quick look at the website. From what I gather at this first look though, it seems already a bit "invasive", in the sense that it seems that to start collecting any information at all about a servlet or tomcat, the least one can do (unless one controls the source code of the application, which I do not), is to configure another classloader etc.. That means changing the Tomcat configuration and restarting Tomcat. That's maybe a bit more than what I am looking for right now. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Simple Tomcat monitoring ?
Caldarale, Charles R wrote: From: André Warnier [mailto:a...@ice-sa.com] Subject: Simple Tomcat monitoring ? 5396 root 20 0 832m 610m 9224 S 100 5.1 12:32.43 java the second task visible above is the interface to the search engine). Which seems to be single-thread and thus a possible bottleneck. The 100% CPU usage is very suspicious. I thought so too, but it does not seem to be. When I use the command ps -p 5396 -Lf I see multiple "instances", all running the same command. They all have the same PID, but different numbers in the "LWP" column (Light-Weight-Process ?). which would be the easiest/quickest way of starting to figure out what this tomcat is doing, and where the system/application bottleneck might be ? This has proven to be a useful tool to look inside a running Tomcat: http://moskito.anotheria.net/ I had a (very) quick look at the website. From what I gather at this first look though, it seems already a bit "invasive", in the sense that it seems that to start collecting any information at all about a servlet or tomcat, the least one can do (unless one controls the source code of the application, which I do not), is to configure another classloader etc.. That means changing the Tomcat configuration and restarting Tomcat. That's maybe a bit more than what I am looking for right now. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Simple Tomcat monitoring ?
On Tue, Apr 12, 2011 at 14:26, André Warnier wrote: > Francis GALIEGUE wrote: >> >> On Tue, Apr 12, 2011 at 12:20, André Warnier wrote: >> [...] >>> >>> Considering all the above, which would be the easiest/quickest way of >>> starting to figure out what this tomcat is doing, and where the >>> system/application bottleneck might be ? >>> >> >> We have the following tools, all very easy to setup and run: >> >> * SNMP monitoring of the JVM (allows to see the amount of memory >> consumed, the number of threads, and others - the MIB is well >> defined); >> * a shell script sending the results of a kill -3 on the JVM (a thread >> dump); >> * a script which triggers when a java.lang.OutOfMemoryError is thrown. >> >> If you are interested in any of it, I can tell you how we do it. >> >> And, considering that your server has 12 GB RAM, you could indeed do >> with a little more heap space! >> > > Thanks. I think I'll accept your offer of details. > > I don't think that OOM is a problem, but the first 2 would be a good start. > > About SNMP monitoring : I already have a "munin agent" installed on that > system, and a munin server collecting stuff like overall cpu and memory > usage. > To your knowledge, does there exist a way to collect this JVM SNMP > information via a munin plugin ? > I don't use Munin, I use cacti, but if Munin can collect SNMP information then it's only a matter of having Munin fetch the appropriate OIDs. More details in a separate mail. > > About the script doing a kill -3 : where does the thread dump go, when you > do that ? and how does the script know which PID is Tomcat ? > Mine does since the environment fetches the PID file and sends the signal to that process. It sends output to either stdout or stderr, I don't know. Script attached. Read the comments carefully, it is somewhat a hack (could be improved by detecting, say, if the output went to stderr). > On that system, the tomcat processes look like this : > > # ps -ef | grep java > root 20997 1 0 12:18 ? 00:00:00 /usr/bin/jsvc -user tomcat55 > -cp > /usr/share/java/commons-daemon.jar:/usr/share/tomcat5.5/bin/bootstrap.jar > -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/tomcat5.5.pid > -Djava.awt.headless=true -Xms128M -Xmx128M > -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed > -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 > -Djava.io.tmpdir=/var/lib/tomcat5.5/temp > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties > org.apache.catalina.startup.Bootstrap > root 20998 20997 0 12:18 ? 00:00:00 /usr/bin/jsvc -user tomcat55 > -cp > /usr/share/java/commons-daemon.jar:/usr/share/tomcat5.5/bin/bootstrap.jar > -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/tomcat5.5.pid > -Djava.awt.headless=true -Xms128M -Xmx128M > -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed > -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 > -Djava.io.tmpdir=/var/lib/tomcat5.5/temp > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties > org.apache.catalina.startup.Bootstrap > tomcat55 20999 20997 0 12:18 ? 00:00:04 /usr/bin/jsvc -user tomcat55 > -cp > /usr/share/java/commons-daemon.jar:/usr/share/tomcat5.5/bin/bootstrap.jar > -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/tomcat5.5.pid > -Djava.awt.headless=true -Xms128M -Xmx128M > -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed > -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 > -Djava.io.tmpdir=/var/lib/tomcat5.5/temp > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties > org.apache.catalina.startup.Bootstrap > vogon2:~# > > So there are 3 Tomcat-related processes, all running under jsvc. I am not > quite sure which one does what. > Wild guess: it's the one in the PID file that matters (-pidfile /xxx). It should be a variable in the Tomcat base configuration file. -- Francis Galiegue ONE2TEAM Ingénieur système Mob : +33 (0) 683 877 875 Tel : +33 (0) 178 945 552 f...@one2team.com 40 avenue Raymond Poincaré 75116 Paris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Simple Tomcat monitoring ?
> From: André Warnier [mailto:a...@ice-sa.com] > Subject: Simple Tomcat monitoring ? > 5396 root 20 0 832m 610m 9224 S 100 5.1 12:32.43 java > the second task visible above is the interface to the search engine). Which seems to be single-thread and thus a possible bottleneck. The 100% CPU usage is very suspicious. > which would be the easiest/quickest way of starting to figure > out what this tomcat is doing, and where the system/application > bottleneck might be ? This has proven to be a useful tool to look inside a running Tomcat: http://moskito.anotheria.net/ - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Simple Tomcat monitoring ?
You could check it apache threads are the problem by using something like: ps -feH | grep httpd | wc (or something else instead httpd to hit only apache processes) If the line number gets close to 150 (Apache (prefork) MaxClients) then that is the problem. On Tue, Apr 12, 2011 at 2:18 PM, André Warnier wrote: > Borut Hadžialić wrote: >> >> I'm gonna try to make a guess here :) >> >>> top - 08:20:02 up 72 days, 9:39, 4 users, load average: 6.83, 5.63, >>> 3.16 >>> Tasks: 265 total, 1 running, 264 sleeping, 0 stopped, 0 zombie >>> Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.2%id, 0.5%wa, 0.0%hi, 0.0%si, >>> 0.0%st >>> Mem: 12328624k total, 11382976k used, 945648k free, 123400k buffers >>> Swap: 4882424k total, 860k used, 4881564k free, 7565012k cached >> >> - somewhat large load average (6.83) >> - but no io waiting (0.5%wa), >> - app is heavily used >> - tomcat requests take long time (sometimes takes few seconds to >> produce the answer). >> >> Could it be a lack of threads at apache or tomcat? > > Spot on for the observation. > > That is one of the things that puzzle me : overall CPU usage is very low, > and there is no i/o wait. Yet the load average is high (which should mean > that a number of processes are constantly waiting for something), and Tomcat > is using 333% of CPU time (which may be very little time in an absolute > sense, but is puzzling anyway). > And the clients are waiting. > So where are the time/resources being spent ? > > About threads : > - Apache (prefork) MaxClients is set to 150 > - Tomcat AJP Connector does not specify any thread parameters, so defaults > apply, which should be : > maxThreads : 200 > maxSpareThreads : 50 > minSpareThreads : 4 > > Considering that Apache handles most requests for static contents, I think > that the numbers should be about right. > But not having so far done any real monitoring, that may be wrong. Which is > one of the things I'd like to check. > > I think I'll start by making the Tomcat AJP Connector parameters explicit, > and raise the minSpareThreads a bit. And also its Heap size. > But I don't like to start changing things left and right, without knowing > exactly why I'm doing it. > > Is there for example any parameter or command-line switch that I can use for > the JVM or Tomcat, which would tell me a bit more (in the logs) about when > Tomcat is starting a new thread, how many are alive on average etc.. ? > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Why? Because YES! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Simple Tomcat monitoring ?
Francis GALIEGUE wrote: On Tue, Apr 12, 2011 at 12:20, André Warnier wrote: [...] Considering all the above, which would be the easiest/quickest way of starting to figure out what this tomcat is doing, and where the system/application bottleneck might be ? We have the following tools, all very easy to setup and run: * SNMP monitoring of the JVM (allows to see the amount of memory consumed, the number of threads, and others - the MIB is well defined); * a shell script sending the results of a kill -3 on the JVM (a thread dump); * a script which triggers when a java.lang.OutOfMemoryError is thrown. If you are interested in any of it, I can tell you how we do it. And, considering that your server has 12 GB RAM, you could indeed do with a little more heap space! Thanks. I think I'll accept your offer of details. I don't think that OOM is a problem, but the first 2 would be a good start. About SNMP monitoring : I already have a "munin agent" installed on that system, and a munin server collecting stuff like overall cpu and memory usage. To your knowledge, does there exist a way to collect this JVM SNMP information via a munin plugin ? About the script doing a kill -3 : where does the thread dump go, when you do that ? and how does the script know which PID is Tomcat ? On that system, the tomcat processes look like this : # ps -ef | grep java root 20997 1 0 12:18 ?00:00:00 /usr/bin/jsvc -user tomcat55 -cp /usr/share/java/commons-daemon.jar:/usr/share/tomcat5.5/bin/bootstrap.jar -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/tomcat5.5.pid -Djava.awt.headless=true -Xms128M -Xmx128M -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 -Djava.io.tmpdir=/var/lib/tomcat5.5/temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties org.apache.catalina.startup.Bootstrap root 20998 20997 0 12:18 ?00:00:00 /usr/bin/jsvc -user tomcat55 -cp /usr/share/java/commons-daemon.jar:/usr/share/tomcat5.5/bin/bootstrap.jar -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/tomcat5.5.pid -Djava.awt.headless=true -Xms128M -Xmx128M -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 -Djava.io.tmpdir=/var/lib/tomcat5.5/temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties org.apache.catalina.startup.Bootstrap tomcat55 20999 20997 0 12:18 ?00:00:04 /usr/bin/jsvc -user tomcat55 -cp /usr/share/java/commons-daemon.jar:/usr/share/tomcat5.5/bin/bootstrap.jar -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/tomcat5.5.pid -Djava.awt.headless=true -Xms128M -Xmx128M -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 -Djava.io.tmpdir=/var/lib/tomcat5.5/temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties org.apache.catalina.startup.Bootstrap vogon2:~# So there are 3 Tomcat-related processes, all running under jsvc. I am not quite sure which one does what. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Simple Tomcat monitoring ?
Borut Hadžialić wrote: I'm gonna try to make a guess here :) top - 08:20:02 up 72 days, 9:39, 4 users, load average: 6.83, 5.63, 3.16 Tasks: 265 total, 1 running, 264 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.2%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 12328624k total, 11382976k used, 945648k free, 123400k buffers Swap: 4882424k total, 860k used, 4881564k free, 7565012k cached - somewhat large load average (6.83) - but no io waiting (0.5%wa), - app is heavily used - tomcat requests take long time (sometimes takes few seconds to produce the answer). Could it be a lack of threads at apache or tomcat? Spot on for the observation. That is one of the things that puzzle me : overall CPU usage is very low, and there is no i/o wait. Yet the load average is high (which should mean that a number of processes are constantly waiting for something), and Tomcat is using 333% of CPU time (which may be very little time in an absolute sense, but is puzzling anyway). And the clients are waiting. So where are the time/resources being spent ? About threads : - Apache (prefork) MaxClients is set to 150 - Tomcat AJP Connector does not specify any thread parameters, so defaults apply, which should be : maxThreads : 200 maxSpareThreads : 50 minSpareThreads : 4 Considering that Apache handles most requests for static contents, I think that the numbers should be about right. But not having so far done any real monitoring, that may be wrong. Which is one of the things I'd like to check. I think I'll start by making the Tomcat AJP Connector parameters explicit, and raise the minSpareThreads a bit. And also its Heap size. But I don't like to start changing things left and right, without knowing exactly why I'm doing it. Is there for example any parameter or command-line switch that I can use for the JVM or Tomcat, which would tell me a bit more (in the logs) about when Tomcat is starting a new thread, how many are alive on average etc.. ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Simple Tomcat monitoring ?
I'm gonna try to make a guess here :) > top - 08:20:02 up 72 days, 9:39, 4 users, load average: 6.83, 5.63, 3.16 > Tasks: 265 total, 1 running, 264 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.2%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st > Mem: 12328624k total, 11382976k used, 945648k free, 123400k buffers > Swap: 4882424k total, 860k used, 4881564k free, 7565012k cached - somewhat large load average (6.83) - but no io waiting (0.5%wa), - app is heavily used - tomcat requests take long time (sometimes takes few seconds to produce the answer). Could it be a lack of threads at apache or tomcat? On Tue, Apr 12, 2011 at 12:20 PM, André Warnier wrote: > Hi. > > I have a Linux application server running Apache + mod_jk + tomcat 5.5 (I > know that it is relatively old; and to compound the heresy, it is installed > from a Debian package). > The hardware is quite OK (quad-core fast CPU, 12 GB RAM). > > Apache is serving most of the static content, and passing some requests to > Tomcat via AJP. Tomcat runs basically a single application, but this > webapp, although itself quite small, can be heavily used : it is an > interface to a text retrieval system (a kind of database specialised for > textual data), and some of the searches made by users can take several > seconds to come back with results, which may themselves be several pages of > data. > > I can access the host remotely via SSH and via HTTP, and have full control > of it. > But it is a productive customer server, so there are limits to what I can > do. > > I am having performance issues with that system : users at times complain > that "they cannot access the application", and I see things like this in a > "top" display : > > top - 08:20:02 up 72 days, 9:39, 4 users, load average: 6.83, 5.63, 3.16 > Tasks: 265 total, 1 running, 264 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.2%id, 0.5%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 12328624k total, 11382976k used, 945648k free, 123400k buffers > Swap: 4882424k total, 860k used, 4881564k free, 7565012k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 2023 tomcat55 20 0 627m 184m 9344 S 333 1.5 32:00.47 jsvc > 5396 root 20 0 832m 610m 9224 S 100 5.1 12:32.43 java > 1 root 20 0 10316 760 624 S 0 0.0 0:33.14 init > > (where I presume that if tomcat is showing 333% CPU usage, it is because it > is running on multiple cores; the second task visible above is the interface > to the search engine). > > Anyway, my question is : > > Considering all the above, which would be the easiest/quickest way of > starting to figure out what this tomcat is doing, and where the > system/application bottleneck might be ? > > My preference would be in an easy tool to install and run, to first get a > rough idea, and then maybe select more specific tools to examine one or the > other area more in-depth. > Or maybe I can just first add some command-line switches to the JVM running > tomcat, to give me more information in the logs ? > > (My workstation is Windows XP, and I can install anything I want on it.) > > Additional details : > > Platform : (Linux 64-bit) > # uname -a > Linux vogon2 2.6.26-2-amd64 #1 SMP Thu Nov 25 04:30:55 UTC 2010 x86_64 > GNU/Linux > > Tomcat and Java : > > Using CATALINA_BASE: /usr/share/tomcat5.5 > Using CATALINA_HOME: /usr/share/tomcat5.5 > Using CATALINA_TMPDIR: /usr/share/tomcat5.5/temp > Using JRE_HOME: /usr/lib/jvm/java-6-sun > Server version: Apache Tomcat/5.5 > Server built: Oct 15 2008 12:57:44 > Server number: 5.5.26.0 > OS Name: Linux > OS Version: 2.6.26-2-amd64 > Architecture: amd64 > JVM Version: 1.6.0_22-b04 > JVM Vendor: Sun Microsystems Inc. > vogon2:/usr/share/tomcat5.5/bin# > > Tomcat is started as : > tomcat55 2023 2021 6 Apr11 ? 00:31:58 /usr/bin/jsvc -user tomcat55 > -cp > /usr/share/java/commons-daemon.jar:/usr/share/tomcat5.5/bin/bootstrap.jar > -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/tomcat5.5.pid > -Djava.awt.headless=true -Xms128M -Xmx128M > -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed > -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 > -Djava.io.tmpdir=/var/lib/tomcat5.5/temp > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties > org.apache.catalina.startup.Bootstrap > > (which I don't think is a lot of heap space, which may be a problem here. > But before I change it, I'd like to know why I change it; the server.xml > settings are pretty much the standard issue, all defaults for threads > etc..). > > Apache and mod_jk : > > Apache/2.2.9 (Debian) DAV/2 mod_jk/1.2.26 mod_apreq2-20051231/2.6.0 > mod_perl/2.0.4 Perl/v5.10.0 configured > > Thanks in advance for any recommendation > > >
Re: Simple Tomcat monitoring ?
On Tue, Apr 12, 2011 at 12:20, André Warnier wrote: [...] > > Considering all the above, which would be the easiest/quickest way of > starting to figure out what this tomcat is doing, and where the > system/application bottleneck might be ? > We have the following tools, all very easy to setup and run: * SNMP monitoring of the JVM (allows to see the amount of memory consumed, the number of threads, and others - the MIB is well defined); * a shell script sending the results of a kill -3 on the JVM (a thread dump); * a script which triggers when a java.lang.OutOfMemoryError is thrown. If you are interested in any of it, I can tell you how we do it. And, considering that your server has 12 GB RAM, you could indeed do with a little more heap space! -- Francis Galiegue ONE2TEAM Ingénieur système Mob : +33 (0) 683 877 875 Tel : +33 (0) 178 945 552 f...@one2team.com 40 avenue Raymond Poincaré 75116 Paris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org