Re: what is it doing ?
On 19/10/2010 16:40, André Warnier wrote: Hi. I have a Linux system like so : Linux arthur 2.6.26-2-686 #1 SMP Mon Aug 30 07:01:57 UTC 2010 i686 GNU/Linux with a tomcat like so : Using CATALINA_BASE: /usr/share/tomcat5.5 Using CATALINA_HOME: /usr/share/tomcat5.5 Using CATALINA_TMPDIR: /usr/share/tomcat5.5/temp Using JRE_HOME: /usr/lib/jvm/java-1.5.0-sun Server version: Apache Tomcat/5.5 Server built: Oct 15 2008 12:57:44 Server number: 5.5.26.0 OS Name:Linux OS Version: 2.6.26-2-686 Architecture: i386 JVM Version:1.5.0_22-b03 JVM Vendor: Sun Microsystems Inc. (Java JDK installed) and where night and day, tomcat seems to be using a fair amount of CPU time, like so : PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2901 tomcat55 20 0 459m 285m 16m S 13 32.3 354:39.13 java although by all means, it should not be doing that much sometimes. How can I figure out what it is actually doing ? I mean, which is the easiest standard tool or procedure not requiring a degree in quantum physics, and which could give me at least a rough idea of which webapp or internal tomcat process is keeping that CPU busy all the time ? I have a nagging feeling that someone is going to tell me to take thread dumps. If so, maybe they would also kindly point out a place where I could find some instructions as to how to interpret such a thing ? You can take sequential dumps and compare states to find out which threads are busy and what they're doing. Look at the thread stacks and see what the code is doing. You can discard idle threads. You're looking to see which threads are active and in which bit of code they are spending time. (Unless something is blocked a single thread dump is of limited use.) You could also attach a JConsole or a profiler and monitor the threads to see what they're doing. Java 6 has better tools for this, though there's a independant download of VisualVM which may work too. p Thanks - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
RE: what is it doing ?
Hello, From a general Linux perspective, if you have a knowledge of Linux system calls, along with what operations are performed by which web application you could use strace (use the -f to follow threads). Depending on what you're interested in you can filter further for operations on files or network activity (i.e. -e trace=file,network,desc). If you see a lot of erroneous activity (such as constantly trying to open/stat a file that doesn't exist) and you know which web app (or Tomcat) is responsible you can address the issue. You could also check top to see whether the CPU is being consumed in usr or sys space, along with whether maybe the memory on your system is maxed out (trigger high CPU load transfer data between RAM and swap space). Regards, Justin Randall Date: Tue, 19 Oct 2010 17:40:17 +0200 From: a...@ice-sa.com To: users@tomcat.apache.org Subject: what is it doing ? Hi. I have a Linux system like so : Linux arthur 2.6.26-2-686 #1 SMP Mon Aug 30 07:01:57 UTC 2010 i686 GNU/Linux with a tomcat like so : Using CATALINA_BASE: /usr/share/tomcat5.5 Using CATALINA_HOME: /usr/share/tomcat5.5 Using CATALINA_TMPDIR: /usr/share/tomcat5.5/temp Using JRE_HOME: /usr/lib/jvm/java-1.5.0-sun Server version: Apache Tomcat/5.5 Server built: Oct 15 2008 12:57:44 Server number: 5.5.26.0 OS Name:Linux OS Version: 2.6.26-2-686 Architecture: i386 JVM Version:1.5.0_22-b03 JVM Vendor: Sun Microsystems Inc. (Java JDK installed) and where night and day, tomcat seems to be using a fair amount of CPU time, like so : PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2901 tomcat55 20 0 459m 285m 16m S 13 32.3 354:39.13 java although by all means, it should not be doing that much sometimes. How can I figure out what it is actually doing ? I mean, which is the easiest standard tool or procedure not requiring a degree in quantum physics, and which could give me at least a rough idea of which webapp or internal tomcat process is keeping that CPU busy all the time ? I have a nagging feeling that someone is going to tell me to take thread dumps. If so, maybe they would also kindly point out a place where I could find some instructions as to how to interpret such a thing ? Thanks - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: what is it doing ?
One more thing, aside from checking whether the CPU consumed is usr or sys you may also want to lookup at iowait (a.k.a wa). This would indicate the CPU is busy waiting for IO operations to complete (potentially a slow disk or storage device). Regards, Justin Randall From: ran...@hotmail.com To: users@tomcat.apache.org Subject: RE: what is it doing ? Date: Tue, 19 Oct 2010 12:10:54 -0400 Hello, From a general Linux perspective, if you have a knowledge of Linux system calls, along with what operations are performed by which web application you could use strace (use the -f to follow threads). Depending on what you're interested in you can filter further for operations on files or network activity (i.e. -e trace=file,network,desc). If you see a lot of erroneous activity (such as constantly trying to open/stat a file that doesn't exist) and you know which web app (or Tomcat) is responsible you can address the issue. You could also check top to see whether the CPU is being consumed in usr or sys space, along with whether maybe the memory on your system is maxed out (trigger high CPU load transfer data between RAM and swap space). Regards, Justin Randall Date: Tue, 19 Oct 2010 17:40:17 +0200 From: a...@ice-sa.com To: users@tomcat.apache.org Subject: what is it doing ? Hi. I have a Linux system like so : Linux arthur 2.6.26-2-686 #1 SMP Mon Aug 30 07:01:57 UTC 2010 i686 GNU/Linux with a tomcat like so : Using CATALINA_BASE: /usr/share/tomcat5.5 Using CATALINA_HOME: /usr/share/tomcat5.5 Using CATALINA_TMPDIR: /usr/share/tomcat5.5/temp Using JRE_HOME: /usr/lib/jvm/java-1.5.0-sun Server version: Apache Tomcat/5.5 Server built: Oct 15 2008 12:57:44 Server number: 5.5.26.0 OS Name:Linux OS Version: 2.6.26-2-686 Architecture: i386 JVM Version:1.5.0_22-b03 JVM Vendor: Sun Microsystems Inc. (Java JDK installed) and where night and day, tomcat seems to be using a fair amount of CPU time, like so : PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2901 tomcat55 20 0 459m 285m 16m S 13 32.3 354:39.13 java although by all means, it should not be doing that much sometimes. How can I figure out what it is actually doing ? I mean, which is the easiest standard tool or procedure not requiring a degree in quantum physics, and which could give me at least a rough idea of which webapp or internal tomcat process is keeping that CPU busy all the time ? I have a nagging feeling that someone is going to tell me to take thread dumps. If so, maybe they would also kindly point out a place where I could find some instructions as to how to interpret such a thing ? Thanks - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: what is it doing ?
Justin Randall wrote: Hello, From a general Linux perspective, if you have a knowledge of Linux system calls, not really along with what operations are performed by which web application not really either. Basically, that is what I am trying to find out. (One might wonder what I really do know, I know. My colleagues also wonder.) I see (with top) that Tomcat as a process is busy doing something, but what I don't know is what within Tomcat is doing it. This Tomcat runs a number of webapps, a minority of which I really have an idea of what they are supposed to be doing. And precisely these should not be doing anything at the times I'm watching. So my suspicion is one of the others, but I may be wrong, and that's what I am trying to figure out. you could use strace (use the -f to follow threads). Depending on what you're interested in you can filter further for operations on files or network activity (i.e. -e trace=file,network,desc). If you see a lot of erroneous activity (such as constantly trying to open/stat a file that doesn't exist) and you know which web app (or Tomcat) is responsible you can address the issue. You could also check top to see whether the CPU is being consumed in usr or sys space, along with whether maybe the memory on your system is maxed out (trigger high CPU load transfer data between RAM and swap space). There may be that too, but at this point I believe not. top and free, each time I look, show the swap space to be virtually unused. The system as a whole is not overloaded, and the Tomcat-centric CPU usage, although high in percentage, is finally a percentage of not very much. Let's just say that I am curious, apart from the fact that some Tomcat applications which should answer quickly don't always. So it's the thread dumps route for me then I guess ? I was hoping someone would just point me to this nice java-top Debian package... - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: what is it doing ?
André Warnier wrote: Justin Randall wrote: Hello, From a general Linux perspective, if you have a knowledge of Linux system calls, not really along with what operations are performed by which web application not really either. Basically, that is what I am trying to find out. (One might wonder what I really do know, I know. My colleagues also wonder.) I see (with top) that Tomcat as a process is busy doing something, but what I don't know is what within Tomcat is doing it. This Tomcat runs a number of webapps, a minority of which I really have an idea of what they are supposed to be doing. And precisely these should not be doing anything at the times I'm watching. So my suspicion is one of the others, but I may be wrong, and that's what I am trying to figure out. you could use strace (use the -f to follow threads). Depending on what you're interested in you can filter further for operations on files or network activity (i.e. -e trace=file,network,desc). If you see a lot of erroneous activity (such as constantly trying to open/stat a file that doesn't exist) and you know which web app (or Tomcat) is responsible you can address the issue. You could also check top to see whether the CPU is being consumed in usr or sys space, along with whether maybe the memory on your system is maxed out (trigger high CPU load transfer data between RAM and swap space). There may be that too, but at this point I believe not. top and free, each time I look, show the swap space to be virtually unused. The system as a whole is not overloaded, and the Tomcat-centric CPU usage, although high in percentage, is finally a percentage of not very much. Let's just say that I am curious, apart from the fact that some Tomcat applications which should answer quickly don't always. So it's the thread dumps route for me then I guess ? I was hoping someone would just point me to this nice java-top Debian package... First thing is to consider the big picture: Tomcat, as designed, does a variety of things that consume CPU cycles - it just depends on how you've got it configured. For example, is it setup to auto-deploy new applications if a WAR file appears in the application root? If so, some slice of it's behavior is checking for changes to the file system - an expensive operation. You MUST understand the configuration you've deployed, and any implications of performance robbing settings. Secondly - one of the BETTER things to know is how to take and review Java thread and memory (heap) dumps. Just behind that (or, sometimes in front) is how to enable remote debugging - get that working and then you can plug your favorite IDE into your webapp(s) for remote debugging. If you know how to do this, you're going to be much better equipped to diagnose a Tomcat/Java - regardless of the operating environment. Third - (and IMO the BEST ) is you need to know is how (and when) to log. Tomcat (as most things Apache) has quite a bit of logging available. I don't know about the apps you are using, but you should check logging facilities are available. You will want to use logging - if you had no other tool excepting logging, you can likely pinpoint the problem. (And obviously, whenever you enable more detailed logging, you're going to incur performance hits...) If you can couple logging with examining thread dumps, you'll be much better to apply a factual triage on what is it doing (Tomcat vs. webapps vs. other things) and in turn, help you decide if you have a real issue. Bottom line: Just because Tomcat is consuming system resource, it does not mean something is wrong. Think about this: If Tomcat wants a slice of CPU time, and there's no contention, why would the system NOT give it 100% of the CPU? When you snapshot system utilization with a tool such as top, it's not enough of a diagnostic tool to determine if any given application (e.g., tomcat) is misbehaving. Especially, as you've indicated, the system is not really in contention for resources... Let me offer an analogy: If you were to just measure wind speed, is that enough information to state there's a storm outside? Obviously, it's not - you have to look at other factors, too. Sorting out how to do these things is a worthwhile effort - try not to shy away from it... -- D Mansfield - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
