Re: tomcat 6.0.29 hung

2010-10-22 Thread Dave Mansfield
This might be related, and don't know if you've read it - it's a bit out 
of date, and depends on versions of stuff you're using, Jason.


http://forums.sun.com/thread.jspa?threadID=343023&start=15&tstart=0

-- D Mansfield

Jason Britton wrote:

Thanks for the insight Dave.  I have several threads sitting spinning
on socket read at the moment and dba reports zero database locks of
any kind on database they're trying to read from (a qa database).
Quite baffled as to what's happening to these database connections and
why every now and then thread sits on socket read while trying to read
from database.  Situation becomes serious and app completely dead
locks when the socket read hang happens inside synchronized block.



On Fri, Oct 22, 2010 at 10:36 AM, Dave Mansfield
 wrote:
  

Well, here's an oversight:  The threads below show concurrent execution of a
prepared statement.  The fact that a thread own the lock, is buried well
into executing that statement, and is waiting for a read from a socket
doesn't mean it's an issue with the AJP - it could well be a DB locking
problem.
You need to examine the Oracle side of things in parallel and start looking
at database lock contention.  I don't know the semantics of Oracle's JDBC
driver when managing a prepared statement - especially if it's done
concurrently, so I can't give you any trails to follow...

I'd focus on forcing the thread holding the lock to abort on the Oracle side
of things - if it frees up the other threads, you have a new point to start
diagnosing...

-- D Mansfield



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


  


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: tomcat 6.0.29 hung

2010-10-22 Thread Dave Mansfield



Jason Britton wrote:

So while working on reproducing tomcat webapp hang as earlier discussed, a
different tomcat serving up a single webapp on same box also stopped
responding, jvm for this tomcat is still running but when trying to reach it
via browser I get error page of unable to connect.

Thread dumps for both problem tomcats seem to exhibit same core problem, at
least in my analysis.  I would really appreciate your feedback on my
analysis of the situation and things anyone thinks I may be over looking.
  
Well, here's an oversight:  The threads below show concurrent execution 
of a prepared statement.  The fact that a thread own the lock, is buried 
well into executing that statement, and is waiting for a read from a 
socket doesn't mean it's an issue with the AJP - it could well be a DB 
locking problem.  

You need to examine the Oracle side of things in parallel and start 
looking at database lock contention.  I don't know the semantics of 
Oracle's JDBC driver when managing a prepared statement - especially if 
it's done concurrently, so I can't give you any trails to follow...


I'd focus on forcing the thread holding the lock to abort on the Oracle 
side of things - if it frees up the other threads, you have a new point 
to start diagnosing...


-- D Mansfield

On both tomcat jvms when they get to the point of locking up, thread dumps

  


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: what is it doing ?

2010-10-19 Thread Dave Mansfield


André Warnier wrote:

Justin Randall wrote:

Hello,

From a general Linux perspective, if you have a knowledge of Linux 
system calls, 


not really

along with what operations are performed by which web application 


not really either.  Basically, that is what I am trying to find out.

(One might wonder what I really do know, I know. My colleagues also 
wonder.)


I see (with top) that Tomcat as a process is busy doing something, but 
what I don't know is what within Tomcat is doing it.
This Tomcat runs a number of webapps, a minority of which I really 
have an idea of what they are supposed to be doing.  And precisely 
these should not be doing anything at the times I'm watching.  So my 
suspicion is one of the others, but I may be wrong, and that's what I 
am trying to figure out.



you could use "strace" (use the -f to follow threads).  Depending on 
what you're interested in you can filter further for operations on 
files or network activity (i.e. "-e trace=file,network,desc").  If you 
see a lot of erroneous activity (such as constantly trying to 
open/stat a file that doesn't exist) and you know which web app (or 
Tomcat) is responsible you can address the issue.


You could also check "top" to see whether the CPU is being consumed 
in usr or sys space, along with whether maybe the memory on your 
system is maxed out (trigger high CPU load transfer data between RAM 
and swap space).


There may be that too, but at this point I believe not.  "top" and 
"free", each time I look, show the swap space to be virtually unused.
The system as a whole is not overloaded, and the Tomcat-centric CPU 
usage, although high in percentage, is finally a percentage of not 
very much.
Let's just say that I am curious, apart from the fact that some Tomcat 
applications which should answer quickly don't always.


So it's the thread dumps route for me then I guess ?
I was hoping someone would just point me to this nice "java-top" 
Debian package...


First thing is to consider the big picture:  Tomcat, as designed, does a 
variety of things that consume CPU cycles - it just depends on how 
you've got it configured.  For example, is it setup to auto-deploy new 
applications if a WAR file appears in the application root?  If so, some 
slice of it's behavior is checking for changes to the file system - an 
expensive operation.  You MUST understand the configuration you've 
deployed, and any implications of performance robbing settings.


Secondly - one of the BETTER things to know is how to take and review 
Java thread and memory (heap) dumps.  Just behind that (or, sometimes in 
front) is how to enable remote debugging - get that working and then you 
can plug your favorite IDE into your webapp(s) for remote debugging.  If 
you know how to do this, you're going to be much better equipped to 
diagnose a Tomcat/Java - regardless of the operating environment.


Third - (and IMO the BEST ) is you need to know is how (and when) to 
log.  Tomcat (as most things Apache) has quite a bit of logging 
available. I don't know about the apps you are using, but you should 
check logging facilities are available.  You will want to use logging - 
if you had no other tool excepting logging, you can likely pinpoint the 
problem.  (And obviously, whenever you enable more detailed logging, 
you're going to incur performance hits...)


If you can couple logging with examining thread dumps, you'll be much 
better to apply a factual "triage" on "what is it doing" (Tomcat vs. 
webapps vs. other things) and in turn, help you decide if you have a 
real issue.


Bottom line: Just because Tomcat is consuming system resource, it does 
not mean something is "wrong".  Think about this:  If Tomcat wants a 
slice of CPU time, and there's no contention, why would the system NOT 
give it 100% of the CPU?  When you "snapshot" system utilization with a 
tool such as "top", it's not enough of a diagnostic tool to determine if 
any given application (e.g., tomcat) is misbehaving.  Especially, as 
you've indicated, the system is not really in contention for resources... 

Let me offer an analogy:  If you were to just measure wind speed, is 
that enough information to state there's a storm outside?  Obviously, 
it's not - you have to look at other factors, too.


Sorting out how to do these things is a worthwhile effort - try not to 
shy away from it...


-- D Mansfield

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org