I certainly agree with what Nathan said about synchronization
primitives, getting
pstacks etc, but maybe I can expand a bit further. Synchronization
primitives
covers not just mutexes but also condition variables. Its quite common in a
multithreaded program to use a condition variable to represent "work
available",
and normal for a bunch of worker threads to be asleep on those condition
variables
waiting for work to arrive.
Mysqld is an example of the challenge in making sense of %LCK. Its a
single process
but heavily multithreaded, and the threads have a variety of purposes.
Normally
the foreground threads that process SQL requests are asleep on a network
read
or waiting for an incoming connection, and spend their time in SLP
state. Several
of the background threads, such as for database i/o wait for work on
condition
variables. What I generally suggest is the people look at the threads
when the
database is idle, to see which state each thread spends most of its time
when it
has nothing to do.
Oracle is implemented around having a large number of processes, only
some of which
are multithreaded. I think you'll find things like the log writer and db
writer
processes are multithreaded, and most of those threads are asleep on
condition
variables waiting for work. Shadow processes are usually
single-threaded, and
spend their time sleeping trying to read from a socket. Here as snippet
from a
dbwr showing an idle i/o thread parked on a condition variable.
<pid>: oracleXXXX (LOCAL=NO)
...
----------------- lwp# 5 / thread# 5 --------------------
ffffffff7c0cd030 lwp_park (0, 0, 0)
ffffffff7c0c6854 cond_wait_queue (106519550, 106519560, 0, 0, 0,
ffffffff7c1f0080) + 28
ffffffff7c0c6e04 cond_wait (106519550, 106519560, 0, 1000,
ffffffff7a500c00, 0) + 10
ffffffff7be039f4 _aio_idle (1065194e0, 1, 106519560, 0,
ffffffff7bf08000, 0) + 28
ffffffff7be0313c _aio_do_request (1065194e0, 10000, 0, 0,
ffffffff7bf08000, ffffffff7bf08000) + c4
ffffffff7c0ccf90 _lwp_start (0, 0, 0, 0, 0, 0)
Specifically in the case of Oracle, it has some interesting V$ tables,
collectively known as Oracle Wait Interface [OWI], that can be used to
get an
idea where threads and processes are spending their time. I have found
v$session_event, v$session, v$sesstat, v$statname useful [typically by
creating
a trigger that logs useful stuff from them when a session ends].
--
============================================================================
,-_|\ Richard Smith Staff Engineer PAE
/ \ Sun Microsystems Phone : +61 3 9869 6200
richard.smith at Sun.COM Direct : +61 3 9869 6224
\_,-._/ 476 St Kilda Road Fax : +61 3 9869 6290
v Melbourne Vic 3004 Australia
===========================================================================
