You could also try to use the Oracle Solaris Studio performance tools,
and profile the application. It does not user DTrace, so
the DTrace lock contention issue won't come up.
http://developers.sun.com/sunstudio/overview/topics/analyzing.jsp
Marty Itzkowitz
Project lead, Oracle Solaris Studio Performance Tools.
On 3/30/10 8:10 AM, Jeremiah Campbell wrote:
> I'm curious if you're seeing a similar problem to what I experienced
recently. We had recently migrated some JVM's onto a brand new Solaris
10 T5220 and saw horrendous performance. We opened a case with Sun and
eventually got to a kernel engineer who discovered we were running into
a variant of bug ID 6815915 which is a C++ or libc bug specifically.
However, there is a Java variant of it that was affecting us (never did
see a specific bug ID for the java variant).
> The issue was discovered when looking at at least a level 2 guds and
looking at the lockstat-H.out file. We were seeing a bunch of mutex
locks bubbling up out of the dtrace helper provider (HotSpot). The
work-around was to disable the Dtrace helper functionality (can be
turned back on via command line if needed) before starting the
application. The work-around was to set an environment variable in your
JVM startup scripts (tomcat profile for instance if your JVMs run under
tomcat). Here's the specific quote from the Sun engineer;
>
> "The system is under heavy lock contention on dtrace_meta_lock
because all java processes are busy in register or destroy dtrace helper
functions. Java should not always load DTrace probes. This is likely a
Java variation of CR 6815915 which is filed for the C++ runtime libray.
> The workaround is to run the application with the environment
variable DTRACE_DOF_INIT_DISABLE set. "
> The below makes me wonder if you're getting some mutex locks out of
Dtrace as well. You might try finding this with guds. I'm no Dtrace
guru, so couldn't tell you how to pull the same information out of
Dtrace as what we found in guds.
>
>> Elapsed Times:
>> SYSCALL TIME (ns)
>> ...
>> lwp_mutex_timedlock 1541179345
>>
>> CPU Times
>> SYSCALL TIME (ns)
>> ...
>> lwp_mutex_timedlock 30490184
>>
_______________________________________________
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
